The Anthropology of Data

I was asked recently where I though data was going; after mulling my answer for a few days I decided I'm not happy with it. My new answer is this - data is moving from being a matter of archeology to anthropology. 

Consider this - we have been collecting electronic data for about 50 years now. The largest organizations now have warehouses that contain petabytes of it. Furthermore, there is no end in sight. As we become better at digitizing information and assigning useful metadata to it; we collect more and more. How will we use all this data in the future?

Even now, the most common use cases involve spending enormous effort moving, cleansing, enriching and transforming data into central stores to generate tabular reports. The value of the data we collect therefore depends on the skill and technology used to clean it, connect it and create reports that provide historical information in the hopes that we can make good decisions about the future. In other words, our data is archeological in nature - wait until its dead, move it into a secure place and inspect it.

Our ability to collect data has always outpaced our ability to make use of it; therefore we will always have more of it than we know what to do with. While the ‘move it to the warehouse once it's dead' approach will continue to be helpful, tomorrow's successful organizations will adopt approaches to data that ire more akin to anthropology. Think about this - the most useful data is not old and dead, it is alive and evolving. While we can make use of old, dead data; the most useful technology will allow us to inspect the culture of living data and use trends and behaviors observed in past to allow predictions of the future.

Compare these two scenarios -

#1 - Archeological Approach

At a certain point in each year, shoppers start buying orange juice and tissues together and in greater quantity. Not-So-SavvyMart takes the archeological approach - they discover this connection by mining the warehouse and concluding that during cold season people need tissue and want the vitamin C in OJ. As a result of this expensive analysis, they eventually co-locate these products to increase sales during certain times of the year. The archeological pattern - collect data, mine data, draw some "ah ha" conclusions, take some action, increase profit. Boy that took a long time.

#2 - Anthropological Approach

This approach recognizes the connection between OJ and tissue is one of many connections between products that come and go; the culture of data (what people are buying and why) is most useful, however, when trends can be immediately recognized and taken advantage of. In this second scenario, SavvyMart employs RFID sensors and smart shopping carts to track what its customers are putting in their shopping carts in near real-time. They are also tracking where their customers are going and the patterns they follow through stores. Rather than wait to store and mine this data, they have designed a predictive model that spots trends in products purchased together. Rather than ask, "why" and take action to rearrange product placement, the monitors on the smart carts simply suggest related products when the first is bought, possibly even offering digital coupons. Managers use a product placement data mash-up application to suggest end-stand arrangements that optimize related product purchase opportunities based on the evolution of cart patterns and product purchases.

Next SavvyMart's procurement system begins noticing sales trends and advertises its desire to purchase more of high demand items via its network of suppliers and the Good Relations Ontology. SavvySupplier notices

Continue Reading