Data Warehouse: Turning Data Into Lead?

The following is an extract from "Knowledge Asset Management and Corporate Memory" a White Paper to be published on the WWW possibly via the Hispacom site in the third week of August 1996...... Data Warehousing may well leverage the rising tide technologies that everyone will want or need, however the current trend in Data Warehousing marketing leaves a lot to be desired. In many organizations there still exists an enormous divide that separates Information Technology and a managers need for Knowledge and Information. It is common currency that there is a whole host of available tools and techniques for locating, scrubbing, sorting, storing, structuring, documenting, processing and presenting information. Unfortunately, tools are tangible and business information and knowledge are not, so they tend to get confused. So why do we still have this confusion? First let's consider how certain companies market Data Warehousing. There are companies that sell database technologies, other companies that sell the platforms (ostensibly consisting of an MPP or SMP architecture), some sell technical Consultancy services, others meta-data tools and services, finally there are the business Consultancy services and the systems integrators - each and everyone with their own particular focus on the critical factors in the success of Data Warehousing projects. In the main, most RDBMS vendors seem to see Data Warehouse projects as a challenge to provide greater performance, greater capacity and greater divergence. With this excuse, most RDBMS products carry functionality that make them about as truly "open" as a UNIVAC 90/30, i.e. No standards for View Partitioning, Bit Mapped Indexing, Histograms, Object Partitioning, SQL query decomposition or SQL evaluation strategies etc. This however is not really the important issue, the real issue is that some vendors sell Data Warehousing as if it just provided a big dumping ground for massive amounts of data with which users are allowed to do anything they like (any this, that or the other - kind of reminds me a little of the history of computing), whilst at the same time freeing up Operational Systems from the need to support end-user informational requirements. Some hardware vendors have a similar approach, i.e. a Data Warehouse platform must inherently have a lot of disks, a lot of memory and a lot of CPUs. However, one of the most successful Data Warehouse projects I have worked on used COMPAQ hardware, which provides an excellent cost/benefit ratio. Some Technical Consultancy Services providers tend to dwell on the performance aspects of Data Warehousing. They see Data Warehousing as a technical challenge, rather than a business opportunity, but the biggest performance payoffs will be brought about when there is a full understanding of how the user wishes to use the information, not before. To my surprise, most Meta-Data Tool vendors seem to have a handle on what Data Warehousing is all about, so there is little to say on this score, apart from the fact that these companies must participate with other vendors in providing a Data Warehousing solution, and this implies a degree of risk. So what is exactly wrong in just putting data into your Warehouse and away you go? Firstly because there are various types of companies, and these companies have various information needs brought about by marketing initiatives that govern the applicability or not of historical data. For instance, a highly innovative company that introduces new but dissimilar products onto the market on a irregular basis may not hope to reap any benefit from analyzing their historical data before launching a new product ... Ask Akio Morita of Sony if a Data Warehouse would have resulted in the launch of the Sony Walkman. A relatively new business in what was once considered a traditional market may not have the amounts of historical data to be even ready to begin segmenting the market. A company that follows trends and uses the past to predict the future may not be fully aware that eventually all markets succumb to the eternal constant: change and furthermore, change that cannot be predicted by solely examining the past. A company that trades in a few products may not have enough historical data to be able to identify precisely the type of clients they have. For example, a pet food manufacturer may discover by using a Data Warehouse that most of their clients are furry, have whiskers and like to go out at night. In diverse corporations there are various kinds of needs and wants, from satisfying the monetary needs of the stakeholders to the psychological needs of the CEO. However, most of these companies cannot readily identify certain market need structures just by analyzing operational data. So, don't go looking for the really important information and knowledge in the operational systems, it will not be there. For example: No amount of legacy data will tell the telephone company that their information service is not providing the service levels that the customer requires. No amount of legacy data will serve to tell the telephone company that their operators need a more customer oriented attitude. No amount of legacy data will tell a supplier that they are meeting a need, but badly, and that this puts them in a potentially risky situation with regard to market incursion from new competitors. No amount of legacy data will tell a company why people do or do not buy from them based on the acceptance or not of the company style, culture, personal traits etc. Maybe companies should hire external consulting companies to tell them just what it feels like to deal with them on a potential client basis. Legacy data that tells a company that it's employees work long hours on various projects will not accurately show that a lot of productive time is actually wasted due to informal or spontaneous business practices, nor will it show the amount of quality time that is actually being wasted. Legacy data that is used to support a Marketing organisation, when the marketing organisation is just a glorified PR exercise can't really be seen to be adding a lot of value to business processes. Legacy data may show that a company is performing well relative to the competition, but will probably not show cases where that very same company is performing badly relative to it's potential. Data Warehousing can handle structured data magnificently, but can it really handle the kind of unstructured information that forms the basis of many business processes and information networking and especially strategic decision making processes? Data Mining will only really work if you have hidden information, that is relevant and pertinent to the enterprise, if you don't have the basic material then no amount of digging will reveal the gold that many aspire to. It doesn't matter what you do with lead, you still can't turn it into a precious metal by pure alchemy