Re: VLDB - for real?

Ok, > or > > > is Jack Parker, > > is Bill Williams... > > > I've noted of late a trend amongst the industry analysts claiming that > > > very large databases are coming. > > > > [useful things clipped] > > > > > I see this as a problem. IMHO a pedabyte database means that someone hasn't > > > done their analysis work very well, or it's something that's chock full of > > > BLOBs. Either way, maybe the Fortune 100 will be worrying about databases > > > of this size, but how about a little thought here... > > > > The regulatory climate toward businesses is bad and getting worse all > > the time. The length of time which corporations are required to keep > > information on file ranges from seven to twenty years. Is it any wonder > > that many of them are going to imaging in order to reduce the number and > > amount of paper which would otherwise have to be kept on file? > > Surely there is no government requirement that historical information be > accessible online? Could not this need be as easily met with an optical > library with a good index? I've been working with BancTec as the architect of a large-scale archiving system of the sort you're describing. Specifically, we're using optical jukeboxes and the like to store massive amounts of imaged information. One of the most compelling reasons that BancTec's customers need such enormous amounts of data stored is that their governments and customers can demand documents on short notice. > What exactly are you storing? Not that you are doing anything wrong - how > should I know - but I can easily see corporations printing reports, form > letters or what not and then scanning it in. Au contraire, mon ami! Many of BancTec's customers are banks. Banks which compete with one another on customer service. Such as responsiveness. Now, consider, if your customers perform 16 million transactions *per day*, including 1 million pieces of paper (checks, deposit slips, etc.), and you want to be able to quickly verify for a customer that they did indeed write the check for $500, not $50, how do you efficiently organize the information? I'm not convinced that Informix et al. can ever implement a truly relational pedabyte database, but COTS is generally easier to sell and support than custom software. The market analysts have noticed that there is big money in VLDBs, and they assume (probably correctly) that Informix, Oracle, and co. will want some of it. Does that mean that the smaller users will suffer? Quite possibly. > Is the data being scanned/stored > in the proper format? Are they spreadsheets that can be more efficiently > stored in an exported form? If they are incoming correspondence, what > happened to OCR? One of BancTec's OpenArchive customers is storing loan-related information. If you have all of the paperwork associated with your last home mortgage, take a look at it. How many of those papers are actually forms? Are they in a font supported by OCR, or have they been repeatedly photocopied to the point of illegibility? As for those that can be OCRed effectively, ask yourself if a court of law would accept an ASCII rendering of your letter confirming the mortgage rate. Yes, we both know that an image can be modified, but paper can be forged, microfilm can be altered, etc. Bottom line: The courts are more accepting of images than they are of raw data. That means lots and lots and lots of BLOBs. > Next question, roughly where does (or would) your client rank in the > Fortune index? You'd be surprised. Many of BancTec's customers are international, and in that context barely register a blip on the Fortune scale. The world is getting smaller, and the markets are getting larger. Your local bank (say 15-20 branches?) may process well over 100,000 transactions per day. At seven years (for US auditing requirements), that's 175,000,000 transactions. And if you have to store images for each transaction, you're probably over 3 terabytes for the life of the archive. Governments generate even more paper, and often have more money. Ironically, I think VLDBs may be coming, but I think they'll be short-lived. The industries that are in such desperate need for VLDBs (banking, real estate, government) have been some of the least efficient industries with regard to adopting and utilizing new technology. In other words, they're using technology to manage their paper, because they haven't learned to use technology to *reduce* their paper. Eventually, the bean-counters will discover that they can cut costs dramatically by eliminating paper, and then the driving requirements for VLDBs will drop to a more manageable level. Sure, the number of records will still be astronomical, but I think there'll be fewer images because there'll be less paper to begin with. So whaddaya think now, Jack? -- Jonathan Crawford Ark Technologies, Inc.