Re: VLDB - for real?

(JP) > I've noted of late a trend amongst the industry analysts claiming that > very large databases are coming. > I see this as a problem. IMHO a pedabyte database means that someone hasn't > done their analysis work very well, or it's something that's chock full of > BLOBs. Either way, maybe the Fortune 100 will be worrying about databases > of this size, but how about a little thought here... > > > (BW) > The regulatory climate toward businesses is bad and getting worse all > the time. The length of time which corporations are required to keep > information on file ranges from seven to twenty years. Is it any wonder > that many of them are going to imaging in order to reduce the number and > amount of paper which would otherwise have to be kept on file? > (JP) > Surely there is no government requirement that historical information be > accessible online? Could not this need be as easily met with an optical > library with a good index? (JC) >I've been working with BancTec as the architect of a large-scale > archiving system of the sort you're describing. Specifically, we're using > optical jukeboxes and the like to store massive amounts of imaged > information. One of the most compelling reasons that BancTec's > customers need such enormous amounts of data stored is that their > governments and customers can demand documents on short notice. ok, how long does it take with a good index and a jukebox to find the right image and print it out? I'd guess not long. Ok, in the online world (with a little 'o') 60 seconds or so may be poor performance - but this sounds like Online 7.x-optical. (JP) > > What exactly are you storing? Not that you are doing anything wrong - how > > should I know - but I can easily see corporations printing reports, form > > letters or what not and then scanning it in. > (JC) > Au contraire, mon ami! Many of BancTec's customers are banks. Banks which > compete with one another on customer service. Such as responsiveness. > Now, consider, if your customers perform 16 million transactions *per day*, > including 1 million pieces of paper (checks, deposit slips, etc.), and > you want to be able to quickly verify for a customer that they did indeed > write the check for $500, not $50, how do you efficiently organize the > information? You mean like AMEX does? They send you an image of each transaction on your bill. Granted its shrunk down a little so they fit 8 or so on a page, but somewhat useful for jogging your memory. I'm not going to store images for my customers for 7 years, they won't care beyond 90 days if that. (JP) > > Is the data being scanned/stored > > in the proper format? Are they spreadsheets that can be more efficiently > > stored in an exported form? If they are incoming correspondence, what > > happened to OCR? > (JC) > One of BancTec's OpenArchive customers is storing loan-related information. > If you have all of the paperwork associated with your last home mortgage, > take a look at it. How many of those papers are actually forms? Are they > in a font supported by OCR, or have they been repeatedly photocopied to the > point of illegibility? As for those that can be OCRed effectively, ask > yourself if a court of law would accept an ASCII rendering of your letter > confirming the mortgage rate. Yes, we both know that an image can be > modified, but paper can be forged, microfilm can be altered, etc. Bottom > line: The courts are more accepting of images than they are of raw data. > That means lots and lots and lots of BLOBs. > (I yield on OCR - never that strong anyway) The last couple of times I went for a loan (we played musical mortgages for a couple of years while remodeling), all of the paperwork DATA was stored electronically, the only part which was handwritten were the signatures and notary seals. Electronic signatures sound a lot cheaper than disk farms. We worked with Seafirst, USBank, and one or fourteen others (I forget half of them). (JP) roughly where does (or would) your client rank in the Fortune index? (JC) > You'd be surprised. Many of BancTec's customers are international, and > in that context barely register a blip on the Fortune scale. The world > is getting smaller, and the markets are getting larger. Your local > bank (say 15-20 branches?) may process well over 100,000 transactions per > day. At seven years (for US auditing requirements), that's 175,000,000 > transactions. And if you have to store images for each transaction, you're > probably over 3 terabytes for the life of the archive. Governments generate > even more paper, and often have more money. Ok, so it's the local banks too - of course these are disappearing rapidly. > > Ironically, I think VLDBs may be coming, but I think they'll be short-lived. > The industries that are in such desperate need for VLDBs (banking, real > estate, government) have been some of the least efficient industries > with regard to adopting and utilizing new technology. In other words, > they're using technology to manage their paper, because they haven't > learned to use technology to *reduce* their paper. Eventually, the > bean-counters will discover that they can cut costs dramatically by > eliminating paper, and then the driving requirements for VLDBs will > drop to a more manageable level. Sure, the number of records will still > be astronomical, but I think there'll be fewer images because there'll be > less paper to begin with. > Good point, with the increase in electronic transactions (internet shopping, credit/debit card, ATM etc.) the day of the paper transaction is going. So knowing this as a bank CEO, I'm going to justify an enormous outlay for bleeding edge software and expensive HW? Not unless my salesman moonlights selling snake oil. If anything, I'm reading of a need for more efficient image storage and retrieval - VLIB vs VLDB if you will. (Hey wait a minute, I just coined a phrase, I hereby copywright it - anybody using the phrase VLIB immediately owes my $20.38 - oops sorry - wrong newsgroup). (JC) > So whaddaya think now, Jack? I think I'll have another one. cheers j. _____________________________________________________________________________ Jack Parker - Hewlett Packard, BSMC Boise, Idaho, USA jparker@hpbs3645.boi.hp.com _____________________________________________________________________________ Back up my hard drive? You mean these things can actually run in reverse? _____________________________________________________________________________ Any opinions expressed herein are my own and not those of my employers. _____________________________________________________________________________