RE: Webcast - Rapid Data Warehouse Development - Build a Pilot in 8 weeks!

I really think that its appropriate to clarify this point. There's this company, that, I know... they are an Oracle shop. They have lots and lots of data. Oracle's RAC couldn't handle it. IBM's Global Services, or whatever they are calling themselves these days, came in, couldn't get the job done. So they went looking and saw that Yahoo!, Twitter, Facebook, etc ... were all using this Hadoop framework from Apache. (Pig, Hive, HBase, etc ...) all on top of a distributed 'cloud' file system (HDFS). So the software is 'free' and you essentially build out a 'cloud' of servers using off the shelf components. So your nodes in your cluster cost around $5K. (Round number. YMMV). On a single 2 socket, 8 core Xeon system, I believe Oracle, DB2 and IDS would cost around $100,000. (Enterprise editions.) I'm guessing $25K a core where the core multiplier is .5. Of course YMMV based on discounts. So for the cost of a traditional RDBMS, licenses alone, you can build a cloud of 20 machines. If you're building this from scratch, you can build 10 nodes w 10GBe and a 10GBe switch. 10 nodes ~40TB of space. (Which means you'll get ~35TB of raw disk for a storage space) The point is that you can build out a functional data warehouse, data mart for the cost of 1 traditional RDBMS machine. Note: You will still want some RDBMS for systems of records and cloud computing shouldn't be considered as a replacement for all of your traditional RDBMS systems. Just that for analytics and DW work, its a very compelling and cost efficient way of handling massive amounts of data. Also, this system will scale. In a presentation, I saw a slide where in 2007, Google processed 30PB (that's P as in 'Petabytes') a month. One caveat. This is leading edge technology. If there wasn't Yahoo! and others using it on a large scale, I'd be really skeptical. I'll be honest, there's a lot of room for improvement, but it does work. For those IDS DBAs who have idle time on their hands because IDS runs so well... I'd check it out. But hey! What do I know? ;-) -G PS. Hey Jerry K. How dem numbers looking? ;-) From: im_gumby@hotmail.com To: lester@iiug.org Subject: RE: Webcast - Rapid Data Warehouse Development - Build a Pilot in 8 weeks! Date: Wed, 27 Jan 2010 11:30:07 +0000 CC: informix-list@iiug.org Its a very good quote and its very appropriate. The key point is that there's a trend now to take advantage of the horsepower in today's systems and networks. The 'bleeding' edge in traditional databases is to create an 'appliance' where the database code is put in to FPGAs. If Sun was still a leader, they could have put a JVM on a FPGA to boost performance. So the idea isn't new, but as FPGAs drop in price and the development tools improve... you will see more appliances. Netezza is Postgres, Oracle has their Exadata. IBM's? Well I'm sure someone is thinking about DB2... :-O The real issue is that there are companies that will scoff and say that taking 8 weeks to build a data mart or data warehouse is way too long. They can be built rather quickly. > Date: Tue, 26 Jan 2010 10:40:23 -0500 > From: lester@iiug.org > To: im_gumby@hotmail.com > CC: informix-list@iiug.org > Subject: Re: Webcast - Rapid Data Warehouse Development - Build a Pilot in 8 weeks! > > Gumby, > > Good quote thanks 'One man's garbage is another man's treasure.' I will use it today in the webcast. I have a great image of a junk yard to go with it;) > > Regards - Lester > > Ian Michael Gumby wrote: > > For some reason, Windows 7 barf'd on my post. (And I know OTC will have > > some smart comment. :-P ) > > > > Just in case the post went out, I'll only respond with a very short comment. > > > > Lester, here's a quote you can attribute to moi.... > > > > "...Remember that there is this saying ... 'One man's garbage is another > > man's treasure.' If you think real hard, you might understand why Google > > is so dangerous..." > > > > -G > > > > > > > Date: Mon, 18 Jan 2010 22:08:06 -0500 > > > From: lester@iiug.org > > > To: im_gumby@hotmail.com > > > CC: obnoxio@serendipita.com; informix-list@iiug.org > > > Subject: Re: Webcast - Rapid Data Warehouse Development - Build a > > Pilot in 8 weeks! > > > > > > Gumby, > > > > > > You are still missing the point, having a couple of petabytes of data > > on fast hardware and software does not make a data warehouse. Most of > > the time it ends up being a data junk yard. > > > > > > The point is how do you quickly design and build something so users > > can gain insight into their data and make meaningful decisions. > > > > > > Regards - Lester > > > > > > Ian Michael Gumby wrote: > > > > Well... > > > > Glad you asked. :-) > > > > > > > > You can use all of the above, or none of above. (Ok, really below, but > > > > you know what I meant.) > > > > > > > > You can write a map/reduce, or series of map/reduce(s) and output to a > > > > flat file, or even to a relational database. > > > > > > > > Then there's pig, but I haven't had time to play with it yet. And > > > > there's now a JDBC conectivity, which I haven't have time to play with > > > > yet, either. > > > > > > > > The point is that you can build data warehouses, data marts, etc to > > > > handle large amounts of data fairly quickly using non traditional > > means. > > > > > > > > Now I do see issues with Hadoop, but I think that over time, it > > will get > > > > better. > > > > > > > > Don't take my word for it, Accenture has already started their own > > cloud > > > > computing practice. Sun's released their vision and well, then there's > > > > IBM.... > > > > > > > > ;-) > > > > > > > > -G > > > > > > > > PS. Here's some food for thought... Back in the 90's Informix's XPS > > had > > > > customers who created 27TB data warehouses. If we rely on Moore's law, > > > > that database would probably be comparable to one of ~2.7 > > petabytes. ;-) > > > > Can DB2 do that? > > > > > > > > > > > > > Date: Tue, 19 Jan 2010 00:29:25 +0000 > > > > > From: obnoxio@serendipita.com > > > > > CC: informix-list@iiug.org > > > > > Subject: Re: Webcast - Rapid Data Warehouse Development - Build a > > > > Pilot in 8 weeks! > > > > > > > > > > Ian Michael Gumby wrote: > > > > > > But I can build a cluster of Hadoop/HBase machines and set up a > > DW in > > > > > > less than 8 days! (Ok, assuming that the machines are already > > built > > > > and > > > > > > are already networked...) > > > > > > > > > > And I can do that in half a day. So fucking what? > > > > > > > > > > And once you have those servers up and running, what are you going to > > > > > use to query them? Cognos? Brio? Business Objects? Crystal Reports? > > > > > > > > > > -- > > > > > Cheers, > > > > > Obnoxio The Clown > > > > > > > > > > http://obotheclown.blogspot.com > > > > > I will now proceed to pleasure myself with this fish. > > > > > > > > > > -- > > > > > This message has been scanned for viruses and > > > > > dangerous content by OpenProtect(http://www.openprotect.com), and is > > > > > believed to be clean. > > > > >