Re: TPC-A and -B Benchmarks

In <21fm68INNjuq@emory.mathcs.emory.edu> inf_bb@hermes1.sps.mot.com (informix users bb) writes: >even if oracle publishes a tpc-c result - can we trust it after the >recent tpc-b fiasco? oracle apparently wrote a "special" version of >the dbms to score high on this test (not a new concept, oracle just >got caught at it). This (rather lucid, I thought) posting on the subject did not get cross-posted here, but i thought you (and others) might find it enlightening, Bob. |In <1993Jul8.143203.8858@jac.nuo.dec.com> berenson@nova.enet.dec.com (Hal Berenson) writes: | | |>In article <1993Jul3.021408.10121@qiclab.scn.rain.com>, tcox@qiclab.scn.rain.com (Thomas Cox) writes... |>>timo@mits.mdata.fi (Timo Talasmaa) writes: |>>2. There appears to be no doubt that all benchmarks of Oracle7 have been |>> completely within the letter of the TPC (Transaction Processing |>> Council) rules, and all have been audited. This is in contrast to |>> DEC's RDB benchmark not long ago that was *withdrawn* for *not* |>> following the rules. | |>Excuse me, but that's bull crap. There was one particular benchmark for |>Rdb where the TPC changed the rules (clarified a technicality) after the |>benchmark was published. The benchmark configuration was obsolete by |>that time, so rather then correct the technicality and spend $10s of |>thousands to re-run and re-audit that particular test it was simply |>withdrawn. This happened SEVERAL years ago by the way. Several other |>vendor tests have been withdrawn as a result of technical rulings, or of |>the vendor discovering a mistake in the benchmark results. | |>Challenges to TPC results, or to their misrepresentation by vendors, |>occur all the time. The TPC has mechanisms for resolving these disputes |>internally and so far, that I can remember, no public censoring or |>ejection from TPC has been required. Just because nothing has reached |>the general public, don't simply assume your favorite vendor is |>guilt-free. | |>The Oracle7 results (and all currently published results) are completely |>valid within TPC rules. Extensive study was done by several parties to |>find a violation of the rules, and the one challenge made was rejected. |>I don't believe that's the issue people should be focusing on. | |>There are two relevent issues here: | |>1) Should there be some mechanism in the TPC process for rejecting |>"benchmark special" features? On the surface the easy answer is yes, but |>as you study the problem you discover "benchmark specials" are in the eye |>of the beholder. Rdb has many tuning features that are designed to help |>application designers optimize for their particular environment. Some of |>these features apply only to high-volume update intensive processing. |>Are they benchmark specials? Hardly. Then why should Oracle7's |>"Discrete Transactions" be considered a benchmark special? | |>The uproar over discrete transactions is that they severely restrict the |>usage of SQL within the transaction and that they severely mess up |>Oracle's standard concurrency control mechanisms (thus making it unlikely |>discrete transactions can be used in any application with substantially |>more complex queries then TPC-A). To a large extent the objection to |>discrete transactions is not based on the validity of the feature, but |>rather on how un-Oracle-like they are and therefor how unlikely it is |>that an Oracle customer could or would use them! | |>So the problem with a "no benchmark special" rule is that determining |>what is or isn't a benchmark special is very subjective. How would a |>vendor prove that a feature wasn't a benchmark special? Would they have |>to produce a customer who would testify they are using the feature? |>Would one be enough? Two? Ten? Should the TPC provide the exact |>program to be run, thus forcing the use of standard SQL syntax and |>preventing the use of vendor-specific features? Would this really |>provide a useful comparison, since customers DO use many non-standard |>features, or would it be just another useless data point? | |>The TPC does have a very strict definition of what the benchmark can and |>can not do. That definition exists at a fairly high level (essentially a |>detailed functional specification) so that any software system on any |>hardware platform can run the benchmark. So, it is possible using TPC |>benchmarks to compare a TPF2 system with the application implemented in |>IBM 370 assembly language accessing a private file structure with |>application implemented recovery mechanisms to a Compaq PC running SCO |>UNIX and ORACLE with the application written in C calling PL/SQL |>procedures. You can also use TPC benchmarks to evaluate the |>suitability of OODB products for TP applications, which you couldn't do |>if the TPC provided an ISO SQL program as the only valid benchmark |>program. | |>2) The real point, the one Standish made but more or less seems to be |>getting lost in the noise between them and Oracle, is that the TPC-A and |>TPC-B benchmarks are obsolete as comparators of database and transaction |>processing systems. When a benchmark is created all vendors will try to |>optimize for that benchmark. At some point the vendors will "crack" the |>benchmark by tuning their systems to the point that the benchmark (a) no |>longer demonstrates user-useful differences between products and/or (b) |>no longer can be extrapolated to predict the performance of customer |>applications. TPC-A and TPC-B have more or less reached this point. |>This doesn't make them bad benchmarks nor does it make vendors who bent |>over backwards to "win" these benchmarks unethical. TPC-A&B have caused |>about an order of magnitude improvement in simple-query pathlength, |>recovery algorithms, and I/O since 1987. Nearly every customer |>application had benefited dramatically from the TPC-A&B benchmark wars. |>1993 pretty much marks the end of those customer benefits from TPC-A&B. |>The TPC-C benchmark not only is representative of a much broader set of |>applications then TPC-A&B, it poses a host of new problems demanding |>vendor attention and optimization. TPC-A&B should be retired as obsolete |>and both customers and vendors should focus on TPC-C (and the soon to |>appear TPC-D). TPC-C&D will provide more useful information to customers, |>and over the next 2-4 years will drive dramatic additional improvements |>in database system performance. |>............................................................................. | |>Hal Berenson | |>Home: 71640.3535@compuserve.com |>Work: berenson@nova.enet.dec.com | |>-- Disclaimer: Opinions expressed here are my own and should not be |>construed as representing the views of my employer or its employees, |>officers, directors, or stockholders. -- graeme -- Disclaimer: The author's opinions are his own, and not necessarily those of Pyramid Technology Ltd. or Pyramid Technology Inc. --------------------------------------------------------------------------- -m------- Graeme Sargent Voice: +44 (0)252 373035 ---mmm----- Senior Database Consultant Fax : +44 (0)252 373135 -----mmmmm--- Pyramid Techno