Tech Tip du Jour: Benchmarking
Not a problem report but a discussion of a blog post advocating that DBAs establish baseline benchmarks — timing a set of common queries (plus ODBC/JDBC network-based tests) regularly — so that vague user claims like "the system is slower than last week" can be checked objectively. Fernando Nunes argued users' perceptions usually signal something real and that activity metrics (onstat/sysmaster counters: disk reads, scans, lock waits) should also be tracked; the author replied that elapsed time is what management understands and both have their place. Ian Gumby suggested JUnit or ESQL/C test harnesses logging results to a database over time. No single conclusion beyond general agreement that baselining is worthwhile.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
http://obotheclown.blogspot.com/2008/07/tech-tip-du-jour-benchmarking.html -- Bye now, Obnoxio http://obotheclown.blogspot.com/
Obnoxio The Clown wrote: > http://obotheclown.blogspot.com/2008/07/tech-tip-du-jour-benchmarking.html > > I would not disagree with the points you make, but one particular point almost makes me angry... Let me quote you: "the system is slower than it was last week", can you refute them? Why the hell should we refute users claims? This reminds me of my "daily wars" with people from all areas (DB, developers, OS, network, SANs etc). Everybody's concern is to make clear that it's not their fault, when everyone should provide information that would make it easier to find out what is causing the issues... I strongly believe that the users don't complaint without a reason... So what we should "benchmark" are activity metrics... disk reads, full scans etc. On IDS 11.x it's easier to collect them, but it's relatively easy to collect them in any informix version. Some simple onstats or sysmaster queries can be very valuable... Regards,
Fernando Nunes said:
> Obnoxio The Clown wrote:
>> http://obotheclown.blogspot.com/2008/07/tech-tip-du-jour-benchmarking.html
>>
>>
>
> I would not disagree with the points you make, but one particular point
> almost
> makes me angry... Let me quote you:
>
> "the system is slower than it was last week", can you refute them?
>
> Why the hell should we refute users claims?
Mostly, because they're talking out of their arseholes. You have to
understand this is only my experience, but after twenty-five years of
doing this for a living, I can count the number of genuine instances of
the system actually being demonstrably slower on the fingers of one hand.
By contrast, I get told that "the system is slower" about once a week. So,
on average, one out of every 260 times I get told the system is slower, it
actually IS slower. The other 259 times, the user is talking shit.
YAMMV, naturally.
> This reminds me of my "daily
> wars"
> with people from all areas (DB, developers, OS, network, SANs etc).
> Everybody's concern is to make clear that it's not their fault, when
> everyone
> should provide information that would make it easier to find out what is
> causing the issues...
> I strongly believe that the users don't complaint without a reason... So
Must be a cultural thing, and I think you're the lucky one. I've had
lusers complain at me on three continents, and they were baseless
complaints 99.6% of the time.
> what
> we should "benchmark" are activity metrics... disk reads, full scans etc.
> On IDS 11.x it's easier to collect them, but it's relatively easy to
> collect
> them in any informix version. Some simple onstats or sysmaster queries can
> be
> very valuable...
Well, I think I highlighted that in my example: I exonerated the database
and the network by demonstrating something that even a CIO can understand:
how long it takes to do a repeatable operation. This was important for
finding the root cause.
onstat information might mean something to you and me, but it don't mean
shit to a CIO. Time is something everyone can understand. Both things have
their place.
--
Bye now,
Obnoxio
http://obotheclown.blogspot.com/
Obnoxio The Clown wrote:
> Fernando Nunes said:
>> Obnoxio The Clown wrote:
>>> http://obotheclown.blogspot.com/2008/07/tech-tip-du-jour-benchmarking.html
>>>
>>>
>> I would not disagree with the points you make, but one particular point
>> almost
>> makes me angry... Let me quote you:
>>
>> "the system is slower than it was last week", can you refute them?
>>
>> Why the hell should we refute users claims?
>
> Mostly, because they're talking out of their arseholes. You have to
> understand this is only my experience, but after twenty-five years of
> doing this for a living, I can count the number of genuine instances of
> the system actually being demonstrably slower on the fingers of one hand.
>
> By contrast, I get told that "the system is slower" about once a week. So,
> on average, one out of every 260 times I get told the system is slower, it
> actually IS slower. The other 259 times, the user is talking shit.
>
How many of those times were they simply waiting on a lock? :)
I didn't mean the users are right about the DB being slower... I meant that
their perception is important... And usually reveals something...
But we're both talking about our experience...
> Must be a cultural thing, and I think you're the lucky one. I've had
> lusers complain at me on three continents, and they were baseless
> complaints 99.6% of the time.
>
Most of the times I could show them that some query was running within the same
time as before... But again, it doesn't mean they didn't perceived some
response slowdown...
>> what
>> we should "benchmark" are activity metrics... disk reads, full scans etc.
>> On IDS 11.x it's easier to collect them, but it's relatively easy to
>> collect
>> them in any informix version. Some simple onstats or sysmaster queries can
>> be
>> very valuable...
>
> Well, I think I highlighted that in my example: I exonerated the database
> and the network by demonstrating something that even a CIO can understand:
> how long it takes to do a repeatable operation. This was important for
> finding the root cause.
>
> onstat information might mean something to you and me, but it don't mean
> shit to a CIO. Time is something everyone can understand. Both things have
> their place.
I agree... the art lies in transforming raw data into flashy graphics :)
Regards
On Jul 20, 6:41 pm, Fernando Nunes <domusonl...@gmail.com> wrote: > Obnoxio The Clown wrote: [SNIP} I wouldn't rely on OAT. And I would recommend looking at the customer's development environment. If they develop in java, then write a series of JUnit tests that can be launched hourly, nightly, etc. You can then dump the output to a database and then track the results over time. If they are not a java shop, then you could do the same in stand alone C/C++/ESQL/C programs that do the same thing. You're going to want to track the number of records currently in your system, and choose records of live data at random. Of course, the tests will vary based on your data, security, and what you're trying to "benchmark". This works well on single table queries. Its when you want to test complex table joins, then you have a bit more work to do. Perhaps a modules where you could pass in the query that you want benchmarkes? ;-) There's more too it, but hey! This should point you down the right track. BTW, I think there may be some generic tools to do this. I don't know if they work against an IDS engine. Perhaps IBM's IDS marketing team may want to reach out and see why there isn't a port ?
Ian Michael Gumby said: > On Jul 20, 6:41 pm, Fernando Nunes <domusonl...@gmail.com> wrote: >> Obnoxio The Clown wrote: > [SNIP} > > I wouldn't rely on OAT. Wouldn't you. Thanks for that. > And I would recommend looking at the customer's development > environment. > If they develop in java, then write a series of JUnit tests that can > be launched hourly, nightly, etc. > You can then dump the output to a database and then track the results > over time. > > If they are not a java shop, then you could do the same in stand alone > C/C++/ESQL/C programs that do the same thing. I said: "Do your users use ODBC or JDBC? Make sure you have a full suite of network-based tests as well." > You're going to want to track the number of records currently in your > system, and choose records of live data at random. > Of course, the tests will vary based on your data, security, and what > you're trying to "benchmark". This works well on single table > queries. > Its when you want to test complex table joins, then you have a bit > more work to do. Perhaps a modules where you could pass in the query > that you want benchmarkes? ;-) I said: "take a number of your most common queries" -- nothing there about single tables. Perhaps you might want to read what's written before rushing off to tell us that you're going to point us down the right track. :op -- Bye now, Obnoxio http://obotheclown.blogspot.com/
Obnoxio The Clown wrote: > http://obotheclown.blogspot.com/2008/07/tech-tip-du-jour-benchmarking.html Keep this up and you'll start to look rational. Do you really want to let that happen? Well written. -- Daniel A. Morgan University of Washington damorgan@x.washington.edu (replace x with u to respond)