Re: Single cpu vp woe
Art S. Kagel wrote:
>
> Andrew, I disagree. NOAGE has nothing to do with allowing IDS to
> take over the "complete allocation of the cpu cycles" of one or more
> processors. The reason for NOAGE is that some OSes (most
> aggressively HPUX and less so Solaris) progressively reduce the OS
> timeslice priority of any process which runs a long time ('long'
> defined by a kernel parameter but typically about an hour IB). For a
> server like IDS which can literally run forever (in hardware speak
> that means until the machine crashes), it means that unless the
> machine is a dedicated server with no other processes running on it,
> IDS will progressively become more and more starved for CPU cycles
> until its oninit processes are rarely in a 'run' state.
>
> Setting NOAGE to 1 notifies the OS that this process is SUPPOSED to
> run for a long time and is not just a runaway or resource hog.
> With NOAGE set to 1 the OS will not reduce the initial priority of
> the oninit processes that make up the server and performance will be
> consistently good over time.
We're in the same ballpark, but my understanding does not match yours.
NOAGE is designed to disable the "niceness" priority aging which penalises
cpu hogs according to their recent cpu usage history. The effect of
priority aging through niceness performs a kind of moving average of
activity only over the last several seconds, and is dependent on total
actvity amongst all the processes.
If a process has NOAGE set, then it's priority is never affected by it's
recent history of activity. Such a process would always get a full
timeslice at all times whenever it was scheduled in, and it will always be
considered a prime candidate for scheduling since it's priorit would never
be downgraded due to it's activity.
In combination with multiple CPU's and process affinity, it's quite
effective for giving a cpuvp virtually 100% ownership of a particular CPU.
Other competing or I/O bound processes tend to drift off to the cpus that
have not been affined with a NOAGE process. Hence, 4GL, java, apache, etc
processes tend to be found on the "spare" cpu that you have graciously
left for other processes.
Viewing the output of the top command can easily demonstrate this effect
on a busy well-balanced multiprocessor machine running a busy IDS. I have
customers, so I sometimes have to deal with a concerned sysadmin who calls
up worried that "oninit is taking 100% of two cpus! Should I kill it?" and
I've found the most effective remedy to this threat is to exclaim with
sickening enthusiasm: "Oh that's fantastic! It means the machine is very
well configured and balanced! Beautiful!" etc etc etc. Since you are your
own customer, you probably don't see that, except maybe when you get a new
jr. admin on staff?
Switching off NOAGE on such a system generally results in removing oninit
from domination, and the difference in performance is quite noticable. The
ability of NOAGE to cause cpu hogging is why it can be so crippling on a
single CPU machine. Once a machine gets busy with database activity, I
almost always see the oninit processes achieve almost 100% domination of
their allocated cpus.
Andrew Hamm wrote:
> Art S. Kagel wrote:
>
>>Andrew, I disagree. NOAGE has nothing to do with allowing IDS to
>>take over the "complete allocation of the cpu cycles" of one or more
>>processors. The reason for NOAGE is that some OSes (most
>>aggressively HPUX and less so Solaris) progressively reduce the OS
>>timeslice priority of any process which runs a long time ('long'
>>defined by a kernel parameter but typically about an hour IB). For a
>>server like IDS which can literally run forever (in hardware speak
>>that means until the machine crashes), it means that unless the
>>machine is a dedicated server with no other processes running on it,
>>IDS will progressively become more and more starved for CPU cycles
>>until its oninit processes are rarely in a 'run' state.
>>
>>Setting NOAGE to 1 notifies the OS that this process is SUPPOSED to
>>run for a long time and is not just a runaway or resource hog.
>>With NOAGE set to 1 the OS will not reduce the initial priority of
>>the oninit processes that make up the server and performance will be
>>consistently good over time.
>
>
> We're in the same ballpark, but my understanding does not match yours.
> NOAGE is designed to disable the "niceness" priority aging which penalises
> cpu hogs according to their recent cpu usage history. The effect of
> priority aging through niceness performs a kind of moving average of
> activity only over the last several seconds, and is dependent on total
> actvity amongst all the processes.
>
> If a process has NOAGE set, then it's priority is never affected by it's
> recent history of activity. Such a process would always get a full
> timeslice at all times whenever it was scheduled in, and it will always be
> considered a prime candidate for scheduling since it's priorit would never
> be downgraded due to it's activity.
>
> In combination with multiple CPU's and process affinity, it's quite
> effective for giving a cpuvp virtually 100% ownership of a particular CPU.
> Other competing or I/O bound processes tend to drift off to the cpus that
> have not been affined with a NOAGE process. Hence, 4GL, java, apache, etc
> processes tend to be found on the "spare" cpu that you have graciously
> left for other processes.
>
> Viewing the output of the top command can easily demonstrate this effect
> on a busy well-balanced multiprocessor machine running a busy IDS. I have
> customers, so I sometimes have to deal with a concerned sysadmin who calls
> up worried that "oninit is taking 100% of two cpus! Should I kill it?" and
> I've found the most effective remedy to this threat is to exclaim with
> sickening enthusiasm: "Oh that's fantastic! It means the machine is very
> well configured and balanced! Beautiful!" etc etc etc. Since you are your
> own customer, you probably don't see that, except maybe when you get a new
> jr. admin on staff?
>
> Switching off NOAGE on such a system generally results in removing oninit
> from domination, and the difference in performance is quite noticable. The
> ability of NOAGE to cause cpu hogging is why it can be so crippling on a
> single CPU machine. Once a machine gets busy with database activity, I
> almost always see the oninit processes achieve almost 100% domination of
> their allocated cpus.
>
>
Andrew, that will only seem to happen when the other activity on the system
is not CPU intensive, which is quite common. Most business applications are
IO bound not CPU bound. We don't see it on our systems because out servers,
much to my chagrin, are sharing the machine with the entire Bloomberg
environment which includes our homegrown global IPC mechanism, Bloomberg
developed database servers, middleware servers, and application servers all
of which are CPU bound. What we see are the oninits spiking into the 90%+
zone and dropping down when not needed.
Art S. Kagel
Art S. Kagel wrote: >> > Andrew, that will only seem to happen when the other activity on the > system is not CPU intensive, which is quite common. Most business > applications are IO bound not CPU bound. very true, and in the systems we play on, the competing processes are 4GL so they generally yield to wait on results from the engine. This is always the perspective I come from. I was going to ask yours, and from what I know of your system, I expected you to say that it's virtually dedicated DSS, so this posting of yours is quite a surprise actually. Still, without NOAGE my type of system somehow manages to starve cpuvps even when the competing processes are only 4GL and the usual standard O/S processes. I'm guessing that Neil's customer is similar. > We don't see it on our > systems because out servers, much to my chagrin, are sharing the > machine with the entire Bloomberg environment which includes our > homegrown global IPC mechanism, Bloomberg developed database servers, > middleware servers, and application servers all of which are CPU > bound. What we see are the oninits spiking into the 90%+ zone and > dropping down when not needed. and you still use NOAGE, or not, to try to gain some edge? The next step is to explicitly mess with the priority settings to give it extra oomph - if you're allowed to of course :-) I'd be interested to see how long the cpuvp queues are during your spikes, and also the quiet periods. Anyway, in summary, I'm not surprised that Neil's customer is finding better performance with more than one cpuvp on a single processor box. I use that config as a legitimate design. Setting three seems to max out the effects, and is borderline compared to setting two. Trial and error.