ontape IO priority

Hello, We have a system that is very busy and as such there is not much disk IO headroom left during normal operation. This is ok for now until new servers can be installed etc with greater IO capacity. However when doing ontape backups, the system becomes very slow for the users. During this time using iostat we can see the IO queue size increasing. We have tried running ontape | gzip with ionice -c3 etc which causes gzip to run with the 'idle' IO priority, but I see that after starting ontape, the work is handed to one of the oninit processes which have a 'best effort class 2' priority which is too high for backup IO. Is there a way of prioritising the IO load imposed when running ontape? The users need the system to be responsive, and it doesn't matter if the backups take a long time to complete. Thanks in advance. Cheers, Andy.

There is nothing you can do to reduce the IO demands that an archive will put on the system. As you point out, it is the oninit process that actually performs the reads on the chunks, the ontape is just taking the output from the archive thread in the engine and writing it out to the back up device or stream. One thing you can do is to eliminate the gzip if you can. Gzip will use 100% of a processor core and that will certainly affect your instance's performance outside of the IO load. Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Fri, Jul 22, 2011 at 9:00 AM, ANDREW LEMIN <a_lemin@hotmail.com> wrote: > Hello, > > We have a system that is very busy and as such there is not much disk IO > headroom left during normal operation. > > This is ok for now until new servers can be installed etc with greater IO > capacity. However when doing ontape backups, the system becomes very slow > for > the users. During this time using iostat we can see the IO queue size > increasing. > > We have tried running ontape | gzip with ionice -c3 etc which causes gzip > to > run with the 'idle' IO priority, but I see that after starting ontape, the > work is handed to one of the oninit processes which have a 'best effort > class > 2' priority which is too high for backup IO. > > Is there a way of prioritising the IO load imposed when running ontape? The > users need the system to be responsive, and it doesn't matter if the > backups > take a long time to complete. > > Thanks in advance. > Cheers, Andy. > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --20cf307d00f05ddcd504a8a8590f

If this is a Linux/Unix OS, would lowering ontape's priority with "nice" help? On 7/22/2011 9:21 AM, Art Kagel wrote: > There is nothing you can do to reduce the IO demands that an archive will > put on the system. As you point out, it is the oninit process that actually > performs the reads on the chunks, the ontape is just taking the output from > the archive thread in the engine and writing it out to the back up device or > stream. One thing you can do is to eliminate the gzip if you can. Gzip > will use 100% of a processor core and that will certainly affect your > instance's performance outside of the IO load. > > Art > > Art S. Kagel > Advanced DataTools (www.advancedatatools.com) > Blog: http://informix-myview.blogspot.com/ > > Disclaimer: Please keep in mind that my own opinions are my own opinions and > do not reflect on my employer, Advanced DataTools, the IIUG, nor any other > organization with which I am associated either explicitly, implicitly, or by > inference. Neither do those opinions reflect those of other individuals > affiliated with any entity with which I am affiliated nor those of the > entities themselves. > > On Fri, Jul 22, 2011 at 9:00 AM, ANDREW LEMIN<a_lemin@hotmail.com> wrote: > >> Hello, >> >> We have a system that is very busy and as such there is not much disk IO >> headroom left during normal operation. >> >> This is ok for now until new servers can be installed etc with greater IO >> capacity. However when doing ontape backups, the system becomes very slow >> for >> the users. During this time using iostat we can see the IO queue size >> increasing. >> >> We have tried running ontape | gzip with ionice -c3 etc which causes gzip >> to >> run with the 'idle' IO priority, but I see that after starting ontape, the >> work is handed to one of the oninit processes which have a 'best effort >> class >> 2' priority which is too high for backup IO. >> >> Is there a way of prioritising the IO load imposed when running ontape? The >> users need the system to be responsive, and it doesn't matter if the >> backups >> take a long time to complete. >> >> Thanks in advance. >> Cheers, Andy. >> >> >> >> > ******************************************************************************* >> Forum Note: Use "Reply" to post a response in the discussion forum. >> >> > --20cf307d00f05ddcd504a8a8590f > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. >

Have you tried running without the gzip? Gzip can kill a CPU without really trying Cheers Paul > If this is a Linux/Unix OS, would lowering ontape's priority with "nice" > help? > > On 7/22/2011 9:21 AM, Art Kagel wrote: >> There is nothing you can do to reduce the IO demands that an archive >> will >> put on the system. As you point out, it is the oninit process that >> actually >> performs the reads on the chunks, the ontape is just taking the output >> from >> the archive thread in the engine and writing it out to the back up >> device or >> stream. One thing you can do is to eliminate the gzip if you can. Gzip >> will use 100% of a processor core and that will certainly affect your >> instance's performance outside of the IO load. >> >> Art >> >> Art S. Kagel >> Advanced DataTools (www.advancedatatools.com) >> Blog: http://informix-myview.blogspot.com/ >> >> Disclaimer: Please keep in mind that my own opinions are my own opinions >> and >> do not reflect on my employer, Advanced DataTools, the IIUG, nor any >> other >> organization with which I am associated either explicitly, implicitly, >> or by >> inference. Neither do those opinions reflect those of other individuals >> affiliated with any entity with which I am affiliated nor those of the >> entities themselves. >> >> On Fri, Jul 22, 2011 at 9:00 AM, ANDREW LEMIN<a_lemin@hotmail.com> >> wrote: >> >>> Hello, >>> >>> We have a system that is very busy and as such there is not much disk >>> IO >>> headroom left during normal operation. >>> >>> This is ok for now until new servers can be installed etc with greater >>> IO >>> capacity. However when doing ontape backups, the system becomes very >>> slow >>> for >>> the users. During this time using iostat we can see the IO queue size >>> increasing. >>> >>> We have tried running ontape | gzip with ionice -c3 etc which causes >>> gzip >>> to >>> run with the 'idle' IO priority, but I see that after starting ontape, >>> the >>> work is handed to one of the oninit processes which have a 'best effort >>> class >>> 2' priority which is too high for backup IO. >>> >>> Is there a way of prioritising the IO load imposed when running ontape? >>> The >>> users need the system to be responsive, and it doesn't matter if the >>> backups >>> take a long time to complete. >>> >>> Thanks in advance. >>> Cheers, Andy. >>> >>> >>> >>> >> > ******************************************************************************* >>> Forum Note: Use "Reply" to post a response in the discussion forum. >>> >>> >> --20cf307d00f05ddcd504a8a8590f >> >> >> > ******************************************************************************* >> Forum Note: Use "Reply" to post a response in the discussion forum. >> > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > -- Paul Watson Tel: +1 913-674-0360 Mob: +1 913-387-7529 Web: www.oninit.com www.advancedatatools.com Failure is not as frightening as regret. If you want to improve, be content to be thought foolish and stupid.

Hi chaps, Thanks for your quick responses. Its disappointing we cannot assign a lower priority to what is effective a long read transaction. We use both 'ionice -c3' and 'nice +19' on both ontape and the gzip processes in a pipeline. If we run the backup without gzip the user performance still crashes (in fact it can be worse as when using it with gzip, gzip seems to slow the IO down slightly). Their is lots of spare CPU capacity and we can leave gzip busying out an entire core without issue, so we found the best way to slow it down was to use maximum gzip compression but we hoped to slow it down some more. Seems like there is no solution. Thanks anyway chaps. cheers, Andy.

Hi Andrew, You didn't told the Informix version you're using, if you're in 11.50.xC4 or earlier, you should use the feature deep compression (compress,repack and shrink) it will compress your data TO around 30%-40%, which 1TB of Data (not index, blob) will be reduced to around 0,3TB and reduce the IO overhead in your system and increase the backup time consequently. It will eliminate the necessity of using the gzip. In 11.50.xC4 or xC5 you have to be aware about old version row in tables. Earlier version is quite simpler. There is a presentation in IIUG2011 about this feature in large database. Best Regards, Celso Cabral Coimbra Administrador de Banco de Dados ClearTech Ltda "Trust at the heart of Communications" Tel. (11) 3576-4509 -----Mensagem original----- De: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] Em nome de ANDREW LEMIN Enviada em: sexta-feira, 22 de julho de 2011 12:48 Para: ids@iiug.org Assunto: Re: ontape IO priority [24442] Hi chaps, Thanks for your quick responses. Its disappointing we cannot assign a lower priority to what is effective a long read transaction. We use both 'ionice -c3' and 'nice +19' on both ontape and the gzip processes in a pipeline. If we run the backup without gzip the user performance still crashes (in fact it can be worse as when using it with gzip, gzip seems to slow the IO down slightly). Their is lots of spare CPU capacity and we can leave gzip busying out an entire core without issue, so we found the best way to slow it down was to use maximum gzip compression but we hoped to slow it down some more. Seems like there is no solution. Thanks anyway chaps. cheers, Andy. ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

We are running IDS 11.50 FC2. I managed to control the IO imposed by adding a 'cstream' section to the pipe. It now looks like this in our backup script; nice -n +${CPUPRIORITY} ${INFORMIXDIR}/bin/ontape -s -L $BACKUPLEVEL -t STDIO -v 2>$ERRORSTRING | nice -n +${CPUPRIORITY} cstream -t $IOTHROTTLE | `nice -n +${CPUPRIORITY} gzip -$COMPRESSION_LEVEL > $TAPEDEV.gz` | tee -a $LOG; x=${PIPESTATUS[0]}; This works perfectly and gives us complete control over the priority and throughput of the backup. As always thank you everyone for your suggestions and thank you Bogdan for the idea. This forum never ceases to impress me with the quality of its community. ;o) Andy