IDS 11 on a RedHat VM

Hi all We are currently in the process of migrating an 850 (ish) Gb database from AIX 5/IDS9.4 to a vm server (ESX 4) running Red hat 2.6.32-431.3.1.el6.x86_64/IDS 11.70.FC7. New server has the same numbers of CPUs, identical memory and the same amount of disk. Shifting the data across from the existing machine to new one is being doen by multiple streams of 'UNLOAD TO `pipe` SELECT * FROM' combined with 'DBLOAD', plus a single stream of ' SELECT * FROM INSERT INTO' for 300 smaller tables (there are around 30 large tables in 6 unload streams). Source and target machines are in the same subnet and have 1Gb end to end network plus the data source is an HDR secondary which is NOT used for any other purpose (such as read-only queries). Basically the load performance sucks on anything more than a single stream and when we have managed a full load, initial testing of the application returns abysmal performance. The disks on the source are not stressed in any way (around 25% utilisation) but the disks on the new server are constantly at 100% but can only seem to shift some 10 Mb/s. The main structural difference between the machines seems to be how the disk is presented and I am trying to find out what is considered 'best practise' or how to get some decent performance. The existing server has direct attached (2Gb fibre, redundant path) storage presenting as 27 x 36Gb disks which are then carved up by AIX LVM into 4 and 8 Gb chunks and assigned to dbspaces. On the new server there is 1 Tb of SAN disk presented through a parascsi adapter to the server as a single disk (/dev/sde) which is then carved up by the RedHat LVM. When loading the data, no matter how many streams run or how many chunks are being hit, the overall thoughput seems to be topping out at 10 Mb/sec. I suppose my queston is, is this the best way of presenting this amount of disk or would it be better to get 10 x 100Gb disks, treat them as separate spindles and divide the data/indexing between them as I would do on tranditionally attached disk. Any thoughts, comments, suggestions gratefully received. Keith ps I only have limited, view accessto the VM environment as it is managed for us but I do hve full root access to the RedHat environment, I have no visibility on the SAN. --089e0115f5183fa01f04fd608efb

Keith, this is EXACTLY the behavior I would expect from a VM, especially an ESX 4 VM. I did extensive testing for a client several years ago on the same systems and not only is the IO throughput fixed on a single VM no matter how many threads are writing to/reading from the disk(s) but on a HUGE machine running multiple VMs gets you the same total throughput across all VMs with the average total throughput about the same as the rated throughput of a single 10K spindle and I was testing against RAID10 arrays with ten 10K drive pairs. Reports from the client since those test indicate that after working with VMWare and EMC^2 for over a year are that if you run the VMs over the vSphere Hypervisor the IO throughput about doubles but is still nailed to a ceiling. My recommendation has always been and continues to be that you run Informix, and all databases, on an OS that is running on bare metal. Art Art S. Kagel, Principal Consultant ASK Database Management Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Fri, Jul 4, 2014 at 12:24 PM, Keith Simmons <smiley73@gmail.com> wrote: > Hi all > > We are currently in the process of migrating an 850 (ish) Gb database from > AIX 5/IDS9.4 to a vm server (ESX 4) running Red hat > 2.6.32-431.3.1.el6.x86_64/IDS 11.70.FC7. > New server has the same numbers of CPUs, identical memory and the same > amount of disk. > > Shifting the data across from the existing machine to new one is being doen > by multiple streams of 'UNLOAD TO `pipe` SELECT * FROM' combined with > 'DBLOAD', plus a single stream of ' SELECT * FROM INSERT INTO' for 300 > smaller tables (there are around 30 large tables in 6 unload streams). > > Source and target machines are in the same subnet and have 1Gb end to end > network plus the data source is an HDR secondary which is NOT used for any > other purpose (such as read-only queries). > > Basically the load performance sucks on anything more than a single stream > and when we have managed a full load, initial testing of the application > returns abysmal performance. > > The disks on the source are not stressed in any way (around 25% > utilisation) but the disks on the new server are constantly at 100% but can > only seem to shift some 10 Mb/s. > > The main structural difference between the machines seems to be how the > disk is presented and I am trying to find out what is considered 'best > practise' or how to get some decent performance. The existing server has > direct attached (2Gb fibre, redundant path) storage presenting as 27 x 36Gb > disks which are then carved up by AIX LVM into 4 and 8 Gb chunks and > assigned to dbspaces. On the new server there is 1 Tb of SAN disk presented > through a parascsi adapter to the server as a single disk (/dev/sde) which > is then carved up by the RedHat LVM. When loading the data, no matter how > many streams run or how many chunks are being hit, the overall thoughput > seems to be topping out at 10 Mb/sec. > > I suppose my queston is, is this the best way of presenting this amount of > disk or would it be better to get 10 x 100Gb disks, treat them as separate > spindles and divide the data/indexing between them as I would do on > tranditionally attached disk. Any thoughts, comments, suggestions > gratefully received. > > Keith > > ps I only have limited, view accessto the VM environment as it is managed > for us but I do hve full root access to the RedHat environment, I have no > visibility on the SAN. > > --089e0115f5183fa01f04fd608efb > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --001a1133a628e0faad04fd624c94

Hi, Firstly, I would defer to Art as the information he has provided appears to be based on a lot of actual "real world" experience which is the only way to get a true understanding ... positive affirmation. Secondly, have you done some actual trials completely outside of IBM Informix?? For example, doing dd N.B DESTRUCTIVE TESTING!!!!!! 1 single stream of ... time dd if=/dev/zero bs=2k|4k|8k|16k of="chunk path count="at least 1 Gig of BS" (BS = Block Size not ... :O ) Then 2 streams of dd and then 4 etc. etc. Thirdly, you mention "a single 1TB /dev/sde, then carved up using LVM" ... how many lvms? what size of lvms? presenting the raw lvm to informix or the block device or a file? KAIO? how may CPU VPs? DIRECT_IO enabled? Now this testing may just confirm the limits you have seen, or you may see different behavior ... Just a quick reply on a Saturday :) JJ

Jon Thanks. dd showed similar behaviour (but not quite so extreme and was not really conclusive) and curiously writes were much quicker than reads (to /dev/null). There are 150 logical volumes, mixture of 6 and 8 Gb with 1 to 5 lvs per dbspace. KAIO is in use. The block device is presented to informix. 4 CPU VPs Keith On 5 July 2014 17:51, JON RITSON <jonritson@sky.com> wrote: > Hi, > > Firstly, I would defer to Art as the information he has provided appears > to be > based on a lot of actual "real world" experience which is the only way to > get > a true understanding ... positive affirmation. > > Secondly, have you done some actual trials completely outside of IBM > Informix?? > > For example, doing dd N.B DESTRUCTIVE TESTING!!!!!! > > 1 single stream of ... > > time dd if=/dev/zero bs=2k|4k|8k|16k of="chunk path count="at least 1 Gig > of > BS" > > (BS = Block Size not ... :O ) > > Then 2 streams of dd and then 4 etc. etc. > > Thirdly, you mention "a single 1TB /dev/sde, then carved up using LVM" ... > how many lvms? > what size of lvms? > presenting the raw lvm to informix or the block device or a file? > KAIO? how may CPU VPs? DIRECT_IO enabled? > > Now this testing may just confirm the limits you have seen, or you may see > different behavior ... > > Just a quick reply on a Saturday :) > > JJ > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --e89a8f921a3ef2928a04fd95c245

Art Many thanks for your insight on this. One further question, if we get some bare metal the disk will still come from a SAN, is it best to still allow this to be presented as a single 1 Tb disk or as mutiple, smaller disks (I've still inclined towards 10 x 100 Gb). Keith On 4 July 2014 19:29, Art Kagel <art.kagel@gmail.com> wrote: > Keith, this is EXACTLY the behavior I would expect from a VM, especially an > ESX 4 VM. I did extensive testing for a client several years ago on the > same systems and not only is the IO throughput fixed on a single VM no > matter how many threads are writing to/reading from the disk(s) but on a > HUGE machine running multiple VMs gets you the same total throughput across > all VMs with the average total throughput about the same as the rated > throughput of a single 10K spindle and I was testing against RAID10 arrays > with ten 10K drive pairs. > > Reports from the client since those test indicate that after working with > VMWare and EMC^2 for over a year are that if you run the VMs over the > vSphere Hypervisor the IO throughput about doubles but is still nailed to a > ceiling. > > My recommendation has always been and continues to be that you run > Informix, and all databases, on an OS that is running on bare metal. > > Art > > Art S. Kagel, Principal Consultant > ASK Database Management > > Blog: http://informix-myview.blogspot.com/ > > Disclaimer: Please keep in mind that my own opinions are my own opinions > and do not reflect on the IIUG, nor any other organization with which I am > associated either explicitly, implicitly, or by inference. Neither do > those opinions reflect those of other individuals affiliated with any > entity with which I am affiliated nor those of the entities themselves. > > On Fri, Jul 4, 2014 at 12:24 PM, Keith Simmons <smiley73@gmail.com> wrote: > > > Hi all > > > > We are currently in the process of migrating an 850 (ish) Gb database > from > > AIX 5/IDS9.4 to a vm server (ESX 4) running Red hat > > 2.6.32-431.3.1.el6.x86_64/IDS 11.70.FC7. > > New server has the same numbers of CPUs, identical memory and the same > > amount of disk. > > > > Shifting the data across from the existing machine to new one is being > doen > > by multiple streams of 'UNLOAD TO `pipe` SELECT * FROM' combined with > > 'DBLOAD', plus a single stream of ' SELECT * FROM INSERT INTO' for 300 > > smaller tables (there are around 30 large tables in 6 unload streams). > > > > Source and target machines are in the same subnet and have 1Gb end to end > > network plus the data source is an HDR secondary which is NOT used for > any > > other purpose (such as read-only queries). > > > > Basically the load performance sucks on anything more than a single > stream > > and when we have managed a full load, initial testing of the application > > returns abysmal performance. > > > > The disks on the source are not stressed in any way (around 25% > > utilisation) but the disks on the new server are constantly at 100% but > can > > only seem to shift some 10 Mb/s. > > > > The main structural difference between the machines seems to be how the > > disk is presented and I am trying to find out what is considered 'best > > practise' or how to get some decent performance. The existing server has > > direct attached (2Gb fibre, redundant path) storage presenting as 27 x > 36Gb > > disks which are then carved up by AIX LVM into 4 and 8 Gb chunks and > > assigned to dbspaces. On the new server there is 1 Tb of SAN disk > presented > > through a parascsi adapter to the server as a single disk (/dev/sde) > which > > is then carved up by the RedHat LVM. When loading the data, no matter how > > many streams run or how many chunks are being hit, the overall thoughput > > seems to be topping out at 10 Mb/sec. > > > > I suppose my queston is, is this the best way of presenting this amount > of > > disk or would it be better to get 10 x 100Gb disks, treat them as > separate > > spindles and divide the data/indexing between them as I would do on > > tranditionally attached disk. Any thoughts, comments, suggestions > > gratefully received. > > > > Keith > > > > ps I only have limited, view accessto the VM environment as it is managed > > for us but I do hve full root access to the RedHat environment, I have no > > visibility on the SAN. > > > > --089e0115f5183fa01f04fd608efb > > > > > > > > > > ******************************************************************************* > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > --001a1133a628e0faad04fd624c94 > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --047d7bdc1a9ca3739104fd9646b8

There is some slight improvement in overhead by dividing the SAN structure into multiple luns on the SAN rather than using a software based LVM to create the multiple LVs. Best practice though would be to not use a single massive array but to break it into three or four arrays so you can isolate high volume tables, indexes, and logs from each other onto separate sets of spindles and carve the luns from these. That you you can keep logical and physical log IO away from each other (since they happen concurrently) as well as your high volume tables and indexes from the logs and from each other. Lower IO volume objects can be spread across the four arrays. If your transaction rates are VERY high, you can spread the logical logs across two luns from different structures and alternate assigning log files between them so that while a log file on one lun is being backed up transactions are writing to a different lun and not competing for throughput or head position. Now, obviously, if your IO volume isn't high, this isn't an issue, but since you were performing throughput testing, I'll assume that your transaction rate is significant. Art Art S. Kagel, Principal Consultant ASK Database Management Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Mon, Jul 7, 2014 at 4:29 AM, Keith Simmons <smiley73@gmail.com> wrote: > Art > > Many thanks for your insight on this. One further question, if we get some > bare metal the disk will still come from a SAN, is it best to still allow > this to be presented as a single 1 Tb disk or as mutiple, smaller disks > (I've still inclined towards 10 x 100 Gb). > > Keith > > On 4 July 2014 19:29, Art Kagel <art.kagel@gmail.com> wrote: > > > Keith, this is EXACTLY the behavior I would expect from a VM, especially > an > > ESX 4 VM. I did extensive testing for a client several years ago on the > > same systems and not only is the IO throughput fixed on a single VM no > > matter how many threads are writing to/reading from the disk(s) but on a > > HUGE machine running multiple VMs gets you the same total throughput > across > > all VMs with the average total throughput about the same as the rated > > throughput of a single 10K spindle and I was testing against RAID10 > arrays > > with ten 10K drive pairs. > > > > Reports from the client since those test indicate that after working with > > VMWare and EMC^2 for over a year are that if you run the VMs over the > > vSphere Hypervisor the IO throughput about doubles but is still nailed > to a > > ceiling. > > > > My recommendation has always been and continues to be that you run > > Informix, and all databases, on an OS that is running on bare metal. > > > > Art > > > > Art S. Kagel, Principal Consultant > > ASK Database Management > > > > Blog: http://informix-myview.blogspot.com/ > > > > Disclaimer: Please keep in mind that my own opinions are my own opinions > > and do not reflect on the IIUG, nor any other organization with which I > am > > associated either explicitly, implicitly, or by inference. Neither do > > those opinions reflect those of other individuals affiliated with any > > entity with which I am affiliated nor those of the entities themselves. > > > > On Fri, Jul 4, 2014 at 12:24 PM, Keith Simmons <smiley73@gmail.com> > wrote: > > > > > Hi all > > > > > > We are currently in the process of migrating an 850 (ish) Gb database > > from > > > AIX 5/IDS9.4 to a vm server (ESX 4) running Red hat > > > 2.6.32-431.3.1.el6.x86_64/IDS 11.70.FC7. > > > New server has the same numbers of CPUs, identical memory and the same > > > amount of disk. > > > > > > Shifting the data across from the existing machine to new one is being > > doen > > > by multiple streams of 'UNLOAD TO `pipe` SELECT * FROM' combined with > > > 'DBLOAD', plus a single stream of ' SELECT * FROM INSERT INTO' for 300 > > > smaller tables (there are around 30 large tables in 6 unload streams). > > > > > > Source and target machines are in the same subnet and have 1Gb end to > end > > > network plus the data source is an HDR secondary which is NOT used for > > any > > > other purpose (such as read-only queries). > > > > > > Basically the load performance sucks on anything more than a single > > stream > > > and when we have managed a full load, initial testing of the > application > > > returns abysmal performance. > > > > > > The disks on the source are not stressed in any way (around 25% > > > utilisation) but the disks on the new server are constantly at 100% but > > can > > > only seem to shift some 10 Mb/s. > > > > > > The main structural difference between the machines seems to be how the > > > disk is presented and I am trying to find out what is considered 'best > > > practise' or how to get some decent performance. The existing server > has > > > direct attached (2Gb fibre, redundant path) storage presenting as 27 x > > 36Gb > > > disks which are then carved up by AIX LVM into 4 and 8 Gb chunks and > > > assigned to dbspaces. On the new server there is 1 Tb of SAN disk > > presented > > > through a parascsi adapter to the server as a single disk (/dev/sde) > > which > > > is then carved up by the RedHat LVM. When loading the data, no matter > how > > > many streams run or how many chunks are being hit, the overall > thoughput > > > seems to be topping out at 10 Mb/sec. > > > > > > I suppose my queston is, is this the best way of presenting this amount > > of > > > disk or would it be better to get 10 x 100Gb disks, treat them as > > separate > > > spindles and divide the data/indexing between them as I would do on > > > tranditionally attached disk. Any thoughts, comments, suggestions > > > gratefully received. > > > > > > Keith > > > > > > ps I only have limited, view accessto the VM environment as it is > managed > > > for us but I do hve full root access to the RedHat environment, I have > no > > > visibility on the SAN. > > > > > > --089e0115f5183fa01f04fd608efb > > > > > > > > > > > > > > > > > > ******************************************************************************* > > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > > > > > --001a1133a628e0faad04fd624c94 > > > > > > > > > > ******************************************************************************* > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > --047d7bdc1a9ca3739104fd9646b8 > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --089e0158c29c99771b04fd99764d

I ran into the same problem, even with physical hardware. Other than the VM issues Art mentioned, having tens or hundreds of chunks and/or LVs against a single presented LUN is suicide. I buried a couple of LUNs on a Clariion array after migrating from 7 spindles and about 25 chunks to 2 LUNs (wasn't my decision) that I partitioned for those 25 chunks, *and* a new faster server. I can only imagine your scenario is much worse. My update stats routine doubled in runtime. I have also seen a system at my current job (luckily I'm not involved) where a system was upgraded and given a 1TB LUN to make many LVs to place Oracle tablespaces. Surprise; performance on the new system is noticeably slower. Bob ----- Original Message ----- From: "Art Kagel" <art.kagel@gmail.com> To: ids@iiug.org Sent: Monday, July 7, 2014 8:17:57 AM Subject: Re: IDS 11 on a RedHat VM [33358] There is some slight improvement in overhead by dividing the SAN structure into multiple luns on the SAN rather than using a software based LVM to create the multiple LVs. Best practice though would be to not use a single massive array but to break it into three or four arrays so you can isolate high volume tables, indexes, and logs from each other onto separate sets of spindles and carve the luns from these. That you you can keep logical and physical log IO away from each other (since they happen concurrently) as well as your high volume tables and indexes from the logs and from each other. Lower IO volume objects can be spread across the four arrays. If your transaction rates are VERY high, you can spread the logical logs across two luns from different structures and alternate assigning log files between them so that while a log file on one lun is being backed up transactions are writing to a different lun and not competing for throughput or head position. Now, obviously, if your IO volume isn't high, this isn't an issue, but since you were performing throughput testing, I'll assume that your transaction rate is significant. Art Art S. Kagel, Principal Consultant ASK Database Management Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Mon, Jul 7, 2014 at 4:29 AM, Keith Simmons <smiley73@gmail.com> wrote: > Art > > Many thanks for your insight on this. One further question, if we get some > bare metal the disk will still come from a SAN, is it best to still allow > this to be presented as a single 1 Tb disk or as mutiple, smaller disks > (I've still inclined towards 10 x 100 Gb). > > Keith > > On 4 July 2014 19:29, Art Kagel <art.kagel@gmail.com> wrote: > > > Keith, this is EXACTLY the behavior I would expect from a VM, especially > an > > ESX 4 VM. I did extensive testing for a client several years ago on the > > same systems and not only is the IO throughput fixed on a single VM no > > matter how many threads are writing to/reading from the disk(s) but on a > > HUGE machine running multiple VMs gets you the same total throughput > across > > all VMs with the average total throughput about the same as the rated > > throughput of a single 10K spindle and I was testing against RAID10 > arrays > > with ten 10K drive pairs. > > > > Reports from the client since those test indicate that after working with > > VMWare and EMC^2 for over a year are that if you run the VMs over the > > vSphere Hypervisor the IO throughput about doubles but is still nailed > to a > > ceiling. > > > > My recommendation has always been and continues to be that you run > > Informix, and all databases, on an OS that is running on bare metal. > > > > Art > > > > Art S. Kagel, Principal Consultant > > ASK Database Management > > > > Blog: http://informix-myview.blogspot.com/ > > > > Disclaimer: Please keep in mind that my own opinions are my own opinions > > and do not reflect on the IIUG, nor any other organization with which I > am > > associated either explicitly, implicitly, or by inference. Neither do > > those opinions reflect those of other individuals affiliated with any > > entity with which I am affiliated nor those of the entities themselves. > > > > On Fri, Jul 4, 2014 at 12:24 PM, Keith Simmons <smiley73@gmail.com> > wrote: > > > > > Hi all > > > > > > We are currently in the process of migrating an 850 (ish) Gb database > > from > > > AIX 5/IDS9.4 to a vm server (ESX 4) running Red hat > > > 2.6.32-431.3.1.el6.x86_64/IDS 11.70.FC7. > > > New server has the same numbers of CPUs, identical memory and the same > > > amount of disk. > > > > > > Shifting the data across from the existing machine to new one is being > > doen > > > by multiple streams of 'UNLOAD TO `pipe` SELECT * FROM' combined with > > > 'DBLOAD', plus a single stream of ' SELECT * FROM INSERT INTO' for 300 > > > smaller tables (there are around 30 large tables in 6 unload streams). > > > > > > Source and target machines are in the same subnet and have 1Gb end to > end > > > network plus the data source is an HDR secondary which is NOT used for > > any > > > other purpose (such as read-only queries). > > > > > > Basically the load performance sucks on anything more than a single > > stream > > > and when we have managed a full load, initial testing of the > application > > > returns abysmal performance. > > > > > > The disks on the source are not stressed in any way (around 25% > > > utilisation) but the disks on the new server are constantly at 100% but > > can > > > only seem to shift some 10 Mb/s. > > > > > > The main structural difference between the machines seems to be how the > > > disk is presented and I am trying to find out what is considered 'best > > > practise' or how to get some decent performance. The existing server > has > > > direct attached (2Gb fibre, redundant path) storage presenting as 27 x > > 36Gb > > > disks which are then carved up by AIX LVM into 4 and 8 Gb chunks and > > > assigned to dbspaces. On the new server there is 1 Tb of SAN disk > > presented > > > through a parascsi adapter to the server as a single disk (/dev/sde) > > which > > > is then carved up by the RedHat LVM. When loading the data, no matter > how > > > many streams run or how many chunks are being hit, the overall > thoughput > > > seems to be topping out at 10 Mb/sec. > > > > > > I suppose my queston is, is this the best way of presenting this amount > > of > > > disk or would it be better to get 10 x 100Gb disks, treat them as > > separate > > > spindles and divide the data/indexing between them as I would do on > > > tranditionally attached disk. Any thoughts, comments, suggestions > > > gratefully received. > > > > > > Keith > > > > > > ps I only have limited, view accessto the VM environment as it is > managed > > > for us but I do hve full root access to the RedHat e

Keith, you will want to take a look at my study presented in Miami, precisely about this matter. Title is "E11-VmTechnology Vs Physical Servers Legends And Facts" Unless "serious" hardware and hypervisors, you will be very disappointed oabout the performance you will have with low/mid range Hypervisors. To be explicit, I had a much better thruput with my 1000$ PC than with a 'Professional' ESX 4 box at one of my customers... If you like VMWare, go at least ESX 5 ( ESX 4 gets limited to 4 cores, AFAIK ) ESX 4 is VERY disappointing in terms of performance. I am at the moment benchmarking IBM Power8 on PowerLinux (Red Hat 6.4) which is a real blast!!! I will publish results in a few weeks. Eric