named pipes - external tables

Aix 5.3 IDS 11.50.fc5 I'm working on moving some data from informix to a Netezzza N1001-05. One of the ways we are looking to do this is by using external tables in informix which are named pipes ... Then on the netezza side the nzload utility is used to insert the data. This seems to be working well if I single thread things. When I attempt to run this a couple of time concurrently I get the following informix error: 26167, all data files are either full or corrupted. I don't get any OS errors. So, does anyone know if this is a limit of informix, to be able to only use one named pipe at a time, or is it a limitation of Aix, or neither and I just have something else messed up? Thanks for any help ... Peter Logan Senior Database Administrator Phone: 616/878-8309

Original post: Aix 5.3 IDS 11.50.fc5 I'm working on moving some data from informix to a Netezzza N1001-05. One of the ways we are looking to do this is by using external tables in informix which are named pipes ... Then on the netezza side the nzload utility is used to insert the data. This seems to be working well if I single thread things. When I attempt to run this a couple of time concurrently I get the following informix error: 26167, all data files are either full or corrupted. I don't get any OS errors. So, does anyone know if this is a limit of informix, to be able to only use one named pipe at a time, or is it a limitation of Aix, or neither and I just have something else messed up? Thanks for any help ... Peter Logan Senior Database Administrator Phone: 616/878-8309 Response: Can you be more specific on "run this a couple of time concurrently"? Do you mean have a couple of informix sessions/threads attempting to unload into the same external table that you have defined as a pipe, or you have a couple of informix sessions/threads attempting to unload into different external tables each with their own named pipe but only 1 informix thread per named pipe? Jacques Renaut IBM Informix Advanced Support APD Team

Sorry ... different tables .. different pipes ...different sessions ... Also, I have tried single threading in a script, and it seems that I get this error for every other table. Almost seems as if it takes the pipe a little bit to really be empty ... Peter Logan Senior Database Administrator Phone: 616/878-8309 From: "JACQUES RENAUT" <jrenaut@us.ibm.com> To: ids@iiug.org, Date: 06/13/2013 09:32 AM Subject: Re: named pipes - external tables [30525] Sent by: ids-bounces@iiug.org Original post: Aix 5.3 IDS 11.50.fc5 I'm working on moving some data from informix to a Netezzza N1001-05. One of the ways we are looking to do this is by using external tables in informix which are named pipes ... Then on the netezza side the nzload utility is used to insert the data. This seems to be working well if I single thread things. When I attempt to run this a couple of time concurrently I get the following informix error: 26167, all data files are either full or corrupted. I don't get any OS errors. So, does anyone know if this is a limit of informix, to be able to only use one named pipe at a time, or is it a limitation of Aix, or neither and I just have something else messed up? Thanks for any help ... Peter Logan Senior Database Administrator Phone: 616/878-8309 Response: Can you be more specific on "run this a couple of time concurrently"? Do you mean have a couple of informix sessions/threads attempting to unload into the same external table that you have defined as a pipe, or you have a couple of informix sessions/threads attempting to unload into different external tables each with their own named pipe but only 1 informix thread per named pipe? Jacques Renaut IBM Informix Advanced Support APD Team ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

Peter: You are aware that if you upgrade to v11.70xC4 or later and slap IWA on top of it you can get Netezza class performance for far less $$ and literally a few minutes effort? Since IWA scales to use as many CPU cores and as much memory as you can pack into a node (IB that would be 64cores and 3TB of memory per node) and that you can cluster as many IWA nodes as you want together to handle truly massive data sets in linear time, I would venture to tell you a secret that IBM doesn't want you to know: You can build your own warehouse system that I believe will outperform any Netezza, Exadata, or Teradata machine for less money using Informix and IWA! And with v12.10 and its Trickle Feed technology, you can be running your BI queries against near-real-time data directly from your production server without expensive and time-consuming (so also $$ consuming) periodic ETL conversions! Netezza can't do that at all! Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Thu, Jun 13, 2013 at 9:10 AM, Peter_Logan@spartanstores.com < Peter_Logan@spartanstores.com> wrote: > Aix 5.3 > IDS 11.50.fc5 > > I'm working on moving some data from informix to a Netezzza N1001-05. One > of the ways we are looking to do this is by using external tables in > informix which are named pipes ... Then on the netezza side the nzload > utility is used to insert the data. This seems to be working well if I > single thread things. When I attempt to run this a couple of time > concurrently I get the following informix error: 26167, all data files > are either full or corrupted. I don't get any OS errors. > > So, does anyone know if this is a limit of informix, to be able to only > use one named pipe at a time, or is it a limitation of Aix, or neither and > I just have something else messed up? > > Thanks for any help ... > > Peter Logan > Senior Database Administrator > Phone: 616/878-8309 > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --001a11c2675ce0cf8004df097d7e

Art, I totally agree with you on this ... This is one of those things where our VP has a friend who has Netezza ... and well, now we have it ... I was working on a POC of IWA and it got pulled out from under me... I wasn't involved with the IBM sales side of it, but it seemed from what I have picked up they were really pushing Netezza and what we got this for was deeply discounted.... I think this was one of those things where the sales guys didn't either understand IWA, or they just didn't care ... Peter Logan Senior Database Administrator Phone: 616/878-8309 From: "Art Kagel" <art.kagel@gmail.com> To: ids@iiug.org, Date: 06/13/2013 09:57 AM Subject: Re: named pipes - external tables [30529] Sent by: ids-bounces@iiug.org Peter: You are aware that if you upgrade to v11.70xC4 or later and slap IWA on top of it you can get Netezza class performance for far less $$ and literally a few minutes effort? Since IWA scales to use as many CPU cores and as much memory as you can pack into a node (IB that would be 64cores and 3TB of memory per node) and that you can cluster as many IWA nodes as you want together to handle truly massive data sets in linear time, I would venture to tell you a secret that IBM doesn't want you to know: You can build your own warehouse system that I believe will outperform any Netezza, Exadata, or Teradata machine for less money using Informix and IWA! And with v12.10 and its Trickle Feed technology, you can be running your BI queries against near-real-time data directly from your production server without expensive and time-consuming (so also $$ consuming) periodic ETL conversions! Netezza can't do that at all! Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Thu, Jun 13, 2013 at 9:10 AM, Peter_Logan@spartanstores.com < Peter_Logan@spartanstores.com> wrote: > Aix 5.3 > IDS 11.50.fc5 > > I'm working on moving some data from informix to a Netezzza N1001-05. One > of the ways we are looking to do this is by using external tables in > informix which are named pipes ... Then on the netezza side the nzload > utility is used to insert the data. This seems to be working well if I > single thread things. When I attempt to run this a couple of time > concurrently I get the following informix error: 26167, all data files > are either full or corrupted. I don't get any OS errors. > > So, does anyone know if this is a limit of informix, to be able to only > use one named pipe at a time, or is it a limitation of Aix, or neither and > I just have something else messed up? > > Thanks for any help ... > > Peter Logan > Senior Database Administrator > Phone: 616/878-8309 > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --001a11c2675ce0cf8004df097d7e ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

I was able to use multiple pipes when I upgraded a 7 engine to a 10 engine, both on AIX hardware, back in May 2008, involving both blobs and non-blob data. I named each pipe differently (albeit close to the name of the table for scripting purposes) and was able to accomplish both the AIX and Informix upgrade. Keep working at it; it is possible ... well, from an Informix and AIX point of view. On Thu, Jun 13, 2013 at 2:10 PM, Peter_Logan@spartanstores.com < Peter_Logan@spartanstores.com> wrote: > Aix 5.3 > IDS 11.50.fc5 > > I'm working on moving some data from informix to a Netezzza N1001-05. One > of the ways we are looking to do this is by using external tables in > informix which are named pipes ... Then on the netezza side the nzload > utility is used to insert the data. This seems to be working well if I > single thread things. When I attempt to run this a couple of time > concurrently I get the following informix error: 26167, all data files > are either full or corrupted. I don't get any OS errors. > > So, does anyone know if this is a limit of informix, to be able to only > use one named pipe at a time, or is it a limitation of Aix, or neither and > I just have something else messed up? > > Thanks for any help ... > > Peter Logan > Senior Database Administrator > Phone: 616/878-8309 > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Take care. Clifton This communication may contain privileged or other confidential information. If you have received it in error, please advise the sender by reply email and immediately delete the message and any attachments without copying or disclosing the contents. Thank you. Consider the environment. Please don't print this e-mail unless you need to. --001a11c338d0dd170b04df093a31

Looks like I got it working ... Not really sure what was different .. I did add removing the pipe and recreating it for each of the tables ... maybe that cleaned something up .. thanks .. Peter Logan Senior Database Administrator Phone: 616/878-8309 From: "Clifton Bean" <clifton.m.bean@gmail.com> To: ids@iiug.org, Date: 06/13/2013 10:09 AM Subject: Re: named pipes - external tables [30532] Sent by: ids-bounces@iiug.org I was able to use multiple pipes when I upgraded a 7 engine to a 10 engine, both on AIX hardware, back in May 2008, involving both blobs and non-blob data. I named each pipe differently (albeit close to the name of the table for scripting purposes) and was able to accomplish both the AIX and Informix upgrade. Keep working at it; it is possible ... well, from an Informix and AIX point of view. On Thu, Jun 13, 2013 at 2:10 PM, Peter_Logan@spartanstores.com < Peter_Logan@spartanstores.com> wrote: > Aix 5.3 > IDS 11.50.fc5 > > I'm working on moving some data from informix to a Netezzza N1001-05. One > of the ways we are looking to do this is by using external tables in > informix which are named pipes ... Then on the netezza side the nzload > utility is used to insert the data. This seems to be working well if I > single thread things. When I attempt to run this a couple of time > concurrently I get the following informix error: 26167, all data files > are either full or corrupted. I don't get any OS errors. > > So, does anyone know if this is a limit of informix, to be able to only > use one named pipe at a time, or is it a limitation of Aix, or neither and > I just have something else messed up? > > Thanks for any help ... > > Peter Logan > Senior Database Administrator > Phone: 616/878-8309 > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Take care. Clifton This communication may contain privileged or other confidential information. If you have received it in error, please advise the sender by reply email and immediately delete the message and any attachments without copying or disclosing the contents. Thank you. Consider the environment. Please don't print this e-mail unless you need to. --001a11c338d0dd170b04df093a31 ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

But you weren't using EXTERNAL TABLES with pipes, not in either 7.xx or 10.00! Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Thu, Jun 13, 2013 at 10:08 AM, Clifton Bean <clifton.m.bean@gmail.com>wrote: > I was able to use multiple pipes when I upgraded a 7 engine to a 10 engine, > both on AIX hardware, back in May 2008, involving both blobs and non-blob > data. I named each pipe differently (albeit close to the name of the table > for scripting purposes) and was able to accomplish both the AIX and > Informix upgrade. > > Keep working at it; it is possible ... well, from an Informix and AIX point > of view. > > On Thu, Jun 13, 2013 at 2:10 PM, Peter_Logan@spartanstores.com < > Peter_Logan@spartanstores.com> wrote: > > > Aix 5.3 > > IDS 11.50.fc5 > > > > I'm working on moving some data from informix to a Netezzza N1001-05. One > > of the ways we are looking to do this is by using external tables in > > informix which are named pipes ... Then on the netezza side the nzload > > utility is used to insert the data. This seems to be working well if I > > single thread things. When I attempt to run this a couple of time > > concurrently I get the following informix error: 26167, all data files > > are either full or corrupted. I don't get any OS errors. > > > > So, does anyone know if this is a limit of informix, to be able to only > > use one named pipe at a time, or is it a limitation of Aix, or neither > and > > I just have something else messed up? > > > > Thanks for any help ... > > > > Peter Logan > > Senior Database Administrator > > Phone: 616/878-8309 > > > > > > > > > > ******************************************************************************* > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > -- > Take care. > Clifton > > This communication may contain privileged or other confidential > information. If you have received it in error, please advise the sender by > reply email and immediately delete the message and any attachments without > copying or disclosing the contents. Thank you. > > Consider the environment. Please don't print this e-mail unless you need > to. > > --001a11c338d0dd170b04df093a31 > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --089e01493c1ef8ce9004df09e209

Offcourse I agree with Art but I know it is not Peter decision.... I would suggest also to use xtrace utility and set XTF_XPL_AIO trace This should help to figure out what error you are getting at OS level. Kind regards Sandor -----------------------------------------------------------------------= -------------------------------------------------------------------- IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Martina Koederitz Gesch=E4ftsf=FChrung: Dirk Wittkopp Sitz der Gesellschaft: B=F6blingen = / Registergericht: Amtsgericht Stuttgart, HRB 243294 = "Art Kagel" = <art.kagel@gmail. = com> = To Sent by: ids@iiug.org, = ids-bounces@iiug. = cc org = Subj= ect Re: named pipes - external table= s 13/06/2013 15:56 [30529] = = = Please respond to = ids@iiug.org = = = Peter: You are aware that if you upgrade to v11.70xC4 or later and slap IWA on= top of it you can get Netezza class performance for far less $$ and literal= ly a few minutes effort? Since IWA scales to use as many CPU cores and as mu= ch memory as you can pack into a node (IB that would be 64cores and 3TB of= memory per node) and that you can cluster as many IWA nodes as you want= together to handle truly massive data sets in linear time, I would venture to tell you a secret that IBM doesn't want you to know:= You can build your own warehouse system that I believe will outperform any Netezza, Exadata, or Teradata machine for less money using Informix and= IWA! And with v12.10 and its Trickle Feed technology, you can be runnin= g your BI queries against near-real-time data directly from your producti= on server without expensive and time-consuming (so also $$ consuming) peri= odic ETL conversions! Netezza can't do that at all! Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinion= s and do not reflect on my employer, Advanced DataTools, the IIUG, nor an= y other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of= other individuals affiliated with any entity with which I am affiliated= nor those of the entities themselves. On Thu, Jun 13, 2013 at 9:10 AM, Peter_Logan@spartanstores.com < Peter_Logan@spartanstores.com> wrote: > Aix 5.3 > IDS 11.50.fc5 > > I'm working on moving some data from informix to a Netezzza N1001-05.= One > of the ways we are looking to do this is by using external tables in > informix which are named pipes ... Then on the netezza side the nzloa= d > utility is used to insert the data. This seems to be working well if = I > single thread things. When I attempt to run this a couple of time > concurrently I get the following informix error: 26167, all data file= s > are either full or corrupted. I don't get any OS errors. > > So, does anyone know if this is a limit of informix, to be able to on= ly > use one named pipe at a time, or is it a limitation of Aix, or neithe= r and > I just have something else messed up? > > Thanks for any help ... > > Peter Logan > Senior Database Administrator > Phone: 616/878-8309 > > > > ***********************************************************************= ******** > Forum Note: Use "Reply" to post a response in the discussion forum. > > --001a11c2675ce0cf8004df097d7e ***********************************************************************= ******** Forum Note: Use "Reply" to post a response in the discussion forum. =

On Thursday, June 13, 2013 03:40:30 pm you wrote: > Sorry ... different tables .. different pipes ...different sessions ... > > Also, I have tried single threading in a script, and it seems that I get > this error for every other table. Almost seems as if it takes the pipe a > little bit to really be empty ... If not already done you could try to increase the number of FIFO VPs, see http://pic.dhe.ibm.com/infocenter/informix/v121/topic/com.ibm.admin.doc/ids_admi n_1308.htm Thanks, Andreas > > Peter Logan > Senior Database Administrator > Phone: 616/878-8309 > > From: "JACQUES RENAUT" <jrenaut@us.ibm.com> > To: ids@iiug.org, > Date: 06/13/2013 09:32 AM > Subject: Re: named pipes - external tables [30525] > Sent by: ids-bounces@iiug.org > > Original post: > > Aix 5.3 > IDS 11.50.fc5 > > I'm working on moving some data from informix to a Netezzza N1001-05. One > of the ways we are looking to do this is by using external tables in > informix which are named pipes ... Then on the netezza side the nzload > utility is used to insert the data. This seems to be working well if I > single thread things. When I attempt to run this a couple of time > concurrently I get the following informix error: 26167, all data files > are either full or corrupted. I don't get any OS errors. > > So, does anyone know if this is a limit of informix, to be able to only > use one named pipe at a time, or is it a limitation of Aix, or neither and > > I just have something else messed up? > > Thanks for any help ... > > Peter Logan > Senior Database Administrator > Phone: 616/878-8309 > > Response: > Can you be more specific on "run this a couple of time concurrently"? Do > you > mean have a couple of informix sessions/threads attempting to unload into > the > same external table that you have defined as a pipe, or you have a couple > of > informix sessions/threads attempting to unload into different external > tables > each with their own named pipe but only 1 informix thread per named pipe? > > Jacques Renaut > IBM Informix Advanced Support > APD Team > > > ******************************************************************************* > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Andreas Breitfeld; Informix Development Munich IBM Deutschland Research & Development GmbH; Vorsitzende des Aufsichtsrats: Martina Koederitz; Geschaeftsfuehrung: Dirk Wittkopp; Sitz der Gesellschaft: Boeblingen; Registergericht: Amtsgericht Stuttgart, HRB 243294