How to Monitor Open Connection Tiime???

Hi, I am having some issues in a new environment we just installed and would like to ask some tips to monitor the open connection time. Informix 1.50 FC7W1GE on Linux RedHat 5.5 Client 3.50 FC4 I am monitoring the transaction time for a transaction from the client perspective, and, some times the transaction time increase 10 times. When I look inside, this increase is at the open connection time. How I can monitor the connection threads (soctcppoll). I didn't see any bottleneck in the Informix side. How I can monitor the Operational System, looking inside Informix? How I can know if I am waiting some Linux function? How I can debug the open connection looking inside the client? I am not using DNS anymore, The Network is stable and fast, and the behavior still happens and the problem is intermittent and I am not able to trace it. Tks -- Grato Miguel Carbone +55 11 96347103 MC Software Ltda. - miguel@mcsoftware.com.br <mailto:miguel@mcsoftware.com.br> - www.mcsoftware.com.br IIUG Board of Directors - International Informix Users Group - miguel@iiug.org <mailto:miguel@iiug.org> - www.iiug.org President - Brazilian Informix Users Group - miguel@briug.org <mailto:miguel@briug.org> - www.briug.org <mailto:www.briug.org> MC SOFTWARE LTDA. Rua Coronel Oscar Porto, 813 - cjto T1 e T2 Paraíso - São Paulo - Brasil - 04003004 +55 11 2594 0048 Localização <http://maps.google.com.br/maps?f=q&source=s_q&hl=pt-br&geocode=&q=mc+software&s ll=-14.179186,-50.449219&sspn=57.748661,108.457031&ie=UTF8&hq=mc+software&hnear= &ll=-23.537865,-46.645203&spn=0.111109,0.21183&z=12&iwloc=A>

How many active connections? What are the NETTYPE settings? Do you have FASTPOLL set? Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) IIUG Board of Directors (art@iiug.org) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Wed, Nov 24, 2010 at 8:47 AM, Miguel Carbone <miguel@mcsoftware.com.br>wrote: > Hi, > > I am having some issues in a new environment we just installed and would > like to ask some tips to monitor the open connection time. > > Informix 1.50 FC7W1GE on Linux RedHat 5.5 > Client 3.50 FC4 > > I am monitoring the transaction time for a transaction from the client > perspective, and, some times the transaction time increase 10 times. > When I look inside, this increase is at the open connection time. > How I can monitor the connection threads (soctcppoll). > I didn't see any bottleneck in the Informix side. > How I can monitor the Operational System, looking inside Informix? How > I can know if I am waiting some Linux function? > How I can debug the open connection looking inside the client? > > I am not using DNS anymore, The Network is stable and fast, and the > behavior still happens and the problem is intermittent and I am not able > to trace it. > > Tks > -- > Grato > > Miguel Carbone > +55 11 96347103 > MC Software Ltda. - miguel@mcsoftware.com.br > <mailto:miguel@mcsoftware.com.br> - www.mcsoftware.com.br > IIUG Board of Directors - International Informix Users Group - > miguel@iiug.org <mailto:miguel@iiug.org> - www.iiug.org > President - Brazilian Informix Users Group - miguel@briug.org > <mailto:miguel@briug.org> - www.briug.org <mailto:www.briug.org> > > MC SOFTWARE LTDA. > Rua Coronel Oscar Porto, 813 - cjto T1 e T2 > Paraíso - São Paulo - Brasil - 04003004 > +55 11 2594 0048 > Localização > > < > http://maps.google.com.br/maps?f=q&source=s_q&hl=pt-br&geocode=&q=mc+software&sl l=-14.179186,-50.449219&sspn=57.748661,108.457031&ie=UTF8&hq=mc+software&hnear=& ll=-23.537865,-46.645203&spn=0.111109,0.21183&z=12&iwloc=A > > > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --20cf30433f2a149ea70495cccd89

On Wed, Nov 24, 2010 at 1:47 PM, Miguel Carbone <miguel@mcsoftware.com.br>wrote: > Hi, > > I am having some issues in a new environment we just installed and would > like to ask some tips to monitor the open connection time. > > Informix 1.50 FC7W1GE on Linux RedHat 5.5 > Client 3.50 FC4 > > I am monitoring the transaction time for a transaction from the client > perspective, and, some times the transaction time increase 10 times. > When I look inside, this increase is at the open connection time. > How I can monitor the connection threads (soctcppoll). > I didn't see any bottleneck in the Informix side. > How I can monitor the Operational System, looking inside Informix? How > I can know if I am waiting some Linux function? > How I can debug the open connection looking inside the client? > > I am not using DNS anymore, The Network is stable and fast, and the > behavior still happens and the problem is intermittent and I am not able > to trace it. > > Tks > Are you sure you're not using DNS? I've seen very weird situation in Linux specially when Linux is using TCP/IP V6. And the settings in /etc/nsswitch.conf sometimes is confusing. Being a sporadic situation is strange... Do you have anything weird at the same time in online.log? Are you doing too many simultaneous connects? If yes, you could check the LISTEN_TIMEOUT parameter and MAX_INCOMPLETE_CONNECTIONS... There are no easy ways to check this in a sporadic situation... If it happened all the time you could trace the adm/msc VPs... Regards. > -- > Grato > > Miguel Carbone > +55 11 96347103 > MC Software Ltda. - miguel@mcsoftware.com.br > <mailto:miguel@mcsoftware.com.br> - www.mcsoftware.com.br > IIUG Board of Directors - International Informix Users Group - > miguel@iiug.org <mailto:miguel@iiug.org> - www.iiug.org > President - Brazilian Informix Users Group - miguel@briug.org > <mailto:miguel@briug.org> - www.briug.org <mailto:www.briug.org> > > MC SOFTWARE LTDA. > Rua Coronel Oscar Porto, 813 - cjto T1 e T2 > Paraíso - São Paulo - Brasil - 04003004 > +55 11 2594 0048 > Localização > > < > http://maps.google.com.br/maps?f=q&source=s_q&hl=pt-br&geocode=&q=mc+software&sl l=-14.179186,-50.449219&sspn=57.748661,108.457031&ie=UTF8&hq=mc+software&hnear=& ll=-23.537865,-46.645203&spn=0.111109,0.21183&z=12&iwloc=A > > > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently... --0015174948102a16570495ce2b3d

Have you looked at things like the switch settings on the network, using auto on ports can case issues. If you know its 100-Full and always will be then set it to 100-full otherwise the switch will periodically check and it can take time. Cheers Paul -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Fernando Nunes Sent: Wednesday, November 24, 2010 9:31 AM To: ids@iiug.org Subject: Re: How to Monitor Open Connection Tiime??? [22035] On Wed, Nov 24, 2010 at 1:47 PM, Miguel Carbone <miguel@mcsoftware.com.br>wrote: > Hi, > > I am having some issues in a new environment we just installed and would > like to ask some tips to monitor the open connection time. > > Informix 1.50 FC7W1GE on Linux RedHat 5.5 > Client 3.50 FC4 > > I am monitoring the transaction time for a transaction from the client > perspective, and, some times the transaction time increase 10 times. > When I look inside, this increase is at the open connection time. > How I can monitor the connection threads (soctcppoll). > I didn't see any bottleneck in the Informix side. > How I can monitor the Operational System, looking inside Informix? How > I can know if I am waiting some Linux function? > How I can debug the open connection looking inside the client? > > I am not using DNS anymore, The Network is stable and fast, and the > behavior still happens and the problem is intermittent and I am not able > to trace it. > > Tks > Are you sure you're not using DNS? I've seen very weird situation in Linux specially when Linux is using TCP/IP V6. And the settings in /etc/nsswitch.conf sometimes is confusing. Being a sporadic situation is strange... Do you have anything weird at the same time in online.log? Are you doing too many simultaneous connects? If yes, you could check the LISTEN_TIMEOUT parameter and MAX_INCOMPLETE_CONNECTIONS... There are no easy ways to check this in a sporadic situation... If it happened all the time you could trace the adm/msc VPs... Regards. > -- > Grato > > Miguel Carbone > +55 11 96347103 > MC Software Ltda. - miguel@mcsoftware.com.br > <mailto:miguel@mcsoftware.com.br> - www.mcsoftware.com.br > IIUG Board of Directors - International Informix Users Group - > miguel@iiug.org <mailto:miguel@iiug.org> - www.iiug.org > President - Brazilian Informix Users Group - miguel@briug.org > <mailto:miguel@briug.org> - www.briug.org <mailto:www.briug.org> > > MC SOFTWARE LTDA. > Rua Coronel Oscar Porto, 813 - cjto T1 e T2 > Paraíso - São Paulo - Brasil - 04003004 > +55 11 2594 0048 > Localização > > < > http://maps.google.com.br/maps?f=q <http://maps.google.com.br/maps?f=q&source=s_q&hl=pt-br&geocode=&q=mc+softwa re&sll=-14.179186,-50.449219&sspn=57.748661,108.457031&ie=UTF8&hq=mc+softwar e&hnear=&ll=-23.537865,-46.645203&spn=0.111109,0.21183&z=12&iwloc=A> &source=s_q&hl=pt-br&geocode=&q=mc+software&sll=-14.179186,-50.449219&sspn=5 7.748661,108.457031&ie=UTF8&hq=mc+software&hnear=&ll=-23.537865,-46.645203&s pn=0.111109,0.21183&z=12&iwloc=A > > > > > > **************************************************************************** *** > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently... --0015174948102a16570495ce2b3d **************************************************************************** *** Forum Note: Use "Reply" to post a response in the discussion forum. _____ avast! Antivirus <http://www.avast.com>; : Outbound message clean. Virus Database (VPS): 101124-0, 11/24/2010 Tested on: 11/24/2010 10:14:49 AM avast! - copyright (c) 1988-2010 ALWIL Software.

NETTYPE soctcp, 8, 200, NET NETTYPE ipcshm 1, 20 , CPU FASTPOLL 1 I have 500 connections, but in my environment, the application open the connection, do the authorization (like a credit card for health insurance), and close the connection. I have the SMTP service active. Can the SMTP service do that? Em 24/11/2010 10:52, Art Kagel escreveu: > How many active connections? What are the NETTYPE settings? Do you have > FASTPOLL set?> > Art > > Art S. Kagel > Advanced DataTools (www.advancedatatools.com) > IIUG Board of Directors (art@iiug.org) > Blog: http://informix-myview.blogspot.com/ > > Disclaimer: Please keep in mind that my own opinions are my own opinions and > do not reflect on my employer, Advanced DataTools, the IIUG, nor any other > organization with which I am associated either explicitly, implicitly, or by > inference. Neither do those opinions reflect those of other individuals > affiliated with any entity with which I am affiliated nor those of the > entities themselves. > > On Wed, Nov 24, 2010 at 8:47 AM, Miguel Carbone > <miguel@mcsoftware.com.br>wrote: > >> Hi, >> >> I am having some issues in a new environment we just installed and would >> like to ask some tips to monitor the open connection time. >> >> Informix 1.50 FC7W1GE on Linux RedHat 5.5 >> Client 3.50 FC4 >> >> I am monitoring the transaction time for a transaction from the client >> perspective, and, some times the transaction time increase 10 times. >> When I look inside, this increase is at the open connection time. >> How I can monitor the connection threads (soctcppoll). >> I didn't see any bottleneck in the Informix side. >> How I can monitor the Operational System, looking inside Informix? How >> I can know if I am waiting some Linux function? >> How I can debug the open connection looking inside the client? >> >> I am not using DNS anymore, The Network is stable and fast, and the >> behavior still happens and the problem is intermittent and I am not able >> to trace it. >> >> Tks >> -- >> Grato >> >> Miguel Carbone >> +55 11 96347103 >> MC Software Ltda. - miguel@mcsoftware.com.br >> <mailto:miguel@mcsoftware.com.br> - www.mcsoftware.com.br >> IIUG Board of Directors - International Informix Users Group - >> miguel@iiug.org<mailto:miguel@iiug.org> - www.iiug.org >> President - Brazilian Informix Users Group - miguel@briug.org >> <mailto:miguel@briug.org> - www.briug.org<mailto:www.briug.org> >> >> MC SOFTWARE LTDA. >> Rua Coronel Oscar Porto, 813 - cjto T1 e T2 >> Paraíso - São Paulo - Brasil - 04003004 >> +55 11 2594 0048 >> Localização >> >> < >> > http://maps.google.com.br/maps?f=q&source=s_q&hl=pt-br&geocode=&q=mc+software&sl l=-14.179186,-50.449219&sspn=57.748661,108.457031&ie=UTF8&hq=mc+software&hnear=& ll=-23.537865,-46.645203&spn=0.111109,0.21183&z=12&iwloc=A >> >> >> > ******************************************************************************* >> Forum Note: Use "Reply" to post a response in the discussion forum. >> >> > --20cf30433f2a149ea70495cccd89 > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Grato Miguel Carbone +55 11 96347103 MC Software Ltda. - miguel@mcsoftware.com.br <mailto:miguel@mcsoftware.com.br> - www.mcsoftware.com.br IIUG Board of Directors - International Informix Users Group - miguel@iiug.org <mailto:miguel@iiug.org> - www.iiug.org President - Brazilian Informix Users Group - miguel@briug.org <mailto:miguel@briug.org> - www.briug.org <mailto:www.briug.org> MC SOFTWARE LTDA. Rua Coronel Oscar Porto, 813 - cjto T1 e T2 Paraíso - São Paulo - Brasil - 04003004 +55 11 2594 0048 Localização <http://maps.google.com.br/maps?f=q&source=s_q&hl=pt-br&geocode=&q=mc+software&s ll=-14.179186,-50.449219&sspn=57.748661,108.457031&ie=UTF8&hq=mc+software&hnear= &ll=-23.537865,-46.645203&spn=0.111109,0.21183&z=12&iwloc=A>

Thanks Fernando When this issue hjappens the msc vp stays runnig and how much msc runs is how much the open connection time increase I have 4 mcs vp's but only one is running at this time Sent by BlackBerry® Miguel Carbone +55 11 96347103 miguel@mcsoftware.com.br MC.Software - CTO miguel@iiug.org IIUG - International Informix Users Group Board of Directors -----Original Message----- From: "Fernando Nunes" <domusonline@gmail.com> Sender: ids-bounces@iiug.org Date: Wed, 24 Nov 2010 10:30:34 To: <ids@iiug.org> Reply-To: ids@iiug.org Subject: Re: How to Monitor Open Connection Tiime??? [22035] On Wed, Nov 24, 2010 at 1:47 PM, Miguel Carbone <miguel@mcsoftware.com.br>wrote: > Hi, > > I am having some issues in a new environment we just installed and would > like to ask some tips to monitor the open connection time. > > Informix 1.50 FC7W1GE on Linux RedHat 5.5 > Client 3.50 FC4 > > I am monitoring the transaction time for a transaction from the client > perspective, and, some times the transaction time increase 10 times. > When I look inside, this increase is at the open connection time. > How I can monitor the connection threads (soctcppoll). > I didn't see any bottleneck in the Informix side. > How I can monitor the Operational System, looking inside Informix? How > I can know if I am waiting some Linux function? > How I can debug the open connection looking inside the client? > > I am not using DNS anymore, The Network is stable and fast, and the > behavior still happens and the problem is intermittent and I am not able > to trace it. > > Tks > Are you sure you're not using DNS? I've seen very weird situation in Linux specially when Linux is using TCP/IP V6. And the settings in /etc/nsswitch.conf sometimes is confusing. Being a sporadic situation is strange... Do you have anything weird at the same time in online.log? Are you doing too many simultaneous connects? If yes, you could check the LISTEN_TIMEOUT parameter and MAX_INCOMPLETE_CONNECTIONS... There are no easy ways to check this in a sporadic situation... If it happened all the time you could trace the adm/msc VPs... Regards. > -- > Grato > > Miguel Carbone > +55 11 96347103 > MC Software Ltda. - miguel@mcsoftware.com.br > <mailto:miguel@mcsoftware.com.br> - www.mcsoftware.com.br > IIUG Board of Directors - International Informix Users Group - > miguel@iiug.org <mailto:miguel@iiug.org> - www.iiug.org > President - Brazilian Informix Users Group - miguel@briug.org > <mailto:miguel@briug.org> - www.briug.org <mailto:www.briug.org> > > MC SOFTWARE LTDA. > Rua Coronel Oscar Porto, 813 - cjto T1 e T2 > Paraíso - São Paulo - Brasil - 04003004 > +55 11 2594 0048 > Localização > > < > http://maps.google.com.br/maps?f=q&source=s_q&hl=pt-br&geocode=&q=mc+software&sll=-14.179186,-50.449219&sspn=57.748661,108.457031&ie=UTF8&hq=mc+software&hnear=&ll=-23.537865,-46.645203&spn=0.111109,0.21183&z=12&iwloc=A > > > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently... --0015174948102a16570495ce2b3d ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

Fernando Thanks, I did some tests and I realize that the msc vp stays running when the issue happens. How much time the msc VP stays running are how much time are incceased at the open connection time. My connection time is something near 0.002 sec and goes to 3.0 sec I will try to understand what this msc VP is doing to take too much time... Miguel Em 24/11/2010 12:30, Fernando Nunes escreveu: > On Wed, Nov 24, 2010 at 1:47 PM, Miguel Carbone > <miguel@mcsoftware.com.br>wrote: > >> Hi, >> >> I am having some issues in a new environment we just installed and would >> like to ask some tips to monitor the open connection time. >> >> Informix 1.50 FC7W1GE on Linux RedHat 5.5 >> Client 3.50 FC4 >> >> I am monitoring the transaction time for a transaction from the client >> perspective, and, some times the transaction time increase 10 times. >> When I look inside, this increase is at the open connection time. >> How I can monitor the connection threads (soctcppoll). >> I didn't see any bottleneck in the Informix side. >> How I can monitor the Operational System, looking inside Informix? How >> I can know if I am waiting some Linux function? >> How I can debug the open connection looking inside the client? >> >> I am not using DNS anymore, The Network is stable and fast, and the >> behavior still happens and the problem is intermittent and I am not able >> to trace it. >> >> Tks >> > Are you sure you're not using DNS? I've seen very weird situation in Linux > specially when Linux is using TCP/IP V6. > And the settings in /etc/nsswitch.conf sometimes is confusing. > Being a sporadic situation is strange... > Do you have anything weird at the same time in online.log? > Are you doing too many simultaneous connects? If yes, you could check the > LISTEN_TIMEOUT parameter and MAX_INCOMPLETE_CONNECTIONS... > There are no easy ways to check this in a sporadic situation... If it > happened all the time you could trace the adm/msc VPs... > > Regards. > >> -- >> Grato >> >> Miguel Carbone >> +55 11 96347103 >> MC Software Ltda. - miguel@mcsoftware.com.br >> <mailto:miguel@mcsoftware.com.br> - www.mcsoftware.com.br >> IIUG Board of Directors - International Informix Users Group - >> miguel@iiug.org<mailto:miguel@iiug.org> - www.iiug.org >> President - Brazilian Informix Users Group - miguel@briug.org >> <mailto:miguel@briug.org> - www.briug.org<mailto:www.briug.org> >> >> MC SOFTWARE LTDA. >> Rua Coronel Oscar Porto, 813 - cjto T1 e T2 >> Paraíso - São Paulo - Brasil - 04003004 >> +55 11 2594 0048 >> Localização >> >> < >> > http://maps.google.com.br/maps?f=q&source=s_q&hl=pt-br&geocode=&q=mc+software&sl l=-14.179186,-50.449219&sspn=57.748661,108.457031&ie=UTF8&hq=mc+software&hnear=& ll=-23.537865,-46.645203&spn=0.111109,0.21183&z=12&iwloc=A >> >> >> > ******************************************************************************* >> Forum Note: Use "Reply" to post a response in the discussion forum. >> >> -- Grato Miguel Carbone +55 11 96347103 MC Software Ltda. - miguel@mcsoftware.com.br <mailto:miguel@mcsoftware.com.br> - www.mcsoftware.com.br IIUG Board of Directors - International Informix Users Group - miguel@iiug.org <mailto:miguel@iiug.org> - www.iiug.org President - Brazilian Informix Users Group - miguel@briug.org <mailto:miguel@briug.org> - www.briug.org <mailto:www.briug.org> MC SOFTWARE LTDA. Rua Coronel Oscar Porto, 813 - cjto T1 e T2 Paraíso - São Paulo - Brasil - 04003004 +55 11 2594 0048 Localização <http://maps.google.com.br/maps?f=q&source=s_q&hl=pt-br&geocode=&q=mc+software&s ll=-14.179186,-50.449219&sspn=57.748661,108.457031&ie=UTF8&hq=mc+software&hnear= &ll=-23.537865,-46.645203&spn=0.111109,0.21183&z=12&iwloc=A>

The misc vp does all the network authentication to and from the OS. Y= ou can have multiple misc VP in version 11 (and maybe later versions of 10). John F. Miller III STSM, Embedability Architect miller3@us.ibm.com 503-578-5645 IBM Informix Dynamic Server (IDS) ids-bounces@iiug.org wrote on 11/24/2010 11:13:11 AM: > From: > > "Miguel Carbone" <miguel@mcsoftware.com.br> > > To: > > ids@iiug.org > > Date: > > 11/24/2010 11:14 AM > > Subject: > > Re: How to Monitor Open Connection Tiime??? [22043] > > Sent by: > > ids-bounces@iiug.org > > Fernando > > Thanks, I did some tests and I realize that the msc vp stays running > when the issue happens. How much time the msc VP stays running are ho= w > much time are incceased at the open connection time. > My connection time is something near 0.002 sec and goes to 3.0 sec > > I will try to understand what this msc VP is doing to take too much time... > > Miguel > > Em 24/11/2010 12:30, Fernando Nunes escreveu: > > On Wed, Nov 24, 2010 at 1:47 PM, Miguel Carbone > > <miguel@mcsoftware.com.br>wrote: > > > >> Hi, > >> > >> I am having some issues in a new environment we just installed and= would > >> like to ask some tips to monitor the open connection time. > >> > >> Informix 1.50 FC7W1GE on Linux RedHat 5.5 > >> Client 3.50 FC4 > >> > >> I am monitoring the transaction time for a transaction from the cl= ient > >> perspective, and, some times the transaction time increase 10 time= s. > >> When I look inside, this increase is at the open connection time. > >> How I can monitor the connection threads (soctcppoll). > >> I didn't see any bottleneck in the Informix side. > >> How I can monitor the Operational System, looking inside Informix?= How > >> I can know if I am waiting some Linux function? > >> How I can debug the open connection looking inside the client? > >> > >> I am not using DNS anymore, The Network is stable and fast, and th= e > >> behavior still happens and the problem is intermittent and I am no= t able > >> to trace it. > >> > >> Tks > >> > > Are you sure you're not using DNS? I've seen very weird situation i= n Linux > > specially when Linux is using TCP/IP V6. > > And the settings in /etc/nsswitch.conf sometimes is confusing. > > Being a sporadic situation is strange... > > Do you have anything weird at the same time in online.log? > > Are you doing too many simultaneous connects? If yes, you could che= ck the > > LISTEN_TIMEOUT parameter and MAX_INCOMPLETE_CONNECTIONS... > > There are no easy ways to check this in a sporadic situation... If = it > > happened all the time you could trace the adm/msc VPs... > > > > Regards. > > > >> -- > >> Grato > >> > >> Miguel Carbone > >> +55 11 96347103 > >> MC Software Ltda. - miguel@mcsoftware.com.br > >> <mailto:miguel@mcsoftware.com.br> - www.mcsoftware.com.br > >> IIUG Board of Directors - International Informix Users Group - > >> miguel@iiug.org<mailto:miguel@iiug.org> - www.iiug.org > >> President - Brazilian Informix Users Group - miguel@briug.org > >> <mailto:miguel@briug.org> - www.briug.org<mailto:www.briug.org> > >> > >> MC SOFTWARE LTDA. > >> Rua Coronel Oscar Porto, 813 - cjto T1 e T2 > >> Para=EDso - S=E3o Paulo - Brasil - 04003004 > >> +55 11 2594 0048 > >> Localiza=E7=E3o > >> > >> < > >> > > > http://maps.google.com.br/maps?f=3Dq&source=3Ds_q&hl=3Dpt-br&geocode=3D= &q=3Dmc > +software&sll=3D-14.179186,-50.449219&sspn=3D57.748661,108. > 457031&ie=3DUTF8&hq=3Dmc+software&hnear=3D&ll=3D-23.537865,-46.645203= &spn=3D0. > 111109,0.21183&z=3D12&iwloc=3DA > >> > >> > >> > > > ***********************************************************************= ******** > >> Forum Note: Use "Reply" to post a response in the discussion forum= . > >> > >> > > -- > Grato > > Miguel Carbone > +55 11 96347103 > MC Software Ltda. - miguel@mcsoftware.com.br > <mailto:miguel@mcsoftware.com.br> - www.mcsoftware.com.br > IIUG Board of Directors - International Informix Users Group - > miguel@iiug.org <mailto:miguel@iiug.org> - www.iiug.org > President - Brazilian Informix Users Group - miguel@briug.org > <mailto:miguel@briug.org> - www.briug.org <mailto:www.briug.org> > > MC SOFTWARE LTDA. > Rua Coronel Oscar Porto, 813 - cjto T1 e T2 > Para=EDso - S=E3o Paulo - Brasil - 04003004 > +55 11 2594 0048 > Localiza=E7=E3o > > <http://maps.google.com.br/maps?f=3Dq&source=3Ds_q&hl=3Dpt- > br&geocode=3D&q=3Dmc+software&sll=3D-14.179186,-50.449219&sspn=3D57.7= 48661, > 108.457031&ie=3DUTF8&hq=3Dmc+software&hnear=3D&ll=3D-23.537865,-46. > 645203&spn=3D0.111109,0.21183&z=3D12&iwloc=3DA> > > > ***********************************************************************= ******** > Forum Note: Use "Reply" to post a response in the discussion forum.= >=

On Wed, Nov 24, 2010 at 8:13 PM, Miguel Carbone <miguel@mcsoftware.com.br>wrote: > Fernando > > Thanks, I did some tests and I realize that the msc vp stays running when > the issue happens. How much time the msc VP stays running are how much time > are incceased at the open connection time. > My connection time is something near 0.002 sec and goes to 3.0 sec > > I will try to understand what this msc VP is doing to take too much time... > > Miguel > > If you're running on Linux you can use strace command (I think you'll require root, but I'd have to check). Note that while stracing a process it will work slower... I would have to revisit the subject but I think MSC will do the reverse DNS... So again... keep an open mind when you say you're not using DNS :) A short strace will show important information... As a side note, it looks like you're making a connection for each client request. That is a very good idea, so even if you solve this issue you should review that way of working... Or at least upgrade to 11.7 and use NS_CACHE :) Regards. -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently... --0015174948103de3250495d353c4

Hi , I'm working with Miguel on this situation . After some debugs with strace, tcpdump, application log we could confirm the problem is the DNS reverse. But is more of one problem together. Let me try explain what happen here: 1) the database was started and at this moment the DNS configuration on Linux (/etc/resolv.conf) isn't correct, they appoint to old DNS. 2) The Linux admin, change the resolv.conf and solve the problem with DNS. 3) the problem into Informix persist and we cannot bounce the instance neither restart the listeners. 4) with strace / tcpdump we detected the Informix still trying do the DNS reverse to old DNS (configured when the database was started) 5) The problem of this long time to open connection occur with connections of application what incoming from a differ network (remote network, behind of a firewall). 6) All other connections what occur at same time, suffer a delay ... the application what we are monitoring is running on the same network and the hostname is in /etc/hosts. 7) The instance was configured with 4 MSC VPs since they up. But only one VP works and when they "freeze" into the DNS reverse, the others MSC VPs just still doing nothing (they should threat the new connections, I suppose) So, here we have 2 problems: - Informix cache the IP of DNS servers and don't update it... - The MSC VP don't parallelize the requests.. (we already open a PMR for this) Reading the manual Admin Guide, I found this: "When a poll thread receives a connection request from a client, it passes the request to the listen thread for the port. The listen thread authenticates the user, establishes the connection to the database server, and starts an sqlexec thread, the session thread that performs the primary processing for the client." So, based on this quote, the listen thread have the "responsibility" the call the MSC VP to do the rest of the job and they "freeze" at the open connection, I deduce.. If I have *more listen threads*, maybe I can soften my problem, where the others listeners will work in parallel and *maybe* work with the others MSC VPs while the first MSC is freeze over the DNS reverse. Then I found a configuration on the manual what I don't have notice before. Configuring more listeners threads for the same service!!!! DBSERVERNAME ifxtestchange to DBSERVERNAME ifxtest-3 # for 3 listener threads (soctcplst) . I tested on v11.70 and works (they create 3 soctcplst)... and appear to be available on version 11.50 too . :) Back to our situation here, how we can not bounce the instance (to update the dbservername), this isn't an available solution at this moment. But is very interesting for new configurations! My questions: 1) Anyone know if more listeners threads will work fine with more MSC VPs? (parallelizing) 2) Know if is stable for production environment? (v11.50 xC7) ? Cesar On 11/24/2010 07:39 PM, Fernando Nunes wrote: > On Wed, Nov 24, 2010 at 8:13 PM, Miguel Carbone > <miguel@mcsoftware.com.br>wrote: > >> Fernando >> >> Thanks, I did some tests and I realize that the msc vp stays running when >> the issue happens. How much time the msc VP stays running are how much time >> are incceased at the open connection time. >> My connection time is something near 0.002 sec and goes to 3.0 sec >> >> I will try to understand what this msc VP is doing to take too much time... >> >> Miguel >> >> > If you're running on Linux you can use strace command (I think you'll > require root, but I'd have to check). > Note that while stracing a process it will work slower... > I would have to revisit the subject but I think MSC will do the reverse > DNS... So again... keep an open mind when you say you're not using DNS :) > A short strace will show important information... > > As a side note, it looks like you're making a connection for each client > request. That is a very good idea, so even if you solve this issue you > should review that way of working... Or at least upgrade to 11.7 and use > NS_CACHE :)> > Regards.

Hello Cesar. Please see the comments below. On Sun, Nov 28, 2010 at 3:20 AM, Cesar Inacio Martins < cesar_inacio_martins@yahoo.com.br> wrote: > Hi , > > I'm working with Miguel on this situation . > After some debugs with strace, tcpdump, application log we could confirm > the problem is the DNS reverse. > But is more of one problem together. > Let me try explain what happen here: > > 1) the database was started and at this moment the DNS configuration on > Linux (/etc/resolv.conf) isn't correct, they appoint to old DNS. > This is an "obscure"/grey area. AFAIK Informix does not cache anything. It only calls gethostbyaddr(). The behavior of this function may differ from OS to OS. > 2) The Linux admin, change the resolv.conf and solve the problem with DNS. > 3) the problem into Informix persist and we cannot bounce the instance > neither restart the listeners. > You don't mention the version (maybe 11.50.xc7?). If your version allows for dynamic listener startup you could launch some other(s). Apparently you want to launch other listeners for the same port. But you dont' have to do that.... You can create several (not many) listeners on the server and configure your clients to use a group that includes them. This will spread the connections for several ports. You could also establish some more agressive values for INFORMIXCONNTIME and INFORMIXCONRETRY. So you could workaround this by launching further listeners and appropriate configuration on the client side... > 4) with strace / tcpdump we detected the Informix still trying do the > DNS reverse to old DNS (configured when the database was started) > 5) The problem of this long time to open connection occur with > connections of application what incoming from a differ network (remote > network, behind of a firewall). > 6) All other connections what occur at same time, suffer a delay ... the > application what we are monitoring is running on the same network and > the hostname is in /etc/hosts. > If you have the entries in the /etc/hosts it should not be talking to the DNS... Unless your settings in /etc/nsswitch.conf are not right. Please check that also. I'm not sure if a modification on this would require a bounce in the engine. If you have doubts please post your /etc/nsswitch.conf > 7) The instance was configured with 4 MSC VPs since they up. But only > one VP works and when they "freeze" into the DNS reverse, the others MSC > VPs just still doing nothing (they should threat the new connections, I > suppose) > I suppose you're becoming "hanged" on the listeners and not on the MSC VPs. > So, here we have 2 problems: > - Informix cache the IP of DNS servers and don't update it... > Already replied... I don't think it's IDS, for the matter is pretty irrelevant.... > - The MSC VP don't parallelize the requests.. > (we already open a PMR for this) > Also replied... I think the problem lies in the listeners, not the MSC > > Reading the manual Admin Guide, I found this: > "When a poll thread receives a connection request from a client, it > passes the > request to the listen thread for the port. The listen thread > authenticates the user, > establishes the connection to the database server, and starts an sqlexec > thread, the > session thread that performs the primary processing for the client." > > So, based on this quote, the listen thread have the "responsibility" the > call the MSC VP to do the rest of the job and they "freeze" at the open > connection, I deduce.. > If I have *more listen threads*, maybe I can soften my problem, where > the others listeners will work in parallel and *maybe* work with the > others MSC VPs while the first MSC is freeze over the DNS reverse. > > That's what I think should happen. So, the idea above (several listeners "grouped" on the client side) could help you. > Then I found a configuration on the manual what I don't have notice > before. Configuring more listeners threads for the same service!!!! > DBSERVERNAME ifxtest> change to > DBSERVERNAME ifxtest-3 # for 3 listener threads (soctcplst) .> > I tested on v11.70 and works (they create 3 soctcplst)... and appear to > be available on version 11.50 too . :) > I'm ashamed, but I haven't noticed this... But now that you mention, I'd say it's there since v10. One of the gurus may be able to confirm this. > Back to our situation here, how we can not bounce the instance (to > update the dbservername), this isn't an available solution at this > moment. But is very interesting for new configurations! > > My questions: > 1) Anyone know if more listeners threads will work fine with more MSC > VPs? (parallelizing) > I suppose so, but I can't give you a definitive answer. > 2) Know if is stable for production environment? (v11.50 xC7) ? > > If you mean several threads for the same service, as I say above I haven't used it. But if it's there since V10 (needs confirmation) than It should be stable. But again, I think you could solve this without bouncing the engine. Start more listeners and use a group in the client side that include some of them. Regards. -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently... --000e0cd1e1640d8b5204962516cc

the IPv6 version DNS name resolution is very slow, you can bring up OS nscd (name service caching daemon) /usr/sbin/nscd or disable IPv6 with Disabling IPv6 Support as below Informix also provides a way to disable IPv6 support when working in IPv4 environments. To disable IPv6 support for all database instances and client applications: * Create an empty file called $INFORMIXDIR/etc/IFX_DISABLE_IPV6. The file should have read permission for user informix. The file is not read from or written to, and does not need to contain any data. To disable IPv6 support for a single database instance or for a single client application: * On the database server instance, or on the machine on which applications are run, create an environment variable named IFX_DISABLE_IPV6 and set its value to yes, as in: IFX_DISABLE_IPV6=yes

Hi Fernando! Thanks for your message! Sorry take to long to answer, I don't know why the last 2 days of messages incoming from IIUG has arrived just now. So, about the gethostbyaddr() , I'm not sure about that because isn't what we see on the strace . The request of DNS reverse lookup, appear to be executed "manually" by Informix or the strace just traced the gethostbyaddr() too. Check the output marked with "##" by me. where: 172.18.0.57 and 172.18.0.119 are the OLD DNS when the server was started. When we run this strace, the resolv.conf already have new values (for at least 2 days): 11:24:28 munmap(0x2abeb6d75000, 4096) = 0 11:24:28 socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 3 ##11:24:28 connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("172.18.0.57")}, 28) = 0 11:24:28 fcntl(3, F_GETFL) = 0x2 (flags O_RDWR) 11:24:28 fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0 11:24:28 poll([{fd=3, events=POLLOUT}], 1, 0) = 1 ([{fd=3, revents=POLLOUT}]) ##11:24:28 sendto(3, "\\\\242;\\\\1\\\\0\\\\0\\\\1\\\\0\\\\0\\\\0\\\\0\\\\0\\\\0\\\\003185\\\\00248\\\\00222\\\\003172\\\\7in-ad"..., 44, MSG_NOSIGNAL, NULL, 0) = 44 ##11:24:28 poll([{fd=3, events=POLLIN}], 1, 5000) = 0 (Timeout) 11:24:33 socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 4 11:24:33 connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("172.18.0.119")}, 28) = 0 11:24:33 fcntl(4, F_GETFL) = 0x2 (flags O_RDWR) ... they continue trying to third DNS server... (check the timestamp,.. 5 seconds of timeout delay) Before you ask, the Linux Admin already try change the timeout (option into resolv.conf) and don't have effect.... Sorry , yes is a 11.50 FC7 GE over Linux Red Hat 5.5 This option to create a group is nice, but I not sure if is viable for this environment because have a lot of applications spread on the company (.net/windows, java/web, C/Linux) what we will need care about to change the SQLHOSTS... The nsswitch.conf is ok, it isn't modified from they default values... and the problem is with connections what become from out of our network (what don't exists into /etc/hosts) and create consequences to local connections (what exists in hosts) when the instance get trouble to try resolve their hostnames (not local clients) And the problem what we detected is over MSC, because is it what freeze... The most weird thing is, when the MSC "freeze" (stuck running in active threads: onstat -g act) they stack dump (onstat -g stk #threadid) don't change anything, still showing in yield process...and just change the status from sleep to running.... About IPv6 I'm answering into the other message... On 11/28/2010 09:12 PM, Fernando Nunes wrote: > Hello Cesar. > Please see the comments below. > > On Sun, Nov 28, 2010 at 3:20 AM, Cesar Inacio Martins< > cesar_inacio_martins@yahoo.com.br> wrote: > >> Hi , >> >> I'm working with Miguel on this situation . >> After some debugs with strace, tcpdump, application log we could confirm >> the problem is the DNS reverse. >> But is more of one problem together. >> Let me try explain what happen here: >> >> 1) the database was started and at this moment the DNS configuration on >> Linux (/etc/resolv.conf) isn't correct, they appoint to old DNS. >> > This is an "obscure"/grey area. AFAIK Informix does not cache anything. It > only calls gethostbyaddr(). The behavior of this function may differ from OS > to OS. > >> 2) The Linux admin, change the resolv.conf and solve the problem with DNS. >> 3) the problem into Informix persist and we cannot bounce the instance >> neither restart the listeners. >> > You don't mention the version (maybe 11.50.xc7?). If your version allows for > dynamic listener startup you could launch some other(s). Apparently you want > to launch other listeners for the same port. But you dont' have to do > that.... You can create several (not many) listeners on the server and > configure your clients to use a group that includes them. This will spread > the connections for several ports. You could also establish some more > agressive values for INFORMIXCONNTIME and INFORMIXCONRETRY. So you could > workaround this by launching further listeners and appropriate configuration > on the client side... > >> 4) with strace / tcpdump we detected the Informix still trying do the >> DNS reverse to old DNS (configured when the database was started) >> 5) The problem of this long time to open connection occur with >> connections of application what incoming from a differ network (remote >> network, behind of a firewall). >> 6) All other connections what occur at same time, suffer a delay ... the >> application what we are monitoring is running on the same network and >> the hostname is in /etc/hosts. >> > If you have the entries in the /etc/hosts it should not be talking to the > DNS... Unless your settings in /etc/nsswitch.conf are not right. > Please check that also. I'm not sure if a modification on this would require > a bounce in the engine. > If you have doubts please post your /etc/nsswitch.conf > >> 7) The instance was configured with 4 MSC VPs since they up. But only >> one VP works and when they "freeze" into the DNS reverse, the others MSC >> VPs just still doing nothing (they should threat the new connections, I >> suppose) >> > I suppose you're becoming "hanged" on the listeners and not on the MSC VPs. > >> So, here we have 2 problems: >> - Informix cache the IP of DNS servers and don't update it... >> > Already replied... I don't think it's IDS, for the matter is pretty > irrelevant.... > >> - The MSC VP don't parallelize the requests.. >> (we already open a PMR for this) >> > Also replied... I think the problem lies in the listeners, not the MSC > >> Reading the manual Admin Guide, I found this: >> "When a poll thread receives a connection request from a client, it >> passes the >> request to the listen thread for the port. The listen thread >> authenticates the user, >> establishes the connection to the database server, and starts an sqlexec >> thread, the >> session thread that performs the primary processing for the client." >> >> So, based on this quote, the listen thread have the "responsibility" the >> call the MSC VP to do the rest of the job and they "freeze" at the open >> connection, I deduce.. >> If I have *more listen threads*, maybe I can soften my problem, where >> the others listeners will work in parallel and *maybe* work with the >> others MSC VPs while the first MSC is freeze over the DNS reverse. >> >> That's what I think should happen. So, the idea above (several listeners > "grouped" on the client side) could help you. > >> Then I found a configuration on the manual what I don't have notice >> before. Configuring more listeners threads for the same service!!!! >> DBSERVERNAME ifxtest>> change to >> DBSERVERNAME ifxtest-3 # for 3 listener threads (soctcplst) .>> >> I tested on v11.70 and works (they create 3 soctcplst)... and appear to >> be available on version 11.50 too . :) >> > I'm ashamed, but I haven't noticed this... > But now that you mention, I'

Hi Jim, Thanks for your message! So, we already think on this possibility (IPv6)... because with netstat -na output I can see somes "::fff:172..." . Despite having values on the network machine configuration, don't exists a DNS for IPv6 and the network of the company don't use it. So, I assume, there is no reason to not disable it... but , if I'm not missing something, for this configuration take effect we need to bounce the instance... And into the strace (what you can see on the other reply my to Fernando), I don't see nothing related to IPv6 (I not sure if this make some difference over the OS calls, what Informix appear don't use it) About nscd we already thinking about that to... but before we want trying other options... and mostly because we don't know if to Informix recognize the nscd service automatically or need bounce the instance...(I don't test this yet, but I will) And we not have sure if this solution will solve the "delay" to Informix resolve the hostname for a IP what become behind of our firwall... So, there is a lot of "trys", but this is a production environment and we can not make any change just to try, we need to be sure! On 11/30/2010 04:23 PM, JIM XIAO wrote: > the IPv6 version DNS name resolution is very slow, you can bring up OS nscd > (name service caching daemon) /usr/sbin/nscd or disable IPv6 with > Disabling IPv6 Support as below > > Informix also provides a way to disable IPv6 support when working in IPv4 > environments. > To disable IPv6 support for all database instances and client applications: > > * Create an empty file called $INFORMIXDIR/etc/IFX_DISABLE_IPV6. > > The file should have read permission for user informix. The file is not read > from or written to, and does not need to contain any data. > > To disable IPv6 support for a single database instance or for a single client > application: > > * On the database server instance, or on the machine on which applications are > run, create an environment variable named IFX_DISABLE_IPV6 and set its value > to yes, as in: > > IFX_DISABLE_IPV6=yes > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. >

On Wed, Dec 1, 2010 at 6:01 PM, Cesar Inacio Martins < cesar_inacio_martins@yahoo.com.br> wrote: > Hi Fernando! > Thanks for your message! > Sorry take to long to answer, I don't know why the last 2 days of > messages incoming from IIUG has arrived just now. > > So, about the gethostbyaddr() , I'm not sure about that because isn't > what we see on the strace . > I wrote a simple program in Fedora, to test your situation.... Several interesting things came up. They're not very helpful, but the prove Informix innocence :) On strace you will not see the reference to gethostbyaddr. Only in a debugger or if you force a stack trace in the precise moment... > The request of DNS reverse lookup, appear to be executed "manually" by > Informix or the strace just traced the gethostbyaddr() too. > Check the output marked with "##" by me. > where: 172.18.0.57 and 172.18.0.119 are the OLD DNS when the server was > started. When we run this strace, the resolv.conf already have new > values (for at least 2 days): > > 11:24:28 munmap(0x2abeb6d75000, 4096) = 0 > 11:24:28 socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 3 > ##11:24:28 connect(3, {sa_family=AF_INET, sin_port=htons(53), > sin_addr=inet_addr("172.18.0.57")}, 28) = 0 > 11:24:28 fcntl(3, F_GETFL) = 0x2 (flags O_RDWR) > 11:24:28 fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0 > 11:24:28 poll([{fd=3, events=POLLOUT}], 1, 0) = 1 ([{fd=3, > revents=POLLOUT}]) > ##11:24:28 sendto(3, > "\\\\242;\\\\1\\\\0\\\\0\\\\1\\\\0\\\\0\\\\0\\\\0\\\\0\\\\0\\\\003185\\\\00248\\\\00222\\\\003172\\\\7in-ad"..., 44, > MSG_NOSIGNAL, NULL, 0) = 44 > ##11:24:28 poll([{fd=3, events=POLLIN}], 1, 5000) = 0 (Timeout) > 11:24:33 socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 4 > 11:24:33 connect(4, {sa_family=AF_INET, sin_port=htons(53), > sin_addr=inet_addr("172.18.0.119")}, 28) = 0 > 11:24:33 fcntl(4, F_GETFL) = 0x2 (flags O_RDWR) > .... they continue trying to third DNS server... > (check the timestamp,.. 5 seconds of timeout delay) > > Before you ask, the Linux Admin already try change the timeout (option > into resolv.conf) and don't have effect.... > > 2nd interesting observation... I put the program in loop... reading an IP from the console and trying to reverse DNS it. Between loops I changed my resolve.conf.... It didn't matter for the running process. I belive gethostbyaddr creates some static structures. First time it runs it reads the resolv.conf, but it doesn't happen again... Probably due to performance reasons. But it's inconvenient in your situation....: open("/etc/resolv.conf", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0644, st_size=55, ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb776e000 read(3, "# Generated by NetworkManager\\ na"..., 4096) = 55 read(3, "", 4096) = 0 close(3) This is why it keeps trying the old DNS... Sorry , yes is a 11.50 FC7 GE over Linux Red Hat 5.5 > This option to create a group is nice, but I not sure if is viable for > this environment because have a lot of applications spread on the > company (.net/windows, java/web, C/Linux) what we will need care about > to change the SQLHOSTS... > > The nsswitch.conf is ok, it isn't modified from they default values... > and the problem is with connections what become from out of our network > (what don't exists into /etc/hosts) and create consequences to local > connections (what exists in hosts) when the instance get trouble to try > resolve their hostnames (not local clients) > Are there many client IPs from outside the network? If the number is low you could put them in /etc/hosts... > > And the problem what we detected is over MSC, because is it what freeze... > The most weird thing is, when the MSC "freeze" (stuck running in active > threads: onstat -g act) they stack dump (onstat -g stk #threadid) don't > change anything, still showing in yield process...and just change the > status from sleep to running.... > > Now... none of these are good news. I really can't see a good solution besides restart... You're a victim of how the TCP/IP name resolution works Apparently nscd will not work also, because gethostbyaddr only tries it the first time... Some other thoughts: 1- You could launch more msc VPs. The new ones should pick up the correct DNS. And before you ask, no, you cannot remove MSC VPs... But with more MSC VPs your chances of getting stuck should be lower 2- (this is very weird...) You could use IPTABLEs to hijack connections to the OLD DNS and redirect them to the new one... Not sure if this is possible, but you would have to create a rule for the old IP, port 53 and protocol UDP. 3- You could "hack" the network to make the old IP available. This could be done by putting a machine on the network with that IP, or eventually by hacking the ARP table 4- you could create another TCP interface on the machine (how?) with the IP of the old DNS server. I tried putting my own IP on the /etc/resolv.conf file and the reverse DNS returns to "quick" (although I don't have anything listening on port 53)... The problem you're facing comes from the fact that the IP cannot be reached. If you make it go to an existing IP it will not resolv, but it will be quick. Naturally 2, 3 and 4 are "crazy" suggestions... But I don't see any "sane" alternative to an instance stop. Regards. -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently... --000e0cd1e1646c4c6404966309a2

Hi Fernando, I wrote a little program here and confirm what you said about not "see" the call of gethostbyaddr... But........ I got new variables.... Testing on my net book , IFX 11.70 xC1 (isn't the same version used on our production) + OpenSuse 11.2 , open connections with changes on the resolv.conf, tracing with strace: - Each open connection, always reread hosts.equivs. (I believed this the res_init() function working inside of the gethostbyaddr). - When I change the resolv.conf , without bounce the instance, the next connection already try solve the DNS using the new configuration. So, now the question, is the Informix or Linux problem ?? I will install the same version what we have the problem on my netbook , ifx 11.50 uc7w1ge (but 32 bits..) and test. Just for curiosity, read this thread : http://fixunix.com/redhat/17199-etc-resolv-conf-how-reload.html Is a similar situation, not with Informix and over RH 4 (not 5.5) . know my suspicious now is over glibc used on this RH. When I finish my test on my netbook with the same version what we use here in production , I will post here the results. On 12/01/2010 11:07 PM, Fernando Nunes wrote: > On Wed, Dec 1, 2010 at 6:01 PM, Cesar Inacio Martins< > cesar_inacio_martins@yahoo.com.br> wrote: > >> Hi Fernando! >> Thanks for your message! >> Sorry take to long to answer, I don't know why the last 2 days of >> messages incoming from IIUG has arrived just now. >> >> So, about the gethostbyaddr() , I'm not sure about that because isn't >> what we see on the strace . >> > I wrote a simple program in Fedora, to test your situation.... Several > interesting things came up. They're not very helpful, but the prove Informix > innocence :) > > On strace you will not see the reference to gethostbyaddr. Only in a > debugger or if you force a stack trace in the precise moment... > >> The request of DNS reverse lookup, appear to be executed "manually" by >> Informix or the strace just traced the gethostbyaddr() too. >> Check the output marked with "##" by me. >> where: 172.18.0.57 and 172.18.0.119 are the OLD DNS when the server was >> started. When we run this strace, the resolv.conf already have new >> values (for at least 2 days): >> >> 11:24:28 munmap(0x2abeb6d75000, 4096) = 0 >> 11:24:28 socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 3 >> ##11:24:28 connect(3, {sa_family=AF_INET, sin_port=htons(53), >> sin_addr=inet_addr("172.18.0.57")}, 28) = 0 >> 11:24:28 fcntl(3, F_GETFL) = 0x2 (flags O_RDWR) >> 11:24:28 fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0 >> 11:24:28 poll([{fd=3, events=POLLOUT}], 1, 0) = 1 ([{fd=3, >> revents=POLLOUT}]) >> ##11:24:28 sendto(3, >> "\\\\242;\\\\1\\\\0\\\\0\\\\1\\\\0\\\\0\\\\0\\\\0\\\\0\\\\0\\\\003185\\\\00248\\\\00222\\\\003172\\\\7in-ad"..., 44, >> MSG_NOSIGNAL, NULL, 0) = 44 >> ##11:24:28 poll([{fd=3, events=POLLIN}], 1, 5000) = 0 (Timeout) >> 11:24:33 socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 4 >> 11:24:33 connect(4, {sa_family=AF_INET, sin_port=htons(53), >> sin_addr=inet_addr("172.18.0.119")}, 28) = 0 >> 11:24:33 fcntl(4, F_GETFL) = 0x2 (flags O_RDWR) >> .... they continue trying to third DNS server... >> (check the timestamp,.. 5 seconds of timeout delay) >> >> Before you ask, the Linux Admin already try change the timeout (option >> into resolv.conf) and don't have effect.... >> >> > 2nd interesting observation... I put the program in loop... reading an IP > from the console and trying to reverse DNS it. > Between loops I changed my resolve.conf.... It didn't matter for the running > process. I belive gethostbyaddr creates some static structures. First time > it runs it reads the resolv.conf, but it doesn't happen again... Probably > due to performance reasons. But it's inconvenient in your situation....: > > open("/etc/resolv.conf", O_RDONLY) = 3 > fstat64(3, {st_mode=S_IFREG|0644, st_size=55, ...}) = 0 > mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = > 0xb776e000 > read(3, "# Generated by NetworkManager\\ na"..., 4096) = 55 > read(3, "", 4096) = 0 > close(3) > > This is why it keeps trying the old DNS... > > Sorry , yes is a 11.50 FC7 GE over Linux Red Hat 5.5 >> This option to create a group is nice, but I not sure if is viable for >> this environment because have a lot of applications spread on the >> company (.net/windows, java/web, C/Linux) what we will need care about >> to change the SQLHOSTS... >> >> The nsswitch.conf is ok, it isn't modified from they default values... >> and the problem is with connections what become from out of our network >> (what don't exists into /etc/hosts) and create consequences to local >> connections (what exists in hosts) when the instance get trouble to try >> resolve their hostnames (not local clients) >> > Are there many client IPs from outside the network? If the number is low you > could put them in /etc/hosts... > >> And the problem what we detected is over MSC, because is it what freeze... >> The most weird thing is, when the MSC "freeze" (stuck running in active >> threads: onstat -g act) they stack dump (onstat -g stk #threadid) don't >> change anything, still showing in yield process...and just change the >> status from sleep to running.... >> >> > Now... none of these are good news. I really can't see a good solution > besides restart... You're a victim of how the TCP/IP name resolution works > Apparently nscd will not work also, because gethostbyaddr only tries it the > first time... > > Some other thoughts: > > 1- You could launch more msc VPs. The new ones should pick up the correct > DNS. And before you ask, no, you cannot remove MSC VPs... > But with more MSC VPs your chances of getting stuck should be lower > 2- (this is very weird...) You could use IPTABLEs to hijack connections to > the OLD DNS and redirect them to the new one... Not sure if this is > possible, but you would have to create a rule for the old IP, port 53 and > protocol UDP. > 3- You could "hack" the network to make the old IP available. This could be > done by putting a machine on the network with that IP, or eventually by > hacking the ARP table > 4- you could create another TCP interface on the machine (how?) with the IP > of the old DNS server. I tried putting my own IP on the /etc/resolv.conf > file and the reverse DNS returns to "quick" (although I don't have anything > listening on port 53)... > The problem you're facing comes from the fact that the IP cannot be reached. > If you make it go to an existing IP it will not resolv, but it will be > quick. > > Naturally 2, 3 and 4 are "crazy" suggestions... But I don't see any "sane" > alternative to an instance stop. > > Regards. >

On Fri, Dec 3, 2010 at 12:44 PM, Cesar Inacio Martins < cesar_inacio_martins@yahoo.com.br> wrote: > Hi Fernando, > > I wrote a little program here and confirm what you said about not "see" > the call of gethostbyaddr... > > But........ I got new variables.... > Testing on my net book , IFX 11.70 xC1 (isn't the same version used on > our production) + OpenSuse 11.2 , open connections with changes on the > resolv.conf, tracing with strace: > - Each open connection, always reread hosts.equivs. (I believed this the > res_init() function working inside of the gethostbyaddr). > hosts.equiv has nothing to do with DNS. It's for trusted connections, and yes, it's read every time (or maybe when it changes, depending on the OS). > - When I change the resolv.conf , without bounce the instance, the next > connection already try solve the DNS using the new configuration. > Not on my system (with the test program). I was using Fedora 13. > > So, now the question, is the Informix or Linux problem ?? > I will install the same version what we have the problem on my netbook , > ifx 11.50 uc7w1ge (but 32 bits..) and test. > On my system, I tested without Informix, so it was definitively a problem with the gethostbyaddr() function. Note that calling this a "problem" is a bit simplistic... I'm not sure we would want it to re-read the file on each request... > > Just for curiosity, read this thread : > http://fixunix.com/redhat/17199-etc-resolv-conf-how-reload.html > > Too long! :) Sorry. Only later. > Is a similar situation, not with Informix and over RH 4 (not 5.5) . > > know my suspicious now is over glibc used on this RH. > When I finish my test on my netbook with the same version what we use > here in production , I will post here the results. > > What about the possibility of adding the hosts to your /etc/hosts file? Are there many connection points in this situation? > On 12/01/2010 11:07 PM, Fernando Nunes wrote: > > On Wed, Dec 1, 2010 at 6:01 PM, Cesar Inacio Martins< > > cesar_inacio_martins@yahoo.com.br> wrote: > > > >> Hi Fernando! > >> Thanks for your message! > >> Sorry take to long to answer, I don't know why the last 2 days of > >> messages incoming from IIUG has arrived just now. > >> > >> So, about the gethostbyaddr() , I'm not sure about that because isn't > >> what we see on the strace . > >> > > I wrote a simple program in Fedora, to test your situation.... Several > > interesting things came up. They're not very helpful, but the prove > Informix > > innocence :) > > > > On strace you will not see the reference to gethostbyaddr. Only in a > > debugger or if you force a stack trace in the precise moment... > > > >> The request of DNS reverse lookup, appear to be executed "manually" by > >> Informix or the strace just traced the gethostbyaddr() too. > >> Check the output marked with "##" by me. > >> where: 172.18.0.57 and 172.18.0.119 are the OLD DNS when the server was > >> started. When we run this strace, the resolv.conf already have new > >> values (for at least 2 days): > >> > >> 11:24:28 munmap(0x2abeb6d75000, 4096) = 0 > >> 11:24:28 socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 3 > >> ##11:24:28 connect(3, {sa_family=AF_INET, sin_port=htons(53), > >> sin_addr=inet_addr("172.18.0.57")}, 28) = 0 > >> 11:24:28 fcntl(3, F_GETFL) = 0x2 (flags O_RDWR) > >> 11:24:28 fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0 > >> 11:24:28 poll([{fd=3, events=POLLOUT}], 1, 0) = 1 ([{fd=3, > >> revents=POLLOUT}]) > >> ##11:24:28 sendto(3, > >> "\\\\242;\\\\1\\\\0\\\\0\\\\1\\\\0\\\\0\\\\0\\\\0\\\\0\\\\0\\\\003185\\\\00248\\\\00222\\\\003172\\\\7in-ad"..., 44, > >> MSG_NOSIGNAL, NULL, 0) = 44 > >> ##11:24:28 poll([{fd=3, events=POLLIN}], 1, 5000) = 0 (Timeout) > >> 11:24:33 socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 4 > >> 11:24:33 connect(4, {sa_family=AF_INET, sin_port=htons(53), > >> sin_addr=inet_addr("172.18.0.119")}, 28) = 0 > >> 11:24:33 fcntl(4, F_GETFL) = 0x2 (flags O_RDWR) > >> .... they continue trying to third DNS server... > >> (check the timestamp,.. 5 seconds of timeout delay) > >> > >> Before you ask, the Linux Admin already try change the timeout (option > >> into resolv.conf) and don't have effect.... > >> > >> > > 2nd interesting observation... I put the program in loop... reading an IP > > from the console and trying to reverse DNS it. > > Between loops I changed my resolve.conf.... It didn't matter for the > running > > process. I belive gethostbyaddr creates some static structures. First > time > > it runs it reads the resolv.conf, but it doesn't happen again... Probably > > due to performance reasons. But it's inconvenient in your situation....: > > > > open("/etc/resolv.conf", O_RDONLY) = 3 > > fstat64(3, {st_mode=S_IFREG|0644, st_size=55, ...}) = 0 > > mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) > = > > 0xb776e000 > > read(3, "# Generated by NetworkManager\\ na"..., 4096) = 55 > > read(3, "", 4096) = 0 > > close(3) > > > > This is why it keeps trying the old DNS... > > > > Sorry , yes is a 11.50 FC7 GE over Linux Red Hat 5.5 > >> This option to create a group is nice, but I not sure if is viable for > >> this environment because have a lot of applications spread on the > >> company (.net/windows, java/web, C/Linux) what we will need care about > >> to change the SQLHOSTS... > >> > >> The nsswitch.conf is ok, it isn't modified from they default values... > >> and the problem is with connections what become from out of our network > >> (what don't exists into /etc/hosts) and create consequences to local > >> connections (what exists in hosts) when the instance get trouble to try > >> resolve their hostnames (not local clients) > >> > > Are there many client IPs from outside the network? If the number is low > you > > could put them in /etc/hosts... > > > >> And the problem what we detected is over MSC, because is it what > freeze... > >> The most weird thing is, when the MSC "freeze" (stuck running in active > >> threads: onstat -g act) they stack dump (onstat -g stk #threadid) don't > >> change anything, still showing in yield process...and just change the > >> status from sleep to running.... > >> > >> > > Now... none of these are good news. I really can't see a good solution > > besides restart... You're a victim of how the TCP/IP name resolution > works > > Apparently nscd will not work also, because gethostbyaddr only tries it > the > > first time... > > > > Some other thoughts: > > > > 1- You could launch more msc VPs. The new ones should pick up the correct > > DNS. And before you ask, no, you cannot remove MSC VPs... > > But with more MSC VPs your chances of getting stuck should be lower > > 2- (this is very weird...) You could use IPTABLEs to hijack connections > to > > the OLD DNS and redirect them to the new one... Not sure if this is > > possible, but you would have to create a rule for the old IP, port 53 and > > protocol UDP. > > 3- You could "hack" the network to make the old IP available. This could > be@

Ooops.. sorry, I confusing my self when I want say resolv.conf and says hosts.equiv... the correct is : "- Each open connection, always reread resolv.conf...". So, I executed some tests here (my netbook), now using the same version of our production (11.50 xC7W1GE). And works too! I started the database with strace and keep a tail over the files. Between this two block I changed the /etc/resolv.conf , the dns from 8.8.8.8 to 1.1.1.1 and try open a new connection from differ host. (check lines marked with "###" by me) ----------------------------------- $tail -n +0 -f ifx* | egrep "add|nscd|host|resol" ###14:01:48 accept(7, {sa_family=AF_INET6, sin6_port=htons(29315), inet_pton(AF_INET6, "::ffff:172.18.0.104",&sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 4 ###14:01:48 stat64("/etc/hosts.equiv", {st_mode=S_IFREG|0644, st_size=230, ...}) = 0 ###14:01:48 open("/etc/hosts.equiv", O_RDONLY|O_LARGEFILE) = 3 ###14:01:48 stat64("/etc/hosts.equiv", {st_mode=S_IFREG|0644, st_size=230, ...}) = 0 14:01:48 read(257, "#\\ # hosts.equiv This file desc"..., 4096) = 230 14:01:48 open("/etc/hosts", O_RDONLY|O_CLOEXEC) = 3 14:01:48 read(3, "#\\ # hosts This file desc"..., 4096) = 3957 14:01:48 stat64("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=850, ...}) = 0 ###14:01:48 connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("8.8.8.8")}, 28) = 0 14:01:48 send(3, "x8\\\\1\\\\0\\\\0\\\\1\\\\0\\\\0\\\\0\\\\0\\\\0\\\\0\\\\003104\\\\0010\\\\00218\\\\003172\\\\7in-add"..., 43, MSG_NOSIGNAL) = 43 14:01:48 recvfrom(3, "x8\\\\201\\\\203\\\\0\\\\1\\\\0\\\\0\\\\0\\\\0\\\\0\\\\0\\\\003104\\\\0010\\\\00218\\\\003172\\\\7in-add"..., 1024, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("8.8.8.8")}, [16]) = 43 ###14:02:09 accept(7, {sa_family=AF_INET6, sin6_port=htons(46297), inet_pton(AF_INET6, "::ffff:172.18.0.105",&sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 4 14:02:09 open("/etc/hosts", O_RDONLY|O_CLOEXEC) = 3 14:02:09 read(3, "#\\ # hosts This file desc"..., 4096) = 3957 ###14:02:09 stat64("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=871, ...}) = 0 ###14:02:09 open("/etc/resolv.conf", O_RDONLY) = 3 ###14:02:09 read(3, "### /etc/resolv.conf file autoge"..., 4096) = 871 ###14:02:09 connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("1.1.1.1")}, 28) = 0 14:02:09 send(3, "\\\\204H\\\\1\\\\0\\\\0\\\\1\\\\0\\\\0\\\\0\\\\0\\\\0\\\\0\\\\003105\\\\0010\\\\00218\\\\003172\\\\7in-add"..., 43, MSG_NOSIGNAL) = 43 14:02:14 send(3, "\\\\204H\\\\1\\\\0\\\\0\\\\1\\\\0\\\\0\\\\0\\\\0\\\\0\\\\0\\\\003105\\\\0010\\\\00218\\\\003172\\\\7in-add"..., 43, MSG_NOSIGNAL) = 43 14:02:19 stat64("/etc/hosts.equiv", {st_mode=S_IFREG|0644, st_size=230, ...}) = 0 14:02:19 open("/etc/hosts.equiv", O_RDONLY|O_LARGEFILE) = 3 14:02:19 stat64("/etc/hosts.equiv", {st_mode=S_IFREG|0644, st_size=230, ...}) = 0 14:02:19 read(257, "#\\ # hosts.equiv This file desc"..., 4096) = 230 ----------------------------------- And this behave, reread the resolv.conf occur only the first time what this IP make a connection, if close and reopen, they don't try resolve the DNS again... only a few minutes later (probably some kind of cache timeout). Include this IPs to /etc/hosts, was the first thing what our Linux Admin does , but is a lot and this can change dynamically...so, isn't a option. Now I'm start to believe the Informix innocence :) and guilty the O.S. , but I need some way to prove this. After some research about the functions gethostbyaddr / gethostbyip , I found them are part of the "resolver" lib, what is from GLIBC. My OpenSuse 11.2 (recent updated) , the Glibc is 2.10 (kernel 2.6.31) The production environment (updated aug/2010) , Red Hat 5.5 , the Glibc is 2.5 Fernando, what's version is your glibc ? I looking for the changelogs / patches for this functions and try identify they are the problem and what version of glibc already able to solve this. Looking the changelog of my glibc (rpm -q --changelog glibc) , have a fews changes over the resolv.conf.. but nothing in particular for this situation. Regards Cesar On 12/03/2010 01:47 PM, Fernando Nunes wrote: > On Fri, Dec 3, 2010 at 12:44 PM, Cesar Inacio Martins< > cesar_inacio_martins@yahoo.com.br> wrote: > >> Hi Fernando, >> >> I wrote a little program here and confirm what you said about not "see" >> the call of gethostbyaddr... >> >> But........ I got new variables.... >> Testing on my net book , IFX 11.70 xC1 (isn't the same version used on >> our production) + OpenSuse 11.2 , open connections with changes on the >> resolv.conf, tracing with strace: >> - Each open connection, always reread hosts.equivs. (I believed this the >> res_init() function working inside of the gethostbyaddr). >> > hosts.equiv has nothing to do with DNS. It's for trusted connections, and > yes, it's read every time (or maybe when it changes, depending on the OS). > >> - When I change the resolv.conf , without bounce the instance, the next >> connection already try solve the DNS using the new configuration. >> > Not on my system (with the test program). I was using Fedora 13. > >> So, now the question, is the Informix or Linux problem ?? >> I will install the same version what we have the problem on my netbook , >> ifx 11.50 uc7w1ge (but 32 bits..) and test. >> > On my system, I tested without Informix, so it was definitively a problem > with the gethostbyaddr() function. > Note that calling this a "problem" is a bit simplistic... I'm not sure we > would want it to re-read the file on each request... > >> Just for curiosity, read this thread : >> http://fixunix.com/redhat/17199-etc-resolv-conf-how-reload.html >> >> > Too long! :) Sorry. Only later. > >> Is a similar situation, not with Informix and over RH 4 (not 5.5) . >> >> know my suspicious now is over glibc used on this RH. >> When I finish my test on my netbook with the same version what we use >> here in production , I will post here the results. >> >> > What about the possibility of adding the hosts to your /etc/hosts file? Are > there many connection points in this situation? > >> On 12/01/2010 11:07 PM, Fernando Nunes wrote: >>> On Wed, Dec 1, 2010 at 6:01 PM, Cesar Inacio Martins< >>> cesar_inacio_martins@yahoo.com.br> wrote: >>> >>>> Hi Fernando! >>>> Thanks for your message! >>>> Sorry take to long to answer, I don't know why the last 2 days of >>>> messages incoming from IIUG has arrived just now. >>>> >>>> So, about the gethostbyaddr() , I'm not sure about that because isn't >>>> what we see on the strace . >>>> >>> I wrote a simple program in Fedora, to test your situation.... Several >>> interesting things came up. They're not very helpful, but the prove >> Informix >>> innocence :) >>> >>> On strace you will not see the reference to gethostbyaddr. Only in a >>> debugger or if you force a stack trace in the precise moment... >>> >>>> The request of DNS re

Damn!! Sorry, this is my "Homer Simpson" moment... :P I marked the wrong lines on the strace output... let's try again... ###14:01:48 accept(7, {sa_family=AF_INET6, sin6_port=htons(29315), inet_pton(AF_INET6, "::ffff:172.18.0.104", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 4 14:01:48 stat64("/etc/hosts.equiv", {st_mode=S_IFREG|0644, st_size=230, ...}) = 0 14:01:48 open("/etc/hosts.equiv", O_RDONLY|O_LARGEFILE) = 3 14:01:48 stat64("/etc/hosts.equiv", {st_mode=S_IFREG|0644, st_size=230, ...}) = 0 14:01:48 read(257, "#\\ # hosts.equiv This file desc"..., 4096) = 230 14:01:48 open("/etc/hosts", O_RDONLY|O_CLOEXEC) = 3 14:01:48 read(3, "#\\ # hosts This file desc"..., 4096) = 3957 ###14:01:48 stat64("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=850, ...}) = 0 ###14:01:48 connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("8.8.8.8")}, 28) = 0 14:01:48 send(3, "x8\\\\1\\\\0\\\\0\\\\1\\\\0\\\\0\\\\0\\\\0\\\\0\\\\0\\\\003104\\\\0010\\\\00218\\\\003172\\\\7in-add"..., 43, MSG_NOSIGNAL) = 43 14:01:48 recvfrom(3, "x8\\\\201\\\\203\\\\0\\\\1\\\\0\\\\0\\\\0\\\\0\\\\0\\\\0\\\\003104\\\\0010\\\\00218\\\\003172\\\\7in-add"..., 1024, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("8.8.8.8")}, [16]) = 43 ###14:02:09 accept(7, {sa_family=AF_INET6, sin6_port=htons(46297), inet_pton(AF_INET6, "::ffff:172.18.0.105", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 4 14:02:09 open("/etc/hosts", O_RDONLY|O_CLOEXEC) = 3 14:02:09 read(3, "#\\ # hosts This file desc"..., 4096) = 3957 ###14:02:09 stat64("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=871, ...}) = 0 ###14:02:09 open("/etc/resolv.conf", O_RDONLY) = 3 ###14:02:09 read(3, "### /etc/resolv.conf file autoge"..., 4096) = 871 ###14:02:09 connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("1.1.1.1")}, 28) = 0 14:02:09 send(3, "\\\\204H\\\\1\\\\0\\\\0\\\\1\\\\0\\\\0\\\\0\\\\0\\\\0\\\\0\\\\003105\\\\0010\\\\00218\\\\003172\\\\7in-add"..., 43, MSG_NOSIGNAL) = 43 14:02:14 send(3, "\\\\204H\\\\1\\\\0\\\\0\\\\1\\\\0\\\\0\\\\0\\\\0\\\\0\\\\0\\\\003105\\\\0010\\\\00218\\\\003172\\\\7in-add"..., 43, MSG_NOSIGNAL) = 43 14:02:19 stat64("/etc/hosts.equiv", {st_mode=S_IFREG|0644, st_size=230, ...}) = 0 14:02:19 open("/etc/hosts.equiv", O_RDONLY|O_LARGEFILE) = 3 14:02:19 stat64("/etc/hosts.equiv", {st_mode=S_IFREG|0644, st_size=230, ...}) = 0 14:02:19 read(257, "#\\ # hosts.equiv This file desc"..., 4096) = 230 I forgot to mention on the last message... The code appear don't reread the resolv.conf every time, appear get the status from the file (stat() function) and if something is different(probably the last modification date), reread the file... On 12/03/2010 03:32 PM, Cesar Inacio Martins wrote: > Ooops.. sorry, I confusing my self when I want say resolv.conf and > says hosts.equiv... > > the correct is : "- Each open connection, always reread resolv.conf...". > > So, I executed some tests here (my netbook), now using the same > version of our production (11.50 xC7W1GE). > And works too! > > I started the database with strace and keep a tail over the files. > Between this two block I changed the /etc/resolv.conf , the dns from > 8.8.8.8 to 1.1.1.1 and try open a new connection from differ host. > > (check lines marked with "###" by me) > ----------------------------------- > $tail -n +0 -f ifx* | egrep "add|nscd|host|resol" > > ###14:01:48 accept(7, {sa_family=AF_INET6, sin6_port=htons(29315), > inet_pton(AF_INET6, "::ffff:172.18.0.104",&sin6_addr), > sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 4 > ###14:01:48 stat64("/etc/hosts.equiv", {st_mode=S_IFREG|0644, > st_size=230, ...}) = 0 > ###14:01:48 open("/etc/hosts.equiv", O_RDONLY|O_LARGEFILE) = 3 > ###14:01:48 stat64("/etc/hosts.equiv", {st_mode=S_IFREG|0644, > st_size=230, ...}) = 0 > 14:01:48 read(257, "#\\ # hosts.equiv This file desc"..., 4096) = 230 > 14:01:48 open("/etc/hosts", O_RDONLY|O_CLOEXEC) = 3 > 14:01:48 read(3, "#\\ # hosts This file desc"..., 4096) = 3957 > 14:01:48 stat64("/etc/resolv.conf", {st_mode=S_IFREG|0644, > st_size=850, ...}) = 0 > ###14:01:48 connect(3, {sa_family=AF_INET, sin_port=htons(53), > sin_addr=inet_addr("8.8.8.8")}, 28) = 0 > 14:01:48 send(3, > "x8\\\\1\\\\0\\\\0\\\\1\\\\0\\\\0\\\\0\\\\0\\\\0\\\\0\\\\003104\\\\0010\\\\00218\\\\003172\\\\7in-add"..., 43, > MSG_NOSIGNAL) = 43 > 14:01:48 recvfrom(3, > "x8\\\\201\\\\203\\\\0\\\\1\\\\0\\\\0\\\\0\\\\0\\\\0\\\\0\\\\003104\\\\0010\\\\00218\\\\003172\\\\7in-add"..., > 1024, 0, {sa_family=AF_INET, sin_port=htons(53), > sin_addr=inet_addr("8.8.8.8")}, [16]) = 43 > > ###14:02:09 accept(7, {sa_family=AF_INET6, sin6_port=htons(46297), > inet_pton(AF_INET6, "::ffff:172.18.0.105",&sin6_addr), > sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 4 > 14:02:09 open("/etc/hosts", O_RDONLY|O_CLOEXEC) = 3 > 14:02:09 read(3, "#\\ # hosts This file desc"..., 4096) = 3957 > ###14:02:09 stat64("/etc/resolv.conf", {st_mode=S_IFREG|0644, > st_size=871, ...}) = 0 > ###14:02:09 open("/etc/resolv.conf", O_RDONLY) = 3 > ###14:02:09 read(3, "### /etc/resolv.conf file autoge"..., 4096) = 871 > ###14:02:09 connect(3, {sa_family=AF_INET, sin_port=htons(53), > sin_addr=inet_addr("1.1.1.1")}, 28) = 0 > 14:02:09 send(3, > "\\\\204H\\\\1\\\\0\\\\0\\\\1\\\\0\\\\0\\\\0\\\\0\\\\0\\\\0\\\\003105\\\\0010\\\\00218\\\\003172\\\\7in-add"..., 43, > MSG_NOSIGNAL) = 43 > 14:02:14 send(3, > "\\\\204H\\\\1\\\\0\\\\0\\\\1\\\\0\\\\0\\\\0\\\\0\\\\0\\\\0\\\\003105\\\\0010\\\\00218\\\\003172\\\\7in-add"..., 43, > MSG_NOSIGNAL) = 43 > 14:02:19 stat64("/etc/hosts.equiv", {st_mode=S_IFREG|0644, > st_size=230, ...}) = 0 > 14:02:19 open("/etc/hosts.equiv", O_RDONLY|O_LARGEFILE) = 3 > 14:02:19 stat64("/etc/hosts.equiv", {st_mode=S_IFREG|0644, > st_size=230, ...}) = 0 > 14:02:19 read(257, "#\\ # hosts.equiv This file desc"..., 4096) = 230 > ----------------------------------- > > And this behave, reread the resolv.conf occur only the first time what > this IP make a connection, if close and reopen, they don't try resolve > the DNS again... only a few minutes later (probably some kind of cache > timeout). > > Include this IPs to /etc/hosts, was the first thing what our Linux > Admin does , but is a lot and this can change dynamically...so, isn't > a option. > > Now I'm start to believe the Informix innocence :) and guilty the O.S. > , but I need some way to prove this. > > After some research about the functions gethostbyaddr / gethostbyip , > I found them are part of the "resolver" lib, what is from GLIBC. > > My OpenSuse 11.2 (recent updated) , the Glibc is 2.10 (kernel 2.6.31) > The production environment (updated aug/2010) , Red Hat 5.5 , the > Glibc is 2.5 > > Fernando, what's version is your glibc ? > > I looking for the changelogs / patches for this functions and try > identify they are the problem and what version of glibc already able@

My glibc is: [root@pacman ~]# rpm -qa | grep glibc glibc-2.12-3.i686 You can prove/test the behavior with the simple C program I sent you... You should see different traces depending on the OS. Assuming your client can understand C (that example is simple...), that you can show it on their system also. Anyway. It's now clear that it will not work until you restart. If you don't want/can't to restart your only option is to "hack" the TCP stack somehow and make the old DNS server IP available. You don't need to have a DNS server there... If it sends the packets and there is nothing on port 53 UDP it will give up quickly... Alternatively you could open a case with RH... Maybe they have a way to deal with that... If not, they may be interested in fixing it... You have a test case, and my past experiences tell me that it's half way to a fix... (or more...) Regards. On Fri, Dec 3, 2010 at 5:33 PM, Cesar Inacio Martins < cesar_inacio_martins@yahoo.com.br> wrote: > Ooops.. sorry, I confusing my self when I want say resolv.conf and says > hosts.equiv... > > the correct is : "- Each open connection, always reread resolv.conf...". > > So, I executed some tests here (my netbook), now using the same version of > our > production (11.50 xC7W1GE). > And works too! > > I started the database with strace and keep a tail over the files. > Between this two block I changed the /etc/resolv.conf , the dns from > 8.8.8.8 > to 1.1.1.1 and try open a new connection from differ host. > > (check lines marked with "###" by me) > ----------------------------------- > $tail -n +0 -f ifx* | egrep "add|nscd|host|resol" > > ###14:01:48 accept(7, {sa_family=AF_INET6, sin6_port=htons(29315), > inet_pton(AF_INET6, "::ffff:172.18.0.104",&sin6_addr), sin6_flowinfo=0, > sin6_scope_id=0}, [28]) = 4 > ###14:01:48 stat64("/etc/hosts.equiv", {st_mode=S_IFREG|0644, st_size=230, > ....}) = 0 > ###14:01:48 open("/etc/hosts.equiv", O_RDONLY|O_LARGEFILE) = 3 > ###14:01:48 stat64("/etc/hosts.equiv", {st_mode=S_IFREG|0644, st_size=230, > ....}) = 0 > 14:01:48 read(257, "#\\ # hosts.equiv This file desc"..., 4096) = 230 > 14:01:48 open("/etc/hosts", O_RDONLY|O_CLOEXEC) = 3 > 14:01:48 read(3, "#\\ # hosts This file desc"..., 4096) = 3957 > 14:01:48 stat64("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=850, > ...}) > = 0 > ###14:01:48 connect(3, {sa_family=AF_INET, sin_port=htons(53), > sin_addr=inet_addr("8.8.8.8")}, 28) = 0 > 14:01:48 send(3, > "x8\\\\1\\\\0\\\\0\\\\1\\\\0\\\\0\\\\0\\\\0\\\\0\\\\0\\\\003104\\\\0010\\\\00218\\\\003172\\\\7in-add"..., > 43, MSG_NOSIGNAL) = 43 > 14:01:48 recvfrom(3, > "x8\\\\201\\\\203\\\\0\\\\1\\\\0\\\\0\\\\0\\\\0\\\\0\\\\0\\\\003104\\\\0010\\\\00218\\\\003172\\\\7in-add"..., 1024, 0, > {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("8.8.8.8")}, > [16]) > = 43 > > ###14:02:09 accept(7, {sa_family=AF_INET6, sin6_port=htons(46297), > inet_pton(AF_INET6, "::ffff:172.18.0.105",&sin6_addr), sin6_flowinfo=0, > sin6_scope_id=0}, [28]) = 4 > 14:02:09 open("/etc/hosts", O_RDONLY|O_CLOEXEC) = 3 > 14:02:09 read(3, "#\\ # hosts This file desc"..., 4096) = 3957 > ###14:02:09 stat64("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=871, > ....}) = 0 > ###14:02:09 open("/etc/resolv.conf", O_RDONLY) = 3 > ###14:02:09 read(3, "### /etc/resolv.conf file autoge"..., 4096) = 871 > ###14:02:09 connect(3, {sa_family=AF_INET, sin_port=htons(53), > sin_addr=inet_addr("1.1.1.1")}, 28) = 0 > 14:02:09 send(3, > "\\\\204H\\\\1\\\\0\\\\0\\\\1\\\\0\\\\0\\\\0\\\\0\\\\0\\\\0\\\\003105\\\\0010\\\\00218\\\\003172\\\\7in-add"..., 43, > MSG_NOSIGNAL) = 43 > 14:02:14 send(3, > "\\\\204H\\\\1\\\\0\\\\0\\\\1\\\\0\\\\0\\\\0\\\\0\\\\0\\\\0\\\\003105\\\\0010\\\\00218\\\\003172\\\\7in-add"..., 43, > MSG_NOSIGNAL) = 43 > 14:02:19 stat64("/etc/hosts.equiv", {st_mode=S_IFREG|0644, st_size=230, > ...}) > = 0 > 14:02:19 open("/etc/hosts.equiv", O_RDONLY|O_LARGEFILE) = 3 > 14:02:19 stat64("/etc/hosts.equiv", {st_mode=S_IFREG|0644, st_size=230, > ...}) > = 0 > 14:02:19 read(257, "#\\ # hosts.equiv This file desc"..., 4096) = 230 > ----------------------------------- > > And this behave, reread the resolv.conf occur only the first time what this > IP > make a connection, if close and reopen, they don't try resolve the DNS > again... only a few minutes later (probably some kind of cache timeout). > > Include this IPs to /etc/hosts, was the first thing what our Linux Admin > does > , but is a lot and this can change dynamically...so, isn't a option. > > Now I'm start to believe the Informix innocence :) and guilty the O.S. , > but I > need some way to prove this. > > After some research about the functions gethostbyaddr / gethostbyip , I > found > them are part of the "resolver" lib, what is from GLIBC. > > My OpenSuse 11.2 (recent updated) , the Glibc is 2.10 (kernel 2.6.31) > The production environment (updated aug/2010) , Red Hat 5.5 , the Glibc is > 2.5 > > Fernando, what's version is your glibc ? > > I looking for the changelogs / patches for this functions and try identify > they are the problem and what version of glibc already able to solve this. > Looking the changelog of my glibc (rpm -q --changelog glibc) , have a fews > changes over the resolv.conf.. but nothing in particular for this > situation. > > Regards > Cesar > > On 12/03/2010 01:47 PM, Fernando Nunes wrote: > > On Fri, Dec 3, 2010 at 12:44 PM, Cesar Inacio Martins< > > cesar_inacio_martins@yahoo.com.br> wrote: > > > >> Hi Fernando, > >> > >> I wrote a little program here and confirm what you said about not "see" > >> the call of gethostbyaddr... > >> > >> But........ I got new variables.... > >> Testing on my net book , IFX 11.70 xC1 (isn't the same version used on > >> our production) + OpenSuse 11.2 , open connections with changes on the > >> resolv.conf, tracing with strace: > >> - Each open connection, always reread hosts.equivs. (I believed this the > >> res_init() function working inside of the gethostbyaddr). > >> > > hosts.equiv has nothing to do with DNS. It's for trusted connections, and > > yes, it's read every time (or maybe when it changes, depending on the > OS). > > > >> - When I change the resolv.conf , without bounce the instance, the next > >> connection already try solve the DNS using the new configuration. > >> > > Not on my system (with the test program). I was using Fedora 13. > > > >> So, now the question, is the Informix or Linux problem ?? > >> I will install the same version what we have the problem on my netbook , > >> ifx 11.50 uc7w1ge (but 32 bits..) and test. > >> > > On my system, I tested without Informix, so it was definitively a problem > > with the gethostbyaddr() function. > > Note that calling this a "problem" is a bit simplistic... I'm not sure we > > would want it to re-read the file on each request... > > > >> Just for curiosity, read this thread : > >> http://fixunix.com/red

Fernando, Thanks a lot for your support. We used your test code to generate information from suse/redhat to open a support request from Red Hat, let's wait for the answer now... Thanks again Cesar On 12/03/2010 03:48 PM, Fernando Nunes wrote: > My glibc is: > > [root@pacman ~]# rpm -qa | grep glibc > glibc-2.12-3.i686 > > You can prove/test the behavior with the simple C program I sent you... You > should see different traces depending on the OS. > Assuming your client can understand C (that example is simple...), that you > can show it on their system also. > > Anyway. It's now clear that it will not work until you restart. If you don't > want/can't to restart your only option is to "hack" the TCP stack somehow > and make the old DNS server IP available. > You don't need to have a DNS server there... If it sends the packets and > there is nothing on port 53 UDP it will give up quickly... > > Alternatively you could open a case with RH... Maybe they have a way to deal > with that... If not, they may be interested in fixing it... You have a test > case, and my past experiences tell me that it's half way to a fix... (or > more...) > > Regards. > > On Fri, Dec 3, 2010 at 5:33 PM, Cesar Inacio Martins< > cesar_inacio_martins@yahoo.com.br> wrote: > >> Ooops.. sorry, I confusing my self when I want say resolv.conf and says >> hosts.equiv... >> >> the correct is : "- Each open connection, always reread resolv.conf...". >> >> So, I executed some tests here (my netbook), now using the same version of >> our >> production (11.50 xC7W1GE). >> And works too! >> >> I started the database with strace and keep a tail over the files. >> Between this two block I changed the /etc/resolv.conf , the dns from >> 8.8.8.8 >> to 1.1.1.1 and try open a new connection from differ host. >> >> (check lines marked with "###" by me) >> ----------------------------------- >> $tail -n +0 -f ifx* | egrep "add|nscd|host|resol" >> >> ###14:01:48 accept(7, {sa_family=AF_INET6, sin6_port=htons(29315), >> inet_pton(AF_INET6, "::ffff:172.18.0.104",&sin6_addr), sin6_flowinfo=0, >> sin6_scope_id=0}, [28]) = 4 >> ###14:01:48 stat64("/etc/hosts.equiv", {st_mode=S_IFREG|0644, st_size=230, >> ....}) = 0 >> ###14:01:48 open("/etc/hosts.equiv", O_RDONLY|O_LARGEFILE) = 3 >> ###14:01:48 stat64("/etc/hosts.equiv", {st_mode=S_IFREG|0644, st_size=230, >> ....}) = 0 >> 14:01:48 read(257, "#\\ # hosts.equiv This file desc"..., 4096) = 230 >> 14:01:48 open("/etc/hosts", O_RDONLY|O_CLOEXEC) = 3 >> 14:01:48 read(3, "#\\ # hosts This file desc"..., 4096) = 3957 >> 14:01:48 stat64("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=850, >> ...}) >> = 0 >> ###14:01:48 connect(3, {sa_family=AF_INET, sin_port=htons(53), >> sin_addr=inet_addr("8.8.8.8")}, 28) = 0 >> 14:01:48 send(3, >> "x8\\\\1\\\\0\\\\0\\\\1\\\\0\\\\0\\\\0\\\\0\\\\0\\\\0\\\\003104\\\\0010\\\\00218\\\\003172\\\\7in-add"..., >> 43, MSG_NOSIGNAL) = 43 >> 14:01:48 recvfrom(3, >> "x8\\\\201\\\\203\\\\0\\\\1\\\\0\\\\0\\\\0\\\\0\\\\0\\\\0\\\\003104\\\\0010\\\\00218\\\\003172\\\\7in-add"..., 1024, 0, >> {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("8.8.8.8")}, >> [16]) >> = 43 >> >> ###14:02:09 accept(7, {sa_family=AF_INET6, sin6_port=htons(46297), >> inet_pton(AF_INET6, "::ffff:172.18.0.105",&sin6_addr), sin6_flowinfo=0, >> sin6_scope_id=0}, [28]) = 4 >> 14:02:09 open("/etc/hosts", O_RDONLY|O_CLOEXEC) = 3 >> 14:02:09 read(3, "#\\ # hosts This file desc"..., 4096) = 3957 >> ###14:02:09 stat64("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=871, >> ....}) = 0 >> ###14:02:09 open("/etc/resolv.conf", O_RDONLY) = 3 >> ###14:02:09 read(3, "### /etc/resolv.conf file autoge"..., 4096) = 871 >> ###14:02:09 connect(3, {sa_family=AF_INET, sin_port=htons(53), >> sin_addr=inet_addr("1.1.1.1")}, 28) = 0 >> 14:02:09 send(3, >> "\\\\204H\\\\1\\\\0\\\\0\\\\1\\\\0\\\\0\\\\0\\\\0\\\\0\\\\0\\\\003105\\\\0010\\\\00218\\\\003172\\\\7in-add"..., 43, >> MSG_NOSIGNAL) = 43 >> 14:02:14 send(3, >> "\\\\204H\\\\1\\\\0\\\\0\\\\1\\\\0\\\\0\\\\0\\\\0\\\\0\\\\0\\\\003105\\\\0010\\\\00218\\\\003172\\\\7in-add"..., 43, >> MSG_NOSIGNAL) = 43 >> 14:02:19 stat64("/etc/hosts.equiv", {st_mode=S_IFREG|0644, st_size=230, >> ...}) >> = 0 >> 14:02:19 open("/etc/hosts.equiv", O_RDONLY|O_LARGEFILE) = 3 >> 14:02:19 stat64("/etc/hosts.equiv", {st_mode=S_IFREG|0644, st_size=230, >> ...}) >> = 0 >> 14:02:19 read(257, "#\\ # hosts.equiv This file desc"..., 4096) = 230 >> ----------------------------------- >> >> And this behave, reread the resolv.conf occur only the first time what this >> IP >> make a connection, if close and reopen, they don't try resolve the DNS >> again... only a few minutes later (probably some kind of cache timeout). >> >> Include this IPs to /etc/hosts, was the first thing what our Linux Admin >> does >> , but is a lot and this can change dynamically...so, isn't a option. >> >> Now I'm start to believe the Informix innocence :) and guilty the O.S. , >> but I >> need some way to prove this. >> >> After some research about the functions gethostbyaddr / gethostbyip , I >> found >> them are part of the "resolver" lib, what is from GLIBC. >> >> My OpenSuse 11.2 (recent updated) , the Glibc is 2.10 (kernel 2.6.31) >> The production environment (updated aug/2010) , Red Hat 5.5 , the Glibc is >> 2.5 >> >> Fernando, what's version is your glibc ? >> >> I looking for the changelogs / patches for this functions and try identify >> they are the problem and what version of glibc already able to solve this. >> Looking the changelog of my glibc (rpm -q --changelog glibc) , have a fews >> changes over the resolv.conf.. but nothing in particular for this >> situation. >> >> Regards >> Cesar >> >> On 12/03/2010 01:47 PM, Fernando Nunes wrote: >>> On Fri, Dec 3, 2010 at 12:44 PM, Cesar Inacio Martins< >>> cesar_inacio_martins@yahoo.com.br> wrote: >>> >>>> Hi Fernando, >>>> >>>> I wrote a little program here and confirm what you said about not "see" >>>> the call of gethostbyaddr... >>>> >>>> But........ I got new variables.... >>>> Testing on my net book , IFX 11.70 xC1 (isn't the same version used on >>>> our production) + OpenSuse 11.2 , open connections with changes on the >>>> resolv.conf, tracing with strace: >>>> - Each open connection, always reread hosts.equivs. (I believed this the >>>> res_init() function working inside of the gethostbyaddr). >>>> >>> hosts.equiv has nothing to do with DNS. It's for trusted connections, and >>> yes, it's read every time (or maybe when it changes, depending on the >> OS). >>>> - When I change the resolv.conf , without bounce the instance, the next >>>> connection already try solve the DNS using the new configuration. >>>> >>> Not on my system (with the test program). I was using Fedora 13. >>> >>>> So, now the question, is the Informix or Linux problem ?? >>>> I will install the same version what we have the problem on my netb

On Mon, Dec 6, 2010 at 1:22 PM, Cesar Inacio Martins < cesar_inacio_martins@yahoo.com.br> wrote: > Fernando, > > Thanks a lot for your support. > > We used your test code to generate information from suse/redhat to open > a support request from Red Hat, let's wait for the answer now... > > Thanks again > Cesar > My pleasure. Let us know how it turns out... Regards -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently... --0015177fd00ce1f2830496c23869