RE: Changed DNS Server and now database connection is very slow
Posted in 2011
Topics: Performance & Tuning, Server Administration, Platform-Specific Issues
If the Jack problem is similar what occur with me a few months ago (with RH 5), the cache on Informix will not solve the problem.
The problem is at OS level.
Some versions of Linux (RH and appear Debian too) cache this damn configuration and just don't refresh it. The only way to solve is the "Windows way"... kick the machine (if you are stress)... and reset it...
With Fernando's program, in my case the Linux administrator agreed with the flaw and said it would open a call at Red Hat
Cesar
--- Em dom, 27/3/11, Ian Michael Gumby <im_gumby@hotmail.com> escreveu:
De: Ian Michael Gumby <im_gumby@hotmail.com>
Assunto: RE: Changed DNS Server and now database connection is very slow
Para: art.kagel@gmail.com, informix-list@iiug.org
Data: Domingo, 27 de Março de 2011, 15:24
Right.
So 900 seconds ~15 minutes.
Then what's the real problem?
You make the change, performance takes a hit for the first 15 minutes and the everything is ok.
Now I know that this is a production system, but...
1) You don't modify your DNS that often.
2) You don't have to bring the system down in order to handle the change.
From: art.kagel@gmail.com
Date: Sun, 27 Mar 2011 09:40:25 -0400
Subject: Re: Changed DNS Server and now database connection is very slow
To: im_gumby@hotmail.com
CC: domusonline@gmail.com; jackson@valu-search.com; informix-list@iiug.org
11.50 doesn't cache network lookup information, that cache was added in 11.70 which refreshes the cache every 900 seconds be default (well, the first time it needs information about users or the network if 900 seconds have expired since the last refresh).
Art
Art S. Kagel
Advanced DataTools (www.advancedatatools.com)
Blog: http://informix-myview.blogspot.com/
Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves.
On Sun, Mar 27, 2011 at 12:38 AM, Ian Michael Gumby <im_gumby@hotmail.com> wrote:
Doesn't Informix age the cache?
That is to say that it will periodically flush the cache and re-fetch the address and insert it in to the cache?
You could then tune the TTL and don't have to worry about it.
(Yes, I agree that a DBA command would also be nice, but if you allow for a TTL setting, then periodically the cache renews itself. Then worst case, you take a performance hit until the cache is reset.)
Date: Sat, 26 Mar 2011 18:51:18 +0000
Subject: Re: Changed DNS Server and now database connection is very slow
From: domusonline@gmail.com
To: jackson@valu-search.com
CC: informix-list@iiug.org
Bad news... There is currently no workaround (except eventually some network tweaking).
The problem lies in the reverse DNS system call - gethostbyaddr() - The first time it runs it creates a cache of your DNS configuration.
If you later change it, it has no way of knowing the change happened. So it keeps trying to talk to your previously configured DNS server.
The delay is the time it takes to give up....
This can be showed with a simple C program. The cache will only be forgotten if the process calls another system function. Currently Informix doesn't do that. A mechanism would have to be implemented to allow the DBA to "ask" Informix to run that function. Note that other solutions like periodically call it, would be bad for performance.
I've recently created a feature request for this. Note that this problem can affect any kind of server software (I've seen it with sendmail a few years ago).
If your configuration of the resolver system included a first try with the files (/etc/hosts) you could try to workaround it by including all the client host IP address/names in the file.
This is usually not advisable, but you may consider it a workaround until you can restart Informix
Another possible option (and I'm getting completely out of my comfort zone here) could be trying to full the network layer to re-write the packet that are sent to the old DNS in a way that they're directed to the new one....
If you feel sufficiently annoyed by this you can also open a PMR (which will not solve your problem) and make sure your need is mapped to the feature I'm asked for. I could provide details.
Regards.
On Sat, Mar 26, 2011 at 6:07 PM, Jackson Thompson <jackson@valu-search.com> wrote:
I changed our DNS server and when I disconnect the old DNS server our
informix ids 11.5 instance takes up to 30 seconds to find the
databases. If you use dbaccess it starts and when is is going to list
the databases it stalls, the connect process and all query seem to run
fine. We I have the old DNS server off line I can get good forward and
reverse DNS lookups so the DNS for the os (debian linux) seems to be
fine. I have not restart the Informix instance, it is a production
system of a 24/7 contact center, so I don't want to restart without
knowing that there might be a fix in it.
Thanks
Jackson
_______________________________________________
Informix-list mailing list
Informix-list@iiug.org
http://www.iiug.org/mailman/listinfo/informix-list
--
Fernando Nunes
Portugal
http://informix-technology.blogspot.com
My email works... but I don't check it frequently...
_______________________________________________
Informix-list mailing list
Informix-list@iiug.org
http://www.iiug.org/mailman/listinfo/informix-list
_______________________________________________
Informix-list mailing list
Informix-list@iiug.org
http://www.iiug.org/mailman/listinfo/informix-list
-----Anexo incorporado-----
_______________________________________________
Informix-list mailing list
Informix-list@iiug.org
http://www.iiug.org/mailman/listinfo/informix-list
Cesar Inacio Martins wrote: > > If the Jack problem is similar what occur with me a few months ago (with > RH 5), the cache on Informix will not solve the problem. > > The problem is at OS level. > Some versions of Linux (RH and appear Debian too) cache this damn > configuration and just don't refresh it. The only way to solve is the > "Windows way"... kick the machine (if you are stress)... and reset it... Not necessarily. arp -d Years ago - about the time of Informix 4.x - we had HP-UX boxes which had to use DEC-Net to talk to a VAX. One of the characteristics of DEC-Net was that it used the lower bits of the MAC as the network address, the upper bits being assumed to be those used by DEC. So when we brought the 3rd party S/W online it rewrote the MAC on the HP box. All the clients, probably Win 3.0 back in those days or maybe just terminal servers, lost connectivity. However, they timed out, re-arped & were back in about a minute without, as far as I know, loosing the actual telnet session. Odd how improvements don't always make things better. -- Ian The Hotmail address is my spam-bin. Real mail address is iang at austonley org uk
Hum... I don't remember if we try this... Fernando, can you test this solution? --- Em dom, 27/3/11, Ian Goddard <goddai01@hotmail.co.uk> escreveu: De: Ian Goddard <goddai01@hotmail.co.uk> Assunto: Re: Changed DNS Server and now database connection is very slow Para: informix-list@iiug.org Data: Domingo, 27 de Março de 2011, 17:19 Cesar Inacio Martins wrote: > > If the Jack problem is similar what occur with me a few months ago (with > RH 5), the cache on Informix will not solve the problem. > > The problem is at OS level. > Some versions of Linux (RH and appear Debian too) cache this damn > configuration and just don't refresh it. The only way to solve is the > "Windows way"... kick the machine (if you are stress)... and reset it... Not necessarily. arp -d Years ago - about the time of Informix 4.x - we had HP-UX boxes which had to use DEC-Net to talk to a VAX. One of the characteristics of DEC-Net was that it used the lower bits of the MAC as the network address, the upper bits being assumed to be those used by DEC. So when we brought the 3rd party S/W online it rewrote the MAC on the HP box. All the clients, probably Win 3.0 back in those days or maybe just terminal servers, lost connectivity. However, they timed out, re-arped & were back in about a minute without, as far as I know, loosing the actual telnet session. Odd how improvements don't always make things better. -- Ian The Hotmail address is my spam-bin. Real mail address is iang at austonley org uk _______________________________________________ Informix-list mailing list Informix-list@iiug.org http://www.iiug.org/mailman/listinfo/informix-list