Intermittent DNS failures

While this is probably not primarily an Informix question, it is impacting our users' ability to establish a database connection and causing a great deal of 'pain' and I am looking for any help I can get. IDS: 11.1FC3 OS: RHEL 5.11 on VMWARE Our configuration has been in place for several years and most of it trouble free. However, recently we started having issues with users logging in to our application or getting kicked-out after trying to apply a transaction. We had some network issues in the past so we redesigned the app to create the connection, use it, close it, as having an open connection and using it all session caused issues if the connection was lost during a network 'hiccup'. This worked well for the last couple of years, but recently ran into more issues. What it appears like it happening is the client is connecting to the server and while doing a reverse lookup back to the client, it is failing/timing out and then the connection fails and user can't login or gets booted out and has to re-launch and re-do the transaction. To mitigate the problem for the short term, I added all of our users to the server's host file, but their IP's could change and cause more problems moving forward with this approach. Couple of questions. Has anyone experienced this issue and resolved? If so, what areas should I be targeting as a starting point of investigation? Is there a way to change the configuration of Informix so it doesn't require the reverse lookup to establish the database connection? TIA, Randy

Randy, Have you looked into using netcd - Network caching deamon? Description The netcd daemon reduces the time taken by the local, DNS, NIS, NIS+ and user loadable module services to respond to a query by caching the response retrieved from resolvers. When the netcd daemon is running and configured for a resolver (for example, DNS) and a map (for example, hosts), the resolution is first made using the cached answers. If it fails, the resolver is called and the response is cached by the netcd daemon. We have implemented it to help with this type of issues. -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Kennedy, Randy Sent: Monday, December 22, 2014 1:06 PM To: ids@iiug.org Subject: Intermittent DNS failures [34379] While this is probably not primarily an Informix question, it is impacting our users' ability to establish a database connection and causing a great deal of 'pain' and I am looking for any help I can get. IDS: 11.1FC3 OS: RHEL 5.11 on VMWARE Our configuration has been in place for several years and most of it trouble free. However, recently we started having issues with users logging in to our application or getting kicked-out after trying to apply a transaction. We had some network issues in the past so we redesigned the app to create the connection, use it, close it, as having an open connection and using it all session caused issues if the connection was lost during a network 'hiccup'. This worked well for the last couple of years, but recently ran into more issues. What it appears like it happening is the client is connecting to the server and while doing a reverse lookup back to the client, it is failing/timing out and then the connection fails and user can't login or gets booted out and has to re-launch and re-do the transaction. To mitigate the problem for the short term, I added all of our users to the server's host file, but their IP's could change and cause more problems moving forward with this approach. Couple of questions. Has anyone experienced this issue and resolved? If so, what areas should I be targeting as a starting point of investigation? Is there a way to change the configuration of Informix so it doesn't require the reverse lookup to establish the database connection? TIA, Randy ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

Randy, I don't know if this idea applies to your problem, but I had a DNS lookup timeout problem. Our Internet was out and Informix/RHEL could not reach the DNS server. It waited 30 seconds before giving up. Then the connection was made. The fix was to set hostname in the sqlhosts file to 127.0.0.1. David On 12/22/2014 2:06 PM, Kennedy, Randy wrote: > While this is probably not primarily an Informix question, it is impacting our > users' ability to establish a database connection and causing a great deal of > 'pain' and I am looking for any help I can get. > > IDS: 11.1FC3 > OS: RHEL 5.11 on VMWARE > > Our configuration has been in place for several years and most of it trouble > free. However, recently we started having issues with users logging in to our > application or getting kicked-out after trying to apply a transaction. We had > some network issues in the past so we redesigned the app to create the > connection, use it, close it, as having an open connection and using it all > session caused issues if the connection was lost during a network 'hiccup'. > This worked well for the last couple of years, but recently ran into more > issues. What it appears like it happening is the client is connecting to the > server and while doing a reverse lookup back to the client, it is > failing/timing out and then the connection fails and user can't login or gets > booted out and has to re-launch and re-do the transaction. To mitigate the > problem for the short term, I added all of our users to the server's host > file, but their IP's could change and cause more problems moving forward with > this approach. > > Couple of questions. > Has anyone experienced this issue and resolved? If so, what areas should I be > targeting as a starting point of investigation? > Is there a way to change the configuration of Informix so it doesn't require > the reverse lookup to establish the database connection? > > TIA, > Randy > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > >

Also you might need to set up Ns_cache on onconfig . In should help in the context of DNS changes whilst Ids is up Sent from my iPhone > On 22 Dec 2014, at 21:32, David Zinder <zinder@ztechz.com> wrote: > > Randy, > > I don't know if this idea applies to your problem, but I had a DNS > lookup timeout problem. Our Internet was out and Informix/RHEL could not > reach the DNS server. It waited 30 seconds before giving up. Then the > connection was made. The fix was to set hostname in the sqlhosts file to > 127.0.0.1. > > David > >> On 12/22/2014 2:06 PM, Kennedy, Randy wrote: >> While this is probably not primarily an Informix question, it is impacting > our >> users' ability to establish a database connection and causing a great deal > of >> 'pain' and I am looking for any help I can get. >> >> IDS: 11.1FC3 >> OS: RHEL 5.11 on VMWARE >> >> Our configuration has been in place for several years and most of it trouble >> free. However, recently we started having issues with users logging in to > our >> application or getting kicked-out after trying to apply a transaction. We > had >> some network issues in the past so we redesigned the app to create the >> connection, use it, close it, as having an open connection and using it all >> session caused issues if the connection was lost during a network 'hiccup'. >> This worked well for the last couple of years, but recently ran into more >> issues. What it appears like it happening is the client is connecting to the >> server and while doing a reverse lookup back to the client, it is >> failing/timing out and then the connection fails and user can't login or > gets >> booted out and has to re-launch and re-do the transaction. To mitigate the >> problem for the short term, I added all of our users to the server's host >> file, but their IP's could change and cause more problems moving forward > with >> this approach. >> >> Couple of questions. >> Has anyone experienced this issue and resolved? If so, what areas should I > be >> targeting as a starting point of investigation? >> Is there a way to change the configuration of Informix so it doesn't require >> the reverse lookup to establish the database connection? >> >> TIA, >> Randy > ******************************************************************************* >> Forum Note: Use "Reply" to post a response in the discussion forum. > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > Stefan Sammut Manager - Solutions & Innovation [PTL Ltd] Nineteen Twenty Three, Valletta Road Marsa, MRS 3000, MT T +35621445566 +356 99405444 stefan.sammut@ptl.com.mt | www.ptl.com.mt<http://www.ptl.com.mt>; [Facebook]<https://www.facebook.com/PTLMalta/app_349313058487732>; [LinkedIn] <https://www.linkedin.com/company/ptl-ltd?trk=tyah&trkInfo=tarId%3A1401716999276 %2Ctas%3APTL%2Cidx%3A2-3-8> [Twitter] <https://twitter.com/PTL_Malta>; [Youtube] <https://www.youtube.com/channel/UCuXrJBO_54kd9HttG8S89iQ/feed?view_as =public> [Google Plus] <https://plus.google.com/+PTLMalta>;

This is what I would like to use, but it looks like it doesn't exist in 11.1, I don't see it listed until 11.7 Thanks, Randy -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Stefan Sammut Sent: Monday, December 22, 2014 2:19 PM To: ids@iiug.org Subject: Re: Intermittent DNS failures [34385] Also you might need to set up Ns_cache on onconfig . In should help in the context of DNS changes whilst Ids is up Sent from my iPhone > On 22 Dec 2014, at 21:32, David Zinder <zinder@ztechz.com> wrote: > > Randy, > > I don't know if this idea applies to your problem, but I had a DNS > lookup timeout problem. Our Internet was out and Informix/RHEL could not > reach the DNS server. It waited 30 seconds before giving up. Then the > connection was made. The fix was to set hostname in the sqlhosts file to > 127.0.0.1. > > David > >> On 12/22/2014 2:06 PM, Kennedy, Randy wrote: >> While this is probably not primarily an Informix question, it is impacting > our >> users' ability to establish a database connection and causing a great deal > of >> 'pain' and I am looking for any help I can get. >> >> IDS: 11.1FC3 >> OS: RHEL 5.11 on VMWARE >> >> Our configuration has been in place for several years and most of it trouble >> free. However, recently we started having issues with users logging in to > our >> application or getting kicked-out after trying to apply a transaction. We > had >> some network issues in the past so we redesigned the app to create the >> connection, use it, close it, as having an open connection and using it all >> session caused issues if the connection was lost during a network 'hiccup'. >> This worked well for the last couple of years, but recently ran into more >> issues. What it appears like it happening is the client is connecting to the >> server and while doing a reverse lookup back to the client, it is >> failing/timing out and then the connection fails and user can't login or > gets >> booted out and has to re-launch and re-do the transaction. To mitigate the >> problem for the short term, I added all of our users to the server's host >> file, but their IP's could change and cause more problems moving forward > with >> this approach. >> >> Couple of questions. >> Has anyone experienced this issue and resolved? If so, what areas should I > be >> targeting as a starting point of investigation? >> Is there a way to change the configuration of Informix so it doesn't require >> the reverse lookup to establish the database connection? >> >> TIA, >> Randy > ******************************************************************************* >> Forum Note: Use "Reply" to post a response in the discussion forum. > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > Stefan Sammut Manager - Solutions & Innovation [PTL Ltd] Nineteen Twenty Three, Valletta Road Marsa, MRS 3000, MT T +35621445566 +356 99405444 stefan.sammut@ptl.com.mt | www.ptl.com.mt<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ptl.com.m t&d=AwIBAg&c=mLnDz0LAxdRWb0m2sWTY_HoX_augE-BnEl0NeG61j9k&r=isPG3qbzM0phicCiOFZNs E4mQCsIMPX73SiQafIpBxM&m=p_M8ZW7srPLjgyQrrF5IePWQoc6RZ_15vOZAlSRqE8A&s=blozT2pwM RNE-HwD0nE00EB8NF5TK3oR6BlDyoXQtCk&e= > [Facebook]<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com _PTLMalta_app-5F349313058487732&d=AwIBAg&c=mLnDz0LAxdRWb0m2sWTY_HoX_augE-BnEl0Ne G61j9k&r=isPG3qbzM0phicCiOFZNsE4mQCsIMPX73SiQafIpBxM&m=p_M8ZW7srPLjgyQrrF5IePWQo c6RZ_15vOZAlSRqE8A&s=9aAcmmWJ6sP8P9xlAiOaq5uBV8xyAuUooQCZrzGRzCE&e= > [LinkedIn] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_p tl-2Dltd-3Ftrk-3Dtyah-26trkInfo-3DtarId-253A1401716999276-252Ctas-253APTL-252Cid x-253A2-2D3-2D8&d=AwIBAg&c=mLnDz0LAxdRWb0m2sWTY_HoX_augE-BnEl0NeG61j9k&r=isPG3qb zM0phicCiOFZNsE4mQCsIMPX73SiQafIpBxM&m=p_M8ZW7srPLjgyQrrF5IePWQoc6RZ_15vOZAlSRqE 8A&s=eLtD_A_ty0qj4KHIfenX79RklEE9TzprpJuDTq4trT8&e= > [Twitter] <https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_PTL-5FMalta&d= AwIBAg&c=mLnDz0LAxdRWb0m2sWTY_HoX_augE-BnEl0NeG61j9k&r=isPG3qbzM0phicCiOFZNsE4mQ CsIMPX73SiQafIpBxM&m=p_M8ZW7srPLjgyQrrF5IePWQoc6RZ_15vOZAlSRqE8A&s=olAKNPIoTxew3 1kkQSj8D5D6-IUIvZvxUVePur5KkHc&e= > [Youtube] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_channel_UC uXrJBO-5F54kd9HttG8S89iQ_feed-3Fview-5Fas-3Dpublic&d=AwIBAg&c=mLnDz0LAxdRWb0m2sW TY_HoX_augE-BnEl0NeG61j9k&r=isPG3qbzM0phicCiOFZNsE4mQCsIMPX73SiQafIpBxM&m=p_M8ZW 7srPLjgyQrrF5IePWQoc6RZ_15vOZAlSRqE8A&s=HlWqVhgTaZ7NprjW3_A10oHo_D__bVs781sN491r 5d8&e= > [Google Plus] <https://urldefense.proofpoint.com/v2/url?u=https-3A__plus.google.com_-2BPTLMalt a&d=AwIBAg&c=mLnDz0LAxdRWb0m2sWTY_HoX_augE-BnEl0NeG61j9k&r=isPG3qbzM0phicCiOFZNs E4mQCsIMPX73SiQafIpBxM&m=p_M8ZW7srPLjgyQrrF5IePWQoc6RZ_15vOZAlSRqE8A&s=SAbMxK8Uj 6Rr9q6qVbvlH0HPZl-ZGXMeUdlpv1UmXHM&e= > ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

It's a well known problem with Informix... or as you correctly put it, with DNS that impacts Informix. Both suggestion I've seen are interesting (ncd daemon and the ns_cache parameter). Unfortunately you cannot consider the later one (you should be aware that 11.1 is out of support) There are some points I'd like to add. First and more important, and eventually I missed some information or detail on this thread: 1- Do you need the reverse DNS to happen? Have you created trusts based on client names? IF not, they you could configure the system to just go to the local files and be immune to any DNS issue. In other words, if your connections are based on USER/PASS and not trusted, then this would be possible 2- If that's not the case, and you have to rely on your DNS, than the DNS must be considered a top priority and be handled like any high availability system. An Informix customer (specially on versions where NS_CACHE cannot be used) that decides to use DNS will become dependent on it. The way we do things make it particularly susceptible to any issues in the DNS. Let's see.... - The reverse DNS is made by the MSC VP or VPs. - Each call is synchronous - The DNS configuration plans and default to very large timeouts (in some systems I believe you cannot even ask for values below 5s) - To add up to all this, the functions of the OS (gethostbyaddr() ) will by design create "caches" of the DNS configurations (if DNS is used, or just files, what DNS etc). Basically the information in files like /etc/nsswitch.conf and /etc/resolv.conf are cached after the first call. This means that you can change this files but the database will not change the behavior unless you restart it. This is one of my "personal battles" that is going nowhere... So, what can we do to improve the situation? - If possible don't use DNS (I have the impression that if using IPV6 this is not an option) - Eventually use more than 1 MSC VP. If the issue with calls to gethostbyaddr() are occasional, this means that at any moment you may have one "hanged" MSC VP while the others will continue to answer requests. So eventually only some of the connection attempts will fail - Use ncd daemon as very well suggested by another person - Use NS_CACHE if the Informix version has it, as Stefan very well reminded us - If you need to change the DNS configuration frequently there is a trick that can be used while IBM does not solve the problem. I described the trick in this article: http://informix-technology.blogspot.pt/2014/02/dns-changes-ok-mudancas-no-dns-ok .html It's relatively simple to follow, but some programming skills will make it easier to implement. It basically takes advantage of the fact that PAM authentication functions also run on the MSC VPs... so we can create a "fake" PAM module that will clear the MSC VP DNS cache by calling the OS function that clears it (res_init() ). The authentication will always return false, so this fake module should be used only in a "maintenance listener" created only for this purpose. Regards. On Mon, Dec 22, 2014 at 7:06 PM, Kennedy, Randy <RKennedy@scottsdaleaz.gov> wrote: > While this is probably not primarily an Informix question, it is impacting > our > users' ability to establish a database connection and causing a great deal > of > 'pain' and I am looking for any help I can get. > > IDS: 11.1FC3 > OS: RHEL 5.11 on VMWARE > > Our configuration has been in place for several years and most of it > trouble > free. However, recently we started having issues with users logging in to > our > application or getting kicked-out after trying to apply a transaction. We > had > some network issues in the past so we redesigned the app to create the > connection, use it, close it, as having an open connection and using it all > session caused issues if the connection was lost during a network 'hiccup'. > This worked well for the last couple of years, but recently ran into more > issues. What it appears like it happening is the client is connecting to > the > server and while doing a reverse lookup back to the client, it is > failing/timing out and then the connection fails and user can't login or > gets > booted out and has to re-launch and re-do the transaction. To mitigate the > problem for the short term, I added all of our users to the server's host > file, but their IP's could change and cause more problems moving forward > with > this approach. > > Couple of questions. > Has anyone experienced this issue and resolved? If so, what areas should I > be > targeting as a starting point of investigation? > Is there a way to change the configuration of Informix so it doesn't > require > the reverse lookup to establish the database connection? > > TIA, > Randy > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently... --001a1140f88edd55e4050aea31e8

So if I understand you correctly, if we use user/pass for all connections (we do), I can just change my "hosts:" entry in nsswitch.conf from "files dns" to just "files" and leave the hosts file blank of all our local PC's and there won't be any problem with users connecting? We do also do some FTPing from this server, but those few could just be added to the hosts file or setup to FTP by IP instead of name. Thanks, Randy -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Fernando Nunes Sent: Tuesday, December 23, 2014 4:04 PM To: ids@iiug.org Subject: Re: Intermittent DNS failures [34390] It's a well known problem with Informix... or as you correctly put it, with DNS that impacts Informix. Both suggestion I've seen are interesting (ncd daemon and the ns_cache parameter). Unfortunately you cannot consider the later one (you should be aware that 11.1 is out of support) There are some points I'd like to add. First and more important, and eventually I missed some information or detail on this thread: 1- Do you need the reverse DNS to happen? Have you created trusts based on client names? IF not, they you could configure the system to just go to the local files and be immune to any DNS issue. In other words, if your connections are based on USER/PASS and not trusted, then this would be possible 2- If that's not the case, and you have to rely on your DNS, than the DNS must be considered a top priority and be handled like any high availability system. An Informix customer (specially on versions where NS_CACHE cannot be used) that decides to use DNS will become dependent on it. The way we do things make it particularly susceptible to any issues in the DNS. Let's see.... - The reverse DNS is made by the MSC VP or VPs. - Each call is synchronous - The DNS configuration plans and default to very large timeouts (in some systems I believe you cannot even ask for values below 5s) - To add up to all this, the functions of the OS (gethostbyaddr() ) will by design create "caches" of the DNS configurations (if DNS is used, or just files, what DNS etc). Basically the information in files like /etc/nsswitch.conf and /etc/resolv.conf are cached after the first call. This means that you can change this files but the database will not change the behavior unless you restart it. This is one of my "personal battles" that is going nowhere... So, what can we do to improve the situation? - If possible don't use DNS (I have the impression that if using IPV6 this is not an option) - Eventually use more than 1 MSC VP. If the issue with calls to gethostbyaddr() are occasional, this means that at any moment you may have one "hanged" MSC VP while the others will continue to answer requests. So eventually only some of the connection attempts will fail - Use ncd daemon as very well suggested by another person - Use NS_CACHE if the Informix version has it, as Stefan very well reminded us - If you need to change the DNS configuration frequently there is a trick that can be used while IBM does not solve the problem. I described the trick in this article: https://urldefense.proofpoint.com/v2/url?u=http-3A__informix-2Dtechnology.blogsp ot.pt_2014_02_dns-2Dchanges-2Dok-2Dmudancas-2Dno-2Ddns-2Dok.html&d=AwIBAg&c=mLnD z0LAxdRWb0m2sWTY_HoX_augE-BnEl0NeG61j9k&r=isPG3qbzM0phicCiOFZNsE4mQCsIMPX73SiQaf IpBxM&m=WlWafOeImTomWL1wbS1v-AQ4Cg5b6x2wb8IOBJeYFfk&s=knycGBk0ij5jWfajiH-Sbn1K81 5EPeay8qE4S9c8LQo&e= It's relatively simple to follow, but some programming skills will make it easier to implement. It basically takes advantage of the fact that PAM authentication functions also run on the MSC VPs... so we can create a "fake" PAM module that will clear the MSC VP DNS cache by calling the OS function that clears it (res_init() ). The authentication will always return false, so this fake module should be used only in a "maintenance listener" created only for this purpose. Regards. On Mon, Dec 22, 2014 at 7:06 PM, Kennedy, Randy <RKennedy@scottsdaleaz.gov> wrote: > While this is probably not primarily an Informix question, it is impacting > our > users' ability to establish a database connection and causing a great deal > of > 'pain' and I am looking for any help I can get. > > IDS: 11.1FC3 > OS: RHEL 5.11 on VMWARE > > Our configuration has been in place for several years and most of it > trouble > free. However, recently we started having issues with users logging in to > our > application or getting kicked-out after trying to apply a transaction. We > had > some network issues in the past so we redesigned the app to create the > connection, use it, close it, as having an open connection and using it all > session caused issues if the connection was lost during a network 'hiccup'. > This worked well for the last couple of years, but recently ran into more > issues. What it appears like it happening is the client is connecting to > the > server and while doing a reverse lookup back to the client, it is > failing/timing out and then the connection fails and user can't login or > gets > booted out and has to re-launch and re-do the transaction. To mitigate the > problem for the short term, I added all of our users to the server's host > file, but their IP's could change and cause more problems moving forward > with > this approach. > > Couple of questions. > Has anyone experienced this issue and resolved? If so, what areas should I > be > targeting as a starting point of investigation? > Is there a way to change the configuration of Informix so it doesn't > require > the reverse lookup to establish the database connection? > > TIA, > Randy > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Fernando Nunes Portugal https://urldefense.proofpoint.com/v2/url?u=http-3A__informix-2Dtechnology.blogsp ot.com&d=AwIBAg&c=mLnDz0LAxdRWb0m2sWTY_HoX_augE-BnEl0NeG61j9k&r=isPG3qbzM0phicCi OFZNsE4mQCsIMPX73SiQafIpBxM&m=WlWafOeImTomWL1wbS1v-AQ4Cg5b6x2wb8IOBJeYFfk&s=mlli c6DNFcnvyd9Jh7fYBv7WMl_aehcXTWPW3vbe0rc&e= My email works... but I don't check it frequently... --001a1140f88edd55e4050aea31e8 ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

Being able to complete reverse DNS is not a requirement for an Informix connection (unless it's a trusted connection and you use names in the hosts.equiv or .rhosts files. So the answer is yes. But of course please test on another system. Don't forget to restart the database after the change or it will not assume the change. And of course, in any place (onstat -g ses for example) where you usually see a name, you'll see an IP address. Regards On Tue, Dec 23, 2014 at 11:21 PM, Kennedy, Randy <RKennedy@scottsdaleaz.gov> wrote: > So if I understand you correctly, if we use user/pass for all connections > (we > do), I can just change my "hosts:" entry in nsswitch.conf from "files dns" > to > just "files" and leave the hosts file blank of all our local PC's and there > won't be any problem with users connecting? > > We do also do some FTPing from this server, but those few could just be > added > to the hosts file or setup to FTP by IP instead of name. > > Thanks, > Randy > > -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > Fernando > Nunes > Sent: Tuesday, December 23, 2014 4:04 PM > To: ids@iiug.org > Subject: Re: Intermittent DNS failures [34390] > > It's a well known problem with Informix... or as you correctly put it, with > DNS that impacts Informix. > Both suggestion I've seen are interesting (ncd daemon and the ns_cache > parameter). Unfortunately you cannot consider the later one (you should be > aware that 11.1 is out of support) > > There are some points I'd like to add. First and more important, and > eventually I missed some information or detail on this thread: > > 1- Do you need the reverse DNS to happen? Have you created trusts based on > client names? IF not, they you could configure the system to just go to the > local files and be immune to any DNS issue. In other words, if your > connections are based on USER/PASS and not trusted, then this would be > possible > > 2- If that's not the case, and you have to rely on your DNS, than the DNS > must be considered a top priority and be handled like any high availability > system. An Informix customer (specially on versions where NS_CACHE cannot > be used) that decides to use DNS will become dependent on it. The way we do > things make it particularly susceptible to any issues in the DNS. Let's > see.... > > - The reverse DNS is made by the MSC VP or VPs. > - Each call is synchronous > - The DNS configuration plans and default to very large timeouts (in some > systems I believe you cannot even ask for values below 5s) > - To add up to all this, the functions of the OS (gethostbyaddr() ) will by > design create "caches" of the DNS configurations (if DNS is used, or just > files, what DNS etc). Basically the information in files like > /etc/nsswitch.conf and /etc/resolv.conf are cached after the first call. > This means that you can change this files but the database will not change > the behavior unless you restart it. This is one of my "personal battles" > that is going nowhere... > > So, what can we do to improve the situation? > > - If possible don't use DNS (I have the impression that if using IPV6 this > is not an option) > - Eventually use more than 1 MSC VP. If the issue with calls to > gethostbyaddr() are occasional, this means that at any moment you may have > one "hanged" MSC VP while the others will continue to answer requests. So > eventually only some of the connection attempts will fail > - Use ncd daemon as very well suggested by another person > - Use NS_CACHE if the Informix version has it, as Stefan very well reminded > us > - If you need to change the DNS configuration frequently there is a trick > that can be used while IBM does not solve the problem. I described the > trick in this article: > > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__informix-2Dtechnology.blogsp ot.pt_2014_02_dns-2Dchanges-2Dok-2Dmudancas-2Dno-2Ddns-2Dok.html&d=AwIBAg&c=mLnD z0LAxdRWb0m2sWTY_HoX_augE-BnEl0NeG61j9k&r=isPG3qbzM0phicCiOFZNsE4mQCsIMPX73SiQaf IpBxM&m=WlWafOeImTomWL1wbS1v-AQ4Cg5b6x2wb8IOBJeYFfk&s=knycGBk0ij5jWfajiH-Sbn1K81 5EPeay8qE4S9c8LQo&e= > It's relatively simple to follow, but some programming skills will make it > easier to implement. It basically takes advantage of the fact that PAM > authentication functions also run on the MSC VPs... so we can create a > "fake" PAM module that will clear the MSC VP DNS cache by calling the OS > function that clears it (res_init() ). The authentication will always > return false, so this fake module should be used only in a "maintenance > listener" created only for this purpose. > > Regards. > > On Mon, Dec 22, 2014 at 7:06 PM, Kennedy, Randy <RKennedy@scottsdaleaz.gov > > > wrote: > > > While this is probably not primarily an Informix question, it is > impacting > > our > > users' ability to establish a database connection and causing a great > deal > > of > > 'pain' and I am looking for any help I can get. > > > > IDS: 11.1FC3 > > OS: RHEL 5.11 on VMWARE > > > > Our configuration has been in place for several years and most of it > > trouble > > free. However, recently we started having issues with users logging in to > > our > > application or getting kicked-out after trying to apply a transaction. We > > had > > some network issues in the past so we redesigned the app to create the > > connection, use it, close it, as having an open connection and using it > all > > session caused issues if the connection was lost during a network > 'hiccup'. > > This worked well for the last couple of years, but recently ran into more > > issues. What it appears like it happening is the client is connecting to > > the > > server and while doing a reverse lookup back to the client, it is > > failing/timing out and then the connection fails and user can't login or > > gets > > booted out and has to re-launch and re-do the transaction. To mitigate > the > > problem for the short term, I added all of our users to the server's host > > file, but their IP's could change and cause more problems moving forward > > with > > this approach. > > > > Couple of questions. > > Has anyone experienced this issue and resolved? If so, what areas should > I > > be > > targeting as a starting point of investigation? > > Is there a way to change the configuration of Informix so it doesn't > > require > > the reverse lookup to establish the database connection? > > > > TIA, > > Randy > > > > > > > > > > > ******************************************************************************* > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > -- > Fernando Nunes > Portugal > > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__informix-2Dtechnology.blogsp ot.com&d=AwIBAg&c=mLnDz0LAxdRWb0m2sWTY_HoX_augE-BnEl0NeG61j9k&r=isPG3qbzM0phicCi OFZNsE4mQCsIMPX73SiQafIpBxM&m=WlWafOeImTomWL1wbS1v-AQ4Cg5b6x2wb8IOBJeYFfk&s=mlli c6DNFcnvyd9Jh7fYBv7WMl_aehcXTW

You could also drop a cache only DNS server on the server Cheers Paul > -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > Fernando Nunes > Sent: Tuesday, December 23, 2014 5:04 PM > To: ids@iiug.org > Subject: Re: Intermittent DNS failures [34390] > > It's a well known problem with Informix... or as you correctly put it, with > DNS that impacts Informix. > Both suggestion I've seen are interesting (ncd daemon and the ns_cache > parameter). Unfortunately you cannot consider the later one (you should be > aware that 11.1 is out of support) > > There are some points I'd like to add. First and more important, and > eventually I missed some information or detail on this thread: > > 1- Do you need the reverse DNS to happen? Have you created trusts based > on > client names? IF not, they you could configure the system to just go to the > local files and be immune to any DNS issue. In other words, if your > connections are based on USER/PASS and not trusted, then this would be > possible > > 2- If that's not the case, and you have to rely on your DNS, than the DNS > must be considered a top priority and be handled like any high availability > system. An Informix customer (specially on versions where NS_CACHE > cannot > be used) that decides to use DNS will become dependent on it. The way we > do > things make it particularly susceptible to any issues in the DNS. Let's > see.... > > - The reverse DNS is made by the MSC VP or VPs. > - Each call is synchronous > - The DNS configuration plans and default to very large timeouts (in some > systems I believe you cannot even ask for values below 5s) > - To add up to all this, the functions of the OS (gethostbyaddr() ) will by > design create "caches" of the DNS configurations (if DNS is used, or just > files, what DNS etc). Basically the information in files like > /etc/nsswitch.conf and /etc/resolv.conf are cached after the first call. > This means that you can change this files but the database will not change > the behavior unless you restart it. This is one of my "personal battles" > that is going nowhere... > > So, what can we do to improve the situation? > > - If possible don't use DNS (I have the impression that if using IPV6 this > is not an option) > - Eventually use more than 1 MSC VP. If the issue with calls to > gethostbyaddr() are occasional, this means that at any moment you may > have > one "hanged" MSC VP while the others will continue to answer requests. So > eventually only some of the connection attempts will fail > - Use ncd daemon as very well suggested by another person > - Use NS_CACHE if the Informix version has it, as Stefan very well reminded > us > - If you need to change the DNS configuration frequently there is a trick > that can be used while IBM does not solve the problem. I described the > trick in this article: > > http://informix-technology.blogspot.pt/2014/02/dns-changes-ok-mudancas- > no-dns-ok.html > It's relatively simple to follow, but some programming skills will make it > easier to implement. It basically takes advantage of the fact that PAM > authentication functions also run on the MSC VPs... so we can create a > "fake" PAM module that will clear the MSC VP DNS cache by calling the OS > function that clears it (res_init() ). The authentication will always > return false, so this fake module should be used only in a "maintenance > listener" created only for this purpose. > > Regards. > > On Mon, Dec 22, 2014 at 7:06 PM, Kennedy, Randy > <RKennedy@scottsdaleaz.gov> > wrote: > > > While this is probably not primarily an Informix question, it is impacting > > our > > users' ability to establish a database connection and causing a great deal > > of > > 'pain' and I am looking for any help I can get. > > > > IDS: 11.1FC3 > > OS: RHEL 5.11 on VMWARE > > > > Our configuration has been in place for several years and most of it > > trouble > > free. However, recently we started having issues with users logging in to > > our > > application or getting kicked-out after trying to apply a transaction. We > > had > > some network issues in the past so we redesigned the app to create the > > connection, use it, close it, as having an open connection and using it all > > session caused issues if the connection was lost during a network 'hiccup'. > > This worked well for the last couple of years, but recently ran into more > > issues. What it appears like it happening is the client is connecting to > > the > > server and while doing a reverse lookup back to the client, it is > > failing/timing out and then the connection fails and user can't login or > > gets > > booted out and has to re-launch and re-do the transaction. To mitigate the > > problem for the short term, I added all of our users to the server's host > > file, but their IP's could change and cause more problems moving forward > > with > > this approach. > > > > Couple of questions. > > Has anyone experienced this issue and resolved? If so, what areas should I > > be > > targeting as a starting point of investigation? > > Is there a way to change the configuration of Informix so it doesn't > > require > > the reverse lookup to establish the database connection? > > > > TIA, > > Randy > > > > > > > > > ********************************************************** > ********************* > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > -- > Fernando Nunes > Portugal > > http://informix-technology.blogspot.com > My email works... but I don't check it frequently... > > --001a1140f88edd55e4050aea31e8 > > > ********************************************************** > ********************* > Forum Note: Use "Reply" to post a response in the discussion forum.