RE: how do you solve this issue
Someone asked about an IBM tech note on IDS running out of internal file descriptors (open partitions/cursors) and wanted to know the hard limit in IDS 10/11, noting IBM support couldn't say. Replies argued that hitting a ~32K limit almost certainly indicates a badly written application (cursors opened and never closed/cleaned up) rather than genuine load, and recommended a code review or re-architecture. Another poster noted PDQ matters too: a parallel query on fragmented tables opened many descriptors, allocated per thread rather than per session (e.g. 22 fds for one join). No definitive limit figure or fix was recorded; the original poster just said he'd have the code reviewed.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
Eeewww! This is what happens when you application developers who don't grok databases. And trust me, there are a lot out there. Is it possible that there can be more than 32K open cursors at a given time? Yes, and it would imply that you have over 30,000 simultaneous users hitting the same database at the *same* time all requiring different cursors. This would imply that you're looking at a flat client/server architecture. It would also imply that you've got to retain the state during the connection. (All of this is *possible* but highly doubtful) More than likely you have a poorly written application. (Memory leak... code grabbing a new cursor while retaining the old one... long running app that doesn't clean up after itself... poorly designed solution...) You solve it by getting in a senior consultant to help either lead a code review or re-architect the solution. What language? C/C++? Java? Python, Perl? etc ... > Subject: RE: how do you solve this issue > Date: Thu, 21 Feb 2008 16:32:55 -0500 > From: Mark.Denham@lfg.com > To: informix-list@iiug.org > > Have you been running into this problem? > > I would be rather suspicious of any app doing this. > > > -----Original Message----- > > From: informix-list-bounces@iiug.org > > [mailto:informix-list-bounces@iiug.org]On Behalf Of > > vomaringo@yahoo.com > > Sent: Thursday, February 21, 2008 3:52 PM > > To: informix-list@iiug.org > > Subject: how do you solve this issue > > > > > > http://www-1.ibm.com/support/docview.wss?rs=0&uid=swg21251537 > > what's the hard limit of the internal file descriptor in ids 10 and > > 11 ? > > thanks > > jj > > _______________________________________________ > > Informix-list mailing list > > Informix-list@iiug.org > > http://www.iiug.org/mailman/listinfo/informix-list > > > > > > > Notice of Confidentiality: **This E-mail and any of its attachments may contain > Lincoln National Corporation proprietary information, which is privileged, confidential, > or subject to copyright belonging to the Lincoln National Corporation family of > companies. This E-mail is intended solely for the use of the individual or entity to > which it is addressed. If you are not the intended recipient of this E-mail, you are > hereby notified that any dissemination, distribution, copying, or action taken in > relation to the contents of and attachments to this E-mail is strictly prohibited > and may be unlawful. If you have received this E-mail in error, please notify the > sender immediately and permanently delete the original and any copy of this E-mail > and any printout. Thank You.** > _______________________________________________ > Informix-list mailing list > Informix-list@iiug.org > http://www.iiug.org/mailman/listinfo/informix-list _________________________________________________________________ Need to know the score, the latest news, or you need your Hotmail®-get your "fix". http://www.msnmobilefix.com/Default.aspx
On Feb 21, 11:36 pm, Ian Michael Gumby <im_gu...@hotmail.com> wrote: > Eeewww! > > This is what happens when you application developers who don't grok databases. > And trust me, there are a lot out there. > > Is it possible that there can be more than 32K open cursors at a given time? > Yes, and it would imply that you have over 30,000 simultaneous users hitting the same database at the *same* time all requiring different cursors. This would imply that you're looking at a flat client/server architecture. It would also imply that you've got to retain the state during the connection. (All of this is *possible* but highly doubtful) > > More than likely you have a poorly written application. > (Memory leak... code grabbing a new cursor while retaining the old one... long running app that doesn't clean up after itself... poorly designed solution...) > > You solve it by getting in a senior consultant to help either lead a code review or re-architect the solution. > > What language? C/C++? Java? Python, Perl? etc ... > > > > > Subject: RE: how do you solve this issue > > Date: Thu, 21 Feb 2008 16:32:55 -0500 > > From: Mark.Den...@lfg.com > > To: informix-l...@iiug.org > > > Have you been running into this problem? > > > I would be rather suspicious of any app doing this. > > > > -----Original Message----- > > > From: informix-list-boun...@iiug.org > > > [mailto:informix-list-boun...@iiug.org]On Behalf Of > > > vomari...@yahoo.com > > > Sent: Thursday, February 21, 2008 3:52 PM > > > To: informix-l...@iiug.org > > > Subject: how do you solve this issue > > > >http://www-1.ibm.com/support/docview.wss?rs=0&uid=swg21251537 > > > what's the hard limit of the internal file descriptor in ids 10 and > > > 11 ? > > > thanks > > > jj > > > _______________________________________________ > > > Informix-list mailing list > > > Informix-l...@iiug.org > > >http://www.iiug.org/mailman/listinfo/informix-list > > > Notice of Confidentiality: **This E-mail and any of its attachments may contain > > Lincoln National Corporation proprietary information, which is privileged, confidential, > > or subject to copyright belonging to the Lincoln National Corporation family of > > companies. This E-mail is intended solely for the use of the individual or entity to > > which it is addressed. If you are not the intended recipient of this E-mail, you are > > hereby notified that any dissemination, distribution, copying, or action taken in > > relation to the contents of and attachments to this E-mail is strictly prohibited > > and may be unlawful. If you have received this E-mail in error, please notify the > > sender immediately and permanently delete the original and any copy of this E-mail > > and any printout. Thank You.** > > _______________________________________________ > > Informix-list mailing list > > Informix-l...@iiug.org > >http://www.iiug.org/mailman/listinfo/informix-list > > _________________________________________________________________ > Need to know the score, the latest news, or you need your Hotmail®-get your "fix".http://www.msnmobilefix.com/Default.aspx thanks. i'll ask the review the code. just curious about the limit. support doesn't know if there's one.
One other thing to look at is PDQ.
Out of curiosity I ran a query joining 2 tables, both with 7 fragments. I ended up with 22 open fds for the session. Interestingly the fd's look to be thread rather than session related. The 2 join threads have 8 fd's apiece.
See below,
Mark
onstat -g sestid name rstcb flags curstk status
18422 sqlexec c00000004eab9908 Y--P--- 61664 cond wait(netnorm)
18521 join_1.0 c00000004eac6100 ------- 66992 sleeping(secs: 1)
18522 join_1.1 c000000051c72790 ------- 66992 sleeping(secs: 3)
18523 scan_2.0 c00000004b7d2050 ------- 66992 sleeping(secs: 3)
18524 scan_2.1 c00000004b7bda70 ------- 66992 sleeping(secs: 3)
onstat -g opn
18524 0xc00000004b7bda70 0 0x00000408 0x00000003 0x01100003 1 1 0
18524 0xc00000004b7bda70 1 0x00000448 0x00000117 0x0160000b 1 1 0
18523 0xc00000004b7d2050 0 0x00000408 0x00000003 0x01100003 1 1 0
18523 0xc00000004b7d2050 1 0x00000448 0x00000117 0x006000ff 1 1 0
18422 0xc00000004eab9908 0 0x00000400 0x00000397 0x00200100 2 2 0
18422 0xc00000004eab9908 1 0x00000002 0x00000003 0x00200100 2 2 0
18521 0xc00000004eac6100 0 0x00000408 0x00000007 0x00200020 1 1 0
18521 0xc00000004eac6100 1 0x00000448 0x00000007 0x00300039 1 1 0
18521 0xc00000004eac6100 2 0x00000448 0x00000007 0x00400004 1 1 0
18521 0xc00000004eac6100 3 0x00000448 0x00000007 0x00500892 1 1 0
18521 0xc00000004eac6100 4 0x00000448 0x00000403 0x00e00004 1 0 0
18521 0xc00000004eac6100 5 0x00000448 0x00000007 0x00f00004 1 1 0
18521 0xc00000004eac6100 6 0x00000448 0x00000007 0x01000004 1 1 0
18521 0xc00000004eac6100 7 0x00000448 0x00000297 0x00d00002 1 1 0
18522 0xc000000051c72790 0 0x00000408 0x00000007 0x00200020 1 1 0
18522 0xc000000051c72790 1 0x00000448 0x00000007 0x00300039 1 1 0
18522 0xc000000051c72790 2 0x00000448 0x00000007 0x00400004 1 1 0
18522 0xc000000051c72790 3 0x00000448 0x00000007 0x00500892 1 1 0
18522 0xc000000051c72790 4 0x00000448 0x00000007 0x00e00004 1 1 0
18522 0xc000000051c72790 5 0x00000448 0x00000007 0x00f00004 1 1 0
18522 0xc000000051c72790 6 0x00000448 0x00000007 0x01000004 1 1 0
18522 0xc000000051c72790 7 0x00000448 0x00000397 0x00d00002 1 1 0
> -----Original Message-----
> From: informix-list-bounces@iiug.org
> [mailto:informix-list-bounces@iiug.org]On Behalf Of
> vomaringo@yahoo.com
> Sent: Friday, February 22, 2008 10:05 AM
> To: informix-list@iiug.org
> Subject: Re: how do you solve this issue
>
>
> On Feb 21, 11:36 pm, Ian Michael Gumby <im_gu...@hotmail.com> wrote:
> > Eeewww!
> >
> > This is what happens when you application developers who
> don't grok databases.
> > And trust me, there are a lot out there.
> >
> > Is it possible that there can be more than 32K open cursors
> at a given time?
> > Yes, and it would imply that you have over 30,000
> simultaneous users hitting the same database at the *same*
> time all requiring different cursors. This would imply that
> you're looking at a flat client/server architecture. It would
> also imply that you've got to retain the state during the
> connection. (All of this is *possible* but highly doubtful)
> >
> > More than likely you have a poorly written application.
> > (Memory leak... code grabbing a new cursor while retaining
> the old one... long running app that doesn't clean up after
> itself... poorly designed solution...)
> >
> > You solve it by getting in a senior consultant to help
> either lead a code review or re-architect the solution.
> >
> > What language? C/C++? Java? Python, Perl? etc ...
> >
> >
> >
> > > Subject: RE: how do you solve this issue
> > > Date: Thu, 21 Feb 2008 16:32:55 -0500
> > > From: Mark.Den...@lfg.com
> > > To: informix-l...@iiug.org
> >
> > > Have you been running into this problem?
> >
> > > I would be rather suspicious of any app doing this.
> >
> > > > -----Original Message-----
> > > > From: informix-list-boun...@iiug.org
> > > > [mailto:informix-list-boun...@iiug.org]On Behalf Of
> > > > vomari...@yahoo.com
> > > > Sent: Thursday, February 21, 2008 3:52 PM
> > > > To: informix-l...@iiug.org
> > > > Subject: how do you solve this issue
> >
> > > >http://www-1.ibm.com/support/docview.wss?rs=0&uid=swg21251537
> > > > what's the hard limit of the internal file descriptor
> in ids 10 and
> > > > 11 ?
> > > > thanks
> > > > jj
> > > > _______________________________________________
> > > > Informix-list mailing list
> > > > Informix-l...@iiug.org
> > > >http://www.iiug.org/mailman/listinfo/informix-list
> >
> > > Notice of Confidentiality: **This E-mail and any of its
> attachments may contain
> > > Lincoln National Corporation proprietary information,
> which is privileged, confidential,
> > > or subject to copyright belonging to the Lincoln National
> Corporation family of
> > > companies. This E-mail is intended solely for the use of
> the individual or entity to
> > > which it is addressed. If you are not the intended
> recipient of this E-mail, you are
> > > hereby notified that any dissemination, distribution,
> copying, or action taken in
> > > relation to the contents of and attachments to this
> E-mail is strictly prohibited
> > > and may be unlawful. If you have received this E-mail in
> error, please notify the
> > > sender immediately and permanently delete the original
> and any copy of this E-mail
> > > and any printout. Thank You.**
> > > _______________________________________________
> > > Informix-list mailing list
> > > Informix-l...@iiug.org
> > >http://www.iiug.org/mailman/listinfo/informix-list
> >
> > _________________________________________________________________
> > Need to know the score, the latest news, or you need your
> Hotmail®-get your "fix".http://www.msnmobilefix.com/Default.aspx
>
>
> thanks. i'll ask the review the code. just curious about the limit.
> support doesn't know if there's one.
> _______________________________________________
> Informix-list mailing list
> Informix-list@iiug.org
> http://www.iiug.org/mailman/listinfo/informix-list
>
Notice of Confidentiality: **This E-mail and any of its attachments may contain
Lincoln National Corporation proprietary information, which is privileged, confidential,
or subject to copyright belonging to the Lincoln National Corporation family of
companies. This E-mail is intended solely for the use of the individual or entity to
which it is addressed. If you are not the intended recipient of this E-mail, you are
hereby notified that any dissemination, distribution, copying, or action taken in
relation to the contents of and attachments to this E-mail is strictly prohibited
and may be unlawful. If you have received this E-mail in error, please notify the
sender immediately and permanently delete the original and any copy of this E-mail
and any printout. Thank You.**