Slow connect times across certain dbserveraliases
Posted in 1999
Has anyone ever seen the initial connection to a database take longer
using one dbserveralias as opposed to another on the same NIC? Here's
a picture of what we're seeing.
We have 1 T500 HP (8 CPU) server (batman) running IDS 7.30.UC6. 8 CPU
VPs are defined. Valid INFORMIXSERVER names are
rimstcp
rimstcp2
rimstcp3
rimstcp4
rimstcp5
All dbservernames/aliases are bound to a single FDDI card.
On an average day, we have about 1,500 connections to the database.
We have two HP K210 (dual processor) application servers (catwoman &
batgirl) running 4GL applications. The applications use sockets to
communicate with the database on batman.
During our peak times, the connection times spike on rimstcp2
(batgirl), and rimstcp3 (catwoman). Here's some output from a script
that calculates connection times across all aliases.
Wed Feb 24 13:05:00 MST 1999
It's takes 5.30 seconds to connect to rims on rimstcp.
It's takes 24.91 seconds to connect to rims on rimstcp2.
It's takes 23.92 seconds to connect to rims on rimstcp3.
It's takes 3.73 seconds to connect to rims on rimstcp4.
It's takes 3.79 seconds to connect to rims on rimstcp5.
I thought it might be the number of connections across the port, so I
wrote a script to "count" the number of connections for a given
dbservername/alias.
Wed Feb 24 13:15:00 MST 1999
Gathering network socket information...
Getting ports...
Getting number of sockets per port...
UNIX port ** sqlexec ** has 413 socket connections. (rimstcp)
UNIX port ** sqlexec2 ** has 269 socket connections. (rimstcp2)
UNIX port ** sqlexec3 ** has 306 socket connections. (rimstcp3)
UNIX port ** sqlexec4 ** has 389 socket connections. (rimstcp4)
UNIX port ** sqlexec5 ** has 33 socket connections. (rimstcp5)
Well, this showed me the opposite if what I expected. Less
connections on rimstcp2&3, yet slower???? Hmmmmm...
I thought it might be a network problem but both the database and app
servers are on the same subnet and all are connected via FDDI. I
thought the "load" on the app servers might the issue but both of them
are sitting between 1.0 and 2.0. I thought it might be the load on
the database server but it's load is sitting around 2ish. I looked
over my kernel settings and I'm no where near 32767 file descriptors.
Glance says all system tables are at most 10% full. All servers have
about 1GB of Available memory during peak hours. I only have two or
three shared memory segments on batman during peak hours. I thought
it might be an HP sockets problem, so during the "slow" connection
period(s), I telneted to the socket directly and was connected almost
instantaneously!
Here's my current NETTYPE config.
NETTYPE soctcp,5,300,NET
Tonight, we're going to add more poll threads so the config will look
like the following.
NETTYPE soctcp,12,150,NET
I'm wondering if this will really help. onstat -g ntm shows me that
"max in box" has been as high as 3 for a given poll thread. To me,
this seems ok but since I have "little" documentation on the command,
I'm winging it. :) I would also expect that if I was "lacking" on
poll threads, ALL dbserveraliases would be "generally" slow when
connecting to the system.
I've changed the UNIX port number, no luck.
There are no errors reported in the online log for the database.
Everyone gets in! (really!) No one times out.
I thought about defining more aliases, then randomly connect the users
from the app servers to those aliases.
I've seen this problem in 7.24.UC5 and 7.30.UC6 for HP-UX.
Here's some more questions for someone to ponder.
Is a poll thread bound to a specific session for the life of the
connection? (I'm thinking No because the "received" column from
onstat -g ntm is balance across all poll threads)
Are there any sort connection/session bindings that occur on a given
dbserveralias?
Is there anyone out there who has a couple of UNIX app servers talking
to a database server with the number of connections described above?
How may connections per poll thread should be used on a midrange
system like this one?
Should I go with more dbserver aliases? Even more poll threads?
If I could find some tools to determine "what" the connection is
spinning on, I'd be much happier.. :(
I hope the above explanation is sufficient.
Brad Coates
COBE Laboratories Inc.
303-239-2298
bcoates@cobe.com