Random Server Slowdown "Redone"
A user on IDS 11.50.FC5 (RedHat 4, 64-bit, 4 CPUs, ~370 connections) reported random, unexplained instance slowdowns: low CPU/memory use, yet clients and OAT couldn't connect. He posted onstat -p/-P output and his full onconfig. Art Kagel suggested changing the ipcshm NETTYPE from NET to CPU class with ~4 poll threads, raising CLEANERS to match the 8 LRU queues, and cutting the oversized bufferpool (over half of 1.2M buffers unused) to free memory for SHMVIRTSIZE/DS_* settings. Others advised checking statistics, NETTYPE/BTSCANNER ratios, kernel parameters, OS and disk logs, network stats, and capturing sar/onstat data during an episode. No confirmed resolution is recorded in the thread.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
Hi,
In mid December I posted about a problem we have with an instance slowing down
at random times/days. Since that we upgraded the instance to IDS 11.5 64-bit
to better use the hardware assigned to Informix.
Well, the problem has returned and neded assistance in solving the puzzle.
At random days/time we experience slow downs in the responce of the instance,
checking the processor utilization shows no problem 10% average (spikes of
100% util in 1 or 2 processor of 4)and there is plenty of available memory
(the server has 8GB and IDS is consuming no more the 3GB), the most strange
issue is that we are monitoring this server with OAT v2.26 and when the slow
down occurs it can't connect, also, the production environment is using a mix
of clients: Oracle Application server, a compiled .NET app using ODBC (8
Machines).
The maximun number of conections we have seen is 370.
Below I include some info about the server that can be of interest.
My server is using RedHat 4 ES update 5
[informix@saih respaldo_conf]$ uname -a
Linux saih.inr.gob.mx 2.6.9-55.ELsmp #1 SMP Fri Apr 20 16:36:54 EDT 2007
x86_64 x86_64 x86_64 GNU/Linux
We are using IDS 11.5 FC5
[informix@saih respaldo_conf]$ onstat -
IBM Informix Dynamic Server Version 11.50.FC5 -- On-Line -- Up 14:44:21 --3296300 Kbytes
[informix@saih ~]$ onstat -p
IBM Informix Dynamic Server Version 11.50.FC5 -- On-Line -- Up 15:02:58 --3296300 Kbytes
Profile
dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits %cached
803051 5116508 1317398798 99.94 863679 1063515 7082328 87.97
isamtot open start read write rewrite delete commit rollbk
974690064 1029061 1914834 937852565 2519305 12637 73794 8051 4
gp_read gp_write gp_rewrt gp_del gp_alloc gp_free gp_curs
0 0 0 0 0 0 0
ovlock ovuserthread ovbuff usercpu syscpu numckpts flushes
0 0 0 3512.02 49.25 302 151
bufwaits lokwaits lockreqs deadlks dltouts ckpwaits compress seqscans
58848 2 2656077359 0 0 1 19463 70959
ixda-RA idx-RA da-RA RA-pgsused lchwaits
32181 2342 400009 434365 46228
And finaly I include an edited copy of my onconfig (to remove comments)
ROOTNAME rootdbs
ROOTPATH $INFORMIXDIR/liga_dbspaces/dbs_rootweb
ROOTOFFSET 0
ROOTSIZE 2008092MIRROR 0
MIRRORPATH $INFORMIXDIR/tmp/demo_on.root_mirror
MIRROROFFSET 0
PHYSFILE 50000
PLOG_OVERFLOW_PATH $INFORMIXDIR/tmp
PHYSBUFF 128
LOGFILES 20
LOGSIZE 10000
DYNAMIC_LOGS 2
LOGBUFF 64
LTXHWM 70
LTXEHWM 80MSGPATH $INFORMIXDIR/inrserver.log
CONSOLE $INFORMIXDIR/inrserver.con
TBLTBLFIRST 0
TBLTBLNEXT 0
TBLSPACE_STATS 1
DBSPACETEMP dbs_tempweb
SBSPACETEMP
SBSPACENAME
SYSSBSPACENAME
ONDBSPACEDOWN 2
SERVERNUM 0
DBSERVERNAME ol_inrserver
DBSERVERALIASES ol_inrservershm
NETTYPE ipcshm,1,50,NET
NETTYPE soctcp,3,150,NET
LISTEN_TIMEOUT 60
MAX_INCOMPLETE_CONNECTIONS 1024
FASTPOLL 1
MULTIPROCESSOR 1
VPCLASS cpu,num=4,noage
VPCLASS shm,num=1,noage
VPCLASS soc,num=3,noage
VP_MEMORY_CACHE_KB 0
SINGLE_CPU_VP 0
#VPCLASS aio,num=1
CLEANERS 4AUTO_AIOVPS 1
DIRECT_IO 0
LOCKS 20000
DEF_TABLE_LOCKMODE page
RESIDENT 2
SHMBASE 0x44000000L
SHMVIRTSIZE 262144
SHMADD 32768
EXTSHMADD 32768
SHMTOTAL 0
SHMVIRT_ALLOCSEG 0,3
SHMNOACCESS
CKPTINTVL 300AUTO_CKPTS 1
RTO_SERVER_RESTART 0
BLOCKTIMEOUT 3600
TXTIMEOUT 300
DEADLOCK_TIMEOUT 60
HETERO_COMMIT 0
TAPEDEV /dev/IBMtape0
TAPEBLK 32
TAPESIZE 0
LTAPEDEV /dev/null
LTAPEBLK 32
LTAPESIZE 0
BAR_ACT_LOG $INFORMIXDIR/tmp/bar_act.log
BAR_DEBUG_LOG $INFORMIXDIR/tmp/bar_dbug.log
BAR_DEBUG 0
BAR_MAX_BACKUP 0
BAR_RETRY 1
BAR_NB_XPORT_COUNT 20
BAR_XFER_BUF_SIZE 31
RESTARTABLE_RESTORE ON
BAR_PROGRESS_FREQ 0
BAR_BSALIB_PATH
BACKUP_FILTER
RESTORE_FILTER
BAR_PERFORMANCE 0ISM_DATA_POOL ISMData
ISM_LOG_POOL ISMLogs
DD_HASHSIZE 31
DD_HASHMAX 10
DS_HASHSIZE 31
DS_POOLSIZE 127
PC_HASHSIZE 31
PC_POOLSIZE 127
STMT_CACHE 0
STMT_CACHE_HITS 0
STMT_CACHE_SIZE 512
STMT_CACHE_NOLIMIT 0
STMT_CACHE_NUMPOOL 1
USEOSTIME 0
STACKSIZE 128
ALLOW_NEWLINE 0
USELASTCOMMITTED NONE
FILLFACTOR 90
MAX_FILL_DATA_PAGES 0
BTSCANNER num=1,threshold=5000,rangesize=-1,alice=6,compression=default
ONLIDX_MAXMEM 5120
MAX_PDQPRIORITY 20
DS_MAX_QUERIES
DS_TOTAL_MEMORY 2048
DS_MAX_SCANS 1048576
DS_NONPDQ_QUERY_MEM 512
DATASKIP
OPTCOMPIND 2
DIRECTIVES 1
EXT_DIRECTIVES 0
OPT_GOAL -1
IFX_FOLDVIEW 0AUTO_REPREPARE 1
RA_PAGES 64
RA_THRESHOLD 16
EXPLAIN_STAT 1
SQLTRACE level=med,ntraces=5000,size=10240,mode=global#DBCREATE_PERMISSION informix
#DB_LIBRARY_PATH
IFX_EXTEND_ROLE 1
SECURITY_LOCALCONNECTION
UNSECURE_ONSTAT
ADMIN_USER_MODE_WITH_DBSA
ADMIN_MODE_USERS
SSL_KEYSTORE_LABEL
PLCY_POOLSIZE 127
PLCY_HASHSIZE 31
USRC_POOLSIZE 127
USRC_HASHSIZE 31STAGEBLOB
OPCACHEMAX 0
ENCRYPT_HDR
ENCRYPT_SMX
ENCRYPT_CDR 0
ENCRYPT_CIPHERS
ENCRYPT_MAC
ENCRYPT_MACFILE
ENCRYPT_SWITCH
CDR_EVALTHREADS 1,2
CDR_DSLOCKWAIT 5
CDR_QUEUEMEM 4096
CDR_NIFCOMPRESS 0
CDR_SERIAL 0
CDR_DBSPACE
CDR_QHDR_DBSPACE
CDR_QDATA_SBSPACE
CDR_MAX_DYNAMIC_LOGS 0
CDR_SUPPRESS_ATSRISWARN
DRAUTO 0
DRINTERVAL 30
DRTIMEOUT 30
HA_ALIASDRLOSTFOUND $INFORMIXDIR/etc/dr.lostfound
DRIDXAUTO 0
LOG_INDEX_BUILDS
SDS_ENABLE
SDS_TIMEOUT 20
SDS_TEMPDBS
SDS_PAGING
UPDATABLE_SECONDARY 0
FAILOVER_CALLBACK
TEMPTAB_NOLOG 0
DELAY_APPLY 0
STOP_APPLY 0
LOG_STAGING_DIR
ON_RECVRY_THREADS 1
OFF_RECVRY_THREADS 10
DUMPDIR $INFORMIXDIR/tmp
DUMPSHMEM 1
DUMPGCORE 0
DUMPCORE 0
DUMPCNT 1ALARMPROGRAM $INFORMIXDIR/etc/alarmprogram.sh
ALRM_ALL_EVENTS 0
STORAGE_FULL_ALARM 600,3SYSALARMPROGRAM $INFORMIXDIR/etc/evidence.sh
RAS_PLOG_SPEED 25000
RAS_LLOG_SPEED 0
EILSEQ_COMPAT_MODE 0
QSTATS 0
WSTATS 0
#VPCLASS jvp,num=1JVPJAVAHOME $INFORMIXDIR/extend/krakatoa/jre
JVPHOME $INFORMIXDIR/extend/krakatoa
JVPPROPFILE $INFORMIXDIR/extend/krakatoa/.jvpprops
JVPLOGFILE $INFORMIXDIR/jvp.log
#JDKVERSION 1.5
JVPJAVALIB /bin/j9vm
JVPJAVAVM jvm
#JVPARGS -verbose:jni#JVPCLASSPATH
$INFORMIXDIR/extend/krakatoa/krakatoa_g.jar:$INFORMIXDIR/extend/krakatoa/jdbc_g.
jar
JVPCLASSPATH
$INFORMIXDIR/extend/krakatoa/krakatoa.jar:$INFORMIXDIR/extend/krakatoa/jdbc.jar
BUFFERPOOL
default,buffers=10000,lrus=8,lru_min_dirty=50.000000,lru_max_dirty=60.500000
BUFFERPOOL
size=2K,buffers=1200000,lrus=8,lru_min_dirty=50.000000,lru_max_dirty=60.000000
AUTO_LRU_TUNING 1
Hope you can help me.
Javier Hernández
While preparing this message It happened again adn per instructions posted
befor I include the output of the commands that Art and Obnoxio recomended.
onstat -p
IBM Informix Dynamic Server Version 11.50.FC5 -- On-Line -- Up 15:04:53 --3296300 Kbytes
Profile
dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits %cached
803375 5116897 1335552146 99.94 864233 1064307 7084189 87.96
isamtot open start read write rewrite delete commit rollbk
989923084 1040237 1935823 952832828 2519774 12713 74071 8118 4
gp_read gp_write gp_rewrt gp_del gp_alloc gp_free gp_curs
0 0 0 0 0 0 0
ovlock ovuserthread ovbuff usercpu syscpu numckpts flushes
0 0 0 3557.66 49.45 304 152
bufwaits lokwaits lockreqs deadlks dltouts ckpwaits compress seqscans
58886 2 2689656890 0 0 1 19509 71786
ixda-RA idx-RA da-RA RA-pgsused lchwaits
32276 2346 400009 434464 46452
onstat -P
IBM Informix Dynamic Server Version 11.50.FC5 -- On-Line -- Up 15:04:53 --3296300 Kbytes
Buffer pool page size: 2048
partnum total btree data other dirty
0 621337 0 19837 601500 1
1048577 34 0 32 2 0
1048578 3 1 1 1 0
1048580 30 11 19 0 0
1048581 54 24 30 0 0
1048582 13 3 10 0 0
1048583 17 8 9 0 0
1048584 4 1 3 0 0
1048585 9 3 6 0 0
1048586 3 1 1 1 0
1048590 1 1 0 0 0
1048593 1 1 0 0 0
1048595 15 10 5 0 0
1048596 4 2 2 0 0
1048597 3 2 1 0 0
1048598 3 2 1 0 0
1048599 1 1 0 0 0
1048600 1 1 0 0 0
1048601 1 1 0 0 0
1048603 1 1 0 0 0
1048604 4 3 1 0 0
1048606 4 3 1 0 0
1048608 2 1 1 0 0
1048612 14 6 8 0 0
1048624 8 5 3 0 0
1048625 3 3 0 0 0
1048740 15 6 8 1 0
1048741 20 7 13 0 0
1048742 12 1 11 0 0
1048743 7 4 3 0 0
1048744 2 1 1 0 0
1048745 3 1 2 0 0
1048746 3 1 1 1 0
1048750 2 1 1 0 0
1048751 2 1 1 0 0
1048752 9 4 5 0 0
1048753 9 1 8 0 0
1048754 2 1 1 0 0
1048755 31 14 17 0 0
1048756 58 13 45 0 0
1048757 19 5 14 0 0
1048759 1 1 0 0 0
1048760 1 1 0 0 0
1048761 2 1 1 0 0
1048762 4 1 3 0 0
1048763 59 7 52 0 0
1048764 8 1 7 0 0
1048766 6 4 2 0 0
1048768 2 1 1 0 0
1048770 1 1 0 0 0
1048772 12 6 6 0 0
1048774 2 1 1 0 0
1048776 2 1 1 0 0
1048784 5 3 2 0 0
1048785 3 3 0 0 0
1048803 1 0 1 0 0
1048806 10 0 9 1 0
1048808 1 1 0 0 0
1048809 1 1 0 0 0
1048810 1 1 0 0 0
1048811 22 0 21 1 0
1048812 7 7 0 0 0
1048813 17 16 0 1 0
1048814 16 16 0 0 0
1048815 45 0 44 1 0
1048816 2 2 0 0 0
1048817 2 2 0 0 0
1048818 3 0 2 1 0
1048821 1 0 1 0 0
1048822 1 1 0 0 0
1048823 18 0 17 1 0
1048824 3 0 2 1 0
1048825 408 0 407 1 0
1048826 27 0 26 1 0
1048827 120 0 119 1 0
1048828 357 343 0 14 0
1048829 7 6 0 1 0
1048830 88 0 87 1 0
1048832 3 0 3 0 0
1048833 5 5 0 0 0
1048879 403 0 402 1 0
1048884 2 0 1 1 0
1048886 36142 0 36132 10 0
1048887 39763 0 39752 11 0
1048888 2387 1975 0 412 0
1048889 365 331 0 34 0
1048890 224 185 0 39 0
1048891 1647 1370 0 277 0
1048892 2868 2404 0 464 0
1048893 461 388 0 73 0
1048894 2131 1792 0 339 0
1048895 1 1 0 0 0
1048898 85 49 35 1 0
1048899 1 0 1 0 0
1048902 2 2 0 0 0
2097153 1 0 0 1 0
3145729 25 0 4 21 2
3145730 16800 496 16299 5 0
3145731 18 1 15 2 0
3145732 1 0 0 1 0
3145733 4 0 3 1 2
4194305 434 0 433 1 8
4194306 108 43 64 1 4
4194307 205 79 125 1 5
4194308 272 18 253 1 0
4194309 44 19 24 1 4
4194310 2 1 1 0 0
4194311 14 4 9 1 5
4194312 3 1 1 1 0
4194313 4 2 1 1 3
4194315 1 1 0 0 0
4194316 153 21 132 0 0
4194317 13 8 5 0 0
4194319 1 1 0 0 0
4194320 33 21 12 0 0
4194321 132 17 114 1 0
4194322 30 14 15 1 0
4194323 14 8 5 1 0
4194324 10 7 2 1 0
4194325 11 5 6 0 0
4194326 1 1 0 0 0
4194327 1 1 0 0 0
4194329 2286 283 2003 0 0
4194330 215 22 193 0 0
4194331 1 1 0 0 0
4194332 6 4 1 1 0
4194333 3 1 1 1 0
4194334 5 3 1 1 0
4194335 1 1 0 0 0
4194336 1 1 0 0 0
4194338 13 5 8 0 0
4194341 1 1 0 0 0
4194347 1 1 0 0 0
4194348 1 1 0 0 0
4194350 281 184 97 0 0
4194351 3 3 0 0 0
4194365 5 3 1 1 0
4194368 1 0 1 0 0
4194369 1 1 0 0 0
4194370 7 0 6 1 0
4194371 1 1 0 0 0
4194376 4 0 3 1 0
4194378 8 0 8 0 0
4194382 11 0 10 1 0
4194383 4 4 0 0 0
4194384 1 0 1 0 0
4194386 1 0 1 0 0
4194388 1 0 1 0 0
4194389 1 1 0 0 0
4194390 1 0 1 0 0
4194392 1 0 1 0 0
4194394 1 0 1 0 0
4194396 1 0 1 0 0
4194398 3 0 2 1 0
4194402 143 0 142 1 0
4194403 30 30 0 0 0
4194404 541 0 540 1 0
4194405 18 18 0 0 0
4194406 70091 0 70073 18 0
4194407 789 786 0 3 0
4194408 2985 0 2984 1 0
4194410 1 0 1 0 0
4194411 1 1 0 0 0
4194412 2 0 2 0 2
4194413 5 5 0 0 0
4194415 1 1 0 0 0
4194416 1 0 1 0 0
4194417 1 1 0 0 0
4194422 2 0 1 1 0
4194423 3 3 0 0 0
4194424 4 0 3 1 0
4194425 5 4 0 1 0
4194428 1 0 1 0 0
4194436 8 0 7 1 0
4194437 15 14 0 1 0
4194438 89 0 88 1 0
4194439 6 5 0 1 0
4194440 24 0 23 1 0
4194441 4 4 0 0 0
4194442 1 0 1 0 0
4194446 1122 0 1121 1 0
4194447 1744 1744 0 0 0
4194454 366 0 365 1 0
4194455 88 88 0 0 0
4194458 2162 0 2161 1 0
4194459 64 62 0 2 0
4194462 238 0 233 5 0
4194463 374 366 0 8 0
4194464 1674 0 1673 1 0
4194465 53 53 0 0 0
4194466 9 0 8 1 0
4194467 9 9 0 0 0
4194468 19 0 18 1 0
4194469 28 28 0 0 0
4194470 3 0 3 0 0
4194476 1 0 1 0 0
4194477 1 1 0 0 0
4194478 3 0 2 1 0
4194479 1 1 0 0 0
4194480 3 0 2 1 0
4194481 1 1 0 0 0
4194482 3 0 2 1 0
4194483 5 5 0 0 0
4194484 1390 0 1389 1 0
4194485 3 3 0 0 0
4194486 4814 0 4812 2 0
4194487 3 3 0 0 0
4194488 1 0 1 0 0
4194489 1 1 0 0 0
4194490 14363 0 14359 4 0
4194491 11 10 0 1 0
4194492 3824 0 3820 4 0
4194493 811 810 0 1 0
4194494 2743 0 2742 1 0
4194495 5 4 0 1 0
4194496 1 0 1 0 0
4194497 1 0 1 0 0
4194498 1 1 0 0 0
4194499 1 0 1 0 0
4194500 1 1 0 0 0
4194501 1 0 1 0 0
4194503 9 0 8 1 0
4194504 2 2 0 0 0
4194506 15 15 0 0 0
4194509 16 0 15 1 0
4194511 1 0 1 0 0
4194512 1 1 0 0 0
4194513 33 0 32 1 0
4194514 37 37 0 0 0
4194517 1 0 1 0 0
4194518 1 1 0 0 0
4194519 1 0 1 0 0
4194520 1 1 0 0 0
4194523 26 0 23 3 0
4194524 5 5 0 0 0
4194527 8 0 8 0 0
4194528 4 4 0 0 0
4194531 26720 0 26705 15 0
4194532 10 10 0 0 0
4194533 1494 0 1487 7 0
4194534 554 548 0 6 0
4194535 14891 0 14876 15 0
4194536 16 16 0 0 0
4194537 6603 0 6600 3 0
4194538 10 10 0 0 0
4194539 163 0 151 12 0
4194540 26 26 0 0 0
4194541 25 0 25 0 0
4194542 3 3 0 0 0
4194545 38459 0 38436 23 0
4194546 4 4 0 0 0
4194554 8 0 7 1 0
4194555 8 0 7 1 0
4194556 3 3 0 0 0
4194557 23 0 22 1 0
4194558 4 4 0 0 0
4194561 3 0 2 1 0
4194562 3 3 0 0 0
4194563 1 0 1 0 0
4194564 1 1 0 0 0
4194565 1 0 1 0 0
4194566 1 1 0 0 0
4194567 1 0 1 0 0
4194568 1 1 0 0 0
4194570 1 1 0 0 0
4194571 1 0 1 0 0
4194572 1 1 0 0 0
4194573 3 0 2 1 0
4194574 1 1 0 0 0
4194585 1 0 1 0 0
4194589 3 0 2 1 0
4194591 1 0 1 0 0
4194592 1 1 0 0 0
4194593 392 0 391 1
There are some usual things you could do for now:
1) calc your database ratios, I think there are several NETTYPE, BUFFERS
and even BTSCANNER options that could be optimized,
you could download Mr. Art script to automatically know these important
numbers!!! It´s on the download section of IIUG site.
Obs: How many cpu cores do you have?? Your vp cpu class is enough for
the applications ???
2) are you updating the database statistics regularly? Is it working fine???
Regards.
Alexandre Marini
Tecnologia da Informação - DBA
SEFAZ-MS / SGI-UIMP / Sistemas IBM-Informix
IIUG Member
<http://www.iiug.org>;
JAVIER HERNáNDEZ escreveu:
> Hi,
>
> In mid December I posted about a problem we have with an instance slowing
down
> at random times/days. Since that we upgraded the instance to IDS 11.5 64-bit
> to better use the hardware assigned to Informix.
>
> Well, the problem has returned and neded assistance in solving the puzzle.
>
> At random days/time we experience slow downs in the responce of the instance,
> checking the processor utilization shows no problem 10% average (spikes of
> 100% util in 1 or 2 processor of 4)and there is plenty of available memory
> (the server has 8GB and IDS is consuming no more the 3GB), the most strange
> issue is that we are monitoring this server with OAT v2.26 and when the slow
> down occurs it can't connect, also, the production environment is using a mix
> of clients: Oracle Application server, a compiled .NET app using ODBC (8
> Machines).
>
> The maximun number of conections we have seen is 370.
>
> Below I include some info about the server that can be of interest.
>
> My server is using RedHat 4 ES update 5
> [informix@saih respaldo_conf]$ uname -a
> Linux saih.inr.gob.mx 2.6.9-55.ELsmp #1 SMP Fri Apr 20 16:36:54 EDT 2007
> x86_64 x86_64 x86_64 GNU/Linux
>
> We are using IDS 11.5 FC5
> [informix@saih respaldo_conf]$ onstat -
>
> IBM Informix Dynamic Server Version 11.50.FC5 -- On-Line -- Up 14:44:21 --> 3296300 Kbytes
>
> [informix@saih ~]$ onstat -p
>
> IBM Informix Dynamic Server Version 11.50.FC5 -- On-Line -- Up 15:02:58 --> 3296300 Kbytes
>
> Profile
> dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits %cached
> 803051 5116508 1317398798 99.94 863679 1063515 7082328 87.97
>
> isamtot open start read write rewrite delete commit rollbk
> 974690064 1029061 1914834 937852565 2519305 12637 73794 8051 4
>
> gp_read gp_write gp_rewrt gp_del gp_alloc gp_free gp_curs
> 0 0 0 0 0 0 0
>
> ovlock ovuserthread ovbuff usercpu syscpu numckpts flushes
> 0 0 0 3512.02 49.25 302 151
>
> bufwaits lokwaits lockreqs deadlks dltouts ckpwaits compress seqscans
> 58848 2 2656077359 0 0 1 19463 70959
>
> ixda-RA idx-RA da-RA RA-pgsused lchwaits
> 32181 2342 400009 434365 46228
>
> And finaly I include an edited copy of my onconfig (to remove comments)
>
> ROOTNAME rootdbs
> ROOTPATH $INFORMIXDIR/liga_dbspaces/dbs_rootweb
> ROOTOFFSET 0
> ROOTSIZE 2008092> MIRROR 0
> MIRRORPATH $INFORMIXDIR/tmp/demo_on.root_mirror
> MIRROROFFSET 0
> PHYSFILE 50000
> PLOG_OVERFLOW_PATH $INFORMIXDIR/tmp
> PHYSBUFF 128
> LOGFILES 20
> LOGSIZE 10000
> DYNAMIC_LOGS 2
> LOGBUFF 64
> LTXHWM 70
> LTXEHWM 80> MSGPATH $INFORMIXDIR/inrserver.log
> CONSOLE $INFORMIXDIR/inrserver.con
> TBLTBLFIRST 0
> TBLTBLNEXT 0
> TBLSPACE_STATS 1
> DBSPACETEMP dbs_tempweb
> SBSPACETEMP
> SBSPACENAME
> SYSSBSPACENAME
> ONDBSPACEDOWN 2
> SERVERNUM 0
> DBSERVERNAME ol_inrserver
> DBSERVERALIASES ol_inrservershm
> NETTYPE ipcshm,1,50,NET
> NETTYPE soctcp,3,150,NET
> LISTEN_TIMEOUT 60
> MAX_INCOMPLETE_CONNECTIONS 1024
> FASTPOLL 1
> MULTIPROCESSOR 1
> VPCLASS cpu,num=4,noage
> VPCLASS shm,num=1,noage
> VPCLASS soc,num=3,noage
> VP_MEMORY_CACHE_KB 0
> SINGLE_CPU_VP 0
> #VPCLASS aio,num=1
> CLEANERS 4> AUTO_AIOVPS 1
> DIRECT_IO 0
> LOCKS 20000
> DEF_TABLE_LOCKMODE page
> RESIDENT 2
> SHMBASE 0x44000000L
> SHMVIRTSIZE 262144
> SHMADD 32768
> EXTSHMADD 32768
> SHMTOTAL 0
> SHMVIRT_ALLOCSEG 0,3
> SHMNOACCESS
> CKPTINTVL 300> AUTO_CKPTS 1
> RTO_SERVER_RESTART 0
> BLOCKTIMEOUT 3600
> TXTIMEOUT 300
> DEADLOCK_TIMEOUT 60
> HETERO_COMMIT 0
> TAPEDEV /dev/IBMtape0
> TAPEBLK 32
> TAPESIZE 0
> LTAPEDEV /dev/null
> LTAPEBLK 32
> LTAPESIZE 0
> BAR_ACT_LOG $INFORMIXDIR/tmp/bar_act.log
> BAR_DEBUG_LOG $INFORMIXDIR/tmp/bar_dbug.log
> BAR_DEBUG 0
> BAR_MAX_BACKUP 0
> BAR_RETRY 1
> BAR_NB_XPORT_COUNT 20
> BAR_XFER_BUF_SIZE 31
> RESTARTABLE_RESTORE ON
> BAR_PROGRESS_FREQ 0
> BAR_BSALIB_PATH
> BACKUP_FILTER
> RESTORE_FILTER
> BAR_PERFORMANCE 0> ISM_DATA_POOL ISMData
> ISM_LOG_POOL ISMLogs
> DD_HASHSIZE 31
> DD_HASHMAX 10
> DS_HASHSIZE 31
> DS_POOLSIZE 127
> PC_HASHSIZE 31
> PC_POOLSIZE 127
> STMT_CACHE 0
> STMT_CACHE_HITS 0
> STMT_CACHE_SIZE 512
> STMT_CACHE_NOLIMIT 0
> STMT_CACHE_NUMPOOL 1
> USEOSTIME 0
> STACKSIZE 128
> ALLOW_NEWLINE 0
> USELASTCOMMITTED NONE
> FILLFACTOR 90
> MAX_FILL_DATA_PAGES 0
> BTSCANNER num=1,threshold=5000,rangesize=-1,alice=6,compression=default
> ONLIDX_MAXMEM 5120
> MAX_PDQPRIORITY 20
> DS_MAX_QUERIES
> DS_TOTAL_MEMORY 2048
> DS_MAX_SCANS 1048576
> DS_NONPDQ_QUERY_MEM 512
> DATASKIP
> OPTCOMPIND 2
> DIRECTIVES 1
> EXT_DIRECTIVES 0
> OPT_GOAL -1
> IFX_FOLDVIEW 0> AUTO_REPREPARE 1
> RA_PAGES 64
> RA_THRESHOLD 16
> EXPLAIN_STAT 1
> SQLTRACE level=med,ntraces=5000,size=10240,mode=global> #DBCREATE_PERMISSION informix
> #DB_LIBRARY_PATH
> IFX_EXTEND_ROLE 1
> SECURITY_LOCALCONNECTION
> UNSECURE_ONSTAT
> ADMIN_USER_MODE_WITH_DBSA
> ADMIN_MODE_USERS
> SSL_KEYSTORE_LABEL
> PLCY_POOLSIZE 127
> PLCY_HASHSIZE 31
> USRC_POOLSIZE 127
> USRC_HASHSIZE 31> STAGEBLOB
> OPCACHEMAX 0
> ENCRYPT_HDR
> ENCRYPT_SMX
> ENCRYPT_CDR 0
> ENCRYPT_CIPHERS
> ENCRYPT_MAC
> ENCRYPT_MACFILE
> ENCRYPT_SWITCH
> CDR_EVALTHREADS 1,2
> CDR_DSLOCKWAIT 5
> CDR_QUEUEMEM 4096
> CDR_NIFCOMPRESS 0
> CDR_SERIAL 0
> CDR_DBSPACE
> CDR_QHDR_DBSPACE
> CDR_QDATA_SBSPACE
> CDR_MAX_DYNAMIC_LOGS 0
> CDR_SUPPRESS_ATSRISWARN
> DRAUTO 0
> DRINTERVAL 30
> DRTIMEOUT 30
> HA_ALIAS> DRLOSTFOUND $INFORMIXDIR/etc/dr.lostfound
> DRIDXAUTO 0
> LOG_INDEX_BUILDS
> SDS_ENABLE
> SDS_TIMEOUT 20
> SDS_TEMPDBS
> SDS_PAGING
> UPDATABLE_SECONDARY 0
> FAILOVER_CALLBACK
> TEMPTAB_NOLOG 0
> DELAY_APPLY 0
> STOP_APPLY 0
> LOG_STAGING_DIR
> ON_RECVRY_THREADS 1
> OFF_RECVRY_THREADS 10
> DUMPDIR $INFORMIXDIR/tmp
> DUMPSHMEM 1
> DUMPGCORE 0
> DUMPCORE 0
> DUMPCNT 1> ALARMPROGRAM $INFORMIXDIR/etc/alarmprogram.sh
> ALRM_ALL_EVENTS 0
> STORAGE_FULL_ALARM 600,3> SYSALARMPROGRAM $INFORMIXDIR/etc/evidence.sh
> RAS_PLOG_SPEED 25000
> RAS_LLOG_SPEED 0
> EILSEQ_COMPAT_MODE 0
> QSTATS 0
> WSTATS 0
> #VPCLASS jvp,num=1
> JVPJAVAHOME $INF
I'm going to suggest that you change your NETTYPE for ipcshm from NET to CPU
and from a single poll thread to 4 (one for each CPU VP) if there are a
significant number of users connecting via shared memory (if this connection
type is for maintenance only then make the change but you can get away with
only two poll threads). Search the forum history or the Informix FAQ for my
reasoning on this.
Also you have 8 LRU queues but only 4 CLEANERS. You should have the greater
of at least one cleaner thread for each LRU queue or 1.25 cleaner threads
per chunk. Adjust accordingly.
Art
Art S. Kagel
Advanced DataTools (www.advancedatatools.com)
IIUG Board of Directors (art@iiug.org)
See you at the 2010 IIUG Informix Conference
April 25-28, 2010
Overland Park (Kansas City), KS
www.iiug.org/conf
Disclaimer: Please keep in mind that my own opinions are my own opinions and
do not reflect on my employer, Advanced DataTools, the IIUG, nor any other
organization with which I am associated either explicitly, implicitly, or by
inference. Neither do those opinions reflect those of other individuals
affiliated with any entity with which I am affiliated nor those of the
entities themselves.
2010/1/13 JAVIER HERNáNDEZ <javier_hdzt@yahoo.com.mx>
> Hi,
>
> In mid December I posted about a problem we have with an instance slowing
> down
> at random times/days. Since that we upgraded the instance to IDS 11.5
> 64-bit
> to better use the hardware assigned to Informix.
>
> Well, the problem has returned and neded assistance in solving the puzzle.
>
> At random days/time we experience slow downs in the responce of the
> instance,
> checking the processor utilization shows no problem 10% average (spikes of
> 100% util in 1 or 2 processor of 4)and there is plenty of available memory
> (the server has 8GB and IDS is consuming no more the 3GB), the most strange
> issue is that we are monitoring this server with OAT v2.26 and when the
> slow
> down occurs it can't connect, also, the production environment is using a
> mix
> of clients: Oracle Application server, a compiled .NET app using ODBC (8
> Machines).
>
> The maximun number of conections we have seen is 370.
>
> Below I include some info about the server that can be of interest.
>
> My server is using RedHat 4 ES update 5
> [informix@saih respaldo_conf]$ uname -a
> Linux saih.inr.gob.mx 2.6.9-55.ELsmp #1 SMP Fri Apr 20 16:36:54 EDT 2007
> x86_64 x86_64 x86_64 GNU/Linux
>
> We are using IDS 11.5 FC5
> [informix@saih respaldo_conf]$ onstat -
>
> IBM Informix Dynamic Server Version 11.50.FC5 -- On-Line -- Up 14:44:21 --> 3296300 Kbytes
>
> [informix@saih ~]$ onstat -p
>
> IBM Informix Dynamic Server Version 11.50.FC5 -- On-Line -- Up 15:02:58 --> 3296300 Kbytes
>
> Profile
> dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits %cached
> 803051 5116508 1317398798 99.94 863679 1063515 7082328 87.97
>
> isamtot open start read write rewrite delete commit rollbk
> 974690064 1029061 1914834 937852565 2519305 12637 73794 8051 4
>
> gp_read gp_write gp_rewrt gp_del gp_alloc gp_free gp_curs
> 0 0 0 0 0 0 0
>
> ovlock ovuserthread ovbuff usercpu syscpu numckpts flushes
> 0 0 0 3512.02 49.25 302 151
>
> bufwaits lokwaits lockreqs deadlks dltouts ckpwaits compress seqscans
> 58848 2 2656077359 0 0 1 19463 70959
>
> ixda-RA idx-RA da-RA RA-pgsused lchwaits
> 32181 2342 400009 434365 46228
>
> And finaly I include an edited copy of my onconfig (to remove comments)
>
> ROOTNAME rootdbs
> ROOTPATH $INFORMIXDIR/liga_dbspaces/dbs_rootweb
> ROOTOFFSET 0
> ROOTSIZE 2008092> MIRROR 0
> MIRRORPATH $INFORMIXDIR/tmp/demo_on.root_mirror
> MIRROROFFSET 0
> PHYSFILE 50000
> PLOG_OVERFLOW_PATH $INFORMIXDIR/tmp
> PHYSBUFF 128
> LOGFILES 20
> LOGSIZE 10000
> DYNAMIC_LOGS 2
> LOGBUFF 64
> LTXHWM 70
> LTXEHWM 80> MSGPATH $INFORMIXDIR/inrserver.log
> CONSOLE $INFORMIXDIR/inrserver.con
> TBLTBLFIRST 0
> TBLTBLNEXT 0
> TBLSPACE_STATS 1
> DBSPACETEMP dbs_tempweb
> SBSPACETEMP
> SBSPACENAME
> SYSSBSPACENAME
> ONDBSPACEDOWN 2
> SERVERNUM 0
> DBSERVERNAME ol_inrserver
> DBSERVERALIASES ol_inrservershm
> NETTYPE ipcshm,1,50,NET
> NETTYPE soctcp,3,150,NET
> LISTEN_TIMEOUT 60
> MAX_INCOMPLETE_CONNECTIONS 1024
> FASTPOLL 1
> MULTIPROCESSOR 1
> VPCLASS cpu,num=4,noage
> VPCLASS shm,num=1,noage
> VPCLASS soc,num=3,noage
> VP_MEMORY_CACHE_KB 0
> SINGLE_CPU_VP 0
> #VPCLASS aio,num=1
> CLEANERS 4> AUTO_AIOVPS 1
> DIRECT_IO 0
> LOCKS 20000
> DEF_TABLE_LOCKMODE page
> RESIDENT 2
> SHMBASE 0x44000000L
> SHMVIRTSIZE 262144
> SHMADD 32768
> EXTSHMADD 32768
> SHMTOTAL 0
> SHMVIRT_ALLOCSEG 0,3
> SHMNOACCESS
> CKPTINTVL 300> AUTO_CKPTS 1
> RTO_SERVER_RESTART 0
> BLOCKTIMEOUT 3600
> TXTIMEOUT 300
> DEADLOCK_TIMEOUT 60
> HETERO_COMMIT 0
> TAPEDEV /dev/IBMtape0
> TAPEBLK 32
> TAPESIZE 0
> LTAPEDEV /dev/null
> LTAPEBLK 32
> LTAPESIZE 0
> BAR_ACT_LOG $INFORMIXDIR/tmp/bar_act.log
> BAR_DEBUG_LOG $INFORMIXDIR/tmp/bar_dbug.log
> BAR_DEBUG 0
> BAR_MAX_BACKUP 0
> BAR_RETRY 1
> BAR_NB_XPORT_COUNT 20
> BAR_XFER_BUF_SIZE 31
> RESTARTABLE_RESTORE ON
> BAR_PROGRESS_FREQ 0
> BAR_BSALIB_PATH
> BACKUP_FILTER
> RESTORE_FILTER
> BAR_PERFORMANCE 0> ISM_DATA_POOL ISMData
> ISM_LOG_POOL ISMLogs
> DD_HASHSIZE 31
> DD_HASHMAX 10
> DS_HASHSIZE 31
> DS_POOLSIZE 127
> PC_HASHSIZE 31
> PC_POOLSIZE 127
> STMT_CACHE 0
> STMT_CACHE_HITS 0
> STMT_CACHE_SIZE 512
> STMT_CACHE_NOLIMIT 0
> STMT_CACHE_NUMPOOL 1
> USEOSTIME 0
> STACKSIZE 128
> ALLOW_NEWLINE 0
> USELASTCOMMITTED NONE
> FILLFACTOR 90
> MAX_FILL_DATA_PAGES 0
> BTSCANNER num=1,threshold=5000,rangesize=-1,alice=6,compression=default
> ONLIDX_MAXMEM 5120
> MAX_PDQPRIORITY 20
> DS_MAX_QUERIES
> DS_TOTAL_MEMORY 2048
> DS_MAX_SCANS 1048576
> DS_NONPDQ_QUERY_MEM 512
> DATASKIP
> OPTCOMPIND 2
> DIRECTIVES 1
> EXT_DIRECTIVES 0
> OPT_GOAL -1
> IFX_FOLDVIEW 0> AUTO_REPREPARE 1
> RA_PAGES 64
> RA_THRESHOLD 16
> EXPLAIN_STAT 1
> SQLTRACE level=med,ntraces=5000,size=10240,mode=global> #DBCREATE_PERMISSION informix
> #DB_LIBRARY_PATH
> IFX_EXTEND_ROLE 1
> SECURITY_LOCALCONNECTION
> UNSECURE_ONSTAT
> ADMIN_USER_MODE_WITH_DBSA
> ADMIN_MODE_USERS
> SSL_KEYSTORE_LABEL
> PLCY_POOLSIZE 127
> PLCY_HASHSIZE 31
> USRC_POOLSIZE 127
> USRC_HASHSIZE 31> STAGEBLOB
> OPCACHEMAX 0
> ENCRYPT_HDR
> ENCRYPT_SMX
> ENCRYPT_CDR 0
> ENCRYPT_CIPHERS
> ENCRYPT_MAC
> ENCRYPT_MACFILE
> ENCRYPT_SWITCH
> CDR_EVALTHREADS 1,2
> CDR_DSLOCKWAIT 5
> CDR_QUEUEMEM 4096
> CDR_NIFCOMPRESS 0
> CDR_SERIAL 0
> CDR_DBSPACE
> CDR_QHDR_DBSPACE
> CDR_QDATA_SBSPACE
> CDR_MAX_DYNAMIC_LOGS 0
> CDR_SUPPRESS_ATSRISWARN
> DRAUTO 0
> DRINTERVAL 30
> DRTIMEOUT 30
> HA_ALIAS> DRLOSTFOUND $INFORMIXDIR/etc/dr.lostfound
> DRIDXAUTO 0
> LOG_INDEX_BUILDS
> SDS_ENABLE
> SDS_TIMEOUT
I see one thing, you have 1,200,000 buffers configured but over half
(601,500) of those buffers are currently unused - see the onstat -P report
on the first line (partnum 0) the 'other' column is overhead pages for most
partnums but also counts unused pages for partnum zero which is all
overhead. Looks like you can safely reduce your BUFFERPOOL to 600000 with
no adverse effect on performance and free up that memory. If you want,
assign some of it to SHMVIRTSIZE and from there to the memory grant manager
using the DS_* ONCONFIG parameters to speed query processing and sorting.
Art
Art S. Kagel
Advanced DataTools (www.advancedatatools.com)
IIUG Board of Directors (art@iiug.org)
See you at the 2010 IIUG Informix Conference
April 25-28, 2010
Overland Park (Kansas City), KS
www.iiug.org/conf
Disclaimer: Please keep in mind that my own opinions are my own opinions and
do not reflect on my employer, Advanced DataTools, the IIUG, nor any other
organization with which I am associated either explicitly, implicitly, or by
inference. Neither do those opinions reflect those of other individuals
affiliated with any entity with which I am affiliated nor those of the
entities themselves.
2010/1/13 JAVIER HERNáNDEZ <javier_hdzt@yahoo.com.mx>
> While preparing this message It happened again adn per instructions posted
> befor I include the output of the commands that Art and Obnoxio recomended.
>
> onstat -p>
> IBM Informix Dynamic Server Version 11.50.FC5 -- On-Line -- Up 15:04:53 --> 3296300 Kbytes
>
> Profile
> dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits %cached
> 803375 5116897 1335552146 99.94 864233 1064307 7084189 87.96
>
> isamtot open start read write rewrite delete commit rollbk
> 989923084 1040237 1935823 952832828 2519774 12713 74071 8118 4
>
> gp_read gp_write gp_rewrt gp_del gp_alloc gp_free gp_curs
> 0 0 0 0 0 0 0
>
> ovlock ovuserthread ovbuff usercpu syscpu numckpts flushes
> 0 0 0 3557.66 49.45 304 152
>
> bufwaits lokwaits lockreqs deadlks dltouts ckpwaits compress seqscans
> 58886 2 2689656890 0 0 1 19509 71786
>
> ixda-RA idx-RA da-RA RA-pgsused lchwaits
> 32276 2346 400009 434464 46452
>
> onstat -P>
> IBM Informix Dynamic Server Version 11.50.FC5 -- On-Line -- Up 15:04:53 --> 3296300 Kbytes
>
> Buffer pool page size: 2048
> partnum total btree data other dirty
> 0 621337 0 19837 601500 1
> 1048577 34 0 32 2 0
> 1048578 3 1 1 1 0
> 1048580 30 11 19 0 0
> 1048581 54 24 30 0 0
> 1048582 13 3 10 0 0
> 1048583 17 8 9 0 0
> 1048584 4 1 3 0 0
> 1048585 9 3 6 0 0
> 1048586 3 1 1 1 0
> 1048590 1 1 0 0 0
> 1048593 1 1 0 0 0
> 1048595 15 10 5 0 0
> 1048596 4 2 2 0 0
> 1048597 3 2 1 0 0
> 1048598 3 2 1 0 0
> 1048599 1 1 0 0 0
> 1048600 1 1 0 0 0
> 1048601 1 1 0 0 0
> 1048603 1 1 0 0 0
> 1048604 4 3 1 0 0
> 1048606 4 3 1 0 0
> 1048608 2 1 1 0 0
> 1048612 14 6 8 0 0
> 1048624 8 5 3 0 0
> 1048625 3 3 0 0 0
> 1048740 15 6 8 1 0
> 1048741 20 7 13 0 0
> 1048742 12 1 11 0 0
> 1048743 7 4 3 0 0
> 1048744 2 1 1 0 0
> 1048745 3 1 2 0 0
> 1048746 3 1 1 1 0
> 1048750 2 1 1 0 0
> 1048751 2 1 1 0 0
> 1048752 9 4 5 0 0
> 1048753 9 1 8 0 0
> 1048754 2 1 1 0 0
> 1048755 31 14 17 0 0
> 1048756 58 13 45 0 0
> 1048757 19 5 14 0 0
> 1048759 1 1 0 0 0
> 1048760 1 1 0 0 0
> 1048761 2 1 1 0 0
> 1048762 4 1 3 0 0
> 1048763 59 7 52 0 0
> 1048764 8 1 7 0 0
> 1048766 6 4 2 0 0
> 1048768 2 1 1 0 0
> 1048770 1 1 0 0 0
> 1048772 12 6 6 0 0
> 1048774 2 1 1 0 0
> 1048776 2 1 1 0 0
> 1048784 5 3 2 0 0
> 1048785 3 3 0 0 0
> 1048803 1 0 1 0 0
> 1048806 10 0 9 1 0
> 1048808 1 1 0 0 0
> 1048809 1 1 0 0 0
> 1048810 1 1 0 0 0
> 1048811 22 0 21 1 0
> 1048812 7 7 0 0 0
> 1048813 17 16 0 1 0
> 1048814 16 16 0 0 0
> 1048815 45 0 44 1 0
> 1048816 2 2 0 0 0
> 1048817 2 2 0 0 0
> 1048818 3 0 2 1 0
> 1048821 1 0 1 0 0
> 1048822 1 1 0 0 0
> 1048823 18 0 17 1 0
> 1048824 3 0 2 1 0
> 1048825 408 0 407 1 0
> 1048826 27 0 26 1 0
> 1048827 120 0 119 1 0
> 1048828 357 343 0 14 0
> 1048829 7 6 0 1 0
> 1048830 88 0 87 1 0
> 1048832 3 0 3 0 0
> 1048833 5 5 0 0 0
> 1048879 403 0 402 1 0
> 1048884 2 0 1 1 0
> 1048886 36142 0 36132 10 0
> 1048887 39763 0 39752 11 0
> 1048888 2387 1975 0 412 0
> 1048889 365 331 0 34 0
> 1048890 224 185 0 39 0
> 1048891 1647 1370 0 277 0
> 1048892 2868 2404 0 464 0
> 1048893 461 388 0 73 0
> 1048894 2131 1792 0 339 0
> 1048895 1 1 0 0 0
> 1048898 85 49 35 1 0
> 1048899 1 0 1 0 0
> 1048902 2 2 0 0 0
> 2097153 1 0 0 1 0
> 3145729 25 0 4 21 2
> 3145730 16800 496 16299 5 0
> 3145731 18 1 15 2 0
> 3145732 1 0 0 1 0
> 3145733 4 0 3 1 2
> 4194305 434 0 433 1 8
> 4194306 108 43 64 1 4
> 4194307 205 79 125 1 5
> 4194308 272 18 253 1 0
> 4194309 44 19 24 1 4
> 4194310 2 1 1 0 0
> 4194311 14 4 9 1 5
> 4194312 3 1 1 1 0
> 4194313 4 2 1 1 3
> 4194315 1 1 0 0 0
> 4194316 153 21 132 0 0
> 4194317 13 8 5 0 0
> 4194319 1 1 0 0 0
> 4194320 33 21 12 0 0
> 4194321 132 17 114 1 0
> 4194322 30 14 15 1 0
> 4194323 14 8 5 1 0
> 4194324 10 7 2 1 0
> 4194325 11 5 6 0 0
> 4194326 1 1 0 0 0
> 4194327 1 1 0 0 0
> 4194329 2286 283 2003 0 0
> 4194330 215 22 193 0 0
> 4194331 1 1 0 0 0
> 4194332 6 4 1 1 0
> 4194333 3 1 1 1 0
> 4194334 5 3 1 1 0
> 4194335 1 1 0 0 0
> 4194336 1 1 0 0 0
> 4194338 13 5 8 0 0
> 4194341 1 1 0 0 0
> 4194347 1 1 0 0 0
> 4194348 1 1 0 0 0
> 4194350 281 184 97 0 0
> 4194351 3 3 0 0 0
> 4194365 5 3 1 1 0
> 4194368 1 0 1 0 0
> 4194369 1 1 0 0 0
> 4194370 7 0 6 1 0
> 4194371 1 1 0 0 0
> 4194376 4 0 3 1 0
> 4194378 8 0 8 0 0
> 4194382 11 0 10 1 0
> 4194383 4 4 0 0 0
> 4194384 1 0 1 0 0
> 4194386 1 0 1 0 0
> 4194388 1 0 1 0 0
> 4194389 1 1 0 0 0
> 4194390 1 0 1 0 0
> 4194392 1 0 1 0 0
> 4194394 1 0 1 0 0
> 4194396 1 0 1 0 0
> 4194398 3 0 2 1 0
> 4194402 143 0 142 1 0
> 4194403 30 30 0 0 0
> 4194404 541 0 540 1 0
> 4194405 18 18 0 0 0
> 4194406 70091 0 70073 18 0
> 4194407 789 786 0 3 0
> 4194408 2985 0 2984 1 0
> 4194410 1 0 1 0 0
> 4194411 1 1 0 0 0
> 4194412 2 0 2 0 2
> 4194413 5 5 0 0 0
> 4194415 1 1 0 0 0
> 4194416 1 0 1 0 0
> 4194417 1 1 0 0 0
> 4194422 2 0 1 1 0
> 4194423 3 3 0 0 0
> 4194424 4 0 3 1 0
> 4194425 5 4 0 1 0
> 4194428 1 0 1 0 0
> 4194436 8 0 7 1 0
> 4194437 15 14 0 1 0
> 4194438 89 0 88 1 0
> 4194439 6 5 0 1 0
> 4194440 24 0 23 1 0
> 4194441 4 4 0 0 0
> 4194442 1 0 1 0 0
> 4194446 1122 0 1121 1 0
> 4194447 1744 1744 0 0 0
> 4194454 366 0 365 1 0
> 4194455 88 88 0 0 0
> 4194458 2162 0 2161 1 0
> 4194459 64 62 0 2 0
> 4194462 238 0 233 5 0
> 4194463 374 366 0 8 0
> 4194464 1674 0 1673 1 0
> 4194465 53 53 0 0 0
> 4194466 9 0 8 1 0
> 4194467 9 9 0 0 0
> 4194468 19 0 18 1 0
> 4194469 28 28 0 0 0
> 4194470 3 0 3 0 0
> 4194476 1 0 1 0 0
> 4194477 1 1 0 0 0
> 4194478 3 0 2 1 0
> 4
Hi Javier ,
Here my opinion and hints...
1) I agree with Art, where have to lot Lockreqs, but I don't consider this
impossible since you have 8 machines running your applications...
2) Active an OS monitoring tool, like sar to collect information for each 10
or 15 minutes... (check with "man sar" or "man sa0")
3) Check your IDS logs (MSGPATH) if have any unsual message.
4) Since IDS isn't block checkpoint any more, maybe a disk device problem are
masquerade. Check if your disks devices are OK (/var/log/messages, dmesg) .
5) Reread the machine notes, check if not forget set any kernel parameter.
The objective is get problems like this:
http://www-01.ibm.com/support/docview.wss?&uid=swg21204197&loc=en_US&cs=utf-8&la
ng=en
6) This suggestion, is a way to capture more information, create as your
desire... if possible, prepare a shell script to capture all information
possible for example: a "ps -lfe" command, vmstat , iostat, onstat -g act ,
onstat -g rea, onstat -g cpu. In this script run all this command in parallel(background & operator + "wait" command) , lopping each 30 seconds, saving all
outputs
Prepare this script to run with "root" and run with a "nice -n -20" (to get
high priority) .
When you detect the problem, execute the script : ssh root@machine "nice -n
-20 yourscript.sh"
7) Pay attention to your network card too, check the netstat -s statistics and
8) Active the sqltrace to identify if any heavy query are executed (Cartesian
sql) overloading your disk and network I/O.
Regards
Cesar
UPDATE After implementing your suggestions in the instance we have encountered a irregularity in the operation of the server and I need to ask for your help. At some point we got the suggestion that it could be a network problem, so I checked the network statistics of the server and foun this when I run the command "netstat --inet -n -e -c" Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State User Inode tcp 471 0 192.168.10.12:1526 192.168.11.25:1319 ESTABLISHED 0 0 tcp 496 0 192.168.10.12:1526 192.168.10.252:1119 ESTABLISHED 500 109189 tcp 0 0 192.168.10.12:1526 192.168.16.73:2389 ESTABLISHED 500 107466 tcp 0 0 192.168.10.12:1526 192.168.10.9:36629 ESTABLISHED 500 79234 tcp 473 0 192.168.10.12:1526 192.168.10.123:1124 ESTABLISHED 0 0 tcp 0 0 192.168.10.12:1526 192.168.10.7:60975 ESTABLISHED 500 109004 tcp 0 0 192.168.10.12:1526 192.168.10.7:60972 ESTABLISHED 500 109002 tcp 0 0 192.168.10.12:1526 192.168.10.7:60970 ESTABLISHED 500 108998 tcp 0 0 192.168.10.12:1526 192.168.10.7:60971 ESTABLISHED 500 109000 tcp 0 0 192.168.10.12:1526 192.168.10.7:60968 ESTABLISHED 500 108993 tcp 0 0 192.168.10.12:1526 192.168.10.7:60969 ESTABLISHED 500 108996 tcp 0 0 192.168.10.12:1526 192.168.16.69:2161 ESTABLISHED 500 53514 tcp 0 0 192.168.10.12:1526 192.168.10.7:60966 ESTABLISHED 500 108989 tcp 0 0 192.168.10.12:1526 192.168.10.9:42792 ESTABLISHED 500 109030 tcp 0 0 192.168.10.12:1526 192.168.10.7:60967 ESTABLISHED 500 108991 tcp 0 0 192.168.10.12:1526 192.168.10.9:42793 ESTABLISHED 500 109032 tcp 0 0 192.168.10.12:1526 192.168.10.7:60964 ESTABLISHED 500 108987 tcp 0 0 192.168.10.12:1526 192.168.10.9:42794 ESTABLISHED 500 109034 tcp 0 0 192.168.10.12:1526 192.168.10.9:42795 ESTABLISHED 500 109036 tcp 0 0 192.168.10.12:1526 192.168.10.7:33314 ESTABLISHED 500 14715 tcp 0 0 192.168.10.12:1526 192.168.10.9:42796 ESTABLISHED 500 109038 tcp 0 0 192.168.10.12:1526 192.168.10.7:33315 ESTABLISHED 500 14717 tcp 0 0 192.168.10.12:1526 192.168.10.9:42797 ESTABLISHED 500 109040 tcp 0 0 192.168.10.12:1526 192.168.10.9:42798 ESTABLISHED 500 109042 tcp 0 0 192.168.10.12:1526 192.168.10.9:37679 ESTABLISHED 500 87222 tcp 0 0 192.168.10.12:1526 192.168.10.9:42799 ESTABLISHED 500 109044 tcp 0 0 192.168.10.12:1526 192.168.10.9:42800 ESTABLISHED 500 109046 tcp 0 0 192.168.10.12:1526 192.168.10.9:42801 ESTABLISHED 500 109048 tcp 0 0 192.168.10.12:1526 192.168.10.9:42802 ESTABLISHED 500 109050 tcp 0 0 192.168.10.12:1526 192.168.10.7:60724 ESTABLISHED 500 108686 tcp 0 0 192.168.10.12:1526 192.168.10.7:60722 ESTABLISHED 500 108682 tcp 0 0 192.168.10.12:1526 192.168.10.7:60723 ESTABLISHED 500 108684 tcp 0 0 192.168.10.12:1526 192.168.10.7:60720 ESTABLISHED 500 108678 tcp 0 0 192.168.10.12:1526 192.168.10.7:60721 ESTABLISHED 500 108680 tcp 0 0 192.168.10.12:1526 192.168.10.9:36961 ESTABLISHED 500 84071 tcp 0 0 192.168.10.12:1526 192.168.10.7:37217 ESTABLISHED 500 52751 tcp 0 0 192.168.10.12:1526 192.168.10.9:42865 ESTABLISHED 500 109171 tcp 0 0 192.168.10.12:1526 192.168.10.9:42866 ESTABLISHED 500 109173 tcp 0 0 192.168.10.12:1526 192.168.10.9:42867 ESTABLISHED 500 109175 tcp 0 0 192.168.10.12:1526 192.168.10.9:42868 ESTABLISHED 500 109177 tcp 0 0 192.168.10.12:1526 192.168.10.9:42869 ESTABLISHED 500 109179 tcp 0 0 192.168.10.12:1526 192.168.10.7:60535 ESTABLISHED 500 108275 tcp 0 0 192.168.10.12:1526 192.168.16.65:1232 ESTABLISHED 500 73047 tcp 0 0 192.168.10.12:1526 192.168.10.9:38035 ESTABLISHED 500 88585 tcp 0 0 192.168.10.12:1526 192.168.10.9:36545 ESTABLISHED 500 78167 tcp 0 0 192.168.10.12:1526 192.168.10.7:40140 ESTABLISHED 500 71267 tcp 0 0 192.168.10.12:1526 192.168.10.9:36546 ESTABLISHED 500 78169 tcp 471 0 192.168.10.12:1526 192.168.11.24:2515 ESTABLISHED 0 0 tcp 0 0 192.168.10.12:1526 192.168.10.7:32966 ESTABLISHED 500 109122 tcp 0 0 192.168.10.12:1526 192.168.10.7:32964 ESTABLISHED 500 109118 tcp 0 0 192.168.10.12:1526 192.168.10.7:32965 ESTABLISHED 500 109120 tcp 0 0 192.168.10.12:1526 192.168.10.7:32962 ESTABLISHED 500 109114 tcp 0 0 192.168.10.12:1526 192.168.10.7:32963 ESTABLISHED 500 109116 tcp 0 0 192.168.10.12:1526 192.168.10.9:37857 ESTABLISHED 500 87999 tcp 0 0 192.168.10.12:1526 192.168.10.9:42721 ESTABLISHED 500 108838 tcp 0 0 192.168.10.12:1526 192.168.10.9:42722 ESTABLISHED 500 108840 tcp 0 0 192.168.10.12:1526 192.168.10.7:40173 ESTABLISHED 500 71282 tcp 0 0 192.168.10.12:1526 192.168.10.9:42723 ESTABLISHED 500 108842 tcp 0 0 192.168.10.12:1526 192.168.10.9:37092 ESTABLISHED 500 85352 tcp 0 0 192.168.10.12:1526 192.168.10.9:42724 ESTABLISHED 500 108844 tcp 0 0 192.168.10.12:1526 192.168.11.28:1528 ESTABLISHED 500 109185 tcp 0 0 192.168.10.12:1526 192.168.10.9:38394 ESTABLISHED 500 90688 tcp 0 0 192.168.10.12:1526 192.168.10.7:40181 ESTABLISHED 500 71316 tcp 0 0 192.168.10.12:1526 192.168.10.9:38395 ESTABLISHED 500 90690 tcp 0 0 192.168.10.12:1526 192.168.10.7:40178 ESTABLISHED 500 71302 tcp 0 0 192.168.10.12:1526 192.168.10.7:40179 ESTABLISHED 500 71310 tcp 0 As you can see, the second column in most of the conections is "0" but for the clients that suffer the slowdown some times we got data in the que for as much as 20 seconds, then suddenly if "flows" again and we get responce from the server. At some point in the day this situation gets so bad that the application responce goes from seconds (normal) to several minutes for each item that is consulted. Have you seen this behabior and if yes, do you know whow to diagnose and/or correct?? For your info, the client was using ODBC in Windows XP to conect to the DB, since yesterday we changed the conection to the native Informix .NET provider, the problem is not as severe as befor but still it presents. Thanks for any help. Javier Hernández
I got this just some minutes ago. Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State User Inode tcp 0 0 192.168.10.12:1526 192.168.11.25:1380 ESTABLISHED 500 110181 tcp 0 0 192.168.10.12:1526 192.168.16.73:2389 ESTABLISHED 500 107466 tcp 0 0 192.168.10.12:1526 192.168.16.73:2638 ESTABLISHED 500 110162 tcp 0 0 192.168.10.12:1526 192.168.10.9:36629 ESTABLISHED 500 79234 tcp 321 0 192.168.10.12:1526 192.168.16.73:2639 ESTABLISHED 0 0 tcp 471 0 192.168.10.12:1526 192.168.11.24:2823 ESTABLISHED 0 0 tcp 0 0 192.168.10.12:1526 192.168.16.69:2161 ESTABLISHED 500 53514 tcp 0 0 192.168.10.12:1526 192.168.10.7:33314 ESTABLISHED 500 14715 tcp 0 0 192.168.10.12:1526 192.168.10.7:33315 ESTABLISHED 500 14717 tcp 0 0 192.168.10.12:1526 192.168.10.9:37679 ESTABLISHED 500 87222 tcp 0 0 192.168.10.12:1526 192.168.10.9:43055 FIN_WAIT2 0 0 tcp 0 0 192.168.10.12:1526 192.168.10.9:43056 FIN_WAIT2 0 0 tcp 0 0 192.168.10.12:1526 192.168.10.9:43057 FIN_WAIT2 0 0 tcp 0 0 192.168.10.12:1526 192.168.10.9:43058 FIN_WAIT2 0 0 tcp 0 0 192.168.10.12:1526 192.168.10.9:43059 ESTABLISHED 500 109738 tcp 0 0 192.168.10.12:1526 192.168.10.9:43060 ESTABLISHED 500 109741 tcp 473 0 192.168.10.12:1526 192.168.11.28:1604 ESTABLISHED 0 0 tcp 0 0 192.168.10.12:1526 192.168.10.9:36961 ESTABLISHED 500 84071 tcp 0 0 192.168.10.12:1526 192.168.10.7:37217 ESTABLISHED 500 52751 tcp 0 0 192.168.10.12:1526 192.168.10.9:43124 ESTABLISHED 500 109877 tcp 0 0 192.168.10.12:1526 192.168.10.9:43125 ESTABLISHED 500 109879 tcp 0 0 192.168.10.12:1526 192.168.10.9:43126 ESTABLISHED 500 109881 tcp 0 0 192.168.10.12:1526 192.168.10.9:43127 ESTABLISHED 500 109883 tcp 0 0 192.168.10.12:1526 192.168.10.9:43128 ESTABLISHED 500 109885 tcp 0 0 192.168.10.12:1526 192.168.10.9:43129 ESTABLISHED 500 109887 tcp 0 0 192.168.10.12:1526 192.168.10.9:43130 ESTABLISHED 500 109890 tcp 0 0 192.168.10.12:1526 192.168.16.63:1188 FIN_WAIT2 500 109470 tcp 0 0 192.168.10.12:1526 192.168.10.7:33925 ESTABLISHED 500 109993 tcp 0 0 192.168.10.12:1526 192.168.10.7:33692 ESTABLISHED 500 109858 tcp 0 0 192.168.10.12:1526 192.168.10.7:33693 ESTABLISHED 500 109860 tcp 0 0 192.168.10.12:1526 192.168.10.9:38035 ESTABLISHED 500 88585 tcp 0 0 192.168.10.12:1526 192.168.10.7:33690 ESTABLISHED 500 109854 tcp 0 0 192.168.10.12:1526 192.168.10.7:33691 ESTABLISHED 500 109856 tcp 0 0 192.168.10.12:1526 192.168.10.7:33454 ESTABLISHED 500 109674 tcp 0 0 192.168.10.12:1526 192.168.10.7:33452 ESTABLISHED 500 109670 tcp 0 0 192.168.10.12:1526 192.168.10.7:33453 ESTABLISHED 500 109672 tcp 0 0 192.168.10.12:1526 192.168.10.7:33451 ESTABLISHED 500 109668 tcp 0 0 192.168.10.12:1526 192.168.16.63:1178 FIN_WAIT2 500 109399 tcp 0 0 192.168.10.12:1526 192.168.10.7:33209 ESTABLISHED 500 109381 tcp 0 0 192.168.10.12:1526 192.168.10.7:33977 ESTABLISHED 500 110040 tcp 0 0 192.168.10.12:1526 192.168.16.63:1174 ESTABLISHED 500 109355 tcp 0 0 192.168.10.12:1526 192.168.16.63:1175 ESTABLISHED 500 109357 tcp 0 0 192.168.10.12:1526 192.168.10.9:36545 ESTABLISHED 500 78167 tcp 0 0 192.168.10.12:1526 192.168.10.7:40140 ESTABLISHED 500 71267 tcp 0 0 192.168.10.12:1526 192.168.10.9:36546 ESTABLISHED 500 78169 tcp 468 0 192.168.10.12:1526 192.168.12.1:1222 ESTABLISHED 0 0 tcp 0 0 192.168.10.12:1526 192.168.10.9:43215 ESTABLISHED 500 110094 tcp 473 0 192.168.10.12:1526 192.168.16.63:1269 ESTABLISHED 0 0 tcp 0 0 192.168.10.12:1526 192.168.10.9:37857 ESTABLISHED 500 87999 tcp 0 0 192.168.10.12:1526 192.168.11.26:2289 ESTABLISHED 500 110123 tcp 0 0 192.168.10.12:1526 192.168.10.7:40173 ESTABLISHED 500 71282 tcp 0 0 192.168.10.12:1526 192.168.10.9:37092 ESTABLISHED 500 85352 tcp 321 0 192.168.10.12:1526 192.168.10.9:43236 ESTABLISHED 0 0 tcp 458 0 192.168.10.12:1526 192.168.11.26:2296 ESTABLISHED 0 0 tcp 460 0 192.168.10.12:1526 192.168.10.123:1156 ESTABLISHED 0 0 tcp 0 0 192.168.10.12:1526 192.168.10.9:38394 ESTABLISHED 500 90688 tcp 0 0 192.168.10.12:1526 192.168.10.7:40181 ESTABLISHED 500 71316 tcp 0 0 192.168.10.12:1526 192.168.10.9:38395 ESTABLISHED 500 90690 tcp 0 0 192.168.10.12:1526 192.168.10.7:40178 ESTABLISHED 500 71302 tcp 0 0 192.168.10.12:1526 192.168.10.7:40179 ESTABLISHED 500 71310 tcp 321 0 192.168.10.12:1526 192.168.10.7:34033 ESTABLISHED 500 110208 tcp 0 0 192.168.10.12:1526 192.168.10.7:40177 ESTABLISHED 500 71293 udp 0 0 192.168.10.12:33424 192.168.10.11:53 ESTABLISHED 0 110209 udp 0 0 192.168.10.12:33425 192.168.10.13:53 ESTABLISHED 0 110210 We have severe aplication slowdowns at this point.
Hi Javier, Based on your explanation, the network could be the problem ... but it is not easy to affirm. If the slowness is only on the hosts where has a value in Q-Send, you should check the network of this hosts. Talk with your network administrator. Now to solve this problem, I would monitor the hosts where is the application and would check the network settings such QOS drivers... Regards Cesar Posted By: JAVIER HERNáNDEZ <Send E-Mail> Date: Friday, 15 January 2010, at 10:53 a.m. In Response To: Random Server Slowdown "Redone" (JAVIER HERNáNDEZ) UPDATE After implementing your suggestions in the instance we have encountered a irregularity in the operation of the server and I need to ask for your help. At some point we got the suggestion that it could be a network problem, so I checked the network statistics of the server and foun this when I run the command "netstat --inet -n -e -c" Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State User Inode tcp 471 0 192.168.10.12:1526 192.168.11.25:1319 ESTABLISHED 0 0 tcp 496 0 192.168.10.12:1526 192.168.10.252:1119 ESTABLISHED 500 109189 tcp 0 0 192.168.10.12:1526 192.168.16.73:2389 ESTABLISHED 500 107466 tcp 0 0 192.168.10.12:1526 192.168.10.9:36629 ESTABLISHED 500 79234 tcp 473 0 192.168.10.12:1526 192.168.10.123:1124 ESTABLISHED 0 0 tcp 0 0 192.168.10.12:1526 192.168.10.7:60975 ESTABLISHED 500 109004 tcp 0 0 192.168.10.12:1526 192.168.10.7:60972 ESTABLISHED 500 109002 tcp 0 0 192.168.10.12:1526 192.168.10.7:60970 ESTABLISHED 500 108998 tcp 0 0 192.168.10.12:1526 192.168.10.7:60971 ESTABLISHED 500 109000 tcp 0 0 192.168.10.12:1526 192.168.10.7:60968 ESTABLISHED 500 108993 tcp 0 0 192.168.10.12:1526 192.168.10.7:60969 ESTABLISHED 500 108996 tcp 0 0 192.168.10.12:1526 192.168.16.69:2161 ESTABLISHED 500 53514 tcp 0 0 192.168.10.12:1526 192.168.10.7:60966 ESTABLISHED 500 108989 tcp 0 0 192.168.10.12:1526 192.168.10.9:42792 ESTABLISHED 500 109030 tcp 0 0 192.168.10.12:1526 192.168.10.7:60967 ESTABLISHED 500 108991 tcp 0 0 192.168.10.12:1526 192.168.10.9:42793 ESTABLISHED 500 109032 tcp 0 0 192.168.10.12:1526 192.168.10.7:60964 ESTABLISHED 500 108987 tcp 0 0 192.168.10.12:1526 192.168.10.9:42794 ESTABLISHED 500 109034 tcp 0 0 192.168.10.12:1526 192.168.10.9:42795 ESTABLISHED 500 109036 tcp 0 0 192.168.10.12:1526 192.168.10.7:33314 ESTABLISHED 500 14715 tcp 0 0 192.168.10.12:1526 192.168.10.9:42796 ESTABLISHED 500 109038 tcp 0 0 192.168.10.12:1526 192.168.10.7:33315 ESTABLISHED 500 14717 tcp 0 0 192.168.10.12:1526 192.168.10.9:42797 ESTABLISHED 500 109040 tcp 0 0 192.168.10.12:1526 192.168.10.9:42798 ESTABLISHED 500 109042 tcp 0 0 192.168.10.12:1526 192.168.10.9:37679 ESTABLISHED 500 87222 tcp 0 0 192.168.10.12:1526 192.168.10.9:42799 ESTABLISHED 500 109044 tcp 0 0 192.168.10.12:1526 192.168.10.9:42800 ESTABLISHED 500 109046 tcp 0 0 192.168.10.12:1526 192.168.10.9:42801 ESTABLISHED 500 109048 tcp 0 0 192.168.10.12:1526 192.168.10.9:42802 ESTABLISHED 500 109050 tcp 0 0 192.168.10.12:1526 192.168.10.7:60724 ESTABLISHED 500 108686 tcp 0 0 192.168.10.12:1526 192.168.10.7:60722 ESTABLISHED 500 108682 tcp 0 0 192.168.10.12:1526 192.168.10.7:60723 ESTABLISHED 500 108684 tcp 0 0 192.168.10.12:1526 192.168.10.7:60720 ESTABLISHED 500 108678 tcp 0 0 192.168.10.12:1526 192.168.10.7:60721 ESTABLISHED 500 108680 tcp 0 0 192.168.10.12:1526 192.168.10.9:36961 ESTABLISHED 500 84071 tcp 0 0 192.168.10.12:1526 192.168.10.7:37217 ESTABLISHED 500 52751 tcp 0 0 192.168.10.12:1526 192.168.10.9:42865 ESTABLISHED 500 109171 tcp 0 0 192.168.10.12:1526 192.168.10.9:42866 ESTABLISHED 500 109173 tcp 0 0 192.168.10.12:1526 192.168.10.9:42867 ESTABLISHED 500 109175 tcp 0 0 192.168.10.12:1526 192.168.10.9:42868 ESTABLISHED 500 109177 tcp 0 0 192.168.10.12:1526 192.168.10.9:42869 ESTABLISHED 500 109179 tcp 0 0 192.168.10.12:1526 192.168.10.7:60535 ESTABLISHED 500 108275 tcp 0 0 192.168.10.12:1526 192.168.16.65:1232 ESTABLISHED 500 73047 tcp 0 0 192.168.10.12:1526 192.168.10.9:38035 ESTABLISHED 500 88585 tcp 0 0 192.168.10.12:1526 192.168.10.9:36545 ESTABLISHED 500 78167 tcp 0 0 192.168.10.12:1526 192.168.10.7:40140 ESTABLISHED 500 71267 tcp 0 0 192.168.10.12:1526 192.168.10.9:36546 ESTABLISHED 500 78169 tcp 471 0 192.168.10.12:1526 192.168.11.24:2515 ESTABLISHED 0 0 tcp 0 0 192.168.10.12:1526 192.168.10.7:32966 ESTABLISHED 500 109122 tcp 0 0 192.168.10.12:1526 192.168.10.7:32964 ESTABLISHED 500 109118 tcp 0 0 192.168.10.12:1526 192.168.10.7:32965 ESTABLISHED 500 109120 tcp 0 0 192.168.10.12:1526 192.168.10.7:32962 ESTABLISHED 500 109114 tcp 0 0 192.168.10.12:1526 192.168.10.7:32963 ESTABLISHED 500 109116 tcp 0 0 192.168.10.12:1526 192.168.10.9:37857 ESTABLISHED 500 87999 tcp 0 0 192.168.10.12:1526 192.168.10.9:42721 ESTABLISHED 500 108838 tcp 0 0 192.168.10.12:1526 192.168.10.9:42722 ESTABLISHED 500 108840 tcp 0 0 192.168.10.12:1526 192.168.10.7:40173 ESTABLISHED 500 71282 tcp 0 0 192.168.10.12:1526 192.168.10.9:42723 ESTABLISHED 500 108842 tcp 0 0 192.168.10.12:1526 192.168.10.9:37092 ESTABLISHED 500 85352 tcp 0 0 192.168.10.12:1526 192.168.10.9:42724 ESTABLISHED 500 108844 tcp 0 0 192.168.10.12:1526 192.168.11.28:1528 ESTABLISHED 500 109185 tcp 0 0 192.168.10.12:1526 192.168.10.9:38394 ESTABLISHED 500 90688 tcp 0 0 192.168.10.12:1526 192.168.10.7:40181 ESTABLISHED 500 71316 tcp 0 0 192.168.10.12:1526 192.168.10.9:38395 ESTABLISHED 500 90690 tcp 0 0 192.168.10.12:1526 192.168.10.7:40178 ESTABLISHED 500 71302 tcp 0 0 192.168.10.12:1526 192.168.10.7:40179 ESTABLISHED 500 71310 tcp 0 As you can see, the second column in most of the conections is "0" but for the clients that suffer the slowdown some times we got data in the que for as much as 20 seconds, then suddenly if "flows" again and we get responce from the server. At some point in the day this situation gets so bad that the application responce goes from seconds (normal) to several minutes for each item that is consulted. Have you seen this behabior and if yes, do you know whow to diagnose and/or correct?? For your info, the client was using ODBC in Windows XP to conect to the DB, since yesterday we changed the conection to the native Informix .NET provider, the problem is not as severe as befor but still it presents. Thanks for any help. Javier Hernández
It TCP protocol sounds like the listeners are busy, You might try adding one or two more. Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) IIUG Board of Directors (art@iiug.org) See you at the 2010 IIUG Informix Conference April 25-28, 2010 Overland Park (Kansas City), KS www.iiug.org/conf Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. 2010/1/15 JAVIER HERNáNDEZ <javier_hdzt@yahoo.com.mx> > UPDATE > > After implementing your suggestions in the instance we have encountered a > irregularity in the operation of the server and I need to ask for your > help. > > At some point we got the suggestion that it could be a network problem, so > I > checked the network statistics of the server and foun this when I run the > command "netstat --inet -n -e -c" > > Active Internet connections (w/o servers) > Proto Recv-Q Send-Q Local Address Foreign Address State User Inode > tcp 471 0 192.168.10.12:1526 192.168.11.25:1319 ESTABLISHED 0 0 > tcp 496 0 192.168.10.12:1526 192.168.10.252:1119 ESTABLISHED 500 109189 > tcp 0 0 192.168.10.12:1526 192.168.16.73:2389 ESTABLISHED 500 107466 > tcp 0 0 192.168.10.12:1526 192.168.10.9:36629 ESTABLISHED 500 79234 > tcp 473 0 192.168.10.12:1526 192.168.10.123:1124 ESTABLISHED 0 0 > tcp 0 0 192.168.10.12:1526 192.168.10.7:60975 ESTABLISHED 500 109004 > tcp 0 0 192.168.10.12:1526 192.168.10.7:60972 ESTABLISHED 500 109002 > tcp 0 0 192.168.10.12:1526 192.168.10.7:60970 ESTABLISHED 500 108998 > tcp 0 0 192.168.10.12:1526 192.168.10.7:60971 ESTABLISHED 500 109000 > tcp 0 0 192.168.10.12:1526 192.168.10.7:60968 ESTABLISHED 500 108993 > tcp 0 0 192.168.10.12:1526 192.168.10.7:60969 ESTABLISHED 500 108996 > tcp 0 0 192.168.10.12:1526 192.168.16.69:2161 ESTABLISHED 500 53514 > tcp 0 0 192.168.10.12:1526 192.168.10.7:60966 ESTABLISHED 500 108989 > tcp 0 0 192.168.10.12:1526 192.168.10.9:42792 ESTABLISHED 500 109030 > tcp 0 0 192.168.10.12:1526 192.168.10.7:60967 ESTABLISHED 500 108991 > tcp 0 0 192.168.10.12:1526 192.168.10.9:42793 ESTABLISHED 500 109032 > tcp 0 0 192.168.10.12:1526 192.168.10.7:60964 ESTABLISHED 500 108987 > tcp 0 0 192.168.10.12:1526 192.168.10.9:42794 ESTABLISHED 500 109034 > tcp 0 0 192.168.10.12:1526 192.168.10.9:42795 ESTABLISHED 500 109036 > tcp 0 0 192.168.10.12:1526 192.168.10.7:33314 ESTABLISHED 500 14715 > tcp 0 0 192.168.10.12:1526 192.168.10.9:42796 ESTABLISHED 500 109038 > tcp 0 0 192.168.10.12:1526 192.168.10.7:33315 ESTABLISHED 500 14717 > tcp 0 0 192.168.10.12:1526 192.168.10.9:42797 ESTABLISHED 500 109040 > tcp 0 0 192.168.10.12:1526 192.168.10.9:42798 ESTABLISHED 500 109042 > tcp 0 0 192.168.10.12:1526 192.168.10.9:37679 ESTABLISHED 500 87222 > tcp 0 0 192.168.10.12:1526 192.168.10.9:42799 ESTABLISHED 500 109044 > tcp 0 0 192.168.10.12:1526 192.168.10.9:42800 ESTABLISHED 500 109046 > tcp 0 0 192.168.10.12:1526 192.168.10.9:42801 ESTABLISHED 500 109048 > tcp 0 0 192.168.10.12:1526 192.168.10.9:42802 ESTABLISHED 500 109050 > tcp 0 0 192.168.10.12:1526 192.168.10.7:60724 ESTABLISHED 500 108686 > tcp 0 0 192.168.10.12:1526 192.168.10.7:60722 ESTABLISHED 500 108682 > tcp 0 0 192.168.10.12:1526 192.168.10.7:60723 ESTABLISHED 500 108684 > tcp 0 0 192.168.10.12:1526 192.168.10.7:60720 ESTABLISHED 500 108678 > tcp 0 0 192.168.10.12:1526 192.168.10.7:60721 ESTABLISHED 500 108680 > tcp 0 0 192.168.10.12:1526 192.168.10.9:36961 ESTABLISHED 500 84071 > tcp 0 0 192.168.10.12:1526 192.168.10.7:37217 ESTABLISHED 500 52751 > tcp 0 0 192.168.10.12:1526 192.168.10.9:42865 ESTABLISHED 500 109171 > tcp 0 0 192.168.10.12:1526 192.168.10.9:42866 ESTABLISHED 500 109173 > tcp 0 0 192.168.10.12:1526 192.168.10.9:42867 ESTABLISHED 500 109175 > tcp 0 0 192.168.10.12:1526 192.168.10.9:42868 ESTABLISHED 500 109177 > tcp 0 0 192.168.10.12:1526 192.168.10.9:42869 ESTABLISHED 500 109179 > tcp 0 0 192.168.10.12:1526 192.168.10.7:60535 ESTABLISHED 500 108275 > tcp 0 0 192.168.10.12:1526 192.168.16.65:1232 ESTABLISHED 500 73047 > tcp 0 0 192.168.10.12:1526 192.168.10.9:38035 ESTABLISHED 500 88585 > tcp 0 0 192.168.10.12:1526 192.168.10.9:36545 ESTABLISHED 500 78167 > tcp 0 0 192.168.10.12:1526 192.168.10.7:40140 ESTABLISHED 500 71267 > tcp 0 0 192.168.10.12:1526 192.168.10.9:36546 ESTABLISHED 500 78169 > tcp 471 0 192.168.10.12:1526 192.168.11.24:2515 ESTABLISHED 0 0 > tcp 0 0 192.168.10.12:1526 192.168.10.7:32966 ESTABLISHED 500 109122 > tcp 0 0 192.168.10.12:1526 192.168.10.7:32964 ESTABLISHED 500 109118 > tcp 0 0 192.168.10.12:1526 192.168.10.7:32965 ESTABLISHED 500 109120 > tcp 0 0 192.168.10.12:1526 192.168.10.7:32962 ESTABLISHED 500 109114 > tcp 0 0 192.168.10.12:1526 192.168.10.7:32963 ESTABLISHED 500 109116 > tcp 0 0 192.168.10.12:1526 192.168.10.9:37857 ESTABLISHED 500 87999 > tcp 0 0 192.168.10.12:1526 192.168.10.9:42721 ESTABLISHED 500 108838 > tcp 0 0 192.168.10.12:1526 192.168.10.9:42722 ESTABLISHED 500 108840 > tcp 0 0 192.168.10.12:1526 192.168.10.7:40173 ESTABLISHED 500 71282 > tcp 0 0 192.168.10.12:1526 192.168.10.9:42723 ESTABLISHED 500 108842 > tcp 0 0 192.168.10.12:1526 192.168.10.9:37092 ESTABLISHED 500 85352 > tcp 0 0 192.168.10.12:1526 192.168.10.9:42724 ESTABLISHED 500 108844 > tcp 0 0 192.168.10.12:1526 192.168.11.28:1528 ESTABLISHED 500 109185 > tcp 0 0 192.168.10.12:1526 192.168.10.9:38394 ESTABLISHED 500 90688 > tcp 0 0 192.168.10.12:1526 192.168.10.7:40181 ESTABLISHED 500 71316 > tcp 0 0 192.168.10.12:1526 192.168.10.9:38395 ESTABLISHED 500 90690 > tcp 0 0 192.168.10.12:1526 192.168.10.7:40178 ESTABLISHED 500 71302 > tcp 0 0 192.168.10.12:1526 192.168.10.7:40179 ESTABLISHED 500 71310 > tcp 0 > > As you can see, the second column in most of the conections is "0" but for > the > clients that suffer the slowdown some times we got data in the que for as > much > as 20 seconds, then suddenly if "flows" again and we get responce from the > server. > > At some point in the day this situation gets so bad that the application > responce goes from seconds (normal) to several minutes for each item that > is > consulted. > > Have you seen this behabior and if yes, do you know whow to diagnose and/or > correct?? > > For your info, the client was using ODBC in Windows XP to conect to the DB, > since yesterday we changed the conection to the native Informix .NET > provider, > the problem is not as severe as befor but still it presents. > > Thanks for any help. > > Javier Hernández > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --00151747688e23af0e047d378db7
I'm glad to inform you that the problem is resolved. Art, Obnoxio, Mike and any that has interest in the problem. About 2 weeks ago we got a strike of luck, the company Win2K3 Domain controler has an issue and the Telecom departament has to reboot the server and the core switch, as improbable as it seams, the problem went away. In order to find if this issue was related to our failure we keep monitoring the Informix instance the Domain controler and the switch. Well, last friday it presented again the problem with the domain controler, specifically with the DNS part and again we got a slow down that was resolved restarting the DNS service in the domain controller. The courius thing is that we are not using host names to conect to Informix, we make the conection directly by IP. Telecom is currently investigating the matter but Informix is now not getting the blame! Jut to keep you informed. Javier Hernández Consultor Soluciones Integrales AMR, S.A. de C.V.
JAVIER HERNáNDEZ wrote: > Telecom is currently investigating the matter but Informix is now not getting > the blame! That makes a change! :o) -- Cheers, Obnoxio The Clown http://obotheclown.blogspot.com I will now proceed to pleasure myself with this fish. -- This message has been scanned for viruses and dangerous content by OpenProtect(http://www.openprotect.com), and is believed to be clean.
Quoting JAVIER HERNXNDEZ <javier_hdzt@yahoo.com.mx>: > In order to find if this issue was related to our failure we keep monitoring > the Informix instance the Domain controler and the switch. > Well, last friday it presented again the problem with the domain controler, > specifically with the DNS part and again we got a slow down that was resolved > restarting the DNS service in the domain controller. > The courius thing is that we are not using host names to conect to Informix, > we make the conection directly by IP. Someone, either the client or the server, is probably doing a reverse look-up on the IP. So you have to wait for that to time-out. I've never seen this with Informix, but over the years I've seen it with many services - basically you are using DNS even when you think you aren't or try not to.