error -27001
Posted in 2006
Hi All,
IDS 10.0.UC3
ELAS3.0, Kernel 2.4.21-4.ELsmp, RAID10
we have got some issues where connections are being dropped out and
online records -27001 error. Database server and application server
(4js) are on the same host. client application uses GDC 1.31.1e and
connecting DB using ODBC (3.82.0000 2.81.TC3 client side). When -27001
occurs, no connection allowed during that time and application
just hangs for about 3-4 mins and comes back to normal after that. We
conatcted application provider and got a response back that they are
testing GDC 1.32.1e which might fix this issue. I am not sure at this
stage whether this read error is at client side or database side,
however, I am suspecting some issues with database side which may or
may not a contributing factor. Here are some snapshots:
onstat -m
13:04:54 listener-thread: err = -27001: oserr = 0: errstr = : Readerror occurred during connection attempt.
onstat -p
IBM Informix Dynamic Server Version 10.00.UC3 -- On-Line -- Up 10
days 05:48:56 -- 1767052 Kbytes
Profile
dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits
%cached
350141411 812650455 72386225639 99.52 5767661 51497061
30441602 90.42
isamtot open start read write rewrite
delete commit rollbk
58499395809 361322842 1054381511 54636023977 41832039 1749311
1049764 1527687 278
gp_read gp_write gp_rewrt gp_del gp_alloc gp_free
gp_curs
31 0 0 0 0 0 52
ovlock ovuserthread ovbuff usercpu syscpu numckpts flushes
0 0 0 286317.00 43055.85 617 1972
bufwaits lokwaits lockreqs deadlks dltouts ckpwaits
compress seqscans
15142198 20 45368091006 0 0 368
1158865 8541139
ixda-RA idx-RA da-RA RA-pgsused lchwaits
42083593 14215347 188961624 241266788 4988764
onstat -g spi
IBM Informix Dynamic Server Version 10.00.UC3 -- On-Line -- Up 10
days 05:50:01 -- 1767052 Kbytes
Spin locks with waits:
Num Waits Num Loops Avg Loop/Wait Name
12 3756897 313074.75 mtcb vproc_sync_lock
232041 89433947 385.42 mtcb sleeping_lock
2 98 49.00 mtcb tcb_queue_lock
17 249360 14668.24 mtcb tcb_queue_lock
69 122322 1772.78 mtcb mutex_list_lock
99 99057 1000.58 mtcb notify_lock
102 22165 217.30 class cl_lock, class = 0
337731 47315091 140.10 vproc vp_lock, id = 1
243903 36604587 150.08 vproc vp_lock, id = 3
1725787 257670671 149.31 vproc vp_lock, id = 4
.....
.....
onstat -g glo
IBM Informix Dynamic Server Version 10.00.UC3 -- On-Line -- Up 10
days 05:51:05 -- 1767052 Kbytes
MT global info:
sessions threads vps lngspins
293 461 31 2671
sched calls thread switches yield 0 yield n yield
forever
total: 2499649188 1174857135 1416527643 4898397
268246790
per sec: 2864 1201 1841 8 53
Virtual processor summary:
class vps usercpu syscpu total
cpu 3 281845.84 3847.27 285693.11
aio 22 3069.21 15118.16 18187.37
lio 1 18.39 95.39 113.78
pio 1 1.78 22.40 24.18
adm 1 10.56 109.81 120.37
soc 2 1327.41 23830.88 25158.29
msc 1 82.00 49.12 131.12
total 31 286355.19 43073.03 329428.22
Individual virtual processors:
vp pid class usercpu syscpu total
1 6103 cpu 51524.72 1693.39 53218.11
2 6104 adm 10.56 109.81 120.37
3 6105 cpu 69208.90 1276.31 70485.21
4 6106 cpu 161112.22 877.57 161989.79
.....
.....
onstat -g ntd
IBM Informix Dynamic Server Version 10.00.UC3 -- On-Line -- Up 10
days 05:53:47 -- 1767052 Kbytes
global network information:
#netscb connects read write q-free q-limits q-exceed
alloc/max
303/ 391 162361 190897185 192280971 88/ 377 379/ 10 0/ 0
382/ 382
Client Type Calls Accepted Rejected Read Write
sqlexec yes 162361 18 174875499 176469418
srvinfx yes 0 0 0 0
onspace yes 0 0 0 0
onlog yes 0 0 0 0
onparam yes 0 0 0 0
oncheck yes 0 0 15435546 15387710
onload yes 0 0 0 0
onunload yes 0 0 0 0
onmonitor yes 0 0 0 0
dr_accept yes 0 0 0 0
cdraccept no 0 0 0 0
ontape yes 0 0 423787 423823
srvstat yes 0 0 10 20
asfecho yes 0 0 0 0
listener yes 0 0 162343 0
crsamexec yes 0 0 0 0
safe yes 0 0 0 0
onutil yes 0 0 0 0
Totals 162361 18 190897185 192280971
onstat -c
.....
.....
NETTYPE soctcp,2,400,NET # Configure poll thread(s) for nettype
NETTYPE ipcshm,2,400,CPU # Configure poll thread(s) for nettype
.....
DEADLOCK_TIMEOUT 60 # Max time to wait of lock indistributed env
.....
RA_PAGES 128 # Number of pages to attempt to readahead
RA_THRESHOLD 100 # Number of pages left before nextgroup
.....
LISTEN_TIMEOUT 25 # Timeout (in Seconds)
.....
.....
BUFFERPOOL
size=2K,buffers=400000,lrus=128,lru_min_dirty=1.000000,lru_max_dirty=3.000000
iostat
avg-cpu: %user %nice %sys %iowait %idle
4.86 0.00 0.97 3.94 90.23
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 11.80 26.55 70.22 25877144 68431806
sda1 0.00 0.00 0.00 560 198
sda2 0.93 3.26 4.45 3176226 4336256
sda3 1.49 1.01 11.05 982554 10765328
sda4 0.00 0.00 0.00 4 0
sda5 1.04 1.11 7.42 1077978 7230352
sda6 1.62 4.82 9.35 4700088 9113072
sda7 4.73 2.29 35.92 2233114 35006264
sda8 0.21 0.07 1.77 63450 1724904@