why so many informix threads on idle systems ?

Hi Folks Hoping someone can help here there may be something onvious I am missing. WE have two test instances that are a replicated pair. Most of the time they are idle yet when I do onstat -u i see 25 and 26 informix threads in use. as below. why are there so many threads running. I assume the more threads the more memory used- the systems are short of memory onconfig performance parameters Multiprocessor Machine [N] LRU Max Dirty [ 15] Num Procs to Affinity [ 0] LRU Min Dirty [5.0] Proc num to start with [ 0] Checkpoint Interval [ 600] Num of Read Ahead Pages [ 32] CPU VPs [ 1] Read Ahead Threshold [ 16] AIO VPs [ 2] Single CPU VP [Y] NETTYPE settings: Use OS Time [N] Protocol Threads Users VP-class Disable Priority Aging [N] [soctcp] [ 1] [ 5] [NET] Off-Line Recovery Threads [ 5] [ ] [ ] [ ] [ ] On-Line Recovery Threads [ 1] [ ] [ ] [ ] [ ] Num of LRUS queues [ 8] [ ] [ ] [ ] [ ] flags sessid user tty wait tout locks nreads nwr ---P--D 1 informix - 0 0 0 38 24 ---P--F 0 informix - 0 0 0 0 184 ---P--F 0 informix - 0 0 0 0 18 ---P--- 7 informix - 0 0 0 0 0 ---P--B 8 informix - 0 0 0 0 0 Y--P--- 32 informix - c0000000b05faf90 0 1 3 0 ---P--D 11 informix - 0 0 0 0 0 ---P--- 23 informix - 0 0 2 68 7 ---P--- 17 informix - 0 0 0 0 0 Y--P--D 16 informix - c0000000a9ebd800 0 0 0 0 ---P--- 19 informix - 0 0 1 0 0 Y--P--- 20 informix - c0000000ae8dd2a8 0 5 0 0 ---P--- 24 informix - 0 0 2 89 144 ---P--- 22 informix - 0 0 1 164 0 ---P--- 25 informix - 0 0 0 3 0 ---P--- 26 informix - 0 0 0 0 0 ---P--- 27 informix - 0 0 0 0 0 ---P--- 28 informix - 0 0 0 0 0 ---P--- 29 informix - 0 0 1 0 0 ---P--- 30 informix - 0 0 0 613 0 ---P--- 31 informix - 0 0 1 8 0 ---P--- 33 informix - 0 0 0 0 1 Y--P--- 34 informix - c0000000ae85c570 0 0 0 0 Y--P--- 35 informix - c0000000ae85c570 0 0 0 0 Y--P--- 36 informix - c0000000ae85c440 0 1 0 0 otal, 27 maximum concurrent

Please post the output of onstat -g ath. It's better than onstat -u wh= en looking at threads. M.P. = From: "KARL OLIVER" <karl.oliver@maf.govt.nz> = = To: ids@iiug.org = = Date: 04/06/2010 06:32 PM = = Subject: why so many informix threads on idle systems ? [19509] = = Sent by: ids-bounces@iiug.org = = Hi Folks Hoping someone can help here there may be something onvious I am missin= g. WE have two test instances that are a replicated pair. Most of the time= they are idle yet when I do onstat -u i see 25 and 26 informix threads in us= e. as below. why are there so many threads running. I assume the more threads= the more memory used- the systems are short of memory onconfig performance parameters Multiprocessor Machine [N] LRU Max Dirty [ 15] Num Procs to Affinity [ 0] LRU Min Dirty [5.0] Proc num to start with [ 0] Checkpoint Interval [ 600] Num of Read Ahead Pages [ 32] CPU VPs [ 1] Read Ahead Threshold [ 16] AIO VPs [ 2] Single CPU VP [Y] NETTYPE settings: Use OS Time [N] Protocol Threads Users VP-class Disable Priority Aging [N] [soctcp] [ 1] [ 5] [NET] Off-Line Recovery Threads [ 5] [ ] [ ] [ ] [ ] On-Line Recovery Threads [ 1] [ ] [ ] [ ] [ ] Num of LRUS queues [ 8] [ ] [ ] [ ] [ ] flags sessid user tty wait tout locks nreads nwr ---P--D 1 informix - 0 0 0 38 24 ---P--F 0 informix - 0 0 0 0 184 ---P--F 0 informix - 0 0 0 0 18 ---P--- 7 informix - 0 0 0 0 0 ---P--B 8 informix - 0 0 0 0 0 Y--P--- 32 informix - c0000000b05faf90 0 1 3 0 ---P--D 11 informix - 0 0 0 0 0 ---P--- 23 informix - 0 0 2 68 7 ---P--- 17 informix - 0 0 0 0 0 Y--P--D 16 informix - c0000000a9ebd800 0 0 0 0 ---P--- 19 informix - 0 0 1 0 0 Y--P--- 20 informix - c0000000ae8dd2a8 0 5 0 0 ---P--- 24 informix - 0 0 2 89 144 ---P--- 22 informix - 0 0 1 164 0 ---P--- 25 informix - 0 0 0 3 0 ---P--- 26 informix - 0 0 0 0 0 ---P--- 27 informix - 0 0 0 0 0 ---P--- 28 informix - 0 0 0 0 0 ---P--- 29 informix - 0 0 1 0 0 ---P--- 30 informix - 0 0 0 613 0 ---P--- 31 informix - 0 0 1 8 0 ---P--- 33 informix - 0 0 0 0 1 Y--P--- 34 informix - c0000000ae85c570 0 0 0 0 Y--P--- 35 informix - c0000000ae85c570 0 0 0 0 Y--P--- 36 informix - c0000000ae85c440 0 1 0 0 otal, 27 maximum concurrent ***********************************************************************= ******** Forum Note: Use "Reply" to post a response in the discussion forum. =

here is the onstat -g ath BM Informix Dynamic Server Version 11.10.FC3 -- On-Line -- Up 01:09:50 -- 123912 Kbytes Threads: tid tcb rstcb prty status vp-class name 2 c0000000ae5ca9a8 0 1 IO Idle 3lio* lio vp 0 3 c0000000ae5ebc88 0 1 IO Idle 4pio* pio vp 0 4 c0000000ae60d450 0 1 IO Idle 5aio* aio vp 0 5 c0000000ae60cc88 0 1 IO Idle 6msc* msc vp 0 6 c0000000ae62ec88 0 1 IO Idle 7adt* adt vp 0 7 c0000000ae681450 0 1 IO Idle 8aio* aio vp 1 8 c0000000ae65fc88 c0000000adafc028 3 sleeping secs: 1 1cpu main_loop() 9 c0000000ae6c58a0 0 1 running 9soc* soctcppoll 10 c0000000ae6e2c60 0 2 sleeping forever 1cpu* soctcplst 11 c0000000ae70c418 0 2 sleeping forever 1cpu* soctcplst 12 c0000000ae70ccd8 c0000000adafc840 1 sleeping secs: 1 1cpu flush_sub(0) 13 c0000000ae751028 c0000000adafd058 1 sleeping secs: 1 1cpu flush_sub(1) 14 c0000000ae771220 c0000000adafd870 2 sleeping secs: 1 1cpu aslogflush 15 c0000000ae815028 c0000000adafe088 1 sleeping secs: 88 1cpu btscanner_0 26 c0000000aea3d9b8 c0000000adaff8d0 3 sleeping secs: 1 1cpu* onmode_mon 31 c0000000ae87c220 c0000000adb01118 1 cond wait bp_cond 1cpu bf_priosweep() 32 c0000000aea920d0 c0000000adb00900 1 sleeping secs: 30 1cpu CDRSchedMgr 34 c0000000aead9028 c0000000adb02148 1 sleeping secs: 92 1cpu CDRDTCleaner 35 c0000000aead9518 c0000000adb02960 1 cond wait CDRCparse 1cpu CDRCparse 37 c0000000aeb64028 c0000000adb03990 1 sleeping secs: 165 1cpu* dbScheduler 38 c0000000aed3a4c0 c0000000adb000e8 1 sleeping forever 1cpu* dbWorker1 39 c0000000aed4a028 c0000000adb03178 1 sleeping forever 1cpu* dbWorker2 40 c0000000aec04b60 c0000000adb041a8 3 sleeping secs: 1 1cpu CDRGfan 41 c0000000aec4e760 c0000000adb049c0 3 sleeping secs: 2 1cpu CDRGeval0 42 c0000000aec4eca0 c0000000adb051d8 3 sleeping secs: 2 1cpu CDRGeval1 43 c0000000aec91220 c0000000adb059f0 3 sleeping secs: 2 1cpu CDRGeval2 44 c0000000aeb38220 c0000000adb06208 1 sleeping secs: 30 1cpu CDRNsT19 45 c0000000aeb38b78 c0000000adb06a20 2 sleeping secs: 1 1cpu ddr_snoopy 46 c0000000ae85c028 c0000000adb07238 1 sleeping secs: 30 1cpu CDRRTBCleaner 47 c0000000aea3dc08 c0000000adaff0b8 1 cond wait netnorm 1cpu CDRNr19 48 c0000000ae85c7d0 c0000000adb07a50 1 sleeping secs: 1 1cpu CDRM_Monitor 49 c0000000ae85ca20 c0000000adb08268 1 cond wait CDRAckslp 1cpu CDRACK_0 50 c0000000ae85cc70 c0000000adb08a80 1 cond wait CDRAckslp 1cpu CDRACK_1 51 c0000000ae8a0028 c0000000adb09298 1 cond wait CDRDssleep 1cpu CDRD_1

Each of the threads have a special purpose. Since the system is idle, there are no 'sqlexec' threads. Those are the 'user' threads and are responsible for the actual query. The ER threads are event driven and = are driven when other things happen, possibly from a log being written or m= aybe in response to having received a replicated transaction. Since you are= using ER, I'll give a break down on what each of those threads are responsible for and what causes them to go into an active state. Notic= e that the threads in this example are all in a waiting/sleeping state. Just to explain a couple of other things. There are two main waiting states for the threads. The 'sleeping' state and the cond wait state. = The sleeping state is normally used when we want to always wake up every so= often just to see what's going on. If a thread is sleeping, that doesn= 't mean that some other thread can't wake it up immediately. The conditio= n wait is a bit different because there no 'timeout' and thus the thread = will only be activated when some other thread activates it. We could respawn these threads, but the cost of the thread setup is a b= it high because of all of the internal querying against the queues and oth= er tables within the syscdr database. That's the reason that we just keep= the threads but place them in a sleep/waiting state when they don't have wo= rk to do. OK - here goes. |------------------------+------------------------+--------------------= ----| |Thread |Purpose |When activated = | |------------------------+------------------------+--------------------= ----| |CDRD_### |ER apply thread |When a replicated = | | | |transaction is recei= ved.| |------------------------+------------------------+--------------------= ----| |CDRACK_### |ER ACK threads - process|When an ACK is recei= ved | | |ACKS from transactions | = | | |applied on other servers| = | | |within the ER domain | = | |------------------------+------------------------+--------------------= ----| |CDRM_Monitor |ER Monitor thread - |One a second = | | |monitors and adjusts ER | = | | |apply parallelism | = | |------------------------+------------------------+--------------------= ----| |CDRNr19 |ER Network Receive |When a new message i= s | | |Thread connected to node|received from ER nod= e 19| | |19 | = | |------------------------+------------------------+--------------------= ----| |CDRRTBCleaner |Cleans up ER objects no |Every few minutes or= so | | |longer in use | = | |------------------------+------------------------+--------------------= ----| |ddr_snoopy |Receives the log records|When a log buffer is= | | | |flushed = | |------------------------+------------------------+--------------------= ----| |CDRNsT19 |ER Network Send Thread |When a message needs= to | | |connected to node 19 |be sent to node 19 = | |------------------------+------------------------+--------------------= ----| |CDRGeval### |Grouper Evaluator |As new log records a= re | | |threads - evaluates and |received = | | |regroups log records for| = | | |replication | = | |------------------------+------------------------+--------------------= ----| |CDRGfan |Grouper Fanout thread - |As new log records a= re | | |acts as a distributor |received = | | |between the snoopy and | = | | |evaluator threads. Also | = | | |has special logic for | = | | |begin work/commit | = | | |work/rollback... | = | |------------------------+------------------------+--------------------= ----| |CDRCparse |ER statement cache |Only as a replicate = is | | |manager |created/removed = | |------------------------+------------------------+--------------------= ----| |CDRDTCleaner |Delete Table Cleaner - |Every 5 minutes or s= o | | |prunes the delete shadow| = | | |tables | = | |------------------------+------------------------+--------------------= ----| |CDRSchedMgr |ER Scheduler - fires |As needed = | | |retry connection | = | | |attempts, timed based | = | | |replication, etc | = | |------------------------+------------------------+--------------------= ----| = From: "KARL OLIVER" <karl.oliver@maf.govt.nz> = = To: ids@iiug.org = = Date: 04/06/2010 06:59 PM = = Subject: Re: why so many informix threads on idle systems ? [1951= 1] = Sent by: ids-bounces@iiug.org = = here is the onstat -g ath BM Informix Dynamic Server Version 11.10.FC3 -- On-Line -- Up 01:09:50 = -- 123912 Kbytes Threads: tid tcb rstcb prty status vp-class name 2 c0000000ae5ca9a8 0 1 IO Idle 3lio* lio vp 0 3 c0000000ae5ebc88 0 1 IO Idle 4pio* pio vp 0 4 c0000000ae60d450 0 1 IO Idle 5aio* aio vp 0 5 c0000000ae60cc88 0 1 IO Idle 6msc* msc vp 0 6 c0000000ae62ec88 0 1 IO Idle 7adt* adt vp 0 7 c0000000ae681450 0 1 IO Idle 8aio* aio vp 1 8 c0000000ae65fc88 c0000000adafc028 3 sleeping secs: 1 1cpu main_loop()= 9 c0000000ae6c58a0 0 1 running 9soc* soctcppoll 10 c0000000ae6e2c60 0 2 sleeping forever 1cpu* soctcplst 11 c0000000ae70c418 0 2 sleeping forever 1cpu* soctcplst 12 c0000000ae70ccd8 c0000000adafc840 1 sleeping secs: 1 1cpu flush_sub(= 0) 13 c0000000ae751028 c0000000adafd058 1 sleeping secs: 1 1cpu flush_sub(= 1) 14 c0000000ae771220 c0000000adafd870 2 sleeping secs: 1 1cpu aslogflush= 15 c0000000ae815028 c0000000adafe088 1 sleeping secs: 88 1cpu btscanner= _0 26 c0000000aea3d9b8 c0000000adaff8d0 3 sleeping secs: 1 1cpu* onmode_mo= n 31 c0000000ae87c220 c0000000adb01118 1 cond wait bp_cond 1cpu bf_priosw= eep () 32 c0000000aea920d0 c0000000adb00900 1 sleeping secs: 30 1cpu CDRSchedM= gr 34 c0000000aead9028 c0000000adb02148 1 sleeping secs: 92 1cpu CDRDTClea= ner 35 c0000000aead9518 c0000000adb02960 1 cond wait CDRCparse 1cpu CDRCpar= se 37 c0000000aeb64028 c0000000adb03990 1 sleeping secs: 165 1cpu* dbSched= uler 38 c0000000aed3a4c0 c0000000adb000e8 1 sleeping forever 1cpu* dbWorker1= 39 c0000000aed4a028 c0000000adb03178 1 sleeping forever 1cpu* dbWorker2= 40 c0000000aec04b60 c0000000adb041a8 3 sleeping secs: 1 1cpu CDRGfan 41 c0000000aec4e760 c0000000adb049c0 3 sleeping secs: 2 1cpu CDRGeval0 42 c0000000aec4eca0 c0000000adb051d8 3 sleeping secs: 2 1cpu CDRGeval1 43 c0000000aec91220 c0000000adb059f0 3 sleeping secs: 2 1cpu CDRGeval2 44 c0000000aeb38220 c0000000adb06208 1 sleeping secs: 30 1cpu CDRNsT19 45 c0000000aeb38b78 c0000000adb06a20 2 sleeping secs: 1 1cpu ddr_snoopy= 46 c0000000ae85c028 c0000000adb07238 1 sleeping secs: 30 1cpu CDRRTBCle= aner 47 c0000000aea3dc08 c0000000adaff0b8 1 cond wait netnorm 1cpu CDRNr19 48 c0000000ae85c7d0 c0000000adb07a50 1 sleeping secs: 1 1cpu CDRM_Monit= or 49 c0000000ae85ca20 c0000000adb08268 1 cond wait CDRAckslp 1cpu C

WOW, is there a cleaner/prettier way of displaying all that info?.. Even if its like a char-based .per screen? I've heard there are some third-party tools for onstat.

The mail tool messed up my email... = From: "FRANK@ FRANKCOMPUTER.COM" <frank@frankcomputer.com> = = To: ids@iiug.org = = Date: 04/06/2010 08:33 PM = = Subject: Re: why so many informix threads on idle systems ? [1951= 5] = Sent by: ids-bounces@iiug.org = = WOW, is there a cleaner/prettier way of displaying all that info?.. Eve= n if its like a char-based .per screen? I've heard there are some third-part= y tools for onstat. ***********************************************************************= ******** Forum Note: Use "Reply" to post a response in the discussion forum. =

Trying again... |------------------------+---------------------+-----------------------= -| |Thread |Purpose |When activated = | |------------------------+---------------------+-----------------------= -| |CDRD_### |ER apply thread |When a replicated = | | | |transaction is received= .| |------------------------+---------------------+-----------------------= -| |CDRACK_### |ER ACK threads - |When an ACK is received= | | |ACKS from txn.applied| = | | |on other servers | = | | |within the ER domain | = | |------------------------+---------------------+-----------------------= -| |CDRM_Monitor |ER Monitor thread - |One a second = | | |monitors and adjusts | = | | |apply parallelism | = | |------------------------+---------------------+-----------------------= -| |CDRNr19 |ER Network Receive |When a new message is = | | |Thread connected |received from ER node 1= 9| | |to node 19 | = | |------------------------+---------------------+-----------------------= -| |CDRRTBCleaner |Cleans up ER obj no |Every few minutes or so= | | |longer in use | = | |------------------------+---------------------+-----------------------= -| |ddr_snoopy |Receives the log recs|When a log buffer is = | | | |flushed = | |------------------------+---------------------+-----------------------= -| |CDRNsT19 |ER Network Send |When a message needs to= | | |thread to node 19 |be sent to node 19 = | |------------------------+---------------------+-----------------------= -| |CDRGeval### |Grouper Evaluator |As new log records are = | | |threads - evaluates |received = | | |and regroups log | = | | |records for | = | | |replication | = | |------------------------+---------------------+-----------------------= -| |CDRGfan |Grouper Fanout thread|As new log records are = | | |acts as a distributor|received = | | |between the snoopy | = | | |and Eval threads. | = | |------------------------+---------------------+-----------------------= -| |CDRCparse |ER statement cache |Only as a replicate is = | | |manager |created/removed = | |------------------------+---------------------+-----------------------= -| |CDRDTCleaner |Delete Table Cleaner |Every 5 minutes or so = | | |prunes the delete | = | | |shadow tables | = | |------------------------+---------------------+-----------------------= -| |CDRSchedMgr |ER Scheduler - fires |As needed = | | |retry connection | = | | |attempts, timed based| = | | |replication, etc | = | |------------------------+---------------------+-----------------------= -| = From: Madison Pruet/Dallas/IBM@IBMUS = = To: ids@iiug.org = = Date: 04/06/2010 07:29 PM = = Subject: Re: why so many informix threads on idle systems ? [1951= 2] = Sent by: ids-bounces@iiug.org = = Each of the threads have a special purpose. Since the system is idle, there are no 'sqlexec' threads. Those are the 'user' threads and are responsible for the actual query. The ER threads are event driven and =3D= are driven when other things happen, possibly from a log being written or m= =3D aybe in response to having received a replicated transaction. Since you are=3D= using ER, I'll give a break down on what each of those threads are responsible for and what causes them to go into an active state. Notic=3D= e that the threads in this example are all in a waiting/sleeping state. Just to explain a couple of other things. There are two main waiting states for the threads. The 'sleeping' state and the cond wait state. =3D= The sleeping state is normally used when we want to always wake up every so= =3D often just to see what's going on. If a thread is sleeping, that doesn=3D= 't mean that some other thread can't wake it up immediately. The conditio=3D= n wait is a bit different because there no 'timeout' and thus the thread = =3D will only be activated when some other thread activates it. We could respawn these threads, but the cost of the thread setup is a b= =3D it high because of all of the internal querying against the queues and oth= =3D er tables within the syscdr database. That's the reason that we just keep=3D= the threads but place them in a sleep/waiting state when they don't have wo= =3D rk to do. OK - here goes. |------------------------+------------------------+--------------------= =3D ----| |Thread |Purpose |When activated =3D | |------------------------+------------------------+--------------------= =3D ----| |CDRD_### |ER apply thread |When a replicated =3D | | | |transaction is recei=3D ved.| |------------------------+------------------------+--------------------= =3D ----| |CDRACK_### |ER ACK threads - process|When an ACK is recei=3D ved | | |ACKS from transactions | =3D | | |applied on other servers| =3D | | |within the ER domain | =3D | |------------------------+------------------------+--------------------= =3D ----| |CDRM_Monitor |ER Monitor thread - |One a second =3D | | |monitors and adjusts ER | =3D | | |apply parallelism | =3D | |------------------------+------------------------+--------------------= =3D ----| |CDRNr19 |ER Network Receive |When a new message i=3D s | | |Thread connected to node|received from ER nod=3D e 19| | |19 | =3D | |------------------------+------------------------+--------------------= =3D ----| |CDRRTBCleaner |Cleans up ER objects no |Every few minutes or=3D so | | |longer in use | =3D | |------------------------+------------------------+--------------------= =3D ----| |ddr_snoopy |Receives the log records|When a log buffer is=3D | | | |flushed =3D | |------------------------+------------------------+--------------------= =3D ----| |CDRNsT19 |ER Network Send Thread |When a message needs=3D to | | |connected to node 19 |be sent to node 19 =3D | |------------------------+------------------------+--------------------= =3D ----| |CDRGeval### |Grouper Evaluator |As new log records a=3D re | | |threads - evaluates and |received =3D | | |regroups log records for| =3D | | |replication | =3D | |------------------------+------------------------+--------------------= =3D ----| |CDRGfan |Grouper Fanout thread - |As new log records a=3D re | | |acts as a distributor |received =3D | | |between the snoopy and | =3D | | |evaluator threads. Also | =3D | | |has special logic for | =3D | | |begin work/commit | =3D | | |work/rollback... | =3D | |------------------------+------------------------+--------------------= =3D ----| |CDRCparse |ER statement cache |Only as a replicate =3D is |@@N

thanks Madison that makes sense