Informix upgrade issues - 11.10 -> 11.50 with ER (

Hey all - Tried to send this through before with a heap of attachments, but the post failed (to big) Trying to migrate from 11.10.UC2W2 to 11.50.UC3 The main upgrade goes fine, but the CDR component don't work! We run the concdr.sh 11.10 11.50 and then when we start CDR the system grabs 1.3GB RAM and then the CDR threads crash. This is all running on RedHat EL4 in our test environment (so VMWare), so we can run the tests before destroying production! Any ideas? Transcript of test attached (Note the jump from 400Mb to 1.7Gb RAM after cdr is started). Looking at onstat -g ath there are no CDR threads running after cdr start has been run - memory is consumed and then not freed, though is avaialble as free blocks in onstat -g seg. Help please - If nothing obvious, we will raise a PMR. Thanks Jarrod Teale Team Lead - Manufacturing Execution Systems Automation & Process Control Group NZ Technical Fonterra Fonterra Extn: 77525 DDI: +64 7 850 7525 Mobile: +64 21 968 364 fax: +64 7 849 7855 email: jarrod.teale@fonterra.com <mailto:jarrod.teale@fonterra.com> ============================ ============================= TRANSCRIPT =================================== ==================================== Informix 11.20.UC2W2 to 11.50.UC3 upgrade process with ER [informix@muz_jboss ~]$ onstat - IBM Informix Dynamic Server Version 11.10.UC2W2 -- On-Line -- Up 2 days 00:35:51 -- 461932 Kbytes [informix@muz_jboss ~]$ cdr list server SERVER ID STATE STATUS QUEUE CONNECTION CHANGED ----------------------------------------------------------------------- g_jay 11 Active Connected 0 Jul 15 16:08:51 g_jboss 10 Active Local 0 [informix@muz_jboss ~]$ cdr list repl brief REPLICATE TABLE SELECT ------------------------------------------------------------------------ ---- test james@g_jboss:sdev.test <mailto:james@g_jboss:sdev.test> SELECT * FROM test test james@g_jay:sdev.test <mailto:james@g_jay:sdev.test> SELECT * FROM test INSTALL INFORMIX 11.50 to /usr/informix11.50.UC3/ (packaged to a RPM for our installs) [root@muz_jboss ~]# rpm -Uvh MES-informix11-11.50.3-2.i386.rpm Preparing... ########################################### [100%] 1:MES-informix11 ########################################### [100%] CP onconfig and sqlhosts to /usr/informix11.50.UC3/ [root@muz_jboss ~]# cp /usr/informix11.10.UC2W2/etc/onconfig /usr/informix11.50.UC3/etc/onconfig [root@muz_jboss ~]# cp /usr/informix11.10.UC2W2/etc/sqlhosts /usr/informix11.50.UC3/etc/sqlhosts [root@muz_jboss ~]# chown informix: /usr/informix11.50.UC3/etc/sqlhosts [root@muz_jboss ~]# chown informix: /usr/informix11.50.UC3/etc/onconfig Shutdown Engine [informix@muz_jboss ~]$ cdr stop [informix@muz_jboss ~]$ onmode -yuck Move links to start up on new Version [root@muz_jboss ~]# rm /usr/informix; [root@muz_jboss ~]# ln -s /usr/informix11.50.UC3/ /usr/Informix [informix@muz_jboss ~]$ oninit -v Checking group membership to determine server run mode...succeeded Reading configuration file '/usr/informix/etc/onconfig'...succeeded Creating /INFORMIXTMP/.infxdirs...succeeded Creating infos file "/usr/informix/etc/.infos.muz_jboss"...succeeded Linking conf file "/usr/informix/etc/.conf.muz_jboss"...succeeded Checking config parameters...succeeded Writing to infos file...succeeded Allocating and attaching to shared memory...succeeded Creating resident pool 100840 kbytes...succeeded Allocating 40016 kbytes for buffer pool of 2K page size...succeeded Initializing rhead structure...succeeded Initialization of Encryption...succeeded Initializing ASF...succeeded Initializing Dictionary Cache and SPL Routine Cache...succeeded Bringing up ADM VP...succeeded Creating VP classes...succeeded Onlining 1 additional cpu vps...succeeded Onlining 10 IO vps...succeeded Forking main_loop thread...succeeded Initializing DR structures...succeeded Forking 1 'soctcp' listener threads...succeeded Forking 1 'soctcp' listener threads...succeeded Starting tracing...succeeded Initializing 127 flushers...succeeded Initializing log/checkpoint information...succeeded Forking conversion threads Initializing dbspaces...succeeded Opening primary chunks...Converting primary chunk reserved page...succeeded Checking for space in partition header pages...succeeded Opening mirror chunks...succeeded Validating chunks...succeeded Initialize Async Log Flusher...succeeded Starting B-tree Scanner...succeeded Initializing DBSPACETEMP list...succeeded Checking database partition index...succeeded Initializing dataskip structure...succeeded Checking for temporary tables to drop...succeeded Forking onmode_mon thread...succeeded Creating periodic thread...succeeded Verbose output complete: mode = 1 [informix@muz_jboss ~]$ onstat - IBM Informix Dynamic Server Version 11.50.UC3 -- On-Line -- Up 00:00:31 -- 461932 Kbytes [informix@muz_jboss ~]$ /usr/informix11.50.UC3/etc/conv/concdr.sh 11.10 11.50Starting 'syscdr' conversion ... 'syscdr' conversion completed successfully. [informix@muz_jboss ~]$ cdr start [informix@muz_jboss ~]$ onstat - IBM Informix Dynamic Server Version 11.50.UC3 -- On-Line -- Up 00:01:08 -- 1728756 Kbytes [informix@muz_jboss ~]$ cdr list servercommand failed -- Enterprise Replication not active (62) [informix@muz_jboss ~]$ cdr start [informix@muz_jboss ~]$ cdr list server SERVER ID STATE STATUS QUEUE CONNECTION CHANGED ----------------------------------------------------------------------- g_jboss 10 Active Local 0 [informix@muz_jboss ~]$ cdr list repl brief - No response [root@muz_jboss ~]# shutdown -r now Broadcast message from root (pts/4) (Wed Jul 15 16:37:36 2009): The system is going down for reboot NOW! [root@muz_jboss ~]# onstat - IBM Informix Dynamic Server Version 11.50.UC3 -- On-Line -- Up 00:03:33 -- 461932 Kbytes [informix@muz_jboss ~]$ cdr start [informix@muz_jboss ~]$ onstat - IBM Informix Dynamic Server Version 11.50.UC3 -- On-Line -- Up 00:05:55 -- 1728756 Kbytes [informix@muz_jboss ~]$ cdr list server SERVER ID STATE STATUS QUEUE CONNECTION CHANGED ----------------------------------------------------------------------- g_jboss 10 Active Local 0 [informix@muz_jboss ~]$ cdr list repl brief REPLICATE TABLE SELECT ------------------------------------------------------------------------ ---- test james@g_jboss:sdev.test <mailto:james@g_jboss:sdev.test> SELECT * FROM test [informix@muz_jboss ~]$ free -m total used free shared buffers cached Mem: 756 740 16 0 3 469 -/+ buffers/cache: 267 488 Swap: 2047 960 1086 [informix@muz_jboss ~]$ onstat -g seg IBM Informix Dynamic Server Version 11.50.UC3 -- On-Line -- Up 00:18:03 -- 1728756 Kbytes Segment Summary: id key addr size ovhd class blkused blkfree 0 52604801 44000000 145338368 1074732 R* 35481 2 163845 52604806 4ca9b000 327680000 1921184 V* 8670 71330 524304 52604810 6031b000 1297227776

Hey - thought I would post the CDR onconfig settings as well. # CDR Variables CDR_EVALTHREADS 2 # evaluator threads (per-cpu-vp,additional) CDR_DSLOCKWAIT 5 # DS lockwait timeout (seconds) CDR_QUEUEMEM 15360 # Maximum amount of memory for any CDR queue (Kbytes)CDR_LOGDELTA 30 # % of log space allowed in queue memory CDR_NUMCONNECT 16 # Expected connections per server CDR_NIFRETRY 300 # Connection retry (seconds) CDR_NIFCOMPRESS 9 # Link level compression (-1 never, 0 none, 9 max) CDR_SERIAL 100,-1 # Serial Column Sequence CDR_DBSPACE er_dbspace # dbspace for syscdr database CDR_QHDR_DBSPACE er_dbspace # CDR queue dbspace (default sameas catalog) CDR_QDATA_SBSPACE er_sbspace # List of CDR queue smart blobspaces CDR_MAX_DYNAMIC_LOGS 10 -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Jarrod Teale Sent: Thursday, 16 July 2009 8:10 a.m. To: ids@iiug.org Subject: Informix upgrade issues - 11.10 -> 11.50 with .... [16410] Hey all - Tried to send this through before with a heap of attachments, but the post failed (to big) Trying to migrate from 11.10.UC2W2 to 11.50.UC3 The main upgrade goes fine, but the CDR component don't work! We run the concdr.sh 11.10 11.50 and then when we start CDR the system grabs 1.3GB RAM and then the CDR threads crash. This is all running on RedHat EL4 in our test environment (so VMWare), so we can run the tests before destroying production! Any ideas? Transcript of test attached (Note the jump from 400Mb to 1.7Gb RAM after cdr is started). Looking at onstat -g ath there are no CDR threads running after cdr start has been run - memory is consumed and then not freed, though is avaialble as free blocks in onstat -g seg. Help please - If nothing obvious, we will raise a PMR. Thanks Jarrod Teale Team Lead - Manufacturing Execution Systems Automation & Process Control Group NZ Technical Fonterra Fonterra Extn: 77525 DDI: +64 7 850 7525 Mobile: +64 21 968 364 fax: +64 7 849 7855 email: jarrod.teale@fonterra.com <mailto:jarrod.teale@fonterra.com> ============================ ============================= TRANSCRIPT =================================== ==================================== Informix 11.20.UC2W2 to 11.50.UC3 upgrade process with ER [informix@muz_jboss ~]$ onstat - IBM Informix Dynamic Server Version 11.10.UC2W2 -- On-Line -- Up 2 days 00:35:51 -- 461932 Kbytes [informix@muz_jboss ~]$ cdr list server SERVER ID STATE STATUS QUEUE CONNECTION CHANGED ----------------------------------------------------------------------- g_jay 11 Active Connected 0 Jul 15 16:08:51 g_jboss 10 Active Local 0 [informix@muz_jboss ~]$ cdr list repl brief REPLICATE TABLE SELECT ------------------------------------------------------------------------ ---- test james@g_jboss:sdev.test <mailto:james@g_jboss:sdev.test> SELECT * FROM test test james@g_jay:sdev.test <mailto:james@g_jay:sdev.test> SELECT * FROM test INSTALL INFORMIX 11.50 to /usr/informix11.50.UC3/ (packaged to a RPM for our installs) [root@muz_jboss ~]# rpm -Uvh MES-informix11-11.50.3-2.i386.rpm Preparing... ########################################### [100%] 1:MES-informix11 ########################################### [100%] CP onconfig and sqlhosts to /usr/informix11.50.UC3/ [root@muz_jboss ~]# cp /usr/informix11.10.UC2W2/etc/onconfig /usr/informix11.50.UC3/etc/onconfig [root@muz_jboss ~]# cp /usr/informix11.10.UC2W2/etc/sqlhosts /usr/informix11.50.UC3/etc/sqlhosts [root@muz_jboss ~]# chown informix: /usr/informix11.50.UC3/etc/sqlhosts [root@muz_jboss ~]# chown informix: /usr/informix11.50.UC3/etc/onconfig Shutdown Engine [informix@muz_jboss ~]$ cdr stop [informix@muz_jboss ~]$ onmode -yuck Move links to start up on new Version [root@muz_jboss ~]# rm /usr/informix; [root@muz_jboss ~]# ln -s /usr/informix11.50.UC3/ /usr/Informix [informix@muz_jboss ~]$ oninit -v Checking group membership to determine server run mode...succeeded Reading configuration file '/usr/informix/etc/onconfig'...succeeded Creating /INFORMIXTMP/.infxdirs...succeeded Creating infos file "/usr/informix/etc/.infos.muz_jboss"...succeeded Linking conf file "/usr/informix/etc/.conf.muz_jboss"...succeeded Checking config parameters...succeeded Writing to infos file...succeeded Allocating and attaching to shared memory...succeeded Creating resident pool 100840 kbytes...succeeded Allocating 40016 kbytes for buffer pool of 2K page size...succeeded Initializing rhead structure...succeeded Initialization of Encryption...succeeded Initializing ASF...succeeded Initializing Dictionary Cache and SPL Routine Cache...succeeded Bringing up ADM VP...succeeded Creating VP classes...succeeded Onlining 1 additional cpu vps...succeeded Onlining 10 IO vps...succeeded Forking main_loop thread...succeeded Initializing DR structures...succeeded Forking 1 'soctcp' listener threads...succeeded Forking 1 'soctcp' listener threads...succeeded Starting tracing...succeeded Initializing 127 flushers...succeeded Initializing log/checkpoint information...succeeded Forking conversion threads Initializing dbspaces...succeeded Opening primary chunks...Converting primary chunk reserved page...succeeded Checking for space in partition header pages...succeeded Opening mirror chunks...succeeded Validating chunks...succeeded Initialize Async Log Flusher...succeeded Starting B-tree Scanner...succeeded Initializing DBSPACETEMP list...succeeded Checking database partition index...succeeded Initializing dataskip structure...succeeded Checking for temporary tables to drop...succeeded Forking onmode_mon thread...succeeded Creating periodic thread...succeeded Verbose output complete: mode = 1 [informix@muz_jboss ~]$ onstat - IBM Informix Dynamic Server Version 11.50.UC3 -- On-Line -- Up 00:00:31 -- 461932 Kbytes [informix@muz_jboss ~]$ /usr/informix11.50.UC3/etc/conv/concdr.sh 11.10 11.50 Starting 'syscdr' conversion ...'syscdr' conversion completed successfully. [informix@muz_jboss ~]$ cdr start [informix@muz_jboss ~]$ onstat - IBM Informix Dynamic Server Version 11.50.UC3 -- On-Line -- Up 00:01:08 -- 1728756 Kbytes [informix@muz_jboss ~]$ cdr list server command failed -- EnterpriseReplication not active (62) [informix@muz_jboss ~]$ cdr start [informix@muz_jboss ~]$ cdr list server SERVER ID STATE STATUS QUEUE CONNECTION CHANGED ----------------------------------------------------------------------- g_jboss 10 Active Local 0 [informix@muz_jboss ~]$ cdr list repl brief - No response [root@muz_jboss ~]# shutdown -r now Broadcast message from root (pts/4) (Wed Jul 15 16:37:36 2009): The system is going down for reboot NOW! [root@muz_jboss ~]# onstat - IBM Informix Dynamic Server Version 11.50.UC3 -- On-Line -- Up 00:03:33 -- 461932 Kbytes [informix@muz_jboss ~]$ cdr start [informix@muz_jboss ~]$ onstat - IBM Informix Dynamic Server Version 11.50.UC3 -- On-Line -- Up 00:05:55 -- 1728756 Kbytes [informix@mu