Informix 11 Upgrade with ER failed - help please - broken sysmaster database
Posted in 2008
Topics: High Availability & Replication, Installation, Setup & Upgrades, Stored Procedures & SPL, Error Codes & Troubleshooting, Logging & Checkpoints
Hi,
We have a number of supposedly identical systems running 10.00.UC3 on
RedHat Linux.
We are in the process of upgrading these to 11.10.UC2W2 - and have hit a
roadblock.
7 servers upgraded fine - no problem
The eighth one has the following error in the online log. It is a leaf
node in the Enterprise Replication set we have (and yes "cdr stop" was
run before the upgrade happened).
The server is online and working - but ER can't connect (even with a
delete server and define server) - we get "This feature of Enterprise
Replication is not yet supported" which is really unhelpful
We are looking at reverting and trying again, but after a quick answer.
Thanks
Jarrod Teale
MES Team Lead
Fonterra New Zealand.
...
...
11:38:23 Error building 'sysmaster' database.
11:38:23 See '/tmp/buildsmi.26669'.
...
...Unfortunately, that file does not exist
...
...
11:38:31 'sysadmin' database built successfully.
11:38:31 SCHAPI: Started dbScheduler thread.
11:38:31 SCHAPI: Started 2 dbWorker threads.
11:38:31 Loading Module <SPLNULL>
11:38:31 SCHAPI: Error -206 The specified table(sysmaster:sysactptnhdr) is not in the database.
11:38:31 SCHAPI: Type: SENSOR, Name: mon_table_profile, Location: NULL.
11:38:31 SCHAPI: Error -111 ISAM error: no record found.
11:38:31 SCHAPI: Type: SENSOR, Name: mon_table_profile, Location: NULL.
11:38:31 SCHAPI: Error -206 The specified table(sysmaster:syscheckpoint) is not in the database.
11:38:31 SCHAPI: Type: SENSOR, Name: mon_checkpoint, Location: NULL.
11:38:31 SCHAPI: Error -111 ISAM error: no record found.
11:38:31 SCHAPI: Type: SENSOR, Name: mon_checkpoint, Location: NULL.
11:38:31 SCHAPI: Error -1214 Value exceeds limit of SMALLINT precision
11:38:31 SCHAPI: Type: SENSOR, Name: mon_vps, Location: NULL.
11:38:31 SCHAPI: Error -206 The specified table(sysmaster:sysnetworkio) is not in the database.
11:38:31 SCHAPI: Type: SENSOR, Name: mon_users, Location: NULL.
11:38:31 SCHAPI: Error -111 ISAM error: no record found.
11:38:31 SCHAPI: Type: SENSOR, Name: mon_users, Location: NULL.
...
...
11:38:46 (32) connection rejected - no calls allowed for cdraccept
11:38:52 Logical Log 4284 Complete, timestamp: 0x4a862e48.
11:38:54 Checkpoint Completed: duration was 0 seconds.
11:38:54 Tue Jul 1 - loguniq 4285, logpos 0x18, timestamp: 0x4a863133Interval: 4
11:38:54 Maximum server connections 2
11:38:54 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txnsblocked 0, Plog used 5737, Llog used 23633
...
...
Tried "cdr start" at this point.
...
...
11:39:07 CDR Initialization failed (sysmaster is not ready).
11:39:30 SCHAPI: Error -206 The specified table (sysmaster:sysenv) isnot in the database.
11:39:30 SCHAPI: Type: STARTUP SENSOR, Name: mon_sysenv, Location:
NULL.
11:39:30 SCHAPI: Error -111 ISAM error: no record found.
11:39:30 SCHAPI: Type: STARTUP SENSOR, Name: mon_sysenv, Location:
NULL.
11:39:35 CDR Initialization failed (sysmaster is not ready).
Jarrod Teale wrote:
> Hi,
> We have a number of supposedly identical systems running 10.00.UC3 on
> RedHat Linux.
> We are in the process of upgrading these to 11.10.UC2W2 - and have hit a
> roadblock.
> 7 servers upgraded fine - no problem
> The eighth one has the following error in the online log. It is a leaf
> node in the Enterprise Replication set we have (and yes "cdr stop" was
> run before the upgrade happened).
> The server is online and working - but ER can't connect (even with a
> delete server and define server) - we get "This feature of Enterprise
> Replication is not yet supported" which is really unhelpful
> We are looking at reverting and trying again, but after a quick answer.
>
> Thanks
> Jarrod Teale
> MES Team Lead
> Fonterra New Zealand.
The problem is not with ER, the sysmaster database was not able to be
built. I'd start by running $INFORMIXDIR/etc/buildsmi as user informix.
>
>
> ...
> ...
> 11:38:23 Error building 'sysmaster' database.
> 11:38:23 See '/tmp/buildsmi.26669'.
> ...
> ...> Unfortunately, that file does not exist
> ...
> ...
> 11:38:31 'sysadmin' database built successfully.
> 11:38:31 SCHAPI: Started dbScheduler thread.
> 11:38:31 SCHAPI: Started 2 dbWorker threads.
> 11:38:31 Loading Module <SPLNULL>
> 11:38:31 SCHAPI: Error -206 The specified table> (sysmaster:sysactptnhdr) is not in the database.
> 11:38:31 SCHAPI: Type: SENSOR, Name: mon_table_profile, Location: NULL.
> 11:38:31 SCHAPI: Error -111 ISAM error: no record found.
> 11:38:31 SCHAPI: Type: SENSOR, Name: mon_table_profile, Location: NULL.
> 11:38:31 SCHAPI: Error -206 The specified table> (sysmaster:syscheckpoint) is not in the database.
> 11:38:31 SCHAPI: Type: SENSOR, Name: mon_checkpoint, Location: NULL.
> 11:38:31 SCHAPI: Error -111 ISAM error: no record found.
> 11:38:31 SCHAPI: Type: SENSOR, Name: mon_checkpoint, Location: NULL.
> 11:38:31 SCHAPI: Error -1214 Value exceeds limit of SMALLINT precision
> 11:38:31 SCHAPI: Type: SENSOR, Name: mon_vps, Location: NULL.
> 11:38:31 SCHAPI: Error -206 The specified table> (sysmaster:sysnetworkio) is not in the database.
> 11:38:31 SCHAPI: Type: SENSOR, Name: mon_users, Location: NULL.
> 11:38:31 SCHAPI: Error -111 ISAM error: no record found.
> 11:38:31 SCHAPI: Type: SENSOR, Name: mon_users, Location: NULL.
> ...
> ...
> 11:38:46 (32) connection rejected - no calls allowed for cdraccept
> 11:38:52 Logical Log 4284 Complete, timestamp: 0x4a862e48.
> 11:38:54 Checkpoint Completed: duration was 0 seconds.
> 11:38:54 Tue Jul 1 - loguniq 4285, logpos 0x18, timestamp: 0x4a863133> Interval: 4
>
> 11:38:54 Maximum server connections 2
> 11:38:54 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns> blocked 0, Plog used 5737, Llog used 23633
>
> ...
> ...
> Tried "cdr start" at this point.
> ...
> ...
>
> 11:39:07 CDR Initialization failed (sysmaster is not ready).
> 11:39:30 SCHAPI: Error -206 The specified table (sysmaster:sysenv) is> not in the database.
> 11:39:30 SCHAPI: Type: STARTUP SENSOR, Name: mon_sysenv, Location: NULL.
> 11:39:30 SCHAPI: Error -111 ISAM error: no record found.
> 11:39:30 SCHAPI: Type: STARTUP SENSOR, Name: mon_sysenv, Location: NULL.
> 11:39:35 CDR Initialization failed (sysmaster is not ready).>
>
Madison Pruet wrote:
> Jarrod Teale wrote:
>> Hi,
>> We have a number of supposedly identical systems running 10.00.UC3 on
>> RedHat Linux.
>> We are in the process of upgrading these to 11.10.UC2W2 - and have hit
>> a roadblock.
>> 7 servers upgraded fine - no problem
>> The eighth one has the following error in the online log. It is a leaf
>> node in the Enterprise Replication set we have (and yes "cdr stop" was
>> run before the upgrade happened).
>> The server is online and working - but ER can't connect (even with a
>> delete server and define server) - we get "This feature of Enterprise
>> Replication is not yet supported" which is really unhelpful
>> We are looking at reverting and trying again, but after a quick answer.
>>
>> Thanks
>> Jarrod Teale
>> MES Team Lead
>> Fonterra New Zealand.
>
> The problem is not with ER, the sysmaster database was not able to be
> built. I'd start by running $INFORMIXDIR/etc/buildsmi as user informix.
>
Try setting CDRFE_TRACE=1:/tmp/cdr.out and then rerun the cdr utility.
Then post the output of /tmp/cdr.out
>>
>>
>> ...
>> ...
>> 11:38:23 Error building 'sysmaster' database.
>> 11:38:23 See '/tmp/buildsmi.26669'.
>> ...
>> ...>> Unfortunately, that file does not exist
>> ...
>> ...
>> 11:38:31 'sysadmin' database built successfully.
>> 11:38:31 SCHAPI: Started dbScheduler thread.
>> 11:38:31 SCHAPI: Started 2 dbWorker threads.
>> 11:38:31 Loading Module <SPLNULL>
>> 11:38:31 SCHAPI: Error -206 The specified table>> (sysmaster:sysactptnhdr) is not in the database.
>> 11:38:31 SCHAPI: Type: SENSOR, Name: mon_table_profile, Location: NULL.
>> 11:38:31 SCHAPI: Error -111 ISAM error: no record found.
>> 11:38:31 SCHAPI: Type: SENSOR, Name: mon_table_profile, Location: NULL.
>> 11:38:31 SCHAPI: Error -206 The specified table>> (sysmaster:syscheckpoint) is not in the database.
>> 11:38:31 SCHAPI: Type: SENSOR, Name: mon_checkpoint, Location: NULL.
>> 11:38:31 SCHAPI: Error -111 ISAM error: no record found.
>> 11:38:31 SCHAPI: Type: SENSOR, Name: mon_checkpoint, Location: NULL.
>> 11:38:31 SCHAPI: Error -1214 Value exceeds limit of SMALLINT precision
>> 11:38:31 SCHAPI: Type: SENSOR, Name: mon_vps, Location: NULL.
>> 11:38:31 SCHAPI: Error -206 The specified table>> (sysmaster:sysnetworkio) is not in the database.
>> 11:38:31 SCHAPI: Type: SENSOR, Name: mon_users, Location: NULL.
>> 11:38:31 SCHAPI: Error -111 ISAM error: no record found.
>> 11:38:31 SCHAPI: Type: SENSOR, Name: mon_users, Location: NULL.
>> ...
>> ...
>> 11:38:46 (32) connection rejected - no calls allowed for cdraccept
>> 11:38:52 Logical Log 4284 Complete, timestamp: 0x4a862e48.
>> 11:38:54 Checkpoint Completed: duration was 0 seconds.
>> 11:38:54 Tue Jul 1 - loguniq 4285, logpos 0x18, timestamp:>> 0x4a863133 Interval: 4
>>
>> 11:38:54 Maximum server connections 2
>> 11:38:54 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns>> blocked 0, Plog used 5737, Llog used 23633
>>
>> ...
>> ...
>> Tried "cdr start" at this point.
>> ...
>> ...
>>
>> 11:39:07 CDR Initialization failed (sysmaster is not ready).
>> 11:39:30 SCHAPI: Error -206 The specified table (sysmaster:sysenv) is>> not in the database.
>> 11:39:30 SCHAPI: Type: STARTUP SENSOR, Name: mon_sysenv, Location: NULL.
>> 11:39:30 SCHAPI: Error -111 ISAM error: no record found.
>> 11:39:30 SCHAPI: Type: STARTUP SENSOR, Name: mon_sysenv, Location: NULL.
>> 11:39:35 CDR Initialization failed (sysmaster is not ready).>>
>>