Enterprise Replication recovery slow on Solaris
A user running Enterprise Replication from a Solaris 10 primary to an AIX 6.1 target (IDS 11.70.FC5) found that after a 12-hour link outage the ~7.6 GB/13M-transaction send queue was draining at only ~2 MB/minute, implying about a week to resync. Suspecting Solaris tuning, he posted onstat -g rsm/rcv output. The key suggestion was that the target had no optimizer statistics after its dbimport, which throttles the apply. Running his update statistics script on the target (the cron job there had been disabled) raised throughput to ~28 MB/minute, clearing the backlog in roughly four hours. A side question about the T5220 being a T-class machine drew no further discussion.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
Hello,
I have an Enterprise replication environment where the primary is running
SunOS 10 and the backup is AIX 6.1, both running IBM Informix Dynamic Server
Version 11.70.FC5GE. I purposely interrupted communications between the
primary and backup for about 12 hours, then re-established the connection. In
this time, I estimate that my application added maybe 10 GB of data to the
database on the primary. After reconnecting the machines, I shut down all of
my apps so that no new data is being inserted or deleted from the database.
The purpose of this is to see how long it takes for the replicated data to
propagate to the backup from the primary. After 1 day, the databases still
were not in sync, so I manually did a row count of the tables being replicated
on the primary and backup. The primary has about 54 million rows and, at the
beginning of day 2, the backup had 38 million rows. At the beginning of day 4
the backup had about 44 million rows. So, with 6 million rows in 2 days and
another 10 million rows to go, I estimate about another 3 days, making the
entire recovery of 10 GB taking 7 days. This is too long, and I am thinking
that this must be some kind of SunOS tuning issue.
On a possibly related note, both machines were synced up from the database of
a third machine before this test began using dbimport. It took 2 hours to
dbimport the AIX machine and 8 hours to dbimport the same data onto the
Solaris machine.
Some info for the SunOS machine:
- The onconfig file has VPCLASS cpu,num=16,noage.
- The OpenAdmin Tool shows no recommended changes to the configuration.
- The OAT shows the following Server Info:
Server Type: Standard
Version: 11.70.FC5GE
Server Time: 09:45:04
Boot Time: 2013-03-19 13:08
Up Time: 1 days 20:36:50
Sessions: 3
Max Users: 544
Operating System
Total Mem: 15.9 GB
Free Mem: 376 MB
# of CPUs: 32
(I restarted informix 2 days ago to tweak some onconfig settings.)
- Other than informix, nothing is running on either the primary or
the backup.
- As of the last informix restart, the online log shows no errors of
any kind, everything is looking normal.
A prstat shows the following on the primary (5 seconds apart):
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
5837 informix 2796M 2365M sleep 59 -10 8:36:57 0.4% oninit/17
5844 informix 2796M 2350M sleep 52 -10 8:18:56 0.5% oninit/19
5851 informix 2796M 2350M sleep 59 -10 5:09:33 0.1% oninit/18
5852 informix 2796M 2349M sleep 59 -10 4:35:21 0.0% oninit/19
5846 informix 2796M 2347M sleep 59 -10 7:44:28 0.5% oninit/18
5847 informix 2796M 2346M sleep 59 -10 7:25:23 0.6% oninit/17
5845 informix 2796M 2346M sleep 52 -10 7:58:06 0.4% oninit/18
5850 informix 2796M 2345M cpu38 30 -10 6:20:15 0.5% oninit/21
5848 informix 2796M 2343M sleep 59 -10 7:08:30 0.6% oninit/17
5849 informix 2796M 2343M sleep 59 -10 6:49:08 0.4% oninit/18
5853 informix 2796M 2342M sleep 59 -10 4:14:47 0.0% oninit/19
5857 informix 2796M 2337M sleep 59 -10 3:19:36 0.0% oninit/17
5854 informix 2796M 2334M sleep 59 -10 3:52:52 0.0% oninit/17
5855 informix 2796M 2334M sleep 59 -10 3:34:57 0.0% oninit/18
5856 informix 2796M 2332M sleep 59 -10 3:26:08 0.0% oninit/19
=====
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
5837 informix 2796M 2365M sleep 59 -10 8:36:57 0.3% oninit/17
5844 informix 2796M 2350M sleep 30 -10 8:18:58 0.6% oninit/19
5851 informix 2796M 2350M sleep 59 -10 5:09:33 0.1% oninit/18
5852 informix 2796M 2349M sleep 59 -10 4:35:21 0.0% oninit/19
5846 informix 2796M 2347M sleep 59 -10 7:44:29 0.5% oninit/18
5847 informix 2796M 2346M sleep 59 -10 7:25:24 0.5% oninit/17
5845 informix 2796M 2346M sleep 50 -10 7:58:07 0.4% oninit/18
5850 informix 2796M 2345M cpu37 40 -10 6:20:17 0.6% oninit/21
5848 informix 2796M 2343M sleep 59 -10 7:08:30 0.5% oninit/17
5849 informix 2796M 2343M sleep 52 -10 6:49:08 0.4% oninit/18
5853 informix 2796M 2342M sleep 59 -10 4:14:47 0.0% oninit/19
5857 informix 2796M 2337M sleep 59 -10 3:19:36 0.0% oninit/17
5854 informix 2796M 2334M sleep 59 -10 3:52:52 0.0% oninit/17
5855 informix 2796M 2334M sleep 59 -10 3:34:57 0.0% oninit/18
5856 informix 2796M 2332M sleep 59 -10 3:26:08 0.0% oninit/19
=====
An "onstat -g rsm sendq | grep queue" 60 seconds apart produced the following:
Txns in queue: 13438400
Log Events in queue: 39
Size of Data in queue: 7601814074 Bytes
Total data queued: 7010242960 Bytes
Total Txns queued: 18842496
===
Txns in queue: 13435039
Log Events in queue: 39
Size of Data in queue: 7599625995 Bytes
Total data queued: 7010242960 Bytes
Total Txns queued: 18842496
===
Currently, it looks like 2,188,079 Bytes per minute are transferring to the
backup. With 7,601,814,074 Bytes in the queue, looks like it will take another
3474 minutes, or about 57 more hours.
Once again, I see that the data is getting replicated, it's just happening
very slowly.
Any suggestions on how to tune my Solaris box would be appreciated.
Thanks,
Jeff
What is the output on the target of
onstat -g ath
onstat -g rcv full
If there was a large backlog on the source node, then most of the
transactional backlog would be in stable storage so there could be an i=
ssue
with reading the replicated transactions from disk. What is the onconf=
ig
of the source? Are you using kernel IO or aiovps?
What is the output of onstat -g rqm brief on both the source and the
target?
Are the statistics on the target up to date? Since you did a dbimport,=
there would have been no statistics gathered and that could cause us to=
throttle the apply.
From: "JEFFREY GENEGA" <jeffrey.genega@spirent.com>
To: ids@iiug.org,
Date: 03/21/2013 09:28 AM
Subject: Enterprise Replication recovery slow on Solaris [29854]
Sent by: ids-bounces@iiug.org
Hello,
I have an Enterprise replication environment where the primary is runni=
ng
SunOS 10 and the backup is AIX 6.1, both running IBM Informix Dynamic
Server
Version 11.70.FC5GE. I purposely interrupted communications between the=
primary and backup for about 12 hours, then re-established the connecti=
on.
In
this time, I estimate that my application added maybe 10 GB of data to =
the
database on the primary. After reconnecting the machines, I shut down a=
ll
of
my apps so that no new data is being inserted or deleted from the datab=
ase.
The purpose of this is to see how long it takes for the replicated data=
to
propagate to the backup from the primary. After 1 day, the databases st=
ill
were not in sync, so I manually did a row count of the tables being
replicated
on the primary and backup. The primary has about 54 million rows and, a=
t
the
beginning of day 2, the backup had 38 million rows. At the beginning of=
day
4
the backup had about 44 million rows. So, with 6 million rows in 2 days=
and
another 10 million rows to go, I estimate about another 3 days, making =
the
entire recovery of 10 GB taking 7 days. This is too long, and I am thin=
king
that this must be some kind of SunOS tuning issue.
On a possibly related note, both machines were synced up from the datab=
ase
of
a third machine before this test began using dbimport. It took 2 hours =
to
dbimport the AIX machine and 8 hours to dbimport the same data onto the=
Solaris machine.
Some info for the SunOS machine:
- The onconfig file has VPCLASS cpu,num=3D16,noage.
- The OpenAdmin Tool shows no recommended changes to the configuration.=
- The OAT shows the following Server Info:
Server Type: Standard
Version: 11.70.FC5GE
Server Time: 09:45:04
Boot Time: 2013-03-19 13:08
Up Time: 1 days 20:36:50
Sessions: 3
Max Users: 544
Operating System
Total Mem: 15.9 GB
Free Mem: 376 MB
# of CPUs: 32
(I restarted informix 2 days ago to tweak some onconfig settings.)
- Other than informix, nothing is running on either the primary or
the backup.
- As of the last informix restart, the online log shows no errors of
any kind, everything is looking normal.
A prstat shows the following on the primary (5 seconds apart):
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
5837 informix 2796M 2365M sleep 59 -10 8:36:57 0.4% oninit/17
5844 informix 2796M 2350M sleep 52 -10 8:18:56 0.5% oninit/19
5851 informix 2796M 2350M sleep 59 -10 5:09:33 0.1% oninit/18
5852 informix 2796M 2349M sleep 59 -10 4:35:21 0.0% oninit/19
5846 informix 2796M 2347M sleep 59 -10 7:44:28 0.5% oninit/18
5847 informix 2796M 2346M sleep 59 -10 7:25:23 0.6% oninit/17
5845 informix 2796M 2346M sleep 52 -10 7:58:06 0.4% oninit/18
5850 informix 2796M 2345M cpu38 30 -10 6:20:15 0.5% oninit/21
5848 informix 2796M 2343M sleep 59 -10 7:08:30 0.6% oninit/17
5849 informix 2796M 2343M sleep 59 -10 6:49:08 0.4% oninit/18
5853 informix 2796M 2342M sleep 59 -10 4:14:47 0.0% oninit/19
5857 informix 2796M 2337M sleep 59 -10 3:19:36 0.0% oninit/17
5854 informix 2796M 2334M sleep 59 -10 3:52:52 0.0% oninit/17
5855 informix 2796M 2334M sleep 59 -10 3:34:57 0.0% oninit/18
5856 informix 2796M 2332M sleep 59 -10 3:26:08 0.0% oninit/19
=3D=3D=3D=3D=3D
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
5837 informix 2796M 2365M sleep 59 -10 8:36:57 0.3% oninit/17
5844 informix 2796M 2350M sleep 30 -10 8:18:58 0.6% oninit/19
5851 informix 2796M 2350M sleep 59 -10 5:09:33 0.1% oninit/18
5852 informix 2796M 2349M sleep 59 -10 4:35:21 0.0% oninit/19
5846 informix 2796M 2347M sleep 59 -10 7:44:29 0.5% oninit/18
5847 informix 2796M 2346M sleep 59 -10 7:25:24 0.5% oninit/17
5845 informix 2796M 2346M sleep 50 -10 7:58:07 0.4% oninit/18
5850 informix 2796M 2345M cpu37 40 -10 6:20:17 0.6% oninit/21
5848 informix 2796M 2343M sleep 59 -10 7:08:30 0.5% oninit/17
5849 informix 2796M 2343M sleep 52 -10 6:49:08 0.4% oninit/18
5853 informix 2796M 2342M sleep 59 -10 4:14:47 0.0% oninit/19
5857 informix 2796M 2337M sleep 59 -10 3:19:36 0.0% oninit/17
5854 informix 2796M 2334M sleep 59 -10 3:52:52 0.0% oninit/17
5855 informix 2796M 2334M sleep 59 -10 3:34:57 0.0% oninit/18
5856 informix 2796M 2332M sleep 59 -10 3:26:08 0.0% oninit/19
=3D=3D=3D=3D=3D
An "onstat -g rsm sendq | grep queue" 60 seconds apart produced the
following:
Txns in queue: 13438400
Log Events in queue: 39
Size of Data in queue: 7601814074 Bytes
Total data queued: 7010242960 Bytes
Total Txns queued: 18842496
=3D=3D=3D
Txns in queue: 13435039
Log Events in queue: 39
Size of Data in queue: 7599625995 Bytes
Total data queued: 7010242960 Bytes
Total Txns queued: 18842496
=3D=3D=3D
Currently, it looks like 2,188,079 Bytes per minute are transferring to=
the
backup. With 7,601,814,074 Bytes in the queue, looks like it will take
another
3474 minutes, or about 57 more hours.
Once again, I see that the data is getting replicated, it's just happen=
ing
very slowly.
Any suggestions on how to tune my Solaris box would be appreciated.
Thanks,
Jeff
***********************************************************************=
********
Forum Note: Use "Reply" to post a response in the discussion forum.
=
I do not recall if update statistics was run on the backup, but we have a script that is run periodically out of cron that is tuned to our application to run update statistics on the primary. Unfortunately, I did not realize until now that this cron job is disabled on the backup. I just ran the update statistics script on the backup, and it improved the transfer rate as follows: === Txns in queue: 13038990 Log Events in queue: 41 Size of Data in queue: 7380906907 Bytes Total data queued: 2671560 Bytes Total Txns queued: 13195517 === Txns in queue: 12991030 Log Events in queue: 41 Size of Data in queue: 7352376430 Bytes Total data queued: 2671560 Bytes Total Txns queued: 13195517 The difference between runs is 60 seconds. So, it is now transferring at about 28,530,477 bytes per minute. At this rate, the queue should clear out in about 4 hours, which is about what I was expecting, or maybe even a little better. Thanks a bunch.
You might want to run 'onstat -z' on the target system. And a few min
later. 'onstat -g rcv full'.
From: "JEFFREY GENEGA" <jeffrey.genega@spirent.com>
To: ids@iiug.org,
Date: 03/21/2013 11:32 AM
Subject: Re: Enterprise Replication recovery slow on Solari [29858]
Sent by: ids-bounces@iiug.org
I do not recall if update statistics was run on the backup, but we have=
a
script that is run periodically out of cron that is tuned to our
application
to run update statistics on the primary. Unfortunately, I did not reali=
ze
until now that this cron job is disabled on the backup. I just ran the
update
statistics script on the backup, and it improved the transfer rate as
follows:
=3D=3D=3D
Txns in queue: 13038990
Log Events in queue: 41
Size of Data in queue: 7380906907 Bytes
Total data queued: 2671560 Bytes
Total Txns queued: 13195517
=3D=3D=3D
Txns in queue: 12991030
Log Events in queue: 41
Size of Data in queue: 7352376430 Bytes
Total data queued: 2671560 Bytes
Total Txns queued: 13195517
The difference between runs is 60 seconds.
So, it is now transferring at about 28,530,477 bytes per minute. At thi=
s
rate,
the queue should clear out in about 4 hours, which is about what I was
expecting, or maybe even a little better.
Thanks a bunch.
***********************************************************************=
********
Forum Note: Use "Reply" to post a response in the discussion forum.
=
The Sun box wouldn't happen to be a T-class machine would it?
Art
Art S. Kagel
Advanced DataTools (www.advancedatatools.com)
Blog: http://informix-myview.blogspot.com/
Disclaimer: Please keep in mind that my own opinions are my own opinions
and do not reflect on my employer, Advanced DataTools, the IIUG, nor any
other organization with which I am associated either explicitly,
implicitly, or by inference. Neither do those opinions reflect those of
other individuals affiliated with any entity with which I am affiliated nor
those of the entities themselves.
On Thu, Mar 21, 2013 at 10:27 AM, JEFFREY GENEGA <jeffrey.genega@spirent.com
> wrote:
> Hello,
>
> I have an Enterprise replication environment where the primary is running
> SunOS 10 and the backup is AIX 6.1, both running IBM Informix Dynamic
> Server
> Version 11.70.FC5GE. I purposely interrupted communications between the
> primary and backup for about 12 hours, then re-established the connection.
> In
> this time, I estimate that my application added maybe 10 GB of data to the
> database on the primary. After reconnecting the machines, I shut down all
> of
> my apps so that no new data is being inserted or deleted from the database.
> The purpose of this is to see how long it takes for the replicated data to
> propagate to the backup from the primary. After 1 day, the databases still
> were not in sync, so I manually did a row count of the tables being
> replicated
> on the primary and backup. The primary has about 54 million rows and, at
> the
> beginning of day 2, the backup had 38 million rows. At the beginning of
> day 4
> the backup had about 44 million rows. So, with 6 million rows in 2 days and
> another 10 million rows to go, I estimate about another 3 days, making the
> entire recovery of 10 GB taking 7 days. This is too long, and I am thinking
> that this must be some kind of SunOS tuning issue.
>
> On a possibly related note, both machines were synced up from the database
> of
> a third machine before this test began using dbimport. It took 2 hours to
> dbimport the AIX machine and 8 hours to dbimport the same data onto the
> Solaris machine.
>
> Some info for the SunOS machine:
> - The onconfig file has VPCLASS cpu,num=16,noage.
> - The OpenAdmin Tool shows no recommended changes to the configuration.
> - The OAT shows the following Server Info:
>
> Server Type: Standard
> Version: 11.70.FC5GE
> Server Time: 09:45:04
> Boot Time: 2013-03-19 13:08
> Up Time: 1 days 20:36:50
> Sessions: 3
> Max Users: 544
> Operating System
> Total Mem: 15.9 GB
> Free Mem: 376 MB
> # of CPUs: 32
>
> (I restarted informix 2 days ago to tweak some onconfig settings.)
>
> - Other than informix, nothing is running on either the primary or
>
> the backup.
> - As of the last informix restart, the online log shows no errors of
>
> any kind, everything is looking normal.
>
> A prstat shows the following on the primary (5 seconds apart):
>
> PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
> 5837 informix 2796M 2365M sleep 59 -10 8:36:57 0.4% oninit/17
> 5844 informix 2796M 2350M sleep 52 -10 8:18:56 0.5% oninit/19
> 5851 informix 2796M 2350M sleep 59 -10 5:09:33 0.1% oninit/18
> 5852 informix 2796M 2349M sleep 59 -10 4:35:21 0.0% oninit/19
> 5846 informix 2796M 2347M sleep 59 -10 7:44:28 0.5% oninit/18
> 5847 informix 2796M 2346M sleep 59 -10 7:25:23 0.6% oninit/17
> 5845 informix 2796M 2346M sleep 52 -10 7:58:06 0.4% oninit/18
> 5850 informix 2796M 2345M cpu38 30 -10 6:20:15 0.5% oninit/21
> 5848 informix 2796M 2343M sleep 59 -10 7:08:30 0.6% oninit/17
> 5849 informix 2796M 2343M sleep 59 -10 6:49:08 0.4% oninit/18
> 5853 informix 2796M 2342M sleep 59 -10 4:14:47 0.0% oninit/19
> 5857 informix 2796M 2337M sleep 59 -10 3:19:36 0.0% oninit/17
> 5854 informix 2796M 2334M sleep 59 -10 3:52:52 0.0% oninit/17
> 5855 informix 2796M 2334M sleep 59 -10 3:34:57 0.0% oninit/18
> 5856 informix 2796M 2332M sleep 59 -10 3:26:08 0.0% oninit/19
> =====
>
> PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
> 5837 informix 2796M 2365M sleep 59 -10 8:36:57 0.3% oninit/17
> 5844 informix 2796M 2350M sleep 30 -10 8:18:58 0.6% oninit/19
> 5851 informix 2796M 2350M sleep 59 -10 5:09:33 0.1% oninit/18
> 5852 informix 2796M 2349M sleep 59 -10 4:35:21 0.0% oninit/19
> 5846 informix 2796M 2347M sleep 59 -10 7:44:29 0.5% oninit/18
> 5847 informix 2796M 2346M sleep 59 -10 7:25:24 0.5% oninit/17
> 5845 informix 2796M 2346M sleep 50 -10 7:58:07 0.4% oninit/18
> 5850 informix 2796M 2345M cpu37 40 -10 6:20:17 0.6% oninit/21
> 5848 informix 2796M 2343M sleep 59 -10 7:08:30 0.5% oninit/17
> 5849 informix 2796M 2343M sleep 52 -10 6:49:08 0.4% oninit/18
> 5853 informix 2796M 2342M sleep 59 -10 4:14:47 0.0% oninit/19
> 5857 informix 2796M 2337M sleep 59 -10 3:19:36 0.0% oninit/17
> 5854 informix 2796M 2334M sleep 59 -10 3:52:52 0.0% oninit/17
> 5855 informix 2796M 2334M sleep 59 -10 3:34:57 0.0% oninit/18
> 5856 informix 2796M 2332M sleep 59 -10 3:26:08 0.0% oninit/19
> =====
>
> An "onstat -g rsm sendq | grep queue" 60 seconds apart produced the
> following:
>
> Txns in queue: 13438400
> Log Events in queue: 39
> Size of Data in queue: 7601814074 Bytes
> Total data queued: 7010242960 Bytes
> Total Txns queued: 18842496
> ===
> Txns in queue: 13435039
> Log Events in queue: 39
> Size of Data in queue: 7599625995 Bytes
> Total data queued: 7010242960 Bytes
> Total Txns queued: 18842496
> ===
>
> Currently, it looks like 2,188,079 Bytes per minute are transferring to the
> backup. With 7,601,814,074 Bytes in the queue, looks like it will take
> another
> 3474 minutes, or about 57 more hours.
>
> Once again, I see that the data is getting replicated, it's just happening
> very slowly.
>
> Any suggestions on how to tune my Solaris box would be appreciated.
>
> Thanks,
> Jeff
>
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--e89a8f22beb964ef5a04d8725824
Not sure how you would tell. uname -a yields: SunOS astro 5.10 Generic_142909-17 sun4v sparc SUNW,SPARC-Enterprise-T5220 If T5220 is the class, then I guess it is a T class machine.
Here it is. I'm not a DBA, so I'm not sure what this is telling me:
=> onstat -z
IBM Informix Dynamic Server Version 11.70.FC5GE -- On-Line (CKPT INP) -- Up 7
days 04:27:34 -- 27
=> onstat -g rcv full
IBM Informix Dynamic Server Version 11.70.FC5GE -- On-Line (CKPT INP) -- Up 7
days 04:38:21 -- 27ServerId: 50
Flags 0x0
Threads:
Id Name State Handle
99 CDRACK_1 Idle 0x700000084ac2a08
98 CDRACK_0 Idle 0x700000084918d50
Receive Manager global block 0x700000084946028
cdrRM_inst_ct: 1
cdrRM_State: 00000040
cdrRM_numSleepers: 11
cdrRM_DsCreated: 12
cdrRM_MinDSThreads: 3
cdrRM_MaxDSThreads: 12
cdrRM_DSBlock 0
cdrRM_DSParallelPL 0
cdrRM_DSFailRate 0.000000
cdrRM_DSNumRun: 540294
cdrRM_DSNumLockTimeout 0
cdrRM_DSNumLockRB 0
cdrRM_DSNumDeadLocks 0
cdrRM_DSNumPCommits 374972
cdrRM_ACKwaiting 37
cdrRM_totSleep: 375
cdrRM_Sleeptime: 852
cdrRM_Workload: 41567
cdrRM_optscale: 4
cdrRM_MinFloatThreads: 2
cdrRM_MaxFloatThreads: 7
cdrRM_AckThreadCount: 2
cdrRM_AckWaiters: 2
cdrRM_AckCreateStamp: Thu Mar 14 09:17:18 2013
cdrRM_DSCreateStamp: Thu Mar 21 13:53:36 2013
cdrRM_acksInList: 0
cdrRM_BlobErrorBufs: 0
Receive Parallelism Statistics
Server Tot.Txn. Pending Active MaxPnd MaxAct AvgPnd AvgAct CommitRt
50 540294 15513 1 21433 12 2475.01 6.34 0.87
Tot Pending:15513 Tot Active:1 Avg Pending:2475.01 Avg Active:6.34
Commit Rate:0.87
Time Spent In RM Parallel Pipeline Levels
Lev. TimeInSec Pcnt.
0 645 100.00%
1 0 0.00%
2 0 0.00%
Statistics by Source
Server 50
Repl Txn Ins Del Upd Last Target Apply Last Source Commit
3276970 684 537 0 147 2013/03/21 13:55:07 2013/03/16 03:56:32
3276976 537 0 0 537 2013/03/21 13:55:06 2013/03/16 03:56:31
3277461 4000 4000 0 0 2013/03/21 13:53:19 2013/03/16 03:53:50
3277464 4000 4000 0 0 2013/03/21 13:53:30 2013/03/16 03:54:11
3277467 3681 3681 0 0 2013/03/21 13:55:07 2013/03/16 03:56:33
3277470 3337 3335 1988 0 2013/03/21 13:55:07 2013/03/16 03:56:33
3277473 4000 4000 0 0 2013/03/21 13:53:55 2013/03/16 03:54:56
3277476 4000 4000 0 0 2013/03/21 13:54:37 2013/03/16 03:56:07
3277479 3841 3840 536 0 2013/03/21 13:52:50 2013/03/16 03:52:57
3277482 8 0 7112 0 2013/03/21 13:45:03 2013/03/16 03:37:56
3277485 3440 3439 240 0 2013/03/21 13:55:07 2013/03/16 03:56:33
3277488 8 0 7112 0 2013/03/21 13:45:03 2013/03/16 03:37:55
3277491 3681 3677 3072 0 2013/03/21 13:55:07 2013/03/16 03:56:33
3277494 4009 4000 8118 0 2013/03/21 13:53:05 2013/03/16 03:53:21
3277497 4005 4000 4020 0 2013/03/21 13:53:20 2013/03/16 03:53:55
3277500 4008 4000 7824 0 2013/03/21 13:55:02 2013/03/16 03:56:29
3277503 3 0 2880 0 2013/03/21 13:45:17 2013/03/16 03:38:45
3277506 4010 4000 9036 0 2013/03/21 13:53:06 2013/03/16 03:53:25
3277509 3845 3840 4404 0 2013/03/21 13:55:07 2013/03/16 03:56:33
3277512 4010 4000 9264 0 2013/03/21 13:53:11 2013/03/16 03:53:35
3277515 4004 4000 3113 0 2013/03/21 13:53:04 2013/03/16 03:53:21
3277518 7 0 6156 0 2013/03/21 13:45:56 2013/03/16 03:40:26
3277521 3685 3684 528 0 2013/03/21 13:55:07 2013/03/16 03:56:33
3277525 3893 3887 5868 0 2013/03/21 13:52:53 2013/03/16 03:53:01
3277527 4002 4000 1104 0 2013/03/21 13:54:03 2013/03/16 03:55:14
3277530 4003 3996 6588 0 2013/03/21 13:52:53 2013/03/16 03:53:01
3277533 3299 3298 828 0 2013/03/21 13:55:07 2013/03/16 03:56:33
3277536 3734 3730 3322 0 2013/03/21 13:55:07 2013/03/16 03:56:33
3277539 4003 4000 2304 0 2013/03/21 13:54:04 2013/03/16 03:55:16
3277542 4001 4000 828 0 2013/03/21 13:52:59 2013/03/16 03:53:13
3277545 4002 4000 1176 0 2013/03/21 13:54:43 2013/03/16 03:56:12
3277548 3 0 2662 0 2013/03/21 13:46:19 2013/03/16 03:41:11
3277551 4003 4000 2388 0 2013/03/21 13:54:11 2013/03/16 03:55:26
3277555 4002 4000 1116 0 2013/03/21 13:54:37 2013/03/16 03:56:07
3277558 4002 4000 1397 0 2013/03/21 13:55:05 2013/03/16 03:56:30
3277560 3727 3724 2820 0 2013/03/21 13:55:07 2013/03/16 03:56:33
3277563 4003 4000 2277 0 2013/03/21 13:54:18 2013/03/16 03:55:39
3277566 4002 4000 1188 0 2013/03/21 13:54:09 2013/03/16 03:55:24
3277569 4002 4000 1668 0 2013/03/21 13:53:29 2013/03/16 03:54:09
3277572 1 0 22 0 2013/03/21 13:46:52 2013/03/16 03:41:52
3277575 4002 4000 1116 0 2013/03/21 13:53:44 2013/03/16 03:54:37
3277578 1 0 572 0 2013/03/21 13:47:08 2013/03/16 03:42:16
3277581 1 0 672 0 2013/03/21 13:47:09 2013/03/16 03:42:16
3277584 3721 3720 372 0 2013/03/21 13:55:07 2013/03/16 03:56:33
3277587 4003 4000 2002 0 2013/03/21 13:54:20 2013/03/16 03:55:43
3277590 4002 4000 1164 0 2013/03/21 13:52:58 2013/03/16 03:53:09
3277593 4001 4000 638 0 2013/03/21 13:53:02 2013/03/16 03:53:19
3277596 3676 3674 1704 0 2013/03/21 13:52:28 2013/03/16 03:52:27
3277599 3958 3956 1488 0 2013/03/21 13:55:07 2013/03/16 03:56:33
3277602 1 0 583 0 2013/03/21 13:47:36 2013/03/16 03:43:08
3277605 4003 4000 2244 0 2013/03/21 13:54:27 2013/03/16 03:55:55
3277608 4002 4000 1441 0 2013/03/21 13:53:20 2013/03/16 03:53:51
3277611 3847 3846 432 0 2013/03/21 13:52:46 2013/03/16 03:52:53
3277614 4005 4000 4896 0 2013/03/21 13:53:37 2013/03/16 03:54:26
3277618 3912 3911 528 0 2013/03/21 13:52:46 2013/03/16 03:52:53
3277620 4004 4000 3732 0 2013/03/21 13:53:05 2013/03/16 03:53:24
3277624 4003 4000 2112 0 2013/03/21 13:54:01 2013/03/16 03:55:11
3277626 4002 4000 1080 0 2013/03/21 13:53:00 2013/03/16 03:53:15
3277629 3532 3531 44 0 2013/03/21 13:55:07 2013/03/16 03:56:33
3277632 3595 3591 3552 0 2013/03/21 13:55:07 2013/03/16 03:56:33
3277635 4002 4000 1034 0 2013/03/21 13:53:35 2013/03/16 03:54:22
3277638 4001 4000 744 0 2013/03/21 13:53:39 2013/03/16 03:54:31
3277641 3849 3848 484 0 2013/03/21 13:52:50 2013/03/16 03:52:58
3277644 4001 4000 720 0 2013/03/21 13:53:32 2013/03/16 03:54:14
3277647 4003 4000 2816 0 2013/03/21 13:53:20 2013/03/16 03:53:54
3277650 4001 4000 108 0 2013/03/21 13:54:49 2013/03/16 03:56:18
3277654 4001 4000 803 0 2013/03/21 13:53:51 2013/03/16 03:54:51
3277656 4003 4000 2596 0 2013/03/21 13:52:52 2013/03/16 03:53:00
3277659 4002 4000 1884 0 2013/03/21 13:53:36 2013/03/16 03:54:23
3277662 3715 3711 3960 0 2013/03/21 13:53:02 2013/03/16 03:53:18
3277665 3772 3769 2472 0 2013/03/21 13:55:07 2013/03/16 03:56:33
3277668 4001 4000 517 0 2013/03/21 13:53:24 2013/03/16 03:54:02
3277672 3650 3646 3212 0 2013/03/21 13:55:07 2013/03/16 03:56:33
3277674 4003 4000 2992 0 2013/03/21 13:54:58 2013/03/16 03:56:25
3277677 4003 4000 2354 0 2013/03/21 13:53:30 2013/03/16 03:54:10
3277680 4002 4000 1416 0 2013/03/21 13:54:07 2013/03/16 03:55:21
3277683 4 0 3552 0 2013/03/21 13:50:32 2013/03/16 03:48:46
3277686 4002 4000 1331 0 2013/03/21 13:53:57 2013/03/16 03:55:02
3277689 3777 3776 504 0 2013/03/21 13:55:07 2013/03/16 03:56:33
3277692 4004 4000 3619 0 2013/03/21 13:53:48 2013/03/16 03:54:43
3277695 3958 3956 1752 0 2013/03/21 13:52:50 2013/03/16 03:52:58
3277698 4003 4000 2145 0 2013/03/21 13:53:23 2013/03/16 03:54:00
3277701 4001 4000 792 0 2013/03/21 13:53:28 2013/03/16 03:54:07
3277704 4002 4000 1661 0 2013/03/21 13:53:39 2013/03/16 03:54:29
3277707 3470 3469 561 0 2013/03/21 13:55:07 2013/03/16 03:56:33@@N
Related threads
- Posting from the Informix-list
- Migrating from IDS 9.40.UC6 to 11.50.UC3
- Ip for a network session
- questions onstat -g