Re: dbimport - 512MB RAM not enough?

More on this, based on data I gathered in tonight's attempt: On Tue, 10 Mar 1998, David Williams wrote: > Probably because AIX is caching writes, hence the buffer is > dynamically growing as needed. As more memory is needed by > applications the buffer cache will skrink again. This is what they > mean by a unified cache. If that's AIX caching it, I'd expect (hope?) that it'd also use swap space - but it doesn't seem to here; the swap space is never fully exhausted when it crashes. (Regarding the write activity I see to the root dbspace device) > That would be the reseved pages and getting updated. Could it be more than that? Maybe also temporary tables, etc? Or maybe (as Wes Loftis has proposed) logs (even though they are being continuously written to tape - maybe the tape can't keep up? However, tonight I even tried sending the logs to /dev/null - made no difference. > DBTEMP under 5.x > DBSPACETEMP under 7.x []> DBSPACETEMP and temporary dbspaces ONLY EXIST under 7.x NOT under > 5.x. 5.x uses DBTEMP. Does that mean that the functions which use the temp dbspace under 7 (as distinct from the filesystem /tmp writes affected by DBTEMP) can't be forced out of the root dbspace in 5.x? Or should NPROC_DBTEMP affect this in 5.x - I have tried setting it to a filesystem with plenty of space - no effect; nothing is ever written there at all. > Run df -k before starting the dbimport. After a few minutes run > another df -k. What changes. Essentially nothing - no significant change in any mounted filesystem; /tmp and also the filesystem specified by DBTEMP seem to be completely unused. > Same with onstat -d. what changes? Again, not much - the number of free pages in the target dbspace of the import declines as the tables are loaded; the rootdbs and other dbspaces show exactly the same numbers of free pages all throughout the attempt. > Run > > ps -ef | grep dbimport > > what is the size of the dbimport process? It fluctuates wildly - from an initial 1700 K to a max of over 40MB, up and down while the import proceeds. This does NOT correlate with overall memory use, which steadily increases until it crashes. > Run swap -l, how much swap is free? Initially, almost all of it; by the time it crashes, still more than 130MB free there. As with real memory, the amount of free swap space gradually but steadily declines. (One note - this machine has less than an ideal amount of swap for the large amount of RAM, but even with the limited amount available, at least 130MB is always free throughout each of my attempts.) Below are snapshots from AIX monitor (a "top"-like utility) and from tbstat -d, at various points throughout tonight's attempt - maybe someone will see something here I'm missing. What bothers me most is the large amount of write activity to my disk1 (and to some extent disk2), even though the target dbspace for the import is on disk4 and its source is on disk5 -- the rootdbs is on disk1, but this attempt had continuous logging on - pointing at /dev/null; further, the stats for used and free pages in rootdbs never change at all, and the filesystem usage doesn't change either. Regards, Don Miller -------------------- Load averages: 0.80, 0.72, 0.70 ouzo Wed Mar 11 19:29:26 1998 Cpu states: 55.7% user 6.9% system 37.4% wait 0.0% idle Logged on: 4 users 2 active 4 remote 04:00 sleep time Real memory: 111.9M procs 103.8M files 296.3M free 512.0M total Virtual memory: 62.6M used 345.4M free 408.0M total DiskIO read write rsize wsize xfers seeks blksize xrate busy hdisk0 0.0 138.0 kB/s 0.0 30.0 kB 4.6 0.0 512 0 10% hdisk1 0.0 134.8 kB/s 0.0 29.3 kB 4.6 0.0 512 0 9% hdisk2 0.0 0.0 kB/s 0.0 0.0 kB 0.0 0.0 512 0 0% hdisk3 0.0 13.6 kB/s 0.0 10.5 kB 1.3 0.0 512 0 1% hdisk4 90.0 188.4 kB/s 1.5 3.2 kB 58.3 0.0 512 0 71% hdisk5 53.2 0.8 kB/s 19.0 0.3 kB 2.8 0.0 512 0 2% cd0 0.0 0.0 kB/s 0.0 0.0 kB 0.0 0.0 512 0 0% Total 143.2 475.6 kB/s 2.0 6.6 kB 71.6 0.0 512 0 93% Active disks: 5 PID USER PRI NICE SIZE RES STAT TIME CPU% COMMAND 516 root 127 21 16K 20K run 6737 37.9/67.4 Kernel (wait) 41450 informix 70 0 1595K 1584K Fslp 2:38 29.8/27.5 sqlturbo 21224 informix 87 0 20M 20M Fslp 3:20 29.3/34.7 dbimport 48424 don 60 0 498K 584K Frun 0:09 1.0/ 1.0 monitor 31426 root 60 0 327K 424K slp 0:01 0.5/ 0.1 rlogind 24098 informix 60 0 803K 776K Fslp 0:02 0.5/ 0.3 tbinit 13604 informix 60 0 803K 772K Fslp 0:02 0.4/ 0.3 tbinit 61222 informix 60 0 803K 768K Frun 0:02 0.4/ 0.3 tbinit Load averages: 0.81, 0.71, 0.69 ouzo Wed Mar 11 19:34:16 1998 Cpu states: 60.8% user 6.4% system 32.8% wait 0.0% idle Logged on: 4 users 3 active 4 remote 05:13 sleep time Real memory: 135.0M procs 116.0M files 261.0M free 512.0M total Virtual memory: 85.7M used 322.3M free 408.0M total DiskIO read write rsize wsize xfers seeks blksize xrate busy hdisk0 0.0 208.0 kB/s 0.0 32.0 kB 6.5 0.0 512 0 14% hdisk1 0.0 211.2 kB/s 0.0 32.5 kB 6.5 0.0 512 0 12% hdisk2 0.0 0.0 kB/s 0.0 0.0 kB 0.0 0.0 512 0 0% hdisk3 0.0 0.0 kB/s 0.0 0.0 kB 0.0 0.0 512 0 0% hdisk4 96.0 194.4 kB/s 1.9 3.8 kB 51.7 0.0 512 0 75% hdisk5 57.6 0.0 kB/s 32.0 0.0 kB 1.8 0.0 512 0 1% cd0 0.0 0.0 kB/s 0.0 0.0 kB 0.0 0.0 512 0 0% Total 153.6 613.5 kB/s 2.3 9.2 kB 66.5 0.0 512 0 102% Active disks: 4 PID USER PRI NICE SIZE RES STAT TIME CPU% COMMAND 21224 informix 80 0 43M 43M Frun 4:40 39.4/32.3 dbimport 516 root 127 21 16K 20K run 6739 33.5/67.3 Kernel (wait) 41450 informix 73 0 1751K 1740K Fslp 4:12 24.3/29.1 sqlturbo 48424 don 60 0 498K 584K Frun 0:12 1.0/ 1.0 monitor 13604 informix 60 0 807K 776K Fslp 0:04 0.7/ 0.3 tbinit 61222 informix 60 0 803K 768K Fslp 0:03 0.5/ 0.3 tbinit 24098 informix 60 0 803K 776K Fslp 0:03 0.3/ 0.3 tbinit 1032 root 37 21 56K 64K slp 1:21 0.1/ 0.0 Kernel (gil) Load averages: 0.83, 0.74, 0.71 ouzo Wed Mar 11 19:38:16 1998 Cpu states: 56.7% user 6.6% system 36.7% wait 0.0% idle Logged on: 4 users 2 active 4 remote 06:00 sleep time Real memory: 156.1M procs 127.6M files 228.3M free 512.0M total Virtual memory: 107.5M used 300.5M free 408.0M total DiskIO read write rsize wsize xfers seeks blksize xrate busy hdisk0 0.0 214.5 kB/s 0.0 31.5 kB 6.8 0.0 512 0 15% hdisk1 0.0 214.5 kB/s 0.0 32.0 kB 6.7 0.0 512 0 14% hdisk2 0.0 0.0 kB/s 0.0 0.0 kB 0.0 0.0 512 0 0% hdisk3 0.0 0.0 kB/s 0.0 0.0 kB 0.0 0.0 512 0 0% hdisk4 102.4 208.1