iostat , sun ufs io rate

Dear Informix Community, Below is my posting to both oracle and sun newsgroups . I need a clarification about iostat . Since i did not get any response from those newsgroups , i am posting here hoping that you can help me. ***I am sorry for this posting , i dont mean to disturb anyone , i just need information. Below is the outputs of iostat , vmstat and sar -b at the time of a sample oracle database query . ( The running query is the only activity .) The file system sd81 is ufs and its maxcontig is 128 , (which is 1mb. cluster size (8kb.page size * 128) iostat -x 1 extended device statistics device r/s w/s kr/s kw/s wait actv svc_t %w %b sd0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 sd16 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 sd81 898.0 0.0 37552.0 0.0 0.0 8.0 8.9 3 99 vmstat 1 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr s0 s1 s8 sd in sy cs us sy id 0 7 0 5043904 2781624 5102 0 37790 0 0 0 0 0 0 944 6 2215 1002 2393 21 20 58 0 7 0 5043904 2782792 6064 0 44388 0 0 0 0 0 0 1255 0 2691 928 2996 23 20 57 1 3 0 5044184 2782968 5730 0 42059 0 0 0 0 0 0 1204 3 2529 815 2884 19 18 63 0 5 0 5044760 2784800 4756 0 35113 0 0 0 0 0 0 1151 0 2465 824 2789 18 17 65 sar -b 1 1000 SunOS verdenfs1 5.9 Generic_117171-12 sun4u 05/31/2005 11:16:40 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s 11:17:35 0 3235 100 0 0 100 00 11:17:36 0 4712 100 0 0 100 00 11:17:37 0 5135 100 0 0 100 00 11:17:38 0 6558 100 0 0 100 00 11:17:39 0 7477 100 0 0 100 00 11:17:40 0 6668 100 0 0 100 00 These are my questions: 1. Although read cache is always %100 , i see large values in pi of vmstat.How could this be possible? The table size is about 1gb. How could my system achive %100 read cache for this 1mb? 2. In iostat output , i see that for each read , my system can read 37552.0KB / 898.0 = 41KB. . But for this file system when i issue a simple mkfile command i can see 800KB. read per second. Why cant the dbserver achive the same io rate? ( By the way , i have tuned the oracle to issue large io . It issues 1mb. reads to the operating system . ) These are the outputs of basic file write and read commands. The io values are very high here: command issued:mkfile 4000M test vmstat 1: kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr s0 s1 s8 sd in sy cs us sy id 0 0 0 4608480 2271488 0 0 0 0 0 0 0 0 0 62 0 528 8197 347 12 23 64 0 0 0 4608480 2227264 0 0 0 0 0 0 0 0 0 74 3 549 9523 367 9 29 62 0 0 0 4608480 2225520 0 12 0 0 0 0 0 0 0 67 0 527 8770 359 15 29 56 0 0 0 4608480 2224496 0 0 0 0 0 0 0 0 0 77 0 543 8567 344 7 29 64 sar -b 1 1000: 16:35:38 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s 16:35:58 0 25394 100 6 13670 100 00 16:35:59 0 35795 100 4 17826 100 00 16:36:00 0 35320 100 5 17591 100 00 16:36:01 0 32713 100 41 16325 100 00 16:36:02 0 32425 100 4 16144 100 00 16:36:03 0 29074 100 3 14480 100 00 16:36:04 0 37605 100 5 18734 100 00 16:36:05 0 30907 100 41 15433 100 00 16:36:06 0 33631 100 4 16749 100 00 16:36:07 0 38721 100 4 19289 100 00 iostat -x 1: extended device statistics device r/s w/s kr/s kw/s wait actv svc_t %w %b sd0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 sd16 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 sd81 0.0 77.0 0.0 69282.0 0.0 13.3 172.7 0 100 ssd0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 899KB per write But why is the %wcache is 100 although i have written 60mb. per second? ------- command:dd if=test of=/dev/null bs=4096 sar -b 1 1000: bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s 17:05:00 0 38 100 0 0 100 00 17:05:01 0 92 100 0 2 100 00 17:05:02 0 42 100 0 0 100 00 17:05:03 0 40 100 0 0 100 00 17:05:04 0 40 100 0 0 100 00 17:05:05 0 59 100 0 0 100 00 17:05:06 0 386 100 0 3 100 00 17:05:07 0 39 100 0 0 100 00 vmstat 1: kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr s0 s1 s8 sd in sy cs us sy id 0 0 0 4600976 2066696 0 0 18250 0 0 0 0 0 0 18 0 312 9640 311 0 6 94 0 0 0 4600976 2066696 81 350 20277 24 24 0 0 0 0 20 3 337 12052 389 1 10 88 0 0 0 4600976 2070472 0 0 21291 0 0 0 0 0 0 21 0 313 11014 285 0 8 92 iostat -x 1 extended device statistics device r/s w/s kr/s kw/s wait actv svc_t %w %b sd0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 sd16 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 sd81 19.0 0.0 19456.8 0.0 0.0 1.9 99.0 0 100 1mb. per read Kind Regards, hope

hopehope_123 wrote: > > Dear Informix Community, > > Below is my posting to both oracle and sun newsgroups . I need a > clarification about iostat . Since i did not get any response from > those newsgroups , i am posting here hoping that you can help me. > > ***I am sorry for this posting , i dont mean to disturb anyone , i > just need information. > > > > > Below is the outputs of iostat , vmstat and sar -b at the time of a > sample oracle database query . ( The running query is the only > activity .) > > > The file system sd81 is ufs and its maxcontig is 128 , (which is 1mb. > cluster size (8kb.page size * 128) > > > iostat -x 1 > extended device statistics > device r/s w/s kr/s kw/s wait actv svc_t %w %b > sd0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 > sd16 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 > sd81 898.0 0.0 37552.0 0.0 0.0 8.0 8.9 3 99 > > > vmstat 1 > > > kthr memory page disk faults > cpu > r b w swap free re mf pi po fr de sr s0 s1 s8 sd in sy cs > us sy id > 0 7 0 5043904 2781624 5102 0 37790 0 0 0 0 0 0 944 6 2215 1002 2393 > 21 20 58 > 0 7 0 5043904 2782792 6064 0 44388 0 0 0 0 0 0 1255 0 2691 928 2996 > 23 20 57 > 1 3 0 5044184 2782968 5730 0 42059 0 0 0 0 0 0 1204 3 2529 815 2884 > 19 18 63 > 0 5 0 5044760 2784800 4756 0 35113 0 0 0 0 0 0 1151 0 2465 824 2789 > 18 17 65 > > > sar -b 1 1000 > SunOS verdenfs1 5.9 Generic_117171-12 sun4u 05/31/2005 > > > 11:16:40 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s > pwrit/s > > > 11:17:35 0 3235 100 0 0 100 0> 0 > 11:17:36 0 4712 100 0 0 100 0> 0 > 11:17:37 0 5135 100 0 0 100 0> 0 > 11:17:38 0 6558 100 0 0 100 0> 0 > 11:17:39 0 7477 100 0 0 100 0> 0 > 11:17:40 0 6668 100 0 0 100 0> 0 > > > These are my questions: > > > 1. Although read cache is always %100 , i see large values in pi of > vmstat.How could this be possible? The table size is about 1gb. > How could my system achive %100 read cache for this 1mb? Pi = page in, you will always see activity here, it's how Solaris works, don't get hung up on pi/po, concentrate on de and sr, these give you some indication of memory shortfall/problems. sd81 is 99%, 20% is a good target. You need to provide the FS details before anyhting meaningful can be deduced. From above vmstat you have 17 jobs on the ready queue, i.e. awaiting CPU and 20 blocked on non CPU resource. Personally I'd run vmstat with a large time interval say, 30 seconds, and then look at the nubmers. sr/de = 0, so it's unlikely to be memory related Gut feel, poorly laid out disk system and/or missing indexes. Check out the Sun performance books by Adrian Cockcroft, and the internals books by Jim Mauro > > > 2. In iostat output , i see that for each read , my system can read > 37552.0KB / 898.0 = 41KB. . But for this file system when i issue a > simple mkfile command i can see 800KB. read per second. Why cant the > dbserver achive the same io rate? ( By the way , i have tuned the > oracle to issue large io . It issues 1mb. reads to the operating system > > > > . ) > > > These are the outputs of basic file write and read commands. The io > values are very high here: > > > command issued:mkfile 4000M test > > > vmstat 1: > kthr memory page disk faults > cpu > r b w swap free re mf pi po fr de sr s0 s1 s8 sd in sy > > > > cs us sy id > 0 > > > 0 0 4608480 2271488 0 0 0 0 0 0 0 0 0 62 0 528 8197 347 > 12 23 64 > 0 0 0 4608480 2227264 0 0 0 0 0 0 0 0 0 74 3 549 9523 367 > 9 29 62 > 0 0 0 4608480 2225520 0 12 0 0 0 0 0 0 0 67 0 527 8770 359 > 15 29 56 > 0 0 0 4608480 2224496 0 0 0 0 0 0 0 0 0 77 0 543 8567 344 > 7 29 64 > > > sar -b 1 1000: > 16:35:38 > > > bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s > pwrit/s > > > 16:35:58 0 25394 100 6 13670 100 0> 0 > 16:35:59 0 35795 100 4 17826 100 0> 0 > 16:36:00 0 35320 100 5 17591 100 0> 0 > 16:36:01 0 32713 100 41 16325 100 0> 0 > 16:36:02 0 32425 100 4 16144 100 0> 0 > 16:36:03 0 29074 100 3 14480 100 0> 0 > 16:36:04 0 37605 100 5 18734 100 0> 0 > 16:36:05 0 30907 100 41 15433 100 0> 0 > 16:36:06 0 33631 100 4 16749 100 0> 0 > 16:36:07 0 38721 100 4 19289 100 0> 0 > > > iostat -x 1: > extended device statistics > device r/s w/s kr/s kw/s wait actv svc_t %w %b > sd0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 > sd16 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 > sd81 > > > 0.0 77.0 0.0 69282.0 0.0 13.3 172.7 0 100 > ssd0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 > > > 899KB per write > > > But why is the %wcache is 100 although i have written 60mb. per second? > > > > ------- > command:dd if=test of=/dev/null bs=4096 > > > sar -b 1 1000: > bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s > > > > pwrit/s > 17:05:00 0 38 100 0 0 100 0> 0 > 17:05:01 0 92 100 0 2 100 0> 0 > 17:05:02 0 42 100 0 0 100 0> 0 > 17:05:03 0 40 100 0 0 100 0> 0 > 17:05:04 0 40 100 0 0 100 0> 0 > 17:05:05 0 59 100 0 0 100 0> 0 > 17:05:06 0 386 100 0 3 100 0> 0 > 17:05:07 0 39 100 0 0 100 0> 0 > > > vmstat 1: > > > kthr memory page disk faults > cpu > r b w swap free re mf pi po fr de sr s0 s1 > s8 sd in sy cs us sy id > 0 0 0 4600976 2066696 0 0 18250 0 0 0 0 0 0 18 0 > 312 9640 311 0 6 94 > 0 0 0 4600976 2066696 81 350 20277 24 24 0 0 0 0 20 3 337 > > > > 12052 389 1 10 88 > 0 0 0 4600976 2070472 0 0 21291 0 0 0 0 0 0 21 0 > 313 11014 285 0 8 92 > > > iostat -x 1 > extended device statistics > device r/s w/s kr/s kw/s wait actv svc_t %w %b > sd0 0

Hi Paul , Thank you very much for your mail. r b w swap free re mf pi po fr de sr s0 s1 s8 sd in sy cs us sy id > 0 7 0 5043904 2781624 5102 0 37790 0 0 0 0 0 0 944 6 2215 1002 2393 21 20 58 > 0 7 0 5043904 2782792 6064 0 44388 0 0 0 0 0 0 1255 0 2691 928 2996 23 20 57 > 1 3 0 5044184 2782968 5730 0 42059 0 0 0 0 0 0 1204 3 2529 815 2884 19 18 63 > 0 5 0 5044760 2784800 4756 0 35113 0 0 0 0 0 0 1151 0 2465 824 2789 18 17 65 I think we have some problems with the number just because of the display format. Based on the vmstat output , there exists 7-7-3-5 blocked processes for each of the 1 sec. intervals. There is very little activity going on this server , and basically when i issue the mkfile command , isnt it better to have 100% than 20% ? Kind Regards, tolga

hopehope_123 wrote: > Hi Paul , > > Thank you very much for your mail. > > r b w swap free re mf pi po fr de sr s0 s1 s8 sd in sy cs > us sy id > >> 0 7 0 5043904 2781624 5102 0 37790 0 0 0 0 0 0 944 6 2215 1002 2393 21 20 58 >> 0 7 0 5043904 2782792 6064 0 44388 0 0 0 0 0 0 1255 0 2691 928 2996 23 20 57 >> 1 3 0 5044184 2782968 5730 0 42059 0 0 0 0 0 0 1204 3 2529 815 2884 19 18 63 >> 0 5 0 5044760 2784800 4756 0 35113 0 0 0 0 0 0 1151 0 2465 824 2789 18 17 65 > > > I think we have some problems with the number just because of the > display format. Based on the vmstat output , > there exists 7-7-3-5 blocked processes for each of the 1 sec. > intervals. > > There is very little activity going on this server , and basically when > i issue the mkfile command , isnt it better to have 100% than 20% ? > > Kind Regards, > tolga > If that is the only thing the box is doing then maybe 100% is best, but that will never ever be the case, even in single user mode you will continously have processes switching in and out. The normal guidelines are 20-25%. The above vmstat, show processes block waiting on non-CPU resources, and once a process waiting on a CPU. I think you are disk bound somewhere. -- Paul Watson # Oninit Ltd # Growing old is mandatory Tel: +44 1436 672201 # Growing up is optional Fax: +44 1436 678693 # Mob: +44 7818 003457 # www.oninit.com #