ReadyNAS Ultra4 Plusに触ってみた その3
主な利用用途であるiSCSIとNFSのベンチマークを比較したい。まずはiSCSIから。
マルチパスについて
iSCSIブートはdm-multipathと相性が悪いのでHDD起動のサーバに2ポートの1000BASE-Tを取り付けてReadyNASのLAN1、LAN2とそれぞれ直接接続。
iSCSIの設定とdm-multipathの設定を行った。
# multipath -l mpathe (360014052e419090150ad002000000000) dm-0 LIO-ORG,FILEIO size=20G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=0 status=active |- 45:0:0:0 sdg 8:96 active undef running `- 44:0:0:0 sdi 8:128 active undef running mpathd (360014052e419090150ad001000000000) dm-3 LIO-ORG,FILEIO size=200G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=0 status=active |- 42:0:0:0 sde 8:64 active undef running `- 43:0:0:0 sdf 8:80 active undef running mpathc (360014052e419090150ad004000000000) dm-1 LIO-ORG,FILEIO size=20G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=0 status=active |- 40:0:0:0 sdc 8:32 active undef running `- 41:0:0:0 sdd 8:48 active undef running mpathf (360014052e419090150ad003000000000) dm-2 LIO-ORG,FILEIO size=20G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=0 status=active |- 47:0:0:0 sdj 8:144 active undef running `- 46:0:0:0 sdh 8:112 active undef running
/dev/sdcと/dev/sddが同一のiSCSIボリュームでそれぞれ別経路から見ていることになる。それをdm-multipathで/dev/mapper/mpathcとしている。
いろいろボリュームが見えているけど気にしない。別に異常では無い。
それではhdparmで簡単にベンチマーク。
# hdparm -tT /dev/sdc /dev/sdc: Timing cached reads: 25690 MB in 2.00 seconds = 12865.15 MB/sec Timing buffered disk reads: 296 MB in 3.02 seconds = 98.00 MB/sec # hdparm -tT /dev/sdd /dev/sdd: Timing cached reads: 26050 MB in 2.00 seconds = 13045.94 MB/sec Timing buffered disk reads: 312 MB in 3.01 seconds = 103.78 MB/sec # hdparm -tT /dev/mapper/mpathc /dev/mapper/mpathc: Timing cached reads: 25616 MB in 2.00 seconds = 12829.08 MB/sec Timing buffered disk reads: 256 MB in 3.02 seconds = 84.76 MB/sec
/dev/sdXに単独にアクセスした場合よりパフォーマンスが低くなっている。どうもマルチパス構成としてうまく動作していない模様。
ReadyNASの方が対応しないようになっていると思われる。
iSCSIのベンチマーク
iSCSIブート設定にしたサーバでReadyNASのiSCSIボリュームから起動してベンチマーク。なんでHDD起動からテストしてないかと言うと、本番設定をした後で思いついたから。
ext3でフォーマットした領域に対してfio 2.0.5を使ってベンチマークを実行。パラメータなんかは複眼中心さんを参考にさせていただきました。
シーケンシャルリード
[seqread] readwrite=read blocksize=1m size=1g directory=./ direct=1 loops=5
# fio seqread.fio
seqread: (g=0): rw=read, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
fio 2.0.5
Starting 1 process
Jobs: 1 (f=1): [R] [100.0% done] [30287K/0K /s] [28 /0 iops] [eta 00m:00s]
seqread: (groupid=0, jobs=1): err= 0: pid=7344
read : io=5120.0MB, bw=66234KB/s, iops=64 , runt= 79157msec
clat (msec): min=11 , max=528 , avg=15.46, stdev=17.15
lat (msec): min=11 , max=528 , avg=15.46, stdev=17.15
clat percentiles (msec):
| 1.00th=[ 12], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 13],
| 30.00th=[ 13], 40.00th=[ 13], 50.00th=[ 13], 60.00th=[ 13],
| 70.00th=[ 13], 80.00th=[ 14], 90.00th=[ 17], 95.00th=[ 29],
| 99.00th=[ 68], 99.50th=[ 120], 99.90th=[ 227]
bw (KB/s) : min= 9582, max=83136, per=100.00%, avg=67208.73, stdev=19041.60
lat (msec) : 20=91.35%, 50=6.91%, 100=1.11%, 250=0.53%, 500=0.08%
lat (msec) : 750=0.02%
cpu : usr=0.05%, sys=0.70%, ctx=5132, majf=0, minf=282
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=5120/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=5120.0MB, aggrb=66233KB/s, minb=67823KB/s, maxb=67823KB/s, mint=79157msec, maxt=79157msec
Disk stats (read/write):
sda: ios=12928/220, merge=12443/118, ticks=201216/19282, in_queue=220501, util=99.71%
シーケンシャルライト
[seqwrite] readwrite=write blocksize=1m size=1g directory=./ direct=1 loops=5
# fio seqwrite.fio
seqwrite: (g=0): rw=write, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
fio 2.0.5
Starting 1 process
seqwrite: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 1 (f=1): [W] [100.0% done] [0K/8355K /s] [0 /7 iops] [eta 00m:00s]
seqwrite: (groupid=0, jobs=1): err= 0: pid=8990
write: io=5120.0MB, bw=36115KB/s, iops=35 , runt=145173msec
clat (msec): min=13 , max=733 , avg=28.26, stdev=36.18
lat (msec): min=13 , max=733 , avg=28.35, stdev=36.18
clat percentiles (msec):
| 1.00th=[ 14], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 15],
| 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 21],
| 70.00th=[ 26], 80.00th=[ 33], 90.00th=[ 48], 95.00th=[ 76],
| 99.00th=[ 165], 99.50th=[ 223], 99.90th=[ 469]
bw (KB/s) : min= 2409, max=63109, per=100.00%, avg=37125.47, stdev=13231.65
lat (msec) : 20=59.77%, 50=30.78%, 100=6.23%, 250=2.75%, 500=0.37%
lat (msec) : 750=0.10%
cpu : usr=0.31%, sys=0.44%, ctx=5179, majf=0, minf=25
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=5120/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=5120.0MB, aggrb=36114KB/s, minb=36981KB/s, maxb=36981KB/s, mint=145173msec, maxt=145173msec
Disk stats (read/write):
sda: ios=2/25363, merge=0/1425, ticks=37/778774, in_queue=778822, util=99.51%
多重度4のシーケンシャルリード
[seqread4] readwrite=read blocksize=1m size=1g directory=./ direct=1 numjobs=4 loops=5
# fio seqread4.fio
seqread4: (g=0): rw=read, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
...
seqread4: (g=0): rw=read, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
fio 2.0.5
Starting 4 processes
seqread4: Laying out IO file(s) (1 file(s) / 1024MB)
seqread4: Laying out IO file(s) (1 file(s) / 1024MB)
seqread4: Laying out IO file(s) (1 file(s) / 1024MB)
seqread4: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 4 (f=4): [RRRR] [300.0% done] [0K/0K /s] [0 /0 iops] [eta 1158050441d:07hJobs: 4 (f=4): [RRRR] [inf% done] [0K/0K /s] [0 /0 iops] [eta 1158050441d:07h:0Jobs: 4 (f=4): [RRRR] [inf% done] [0K/0K /s] [0 /0 iops] [eta 1158050441d:07h:0Jobs: 4 (f=4): [RRRR] [inf% done] [0K/0K /s] [0 /0 iops] [eta 1158050441d:07h:0Jobs: 4 (f=4): [RRRR] [400.0% done] [0K/0K /s] [0 /0 iops] [eta 1158050441d:07hJobs: 4 (f=4): [RRRR] [inf% done] [963K/0K /s] [0 /0 iops] [eta 1158050441d:07hJobs: 4 (f=4): [RRRR] [inf% done] [5221K/0K /s] [4 /0 iops] [eta 1158050441d:07Jobs: 4 (f=4): [RRRR] [inf% done] [13577K/0K /s] [12 /0 iops] [eta 1158050441d:Jobs: 4 (f=4): [RRRR] [0.0% done] [7310K/0K /s] [6 /0 iops] [eta 09h:57m:13s] Jobs: 1 (f=1): [__R_] [98.9% done] [63708K/0K /s] [60 /0 iops] [eta 00m:05s]
seqread4: (groupid=0, jobs=1): err= 0: pid=9002
read : io=5120.0MB, bw=13432KB/s, iops=13 , runt=390342msec
clat (msec): min=11 , max=2952 , avg=76.23, stdev=143.01
lat (msec): min=11 , max=2952 , avg=76.23, stdev=143.01
clat percentiles (msec):
| 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 13],
| 30.00th=[ 13], 40.00th=[ 13], 50.00th=[ 14], 60.00th=[ 19],
| 70.00th=[ 30], 80.00th=[ 115], 90.00th=[ 318], 95.00th=[ 347],
| 99.00th=[ 424], 99.50th=[ 490], 99.90th=[ 1975]
bw (KB/s) : min= 346, max=30658, per=29.96%, avg=14279.41, stdev=6330.28
lat (msec) : 20=61.35%, 50=15.51%, 100=2.79%, 250=7.29%, 500=12.60%
lat (msec) : 750=0.31%, 1000=0.02%, 2000=0.04%, >=2000=0.10%
cpu : usr=0.01%, sys=0.13%, ctx=5218, majf=0, minf=282
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=5120/w=0/d=0, short=r=0/w=0/d=0
seqread4: (groupid=0, jobs=1): err= 0: pid=9003
read : io=5120.0MB, bw=12326KB/s, iops=12 , runt=425368msec
clat (msec): min=11 , max=5712 , avg=83.08, stdev=175.50
lat (msec): min=11 , max=5712 , avg=83.08, stdev=175.50
clat percentiles (msec):
| 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 13],
| 30.00th=[ 13], 40.00th=[ 15], 50.00th=[ 22], 60.00th=[ 29],
| 70.00th=[ 43], 80.00th=[ 124], 90.00th=[ 302], 95.00th=[ 343],
| 99.00th=[ 445], 99.50th=[ 840], 99.90th=[ 1549]
bw (KB/s) : min= 179, max=50712, per=28.47%, avg=13571.64, stdev=7380.59
lat (msec) : 20=48.46%, 50=24.39%, 100=5.72%, 250=9.14%, 500=11.50%
lat (msec) : 750=0.27%, 1000=0.10%, 2000=0.37%, >=2000=0.04%
cpu : usr=0.01%, sys=0.12%, ctx=5225, majf=0, minf=283
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=5120/w=0/d=0, short=r=0/w=0/d=0
seqread4: (groupid=0, jobs=1): err= 0: pid=9004
read : io=5120.0MB, bw=11917KB/s, iops=11 , runt=439947msec
clat (msec): min=11 , max=9006 , avg=85.92, stdev=309.61
lat (msec): min=11 , max=9006 , avg=85.92, stdev=309.61
clat percentiles (msec):
| 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 13],
| 30.00th=[ 13], 40.00th=[ 13], 50.00th=[ 15], 60.00th=[ 21],
| 70.00th=[ 32], 80.00th=[ 64], 90.00th=[ 314], 95.00th=[ 347],
| 99.00th=[ 465], 99.50th=[ 1188], 99.90th=[ 4621]
bw (KB/s) : min= 173, max=76148, per=31.86%, avg=15188.06, stdev=11432.61
lat (msec) : 20=59.73%, 50=18.26%, 100=3.52%, 250=6.52%, 500=11.05%
lat (msec) : 750=0.33%, 1000=0.08%, 2000=0.14%, >=2000=0.37%
cpu : usr=0.01%, sys=0.13%, ctx=5241, majf=0, minf=283
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=5120/w=0/d=0, short=r=0/w=0/d=0
seqread4: (groupid=0, jobs=1): err= 0: pid=9005
read : io=5120.0MB, bw=13183KB/s, iops=12 , runt=397707msec
clat (msec): min=11 , max=2951 , avg=77.67, stdev=147.40
lat (msec): min=11 , max=2951 , avg=77.67, stdev=147.40
clat percentiles (msec):
| 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 13],
| 30.00th=[ 13], 40.00th=[ 14], 50.00th=[ 17], 60.00th=[ 21],
| 70.00th=[ 32], 80.00th=[ 106], 90.00th=[ 314], 95.00th=[ 347],
| 99.00th=[ 441], 99.50th=[ 619], 99.90th=[ 1795]
bw (KB/s) : min= 347, max=33851, per=29.67%, avg=14144.82, stdev=6534.68
lat (msec) : 20=58.89%, 50=18.38%, 100=2.62%, 250=7.17%, 500=12.23%
lat (msec) : 750=0.37%, 1000=0.14%, 2000=0.12%, >=2000=0.10%
cpu : usr=0.01%, sys=0.13%, ctx=5220, majf=0, minf=282
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=5120/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=20480MB, aggrb=47668KB/s, minb=12203KB/s, maxb=13753KB/s, mint=390342msec, maxt=439947msec
Disk stats (read/write):
sda: ios=59655/1473, merge=46382/960, ticks=9162485/989380, in_queue=10027085, util=100.00%
多重度4のシーケンシャルライト
[seqwrite4] readwrite=write blocksize=1m size=1g directory=./ direct=1 numjobs=4 loops=5
# fio seqwrite4.fio
seqwrite4: (g=0): rw=write, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
...
seqwrite4: (g=0): rw=write, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
fio 2.0.5
Starting 4 processes
seqwrite4: Laying out IO file(s) (1 file(s) / 1024MB)
seqwrite4: Laying out IO file(s) (1 file(s) / 1024MB)
seqwrite4: Laying out IO file(s) (1 file(s) / 1024MB)
seqwrite4: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 1 (f=1): [W___] [100.0% done] [0K/42820K /s] [0 /40 iops] [eta 00m:00s]
seqwrite4: (groupid=0, jobs=1): err= 0: pid=9029
write: io=5120.0MB, bw=9273.5KB/s, iops=9 , runt=565367msec
clat (msec): min=14 , max=2243 , avg=110.33, stdev=141.81
lat (msec): min=14 , max=2243 , avg=110.42, stdev=141.81
clat percentiles (msec):
| 1.00th=[ 16], 5.00th=[ 17], 10.00th=[ 18], 20.00th=[ 20],
| 30.00th=[ 27], 40.00th=[ 41], 50.00th=[ 61], 60.00th=[ 84],
| 70.00th=[ 122], 80.00th=[ 180], 90.00th=[ 273], 95.00th=[ 343],
| 99.00th=[ 627], 99.50th=[ 906], 99.90th=[ 1319]
bw (KB/s) : min= 802, max=50906, per=26.56%, avg=9852.55, stdev=5032.29
lat (msec) : 20=20.68%, 50=23.42%, 100=21.37%, 250=22.73%, 500=10.04%
lat (msec) : 750=1.05%, 1000=0.25%, 2000=0.43%, >=2000=0.02%
cpu : usr=0.08%, sys=0.10%, ctx=5753, majf=0, minf=25
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=5120/d=0, short=r=0/w=0/d=0
seqwrite4: (groupid=0, jobs=1): err= 0: pid=9030
write: io=5120.0MB, bw=9315.6KB/s, iops=9 , runt=562809msec
clat (msec): min=13 , max=4681 , avg=109.83, stdev=164.63
lat (msec): min=13 , max=4681 , avg=109.92, stdev=164.63
clat percentiles (msec):
| 1.00th=[ 15], 5.00th=[ 17], 10.00th=[ 18], 20.00th=[ 19],
| 30.00th=[ 25], 40.00th=[ 39], 50.00th=[ 60], 60.00th=[ 81],
| 70.00th=[ 116], 80.00th=[ 180], 90.00th=[ 269], 95.00th=[ 343],
| 99.00th=[ 627], 99.50th=[ 881], 99.90th=[ 1860]
bw (KB/s) : min= 344, max=34403, per=27.01%, avg=10020.61, stdev=4993.68
lat (msec) : 20=22.99%, 50=22.42%, 100=21.11%, 250=21.99%, 500=9.80%
lat (msec) : 750=0.92%, 1000=0.35%, 2000=0.33%, >=2000=0.08%
cpu : usr=0.08%, sys=0.09%, ctx=5741, majf=0, minf=25
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=5120/d=0, short=r=0/w=0/d=0
seqwrite4: (groupid=0, jobs=1): err= 0: pid=9031
write: io=5120.0MB, bw=9333.9KB/s, iops=9 , runt=561710msec
clat (msec): min=13 , max=2428 , avg=109.62, stdev=145.08
lat (msec): min=13 , max=2429 , avg=109.70, stdev=145.08
clat percentiles (msec):
| 1.00th=[ 15], 5.00th=[ 17], 10.00th=[ 18], 20.00th=[ 19],
| 30.00th=[ 26], 40.00th=[ 39], 50.00th=[ 60], 60.00th=[ 82],
| 70.00th=[ 119], 80.00th=[ 180], 90.00th=[ 269], 95.00th=[ 347],
| 99.00th=[ 619], 99.50th=[ 906], 99.90th=[ 1434]
bw (KB/s) : min= 678, max=31507, per=26.78%, avg=9933.07, stdev=4731.98
lat (msec) : 20=22.29%, 50=23.09%, 100=20.55%, 250=22.38%, 500=9.92%
lat (msec) : 750=1.05%, 1000=0.37%, 2000=0.31%, >=2000=0.04%
cpu : usr=0.08%, sys=0.10%, ctx=5765, majf=0, minf=26
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=5120/d=0, short=r=0/w=0/d=0
seqwrite4: (groupid=0, jobs=1): err= 0: pid=9032
write: io=5120.0MB, bw=9325.5KB/s, iops=9 , runt=562214msec
clat (msec): min=14 , max=4000 , avg=109.72, stdev=170.30
lat (msec): min=14 , max=4000 , avg=109.80, stdev=170.30
clat percentiles (msec):
| 1.00th=[ 15], 5.00th=[ 17], 10.00th=[ 18], 20.00th=[ 20],
| 30.00th=[ 26], 40.00th=[ 39], 50.00th=[ 59], 60.00th=[ 81],
| 70.00th=[ 116], 80.00th=[ 174], 90.00th=[ 265], 95.00th=[ 338],
| 99.00th=[ 660], 99.50th=[ 938], 99.90th=[ 2147]
bw (KB/s) : min= 508, max=35929, per=27.24%, avg=10105.39, stdev=4912.41
lat (msec) : 20=21.64%, 50=23.79%, 100=21.00%, 250=22.68%, 500=9.22%
lat (msec) : 750=0.86%, 1000=0.39%, 2000=0.29%, >=2000=0.14%
cpu : usr=0.08%, sys=0.09%, ctx=5807, majf=0, minf=25
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=5120/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=20480MB, aggrb=37093KB/s, minb=9495KB/s, maxb=9557KB/s, mint=561710msec, maxt=565367msec
Disk stats (read/write):
sda: ios=0/72719, merge=0/44002, ticks=0/10909055, in_queue=10909124, util=99.98%
ランダムリード512k
[randread512k] readwrite=randread blocksize=512k size=1g directory=./ direct=1 loops=5
# fio randread512k.fio
randread512k: (g=0): rw=randread, bs=512K-512K/512K-512K, ioengine=sync, iodepth=1
fio 2.0.5
Starting 1 process
randread512k: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 1 (f=1): [r] [100.0% done] [10966K/0K /s] [20 /0 iops] [eta 00m:00s]
randread512k: (groupid=0, jobs=1): err= 0: pid=9073
read : io=5120.0MB, bw=11334KB/s, iops=22 , runt=462585msec
clat (msecin=5 , max=1213 , avg=45.17, stdev=35.23
lat (msec): min=5 , max=1213 , avg=45.17, stdev=35.23
clat perntiles (msec):
| 1.00th=[ 6], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 18],
| 30.00th=[ 28], 40.00th=[ 35], 50.00th=[ 42], 60.00th=[ 49],
| 70.00th=[ 57], 80.00th=[ 67], 90.00th=[ 83], 95.00th=[ 102],
| 99.00th=[ 151], 99.50th=[ 178], 99.90th=[ 285]
bw (KB/s) : min= 640, max=20480, per=100.00%, avg=11392.76, stdev=2561.48
lat (msec) : 10=17.86%, 20=3.89%, 50=40.21%, 100=32.78%, 250=5.01%
lat (msec) : 500=0.23%, 750=0.01%, 2000=0.01%
cpu : usr=0.02%, sys=0.15%, ctx=10292, majf=0, minf=154
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=10240/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=5120.0MB, aggrb=11333KB/s, minb=11605KB/s, maxb=11605KB/s, mint=462585msec, maxt=462585msec
Disk stats (read/write):
sda: ios=12996/1670, merge=12779/625, ticks=673643/430788, in_queue=1104454, util=99.92%
ランダムライト512k
[randwrite512k] readwrite=randwrite blocksize=512k size=1g directory=./ direct=1 loops=5
# fio randwrite512k.fio
randwrite512k: (g=0): rw=randwrite, bs=512K-512K/512K-512K, ioengine=sync, iodepth=1
fio 2.0.5
Starting 1 process
randwrite512k: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 1 (f=1): [w] [100.0% done] [0K/29765K /s] [0 /56 iops] [eta 00m:00s]
randwrite512k: (groupid=0, jobs=1): err= 0: pid=22853
write: io=5120.0MB, bw=34276KB/s, iops=66 , runt=152959msec
clat (msec): min=7 , max=1158 , avg=14.89, stdev=28.56
lat (msec): min=7 , max=1158 , avg=14.93, stdev=28.56
clat percentiles (msec):
| 1.00th=[ 9], 5.00th=[ 9], 10.00th=[ 9], 20.00th=[ 9],
| 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 10], 60.00th=[ 11],
| 70.00th=[ 11], 80.00th=[ 14], 90.00th=[ 23], 95.00th=[ 35],
| 99.00th=[ 90], 99.50th=[ 128], 99.90th=[ 416]
bw (KB/s) : min= 873, max=56440, per=100.00%, avg=35549.34, stdev=12392.43
lat (msec) : 10=60.17%, 20=26.65%, 50=10.36%, 100=2.06%, 250=0.58%
lat (msec) : 500=0.14%, 750=0.01%, 1000=0.02%, 2000=0.02%
cpu : usr=0.27%, sys=1.67%, ctx=10951, majf=0, minf=25
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=10240/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=5120.0MB, aggrb=34276KB/s, minb=35099KB/s, maxb=35099KB/s, mint=152959msec, maxt=152959msec
Disk stats (read/write):
sda: ios=0/23917, merge=0/260483, ticks=0/773513, in_queue=773513, util=98.40%
ランダムリード4k
[randread4k] readwrite=randread blocksize=4k size=100m directory=./ direct=1 loops=3
# fio randread4k.fio
randread4k: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
fio 2.0.5
Starting 1 process
randread4k: Laying out IO file(s) (1 file(s) / 100MB)
Jobs: 1 (f=1): [r] [100.0% done] [11704K/0K /s] [2857 /0 iops] [eta 00m:00s]
randread4k: (groupid=0, jobs=1): err= 0: pid=22895
read : io=307200KB, bw=10796KB/s, iops=2699 , runt= 28454msec
clat (usec): min=256 , max=128032 , avg=365.92, stdev=839.38
lat (usec): min=256 , max=128033 , avg=366.17, stdev=839.38
clat percentiles (usec):
| 1.00th=[ 330], 5.00th=[ 334], 10.00th=[ 338], 20.00th=[ 338],
| 30.00th=[ 342], 40.00th=[ 342], 50.00th=[ 346], 60.00th=[ 350],
| 70.00th=[ 354], 80.00th=[ 362], 90.00th=[ 374], 95.00th=[ 382],
| 99.00th=[ 402], 99.50th=[ 426], 99.90th=[ 1020]
bw (KB/s) : min= 7024, max=11440, per=99.96%, avg=10791.29, stdev=955.92
lat (usec) : 500=99.79%, 750=0.07%, 1000=0.04%
lat (msec) : 2=0.03%, 4=0.01%, 10=0.02%, 20=0.03%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=1.39%, sys=3.71%, ctx=76802, majf=0, minf=27
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=76800/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=307200KB, aggrb=10796KB/s, minb=11055KB/s, maxb=11055KB/s, mint=28454msec, maxt=28454msec
Disk stats (read/write):
sda: ios=76084/241, merge=0/87, ticks=27026/15233, in_queue=42258, util=95.51%
ランダムライト4k
[randwrite4k] readwrite=randwrite blocksize=4k size=100m directory=./ direct=1 loops=3
# fio randwrite4k.fio
randwrite4k: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
fio 2.0.5
Starting 1 process
randwrite4k: Laying out IO file(s) (1 file(s) / 100MB)
Jobs: 1 (f=1): [w] [100.0% done] [0K/9893K /s] [0 /2415 iops] [eta 00m:00s]
randwrite4k: (groupid=0, jobs=1): err= 0: pid=22907
write: io=307200KB, bw=8351.3KB/s, iops=2087 , runt= 36786msec
clat (usec): min=323 , max=131784 , avg=474.03, stdev=1177.03
lat (usec): min=323 , max=131785 , avg=474.44, stdev=1177.03
clat percentiles (usec):
| 1.00th=[ 350], 5.00th=[ 354], 10.00th=[ 358], 20.00th=[ 362],
| 30.00th=[ 370], 40.00th=[ 374], 50.00th=[ 386], 60.00th=[ 398],
| 70.00th=[ 502], 80.00th=[ 532], 90.00th=[ 588], 95.00th=[ 604],
| 99.00th=[ 724], 99.50th=[ 884], 99.90th=[14272]
bw (KB/s) : min= 296, max=10664, per=99.71%, avg=8327.08, stdev=2390.60
lat (usec) : 500=68.98%, 750=30.12%, 1000=0.50%
lat (msec) : 2=0.20%, 4=0.04%, 10=0.04%, 20=0.05%, 50=0.05%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=1.30%, sys=4.31%, ctx=76814, majf=0, minf=25
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=76800/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=307200KB, aggrb=8351KB/s, minb=8551KB/s, maxb=8551KB/s, mint=36786msec, maxt=36786msec
Disk stats (read/write):
sda: ios=0/76445, merge=0/392, ticks=0/47345, in_queue=47412, util=95.54%
多重度32のランダムライト4
[randread4k32] readwrite=randread blocksize=4k size=10m directory=./ direct=1 numjobs=32 loops=3
# fio randread4k32.fio
randread4k32: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
...
randread4k32: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
fio 2.0.5
Starting 32 processes
（略）
randread4k32: (groupid=0, jobs=1): err= 0: pid=9275
read : io=30720KB, bw=564134 B/s, iops=137 , runt= 55762msec
clat (usec): min=250 , max=206707 , avg=7254.77, stdev=14087.63
lat (usec): min=250 , max=206707 , avg=7255.13, stdev=14087.65
clat percentiles (usec):
| 1.00th=[ 318], 5.00th=[ 334], 10.00th=[ 334], 20.00th=[ 342],
| 30.00th=[ 346], 40.00th=[ 354], 50.00th=[ 374], 60.00th=[ 1304],
| 70.00th=[ 4448], 80.00th=[11968], 90.00th=[25216], 95.00th=[36608],
| 99.00th=[63744], 99.50th=[79360], 99.90th=[109056]
bw (KB/s) : min= 87, max= 2216, per=3.09%, avg=528.11, stdev=462.15
lat (usec) : 500=56.29%, 750=1.65%, 1000=1.13%
lat (msec) : 2=3.57%, 4=6.48%, 10=8.53%, 20=9.23%, 50=11.05%
lat (msec) : 100=1.88%, 250=0.18%
cpu : usr=0.10%, sys=0.25%, ctx=8095, majf=0, minf=27
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=7680/w=0/d=0, short=r=0/w=0/d=0
randread4k32: (groupid=0, jobs=1): err= 0: pid=9276
read : io=30720KB, bw=640117 B/s, iops=156 , runt= 49143msec
clat (usec): min=267 , max=233232 , avg=6393.08, stdev=13452.57
lat (usec): min=267 , max=233232 , avg=6393.42, stdev=13452.63
clat percentiles (usec):
| 1.00th=[ 322], 5.00th=[ 334], 10.00th=[ 338], 20.00th=[ 338],
| 30.00th=[ 342], 40.00th=[ 350], 50.00th=[ 362], 60.00th=[ 386],
| 70.00th=[ 2448], 80.00th=[ 9664], 90.00th=[22656], 95.00th=[34048],
| 99.00th=[60672], 99.50th=[77312], 99.90th=[111104]
bw (KB/s) : min= 62, max= 1801, per=3.66%, avg=625.30, stdev=460.94
lat (usec) : 500=63.33%, 750=0.64%, 1000=0.69%
lat (msec) : 2=2.93%, 4=5.48%, 10=7.11%, 20=8.16%, 50=9.88%
lat (msec) : 100=1.59%, 250=0.18%
cpu : usr=0.12%, sys=0.29%, ctx=8059, majf=0, minf=28
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=7680/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=983040KB, aggrb=17083KB/s, minb=546KB/s, maxb=807KB/s, mint=38957msec, maxt=57543msec
Disk stats (read/write):
sda: ios=245065/214, merge=0/123, ticks=1641829/5071, in_queue=1646895, util=99.88%
多重度32のランダムライト4k
[randwrite4k32] readwrite=randwrite blocksize=4k size=10m directory=/c direct=1 numjobs=32 loops=3
# fio randwrite4k32.fio
randwrite4k32: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
...
randwrite4k32: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
fio 2.0.5
Starting 32 processes
（略）
randwrite4k32: (groupid=0, jobs=1): err= 0: pid=9386
write: io=30720KB, bw=261951 B/s, iops=63 , runt=120088msec
clat (usec): min=327 , max=5199.8K, avg=15631.22, stdev=137044.57
lat (usec): min=328 , max=5199.8K, avg=15631.68, stdev=137044.60
clat percentiles (usec):
| 1.00th=[ 350], 5.00th=[ 358], 10.00th=[ 362], 20.00th=[ 366],
| 30.00th=[ 370], 40.00th=[ 378], 50.00th=[ 386], 60.00th=[ 398],
| 70.00th=[ 410], 80.00th=[ 466], 90.00th=[ 716], 95.00th=[ 756],
| 99.00th=[460800], 99.50th=[684032], 99.90th=[1810432]
bw (KB/s) : min= 0, max= 2424, per=4.43%, avg=303.23, stdev=588.61
lat (usec) : 500=81.11%, 750=13.41%, 1000=2.02%
lat (msec) : 2=0.12%, 4=0.17%, 10=0.03%, 20=0.05%, 50=0.13%
lat (msec) : 100=0.14%, 250=0.89%, 500=1.07%, 750=0.44%, 1000=0.12%
lat (msec) : 2000=0.22%, >=2000=0.09%
cpu : usr=0.04%, sys=0.06%, ctx=7711, majf=0, minf=26
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=7680/d=0, short=r=0/w=0/d=0
randwrite4k32: (groupid=0, jobs=1): err= 0: pid=9387
write: io=30720KB, bw=340819 B/s, iops=83 , runt= 92299msec
clat (usec): min=313 , max=5199.1K, avg=12012.91, stdev=122783.50
lat (usec): min=314 , max=5199.1K, avg=12013.35, stdev=122783.51
clat percentiles (usec):
| 1.00th=[ 350], 5.00th=[ 354], 10.00th=[ 358], 20.00th=[ 366],
| 30.00th=[ 374], 40.00th=[ 378], 50.00th=[ 386], 60.00th=[ 394],
| 70.00th=[ 410], 80.00th=[ 438], 90.00th=[ 580], 95.00th=[ 716],
| 99.00th=[350208], 99.50th=[593920], 99.90th=[1810432]
bw (KB/s) : min= 0, max= 3847, per=6.04%, avg=414.00, stdev=771.07
lat (usec) : 500=86.95%, 750=9.61%, 1000=0.72%
lat (msec) : 2=0.10%, 4=0.17%, 10=0.05%, 20=0.03%, 50=0.10%
lat (msec) : 100=0.09%, 250=0.68%, 500=0.92%, 750=0.20%, 1000=0.10%
lat (msec) : 2000=0.20%, >=2000=0.08%
cpu : usr=0.05%, sys=0.11%, ctx=7721, majf=0, minf=26
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=7680/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=983040KB, aggrb=6852KB/s, minb=219KB/s, maxb=643KB/s, mint=48878msec, maxt=143459msec
Disk stats (read/write):
sda: ios=0/245669, merge=0/380, ticks=0/3939189, in_queue=3939238, util=100.00%
上記の結果をまとめた物が下の表になる。多重化した時のIOPSがよく分からないので空白、たぶん省略した場所の数字が必要なのだろうけど、文字数制限を軽く超えるため割愛。
| Type | Speed (MB/s) | IOPS |
|---|---|---|
| seq read | 66.2 | 64 |
| seq write | 36.1 | 35 |
| seq read*4 | 47.7 | |
| seq write*4 | 37.1 | |
| rand read 512k | 11.3 | 22 |
| rand write 512k | 34.3 | 66 |
| rand read 4k | 10.8 | 2699 |
| rand write 4k | 8.4 | 2087 |
| rand read 4k*32 | 17.0 | |
| rand write 4k*32 | 6.9 |
ランダム4kの性能が非常に良いように見えるが、これはReadyNASのキャッシュが効いているためだと思われる。
逆にシーケンシャルの速度が見劣りするが、それがX-RAID2のせいなのかが気になる。
そのため、X-RAID2のベンチマークを取ってみる。NFSは後回し。
