読者です 読者をやめる 読者になる 読者になる

ReadyNAS Ultra4 Plusに触ってみた その3

主な利用用途であるiSCSINFSベンチマークを比較したい。まずはiSCSIから。

マルチパスについて

iSCSIブートはdm-multipathと相性が悪いのでHDD起動のサーバに2ポートの1000BASE-Tを取り付けてReadyNASのLAN1、LAN2とそれぞれ直接接続。
iSCSIの設定とdm-multipathの設定を行った。

# multipath -l
mpathe (360014052e419090150ad002000000000) dm-0 LIO-ORG,FILEIO
size=20G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
  |- 45:0:0:0 sdg 8:96  active undef running
  `- 44:0:0:0 sdi 8:128 active undef running
mpathd (360014052e419090150ad001000000000) dm-3 LIO-ORG,FILEIO
size=200G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
  |- 42:0:0:0 sde 8:64  active undef running
  `- 43:0:0:0 sdf 8:80  active undef running
mpathc (360014052e419090150ad004000000000) dm-1 LIO-ORG,FILEIO
size=20G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
  |- 40:0:0:0 sdc 8:32  active undef running
  `- 41:0:0:0 sdd 8:48  active undef running
mpathf (360014052e419090150ad003000000000) dm-2 LIO-ORG,FILEIO
size=20G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
  |- 47:0:0:0 sdj 8:144 active undef running
  `- 46:0:0:0 sdh 8:112 active undef running

/dev/sdcと/dev/sddが同一のiSCSIボリュームでそれぞれ別経路から見ていることになる。それをdm-multipathで/dev/mapper/mpathcとしている。
いろいろボリュームが見えているけど気にしない。別に異常では無い。

それではhdparmで簡単にベンチマーク

# hdparm -tT /dev/sdc
/dev/sdc:
 Timing cached reads:   25690 MB in  2.00 seconds = 12865.15 MB/sec
 Timing buffered disk reads:  296 MB in  3.02 seconds =  98.00 MB/sec

# hdparm -tT /dev/sdd
/dev/sdd:
 Timing cached reads:   26050 MB in  2.00 seconds = 13045.94 MB/sec
 Timing buffered disk reads:  312 MB in  3.01 seconds = 103.78 MB/sec

# hdparm -tT /dev/mapper/mpathc
/dev/mapper/mpathc:
 Timing cached reads:   25616 MB in  2.00 seconds = 12829.08 MB/sec
 Timing buffered disk reads:  256 MB in  3.02 seconds =  84.76 MB/sec

/dev/sdXに単独にアクセスした場合よりパフォーマンスが低くなっている。どうもマルチパス構成としてうまく動作していない模様。
ReadyNASの方が対応しないようになっていると思われる。

iSCSIベンチマーク

iSCSIブート設定にしたサーバでReadyNASのiSCSIボリュームから起動してベンチマーク。なんでHDD起動からテストしてないかと言うと、本番設定をした後で思いついたから。

ext3でフォーマットした領域に対してfio 2.0.5を使ってベンチマークを実行。パラメータなんかは複眼中心さんを参考にさせていただきました。

シーケンシャルリード

[seqread]
readwrite=read
blocksize=1m
size=1g
directory=./
direct=1
loops=5
# fio seqread.fio
seqread: (g=0): rw=read, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
fio 2.0.5
Starting 1 process
Jobs: 1 (f=1): [R] [100.0% done] [30287K/0K /s] [28 /0  iops] [eta 00m:00s]
seqread: (groupid=0, jobs=1): err= 0: pid=7344
  read : io=5120.0MB, bw=66234KB/s, iops=64 , runt= 79157msec
    clat (msec): min=11 , max=528 , avg=15.46, stdev=17.15
     lat (msec): min=11 , max=528 , avg=15.46, stdev=17.15
    clat percentiles (msec):
     |  1.00th=[   12],  5.00th=[   12], 10.00th=[   12], 20.00th=[   13],
     | 30.00th=[   13], 40.00th=[   13], 50.00th=[   13], 60.00th=[   13],
     | 70.00th=[   13], 80.00th=[   14], 90.00th=[   17], 95.00th=[   29],
     | 99.00th=[   68], 99.50th=[  120], 99.90th=[  227]
    bw (KB/s)  : min= 9582, max=83136, per=100.00%, avg=67208.73, stdev=19041.60
    lat (msec) : 20=91.35%, 50=6.91%, 100=1.11%, 250=0.53%, 500=0.08%
    lat (msec) : 750=0.02%
  cpu          : usr=0.05%, sys=0.70%, ctx=5132, majf=0, minf=282
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=5120/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=5120.0MB, aggrb=66233KB/s, minb=67823KB/s, maxb=67823KB/s, mint=79157msec, maxt=79157msec

Disk stats (read/write):
  sda: ios=12928/220, merge=12443/118, ticks=201216/19282, in_queue=220501, util=99.71%

シーケンシャルライト

[seqwrite]
readwrite=write
blocksize=1m
size=1g
directory=./
direct=1
loops=5
# fio seqwrite.fio
seqwrite: (g=0): rw=write, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
fio 2.0.5
Starting 1 process
seqwrite: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 1 (f=1): [W] [100.0% done] [0K/8355K /s] [0 /7  iops] [eta 00m:00s]
seqwrite: (groupid=0, jobs=1): err= 0: pid=8990
  write: io=5120.0MB, bw=36115KB/s, iops=35 , runt=145173msec
    clat (msec): min=13 , max=733 , avg=28.26, stdev=36.18
     lat (msec): min=13 , max=733 , avg=28.35, stdev=36.18
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   15], 10.00th=[   15], 20.00th=[   15],
     | 30.00th=[   15], 40.00th=[   16], 50.00th=[   17], 60.00th=[   21],
     | 70.00th=[   26], 80.00th=[   33], 90.00th=[   48], 95.00th=[   76],
     | 99.00th=[  165], 99.50th=[  223], 99.90th=[  469]
    bw (KB/s)  : min= 2409, max=63109, per=100.00%, avg=37125.47, stdev=13231.65
    lat (msec) : 20=59.77%, 50=30.78%, 100=6.23%, 250=2.75%, 500=0.37%
    lat (msec) : 750=0.10%
  cpu          : usr=0.31%, sys=0.44%, ctx=5179, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=5120/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=5120.0MB, aggrb=36114KB/s, minb=36981KB/s, maxb=36981KB/s, mint=145173msec, maxt=145173msec

Disk stats (read/write):
  sda: ios=2/25363, merge=0/1425, ticks=37/778774, in_queue=778822, util=99.51%

多重度4のシーケンシャルリード

[seqread4]
readwrite=read
blocksize=1m
size=1g
directory=./
direct=1
numjobs=4
loops=5
# fio seqread4.fio
seqread4: (g=0): rw=read, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
...
seqread4: (g=0): rw=read, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
fio 2.0.5
Starting 4 processes
seqread4: Laying out IO file(s) (1 file(s) / 1024MB)
seqread4: Laying out IO file(s) (1 file(s) / 1024MB)
seqread4: Laying out IO file(s) (1 file(s) / 1024MB)
seqread4: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 4 (f=4): [RRRR] [300.0% done] [0K/0K /s] [0 /0  iops] [eta 1158050441d:07hJobs: 4 (f=4): [RRRR] [inf% done] [0K/0K /s] [0 /0  iops] [eta 1158050441d:07h:0Jobs: 4 (f=4): [RRRR] [inf% done] [0K/0K /s] [0 /0  iops] [eta 1158050441d:07h:0Jobs: 4 (f=4): [RRRR] [inf% done] [0K/0K /s] [0 /0  iops] [eta 1158050441d:07h:0Jobs: 4 (f=4): [RRRR] [400.0% done] [0K/0K /s] [0 /0  iops] [eta 1158050441d:07hJobs: 4 (f=4): [RRRR] [inf% done] [963K/0K /s] [0 /0  iops] [eta 1158050441d:07hJobs: 4 (f=4): [RRRR] [inf% done] [5221K/0K /s] [4 /0  iops] [eta 1158050441d:07Jobs: 4 (f=4): [RRRR] [inf% done] [13577K/0K /s] [12 /0  iops] [eta 1158050441d:Jobs: 4 (f=4): [RRRR] [0.0% done] [7310K/0K /s] [6 /0  iops] [eta 09h:57m:13s]  Jobs: 1 (f=1): [__R_] [98.9% done] [63708K/0K /s] [60 /0  iops] [eta 00m:05s]
seqread4: (groupid=0, jobs=1): err= 0: pid=9002
  read : io=5120.0MB, bw=13432KB/s, iops=13 , runt=390342msec
    clat (msec): min=11 , max=2952 , avg=76.23, stdev=143.01
     lat (msec): min=11 , max=2952 , avg=76.23, stdev=143.01
    clat percentiles (msec):
     |  1.00th=[   12],  5.00th=[   13], 10.00th=[   13], 20.00th=[   13],
     | 30.00th=[   13], 40.00th=[   13], 50.00th=[   14], 60.00th=[   19],
     | 70.00th=[   30], 80.00th=[  115], 90.00th=[  318], 95.00th=[  347],
     | 99.00th=[  424], 99.50th=[  490], 99.90th=[ 1975]
    bw (KB/s)  : min=  346, max=30658, per=29.96%, avg=14279.41, stdev=6330.28
    lat (msec) : 20=61.35%, 50=15.51%, 100=2.79%, 250=7.29%, 500=12.60%
    lat (msec) : 750=0.31%, 1000=0.02%, 2000=0.04%, >=2000=0.10%
  cpu          : usr=0.01%, sys=0.13%, ctx=5218, majf=0, minf=282
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=5120/w=0/d=0, short=r=0/w=0/d=0
seqread4: (groupid=0, jobs=1): err= 0: pid=9003
  read : io=5120.0MB, bw=12326KB/s, iops=12 , runt=425368msec
    clat (msec): min=11 , max=5712 , avg=83.08, stdev=175.50
     lat (msec): min=11 , max=5712 , avg=83.08, stdev=175.50
    clat percentiles (msec):
     |  1.00th=[   12],  5.00th=[   13], 10.00th=[   13], 20.00th=[   13],
     | 30.00th=[   13], 40.00th=[   15], 50.00th=[   22], 60.00th=[   29],
     | 70.00th=[   43], 80.00th=[  124], 90.00th=[  302], 95.00th=[  343],
     | 99.00th=[  445], 99.50th=[  840], 99.90th=[ 1549]
    bw (KB/s)  : min=  179, max=50712, per=28.47%, avg=13571.64, stdev=7380.59
    lat (msec) : 20=48.46%, 50=24.39%, 100=5.72%, 250=9.14%, 500=11.50%
    lat (msec) : 750=0.27%, 1000=0.10%, 2000=0.37%, >=2000=0.04%
  cpu          : usr=0.01%, sys=0.12%, ctx=5225, majf=0, minf=283
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=5120/w=0/d=0, short=r=0/w=0/d=0
seqread4: (groupid=0, jobs=1): err= 0: pid=9004
  read : io=5120.0MB, bw=11917KB/s, iops=11 , runt=439947msec
    clat (msec): min=11 , max=9006 , avg=85.92, stdev=309.61
     lat (msec): min=11 , max=9006 , avg=85.92, stdev=309.61
    clat percentiles (msec):
     |  1.00th=[   12],  5.00th=[   13], 10.00th=[   13], 20.00th=[   13],
     | 30.00th=[   13], 40.00th=[   13], 50.00th=[   15], 60.00th=[   21],
     | 70.00th=[   32], 80.00th=[   64], 90.00th=[  314], 95.00th=[  347],
     | 99.00th=[  465], 99.50th=[ 1188], 99.90th=[ 4621]
    bw (KB/s)  : min=  173, max=76148, per=31.86%, avg=15188.06, stdev=11432.61
    lat (msec) : 20=59.73%, 50=18.26%, 100=3.52%, 250=6.52%, 500=11.05%
    lat (msec) : 750=0.33%, 1000=0.08%, 2000=0.14%, >=2000=0.37%
  cpu          : usr=0.01%, sys=0.13%, ctx=5241, majf=0, minf=283
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=5120/w=0/d=0, short=r=0/w=0/d=0
seqread4: (groupid=0, jobs=1): err= 0: pid=9005
  read : io=5120.0MB, bw=13183KB/s, iops=12 , runt=397707msec
    clat (msec): min=11 , max=2951 , avg=77.67, stdev=147.40
     lat (msec): min=11 , max=2951 , avg=77.67, stdev=147.40
    clat percentiles (msec):
     |  1.00th=[   12],  5.00th=[   13], 10.00th=[   13], 20.00th=[   13],
     | 30.00th=[   13], 40.00th=[   14], 50.00th=[   17], 60.00th=[   21],
     | 70.00th=[   32], 80.00th=[  106], 90.00th=[  314], 95.00th=[  347],
     | 99.00th=[  441], 99.50th=[  619], 99.90th=[ 1795]
    bw (KB/s)  : min=  347, max=33851, per=29.67%, avg=14144.82, stdev=6534.68
    lat (msec) : 20=58.89%, 50=18.38%, 100=2.62%, 250=7.17%, 500=12.23%
    lat (msec) : 750=0.37%, 1000=0.14%, 2000=0.12%, >=2000=0.10%
  cpu          : usr=0.01%, sys=0.13%, ctx=5220, majf=0, minf=282
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=5120/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=20480MB, aggrb=47668KB/s, minb=12203KB/s, maxb=13753KB/s, mint=390342msec, maxt=439947msec

Disk stats (read/write):
  sda: ios=59655/1473, merge=46382/960, ticks=9162485/989380, in_queue=10027085, util=100.00%

多重度4のシーケンシャルライト

[seqwrite4]
readwrite=write
blocksize=1m
size=1g
directory=./
direct=1
numjobs=4
loops=5
# fio seqwrite4.fio
seqwrite4: (g=0): rw=write, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
...
seqwrite4: (g=0): rw=write, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
fio 2.0.5
Starting 4 processes
seqwrite4: Laying out IO file(s) (1 file(s) / 1024MB)
seqwrite4: Laying out IO file(s) (1 file(s) / 1024MB)
seqwrite4: Laying out IO file(s) (1 file(s) / 1024MB)
seqwrite4: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 1 (f=1): [W___] [100.0% done] [0K/42820K /s] [0 /40  iops] [eta 00m:00s]
seqwrite4: (groupid=0, jobs=1): err= 0: pid=9029
  write: io=5120.0MB, bw=9273.5KB/s, iops=9 , runt=565367msec
    clat (msec): min=14 , max=2243 , avg=110.33, stdev=141.81
     lat (msec): min=14 , max=2243 , avg=110.42, stdev=141.81
    clat percentiles (msec):
     |  1.00th=[   16],  5.00th=[   17], 10.00th=[   18], 20.00th=[   20],
     | 30.00th=[   27], 40.00th=[   41], 50.00th=[   61], 60.00th=[   84],
     | 70.00th=[  122], 80.00th=[  180], 90.00th=[  273], 95.00th=[  343],
     | 99.00th=[  627], 99.50th=[  906], 99.90th=[ 1319]
    bw (KB/s)  : min=  802, max=50906, per=26.56%, avg=9852.55, stdev=5032.29
    lat (msec) : 20=20.68%, 50=23.42%, 100=21.37%, 250=22.73%, 500=10.04%
    lat (msec) : 750=1.05%, 1000=0.25%, 2000=0.43%, >=2000=0.02%
  cpu          : usr=0.08%, sys=0.10%, ctx=5753, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=5120/d=0, short=r=0/w=0/d=0
seqwrite4: (groupid=0, jobs=1): err= 0: pid=9030
  write: io=5120.0MB, bw=9315.6KB/s, iops=9 , runt=562809msec
    clat (msec): min=13 , max=4681 , avg=109.83, stdev=164.63
     lat (msec): min=13 , max=4681 , avg=109.92, stdev=164.63
    clat percentiles (msec):
     |  1.00th=[   15],  5.00th=[   17], 10.00th=[   18], 20.00th=[   19],
     | 30.00th=[   25], 40.00th=[   39], 50.00th=[   60], 60.00th=[   81],
     | 70.00th=[  116], 80.00th=[  180], 90.00th=[  269], 95.00th=[  343],
     | 99.00th=[  627], 99.50th=[  881], 99.90th=[ 1860]
    bw (KB/s)  : min=  344, max=34403, per=27.01%, avg=10020.61, stdev=4993.68
    lat (msec) : 20=22.99%, 50=22.42%, 100=21.11%, 250=21.99%, 500=9.80%
    lat (msec) : 750=0.92%, 1000=0.35%, 2000=0.33%, >=2000=0.08%
  cpu          : usr=0.08%, sys=0.09%, ctx=5741, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=5120/d=0, short=r=0/w=0/d=0
seqwrite4: (groupid=0, jobs=1): err= 0: pid=9031
  write: io=5120.0MB, bw=9333.9KB/s, iops=9 , runt=561710msec
    clat (msec): min=13 , max=2428 , avg=109.62, stdev=145.08
     lat (msec): min=13 , max=2429 , avg=109.70, stdev=145.08
    clat percentiles (msec):
     |  1.00th=[   15],  5.00th=[   17], 10.00th=[   18], 20.00th=[   19],
     | 30.00th=[   26], 40.00th=[   39], 50.00th=[   60], 60.00th=[   82],
     | 70.00th=[  119], 80.00th=[  180], 90.00th=[  269], 95.00th=[  347],
     | 99.00th=[  619], 99.50th=[  906], 99.90th=[ 1434]
    bw (KB/s)  : min=  678, max=31507, per=26.78%, avg=9933.07, stdev=4731.98
    lat (msec) : 20=22.29%, 50=23.09%, 100=20.55%, 250=22.38%, 500=9.92%
    lat (msec) : 750=1.05%, 1000=0.37%, 2000=0.31%, >=2000=0.04%
  cpu          : usr=0.08%, sys=0.10%, ctx=5765, majf=0, minf=26
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=5120/d=0, short=r=0/w=0/d=0
seqwrite4: (groupid=0, jobs=1): err= 0: pid=9032
  write: io=5120.0MB, bw=9325.5KB/s, iops=9 , runt=562214msec
    clat (msec): min=14 , max=4000 , avg=109.72, stdev=170.30
     lat (msec): min=14 , max=4000 , avg=109.80, stdev=170.30
    clat percentiles (msec):
     |  1.00th=[   15],  5.00th=[   17], 10.00th=[   18], 20.00th=[   20],
     | 30.00th=[   26], 40.00th=[   39], 50.00th=[   59], 60.00th=[   81],
     | 70.00th=[  116], 80.00th=[  174], 90.00th=[  265], 95.00th=[  338],
     | 99.00th=[  660], 99.50th=[  938], 99.90th=[ 2147]
    bw (KB/s)  : min=  508, max=35929, per=27.24%, avg=10105.39, stdev=4912.41
    lat (msec) : 20=21.64%, 50=23.79%, 100=21.00%, 250=22.68%, 500=9.22%
    lat (msec) : 750=0.86%, 1000=0.39%, 2000=0.29%, >=2000=0.14%
  cpu          : usr=0.08%, sys=0.09%, ctx=5807, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=5120/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=20480MB, aggrb=37093KB/s, minb=9495KB/s, maxb=9557KB/s, mint=561710msec, maxt=565367msec

Disk stats (read/write):
  sda: ios=0/72719, merge=0/44002, ticks=0/10909055, in_queue=10909124, util=99.98%

ランダムリード512k

[randread512k]
readwrite=randread
blocksize=512k
size=1g
directory=./
direct=1
loops=5
# fio randread512k.fio
randread512k: (g=0): rw=randread, bs=512K-512K/512K-512K, ioengine=sync, iodepth=1
fio 2.0.5
Starting 1 process
randread512k: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 1 (f=1): [r] [100.0% done] [10966K/0K /s] [20 /0  iops] [eta 00m:00s]
randread512k: (groupid=0, jobs=1): err= 0: pid=9073
  read : io=5120.0MB, bw=11334KB/s, iops=22 , runt=462585msec
    clat (msecin=5 , max=1213 , avg=45.17, stdev=35.23
     lat (msec): min=5 , max=1213 , avg=45.17, stdev=35.23
    clat perntiles (msec):
     |  1.00th=[    6],  5.00th=[    6], 10.00th=[    7], 20.00th=[   18],
     | 30.00th=[   28], 40.00th=[   35], 50.00th=[   42], 60.00th=[   49],
     | 70.00th=[   57], 80.00th=[   67], 90.00th=[   83], 95.00th=[  102],
     | 99.00th=[  151], 99.50th=[  178], 99.90th=[  285]
    bw (KB/s)  : min=  640, max=20480, per=100.00%, avg=11392.76, stdev=2561.48
    lat (msec) : 10=17.86%, 20=3.89%, 50=40.21%, 100=32.78%, 250=5.01%
    lat (msec) : 500=0.23%, 750=0.01%, 2000=0.01%
  cpu          : usr=0.02%, sys=0.15%, ctx=10292, majf=0, minf=154
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=10240/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=5120.0MB, aggrb=11333KB/s, minb=11605KB/s, maxb=11605KB/s, mint=462585msec, maxt=462585msec

Disk stats (read/write):
  sda: ios=12996/1670, merge=12779/625, ticks=673643/430788, in_queue=1104454, util=99.92%

ランダムライト512k

[randwrite512k]
readwrite=randwrite
blocksize=512k
size=1g
directory=./
direct=1
loops=5
# fio randwrite512k.fio
randwrite512k: (g=0): rw=randwrite, bs=512K-512K/512K-512K, ioengine=sync, iodepth=1
fio 2.0.5
Starting 1 process
randwrite512k: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 1 (f=1): [w] [100.0% done] [0K/29765K /s] [0 /56  iops] [eta 00m:00s]
randwrite512k: (groupid=0, jobs=1): err= 0: pid=22853
  write: io=5120.0MB, bw=34276KB/s, iops=66 , runt=152959msec
    clat (msec): min=7 , max=1158 , avg=14.89, stdev=28.56
     lat (msec): min=7 , max=1158 , avg=14.93, stdev=28.56
    clat percentiles (msec):
     |  1.00th=[    9],  5.00th=[    9], 10.00th=[    9], 20.00th=[    9],
     | 30.00th=[    9], 40.00th=[   10], 50.00th=[   10], 60.00th=[   11],
     | 70.00th=[   11], 80.00th=[   14], 90.00th=[   23], 95.00th=[   35],
     | 99.00th=[   90], 99.50th=[  128], 99.90th=[  416]
    bw (KB/s)  : min=  873, max=56440, per=100.00%, avg=35549.34, stdev=12392.43
    lat (msec) : 10=60.17%, 20=26.65%, 50=10.36%, 100=2.06%, 250=0.58%
    lat (msec) : 500=0.14%, 750=0.01%, 1000=0.02%, 2000=0.02%
  cpu          : usr=0.27%, sys=1.67%, ctx=10951, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=10240/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=5120.0MB, aggrb=34276KB/s, minb=35099KB/s, maxb=35099KB/s, mint=152959msec, maxt=152959msec

Disk stats (read/write):
  sda: ios=0/23917, merge=0/260483, ticks=0/773513, in_queue=773513, util=98.40%

ランダムリード4k

[randread4k]
readwrite=randread
blocksize=4k
size=100m
directory=./
direct=1
loops=3
# fio randread4k.fio
randread4k: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
fio 2.0.5
Starting 1 process
randread4k: Laying out IO file(s) (1 file(s) / 100MB)
Jobs: 1 (f=1): [r] [100.0% done] [11704K/0K /s] [2857 /0  iops] [eta 00m:00s]
randread4k: (groupid=0, jobs=1): err= 0: pid=22895
  read : io=307200KB, bw=10796KB/s, iops=2699 , runt= 28454msec
    clat (usec): min=256 , max=128032 , avg=365.92, stdev=839.38
     lat (usec): min=256 , max=128033 , avg=366.17, stdev=839.38
    clat percentiles (usec):
     |  1.00th=[  330],  5.00th=[  334], 10.00th=[  338], 20.00th=[  338],
     | 30.00th=[  342], 40.00th=[  342], 50.00th=[  346], 60.00th=[  350],
     | 70.00th=[  354], 80.00th=[  362], 90.00th=[  374], 95.00th=[  382],
     | 99.00th=[  402], 99.50th=[  426], 99.90th=[ 1020]
    bw (KB/s)  : min= 7024, max=11440, per=99.96%, avg=10791.29, stdev=955.92
    lat (usec) : 500=99.79%, 750=0.07%, 1000=0.04%
    lat (msec) : 2=0.03%, 4=0.01%, 10=0.02%, 20=0.03%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=1.39%, sys=3.71%, ctx=76802, majf=0, minf=27
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=76800/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=307200KB, aggrb=10796KB/s, minb=11055KB/s, maxb=11055KB/s, mint=28454msec, maxt=28454msec

Disk stats (read/write):
  sda: ios=76084/241, merge=0/87, ticks=27026/15233, in_queue=42258, util=95.51%

ランダムライト4k

[randwrite4k]
readwrite=randwrite
blocksize=4k
size=100m
directory=./
direct=1
loops=3
# fio randwrite4k.fio
randwrite4k: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
fio 2.0.5
Starting 1 process
randwrite4k: Laying out IO file(s) (1 file(s) / 100MB)
Jobs: 1 (f=1): [w] [100.0% done] [0K/9893K /s] [0 /2415  iops] [eta 00m:00s]
randwrite4k: (groupid=0, jobs=1): err= 0: pid=22907
  write: io=307200KB, bw=8351.3KB/s, iops=2087 , runt= 36786msec
    clat (usec): min=323 , max=131784 , avg=474.03, stdev=1177.03
     lat (usec): min=323 , max=131785 , avg=474.44, stdev=1177.03
    clat percentiles (usec):
     |  1.00th=[  350],  5.00th=[  354], 10.00th=[  358], 20.00th=[  362],
     | 30.00th=[  370], 40.00th=[  374], 50.00th=[  386], 60.00th=[  398],
     | 70.00th=[  502], 80.00th=[  532], 90.00th=[  588], 95.00th=[  604],
     | 99.00th=[  724], 99.50th=[  884], 99.90th=[14272]
    bw (KB/s)  : min=  296, max=10664, per=99.71%, avg=8327.08, stdev=2390.60
    lat (usec) : 500=68.98%, 750=30.12%, 1000=0.50%
    lat (msec) : 2=0.20%, 4=0.04%, 10=0.04%, 20=0.05%, 50=0.05%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=1.30%, sys=4.31%, ctx=76814, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=76800/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=307200KB, aggrb=8351KB/s, minb=8551KB/s, maxb=8551KB/s, mint=36786msec, maxt=36786msec

Disk stats (read/write):
  sda: ios=0/76445, merge=0/392, ticks=0/47345, in_queue=47412, util=95.54%

多重度32のランダムライト4

[randread4k32]
readwrite=randread
blocksize=4k
size=10m
directory=./
direct=1
numjobs=32
loops=3
# fio randread4k32.fio
randread4k32: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
...
randread4k32: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
fio 2.0.5
Starting 32 processes
(略)
randread4k32: (groupid=0, jobs=1): err= 0: pid=9275
  read : io=30720KB, bw=564134 B/s, iops=137 , runt= 55762msec
    clat (usec): min=250 , max=206707 , avg=7254.77, stdev=14087.63
     lat (usec): min=250 , max=206707 , avg=7255.13, stdev=14087.65
    clat percentiles (usec):
     |  1.00th=[  318],  5.00th=[  334], 10.00th=[  334], 20.00th=[  342],
     | 30.00th=[  346], 40.00th=[  354], 50.00th=[  374], 60.00th=[ 1304],
     | 70.00th=[ 4448], 80.00th=[11968], 90.00th=[25216], 95.00th=[36608],
     | 99.00th=[63744], 99.50th=[79360], 99.90th=[109056]
    bw (KB/s)  : min=   87, max= 2216, per=3.09%, avg=528.11, stdev=462.15
    lat (usec) : 500=56.29%, 750=1.65%, 1000=1.13%
    lat (msec) : 2=3.57%, 4=6.48%, 10=8.53%, 20=9.23%, 50=11.05%
    lat (msec) : 100=1.88%, 250=0.18%
  cpu          : usr=0.10%, sys=0.25%, ctx=8095, majf=0, minf=27
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=7680/w=0/d=0, short=r=0/w=0/d=0
randread4k32: (groupid=0, jobs=1): err= 0: pid=9276
  read : io=30720KB, bw=640117 B/s, iops=156 , runt= 49143msec
    clat (usec): min=267 , max=233232 , avg=6393.08, stdev=13452.57
     lat (usec): min=267 , max=233232 , avg=6393.42, stdev=13452.63
    clat percentiles (usec):
     |  1.00th=[  322],  5.00th=[  334], 10.00th=[  338], 20.00th=[  338],
     | 30.00th=[  342], 40.00th=[  350], 50.00th=[  362], 60.00th=[  386],
     | 70.00th=[ 2448], 80.00th=[ 9664], 90.00th=[22656], 95.00th=[34048],
     | 99.00th=[60672], 99.50th=[77312], 99.90th=[111104]
    bw (KB/s)  : min=   62, max= 1801, per=3.66%, avg=625.30, stdev=460.94
    lat (usec) : 500=63.33%, 750=0.64%, 1000=0.69%
    lat (msec) : 2=2.93%, 4=5.48%, 10=7.11%, 20=8.16%, 50=9.88%
    lat (msec) : 100=1.59%, 250=0.18%
  cpu          : usr=0.12%, sys=0.29%, ctx=8059, majf=0, minf=28
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=7680/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=983040KB, aggrb=17083KB/s, minb=546KB/s, maxb=807KB/s, mint=38957msec, maxt=57543msec

Disk stats (read/write):
  sda: ios=245065/214, merge=0/123, ticks=1641829/5071, in_queue=1646895, util=99.88%

多重度32のランダムライト4k

[randwrite4k32]
readwrite=randwrite
blocksize=4k
size=10m
directory=/c
direct=1
numjobs=32
loops=3
# fio randwrite4k32.fio
randwrite4k32: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
...
randwrite4k32: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
fio 2.0.5
Starting 32 processes
(略)
randwrite4k32: (groupid=0, jobs=1): err= 0: pid=9386
  write: io=30720KB, bw=261951 B/s, iops=63 , runt=120088msec
    clat (usec): min=327 , max=5199.8K, avg=15631.22, stdev=137044.57
     lat (usec): min=328 , max=5199.8K, avg=15631.68, stdev=137044.60
    clat percentiles (usec):
     |  1.00th=[  350],  5.00th=[  358], 10.00th=[  362], 20.00th=[  366],
     | 30.00th=[  370], 40.00th=[  378], 50.00th=[  386], 60.00th=[  398],
     | 70.00th=[  410], 80.00th=[  466], 90.00th=[  716], 95.00th=[  756],
     | 99.00th=[460800], 99.50th=[684032], 99.90th=[1810432]
    bw (KB/s)  : min=    0, max= 2424, per=4.43%, avg=303.23, stdev=588.61
    lat (usec) : 500=81.11%, 750=13.41%, 1000=2.02%
    lat (msec) : 2=0.12%, 4=0.17%, 10=0.03%, 20=0.05%, 50=0.13%
    lat (msec) : 100=0.14%, 250=0.89%, 500=1.07%, 750=0.44%, 1000=0.12%
    lat (msec) : 2000=0.22%, >=2000=0.09%
  cpu          : usr=0.04%, sys=0.06%, ctx=7711, majf=0, minf=26
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=7680/d=0, short=r=0/w=0/d=0
randwrite4k32: (groupid=0, jobs=1): err= 0: pid=9387
  write: io=30720KB, bw=340819 B/s, iops=83 , runt= 92299msec
    clat (usec): min=313 , max=5199.1K, avg=12012.91, stdev=122783.50
     lat (usec): min=314 , max=5199.1K, avg=12013.35, stdev=122783.51
    clat percentiles (usec):
     |  1.00th=[  350],  5.00th=[  354], 10.00th=[  358], 20.00th=[  366],
     | 30.00th=[  374], 40.00th=[  378], 50.00th=[  386], 60.00th=[  394],
     | 70.00th=[  410], 80.00th=[  438], 90.00th=[  580], 95.00th=[  716],
     | 99.00th=[350208], 99.50th=[593920], 99.90th=[1810432]
    bw (KB/s)  : min=    0, max= 3847, per=6.04%, avg=414.00, stdev=771.07
    lat (usec) : 500=86.95%, 750=9.61%, 1000=0.72%
    lat (msec) : 2=0.10%, 4=0.17%, 10=0.05%, 20=0.03%, 50=0.10%
    lat (msec) : 100=0.09%, 250=0.68%, 500=0.92%, 750=0.20%, 1000=0.10%
    lat (msec) : 2000=0.20%, >=2000=0.08%
  cpu          : usr=0.05%, sys=0.11%, ctx=7721, majf=0, minf=26
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=7680/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=983040KB, aggrb=6852KB/s, minb=219KB/s, maxb=643KB/s, mint=48878msec, maxt=143459msec

Disk stats (read/write):
  sda: ios=0/245669, merge=0/380, ticks=0/3939189, in_queue=3939238, util=100.00%

上記の結果をまとめた物が下の表になる。多重化した時のIOPSがよく分からないので空白、たぶん省略した場所の数字が必要なのだろうけど、文字数制限を軽く超えるため割愛。

Type Speed (MB/s) IOPS
seq read 66.2 64
seq write 36.1 35
seq read*4 47.7
seq write*4 37.1
rand read 512k 11.3 22
rand write 512k 34.3 66
rand read 4k 10.8 2699
rand write 4k 8.4 2087
rand read 4k*32 17.0
rand write 4k*32 6.9

ランダム4kの性能が非常に良いように見えるが、これはReadyNASのキャッシュが効いているためだと思われる。
逆にシーケンシャルの速度が見劣りするが、それがX-RAID2のせいなのかが気になる。
そのため、X-RAID2のベンチマークを取ってみる。NFSは後回し。