ReadyNAS Ultra4 Plusに触ってみた その1
はてなダイアリーから移行中。
NETGEARのReadyNAS Ultra4 Plus(RNDP400U-100AJS)に触ってみることがあったのでメモ代わりに。
複数のLinuxサーバをiSCSIブートさせたい&NFSでデータ共有させたいという目的のためにストレージとしてこの機種を選定。
各サーバにRAID設定したりバックアップするのがめんどくさかったのと、今後台数が増える可能性も考慮してのストレージを導入した。
QNAPも考えたが、今回はNETGEARにすることに。なお、予算的な制約でUltra6以上にすることは不可能だった。
Ultra4 Plusは無印のUltra4からCPUが変更されて、シングルコアからデュアルコアになったものとのこと。
まずは開封から。
ReadyNAS本体。
磁石で扉が固定されており、開けるとHDDトレイが見える。
一つ外してみたところ、扉もトレイも鍵のようなものでロックされていたりはしない。
トレイの入るスロットの下側にドライブ番号が彫ってあるが、確認しづらい。
全部のトレイを外してみたところ。FOXCONNのシルク印刷が見える。コネクタ部分はSATA専用でSASのドライブは取り付け不可能。
後ろから見たところ。
取っ手(?)に隠れて見えないが、1000BASE-T対応のGbEが2つ備わっている。その隣にUSB 2.0が2ポートとリセットスイッチが見える。
外したHDDトレイ。それぞれ別アングルから。
他の付属品。説明書の他にHDD取り付け用のネジとかSTPケーブル、電源ケーブル。
なぜか3本ある電源ケーブル。日本国内で一般的な形状の他、ドイツ/フランス用、オーストラリア用のケーブルが付属している。
本体側はPC用電源で一般的形状のため、一般的な電源ケーブルも利用可能と思われる。責任は取れませんが。
付属のHDD取り付け用のネジ。余りは無いので無くすと面倒。
今回はSeagateのST2000DL003を使用。どうせGbE経由なので速度は期待してない。
トレイに取り付けると下のようになる。
HDD2台をスロットに納めてみたところ。奥まで押し込んだ後にレバーを下げると固定される。
電源を入れると初期化が始まる。アレイを設定しているわけでは無いので、すぐ終わった。
続いてRAIDarからの接続を待ち始める。10分以内にPCにインストールしたRAIDerからアクセスすると良いのだが、面倒なのでタイムアウトまで待つ。このあいだにDHCPサーバとLANで接続しておく。
タイムアウト後にIPアドレスが表示されるので http://IPアドレス/admin/ にWebブラウザでアクセスすると管理画面が表示される。id:admin、初期パスワード:netgear1。
同時にアレイの初期化が始まる。
つづく
ReadyNAS Ultra4 Plusに触ってみた その2
ReadyNASはブラウザからだいたいの設定を行う事ができる。
ReadyNASのWebUIにアクセスしたところから続き。
下のような画面が表示される。RAIDiator 4.2.19は現時点での最新Ver。
HDD4台の場合は、RAIDarを使わないと自動的にX-RAID2になり、このWebUIではRAID5等に変更できない。
初期状態で216MBほどディスク領域が利用されていた。2TBx4ではアレイの初期化に5時間程度かかる模様。
UPSが検出されないとシステムの電源タブはこのような表示になっており、UPSが接続された他のReadyNASと連動させる設定ができるようになっている。
UPSを接続すると項目内容が変化する。
また、他のReadyNASを連動させることもできるようになる。
PCからの参照も可能ではないかと思ったが、それは後に調査することに。
UPSを接続すると動作状況にUPSの項目が表示される。今回はAPCのSmart-UPS 500をUSBで接続した。
ベンチマークにつづく。
ReadyNAS Ultra4 Plusに触ってみた その3
主な利用用途であるiSCSIとNFSのベンチマークを比較したい。まずはiSCSIから。
マルチパスについて
iSCSIブートはdm-multipathと相性が悪いのでHDD起動のサーバに2ポートの1000BASE-Tを取り付けてReadyNASのLAN1、LAN2とそれぞれ直接接続。
iSCSIの設定とdm-multipathの設定を行った。
# multipath -l mpathe (360014052e419090150ad002000000000) dm-0 LIO-ORG,FILEIO size=20G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=0 status=active |- 45:0:0:0 sdg 8:96 active undef running `- 44:0:0:0 sdi 8:128 active undef running mpathd (360014052e419090150ad001000000000) dm-3 LIO-ORG,FILEIO size=200G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=0 status=active |- 42:0:0:0 sde 8:64 active undef running `- 43:0:0:0 sdf 8:80 active undef running mpathc (360014052e419090150ad004000000000) dm-1 LIO-ORG,FILEIO size=20G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=0 status=active |- 40:0:0:0 sdc 8:32 active undef running `- 41:0:0:0 sdd 8:48 active undef running mpathf (360014052e419090150ad003000000000) dm-2 LIO-ORG,FILEIO size=20G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=0 status=active |- 47:0:0:0 sdj 8:144 active undef running `- 46:0:0:0 sdh 8:112 active undef running
/dev/sdcと/dev/sddが同一のiSCSIボリュームでそれぞれ別経路から見ていることになる。それをdm-multipathで/dev/mapper/mpathcとしている。
いろいろボリュームが見えているけど気にしない。別に異常では無い。
それではhdparmで簡単にベンチマーク。
# hdparm -tT /dev/sdc /dev/sdc: Timing cached reads: 25690 MB in 2.00 seconds = 12865.15 MB/sec Timing buffered disk reads: 296 MB in 3.02 seconds = 98.00 MB/sec # hdparm -tT /dev/sdd /dev/sdd: Timing cached reads: 26050 MB in 2.00 seconds = 13045.94 MB/sec Timing buffered disk reads: 312 MB in 3.01 seconds = 103.78 MB/sec # hdparm -tT /dev/mapper/mpathc /dev/mapper/mpathc: Timing cached reads: 25616 MB in 2.00 seconds = 12829.08 MB/sec Timing buffered disk reads: 256 MB in 3.02 seconds = 84.76 MB/sec
/dev/sdXに単独にアクセスした場合よりパフォーマンスが低くなっている。どうもマルチパス構成としてうまく動作していない模様。
ReadyNASの方が対応しないようになっていると思われる。
iSCSIのベンチマーク
iSCSIブート設定にしたサーバでReadyNASのiSCSIボリュームから起動してベンチマーク。なんでHDD起動からテストしてないかと言うと、本番設定をした後で思いついたから。
ext3でフォーマットした領域に対してfio 2.0.5を使ってベンチマークを実行。パラメータなんかは複眼中心さんを参考にさせていただきました。
シーケンシャルリード
[seqread] readwrite=read blocksize=1m size=1g directory=./ direct=1 loops=5
# fio seqread.fio seqread: (g=0): rw=read, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1 fio 2.0.5 Starting 1 process Jobs: 1 (f=1): [R] [100.0% done] [30287K/0K /s] [28 /0 iops] [eta 00m:00s] seqread: (groupid=0, jobs=1): err= 0: pid=7344 read : io=5120.0MB, bw=66234KB/s, iops=64 , runt= 79157msec clat (msec): min=11 , max=528 , avg=15.46, stdev=17.15 lat (msec): min=11 , max=528 , avg=15.46, stdev=17.15 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 13], 40.00th=[ 13], 50.00th=[ 13], 60.00th=[ 13], | 70.00th=[ 13], 80.00th=[ 14], 90.00th=[ 17], 95.00th=[ 29], | 99.00th=[ 68], 99.50th=[ 120], 99.90th=[ 227] bw (KB/s) : min= 9582, max=83136, per=100.00%, avg=67208.73, stdev=19041.60 lat (msec) : 20=91.35%, 50=6.91%, 100=1.11%, 250=0.53%, 500=0.08% lat (msec) : 750=0.02% cpu : usr=0.05%, sys=0.70%, ctx=5132, majf=0, minf=282 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=5120/w=0/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): READ: io=5120.0MB, aggrb=66233KB/s, minb=67823KB/s, maxb=67823KB/s, mint=79157msec, maxt=79157msec Disk stats (read/write): sda: ios=12928/220, merge=12443/118, ticks=201216/19282, in_queue=220501, util=99.71%
シーケンシャルライト
[seqwrite] readwrite=write blocksize=1m size=1g directory=./ direct=1 loops=5
# fio seqwrite.fio seqwrite: (g=0): rw=write, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1 fio 2.0.5 Starting 1 process seqwrite: Laying out IO file(s) (1 file(s) / 1024MB) Jobs: 1 (f=1): [W] [100.0% done] [0K/8355K /s] [0 /7 iops] [eta 00m:00s] seqwrite: (groupid=0, jobs=1): err= 0: pid=8990 write: io=5120.0MB, bw=36115KB/s, iops=35 , runt=145173msec clat (msec): min=13 , max=733 , avg=28.26, stdev=36.18 lat (msec): min=13 , max=733 , avg=28.35, stdev=36.18 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 15], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 21], | 70.00th=[ 26], 80.00th=[ 33], 90.00th=[ 48], 95.00th=[ 76], | 99.00th=[ 165], 99.50th=[ 223], 99.90th=[ 469] bw (KB/s) : min= 2409, max=63109, per=100.00%, avg=37125.47, stdev=13231.65 lat (msec) : 20=59.77%, 50=30.78%, 100=6.23%, 250=2.75%, 500=0.37% lat (msec) : 750=0.10% cpu : usr=0.31%, sys=0.44%, ctx=5179, majf=0, minf=25 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=5120/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): WRITE: io=5120.0MB, aggrb=36114KB/s, minb=36981KB/s, maxb=36981KB/s, mint=145173msec, maxt=145173msec Disk stats (read/write): sda: ios=2/25363, merge=0/1425, ticks=37/778774, in_queue=778822, util=99.51%
多重度4のシーケンシャルリード
[seqread4] readwrite=read blocksize=1m size=1g directory=./ direct=1 numjobs=4 loops=5
# fio seqread4.fio seqread4: (g=0): rw=read, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1 ... seqread4: (g=0): rw=read, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1 fio 2.0.5 Starting 4 processes seqread4: Laying out IO file(s) (1 file(s) / 1024MB) seqread4: Laying out IO file(s) (1 file(s) / 1024MB) seqread4: Laying out IO file(s) (1 file(s) / 1024MB) seqread4: Laying out IO file(s) (1 file(s) / 1024MB) Jobs: 4 (f=4): [RRRR] [300.0% done] [0K/0K /s] [0 /0 iops] [eta 1158050441d:07hJobs: 4 (f=4): [RRRR] [inf% done] [0K/0K /s] [0 /0 iops] [eta 1158050441d:07h:0Jobs: 4 (f=4): [RRRR] [inf% done] [0K/0K /s] [0 /0 iops] [eta 1158050441d:07h:0Jobs: 4 (f=4): [RRRR] [inf% done] [0K/0K /s] [0 /0 iops] [eta 1158050441d:07h:0Jobs: 4 (f=4): [RRRR] [400.0% done] [0K/0K /s] [0 /0 iops] [eta 1158050441d:07hJobs: 4 (f=4): [RRRR] [inf% done] [963K/0K /s] [0 /0 iops] [eta 1158050441d:07hJobs: 4 (f=4): [RRRR] [inf% done] [5221K/0K /s] [4 /0 iops] [eta 1158050441d:07Jobs: 4 (f=4): [RRRR] [inf% done] [13577K/0K /s] [12 /0 iops] [eta 1158050441d:Jobs: 4 (f=4): [RRRR] [0.0% done] [7310K/0K /s] [6 /0 iops] [eta 09h:57m:13s] Jobs: 1 (f=1): [__R_] [98.9% done] [63708K/0K /s] [60 /0 iops] [eta 00m:05s] seqread4: (groupid=0, jobs=1): err= 0: pid=9002 read : io=5120.0MB, bw=13432KB/s, iops=13 , runt=390342msec clat (msec): min=11 , max=2952 , avg=76.23, stdev=143.01 lat (msec): min=11 , max=2952 , avg=76.23, stdev=143.01 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 13], | 30.00th=[ 13], 40.00th=[ 13], 50.00th=[ 14], 60.00th=[ 19], | 70.00th=[ 30], 80.00th=[ 115], 90.00th=[ 318], 95.00th=[ 347], | 99.00th=[ 424], 99.50th=[ 490], 99.90th=[ 1975] bw (KB/s) : min= 346, max=30658, per=29.96%, avg=14279.41, stdev=6330.28 lat (msec) : 20=61.35%, 50=15.51%, 100=2.79%, 250=7.29%, 500=12.60% lat (msec) : 750=0.31%, 1000=0.02%, 2000=0.04%, >=2000=0.10% cpu : usr=0.01%, sys=0.13%, ctx=5218, majf=0, minf=282 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=5120/w=0/d=0, short=r=0/w=0/d=0 seqread4: (groupid=0, jobs=1): err= 0: pid=9003 read : io=5120.0MB, bw=12326KB/s, iops=12 , runt=425368msec clat (msec): min=11 , max=5712 , avg=83.08, stdev=175.50 lat (msec): min=11 , max=5712 , avg=83.08, stdev=175.50 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 13], | 30.00th=[ 13], 40.00th=[ 15], 50.00th=[ 22], 60.00th=[ 29], | 70.00th=[ 43], 80.00th=[ 124], 90.00th=[ 302], 95.00th=[ 343], | 99.00th=[ 445], 99.50th=[ 840], 99.90th=[ 1549] bw (KB/s) : min= 179, max=50712, per=28.47%, avg=13571.64, stdev=7380.59 lat (msec) : 20=48.46%, 50=24.39%, 100=5.72%, 250=9.14%, 500=11.50% lat (msec) : 750=0.27%, 1000=0.10%, 2000=0.37%, >=2000=0.04% cpu : usr=0.01%, sys=0.12%, ctx=5225, majf=0, minf=283 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=5120/w=0/d=0, short=r=0/w=0/d=0 seqread4: (groupid=0, jobs=1): err= 0: pid=9004 read : io=5120.0MB, bw=11917KB/s, iops=11 , runt=439947msec clat (msec): min=11 , max=9006 , avg=85.92, stdev=309.61 lat (msec): min=11 , max=9006 , avg=85.92, stdev=309.61 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 13], | 30.00th=[ 13], 40.00th=[ 13], 50.00th=[ 15], 60.00th=[ 21], | 70.00th=[ 32], 80.00th=[ 64], 90.00th=[ 314], 95.00th=[ 347], | 99.00th=[ 465], 99.50th=[ 1188], 99.90th=[ 4621] bw (KB/s) : min= 173, max=76148, per=31.86%, avg=15188.06, stdev=11432.61 lat (msec) : 20=59.73%, 50=18.26%, 100=3.52%, 250=6.52%, 500=11.05% lat (msec) : 750=0.33%, 1000=0.08%, 2000=0.14%, >=2000=0.37% cpu : usr=0.01%, sys=0.13%, ctx=5241, majf=0, minf=283 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=5120/w=0/d=0, short=r=0/w=0/d=0 seqread4: (groupid=0, jobs=1): err= 0: pid=9005 read : io=5120.0MB, bw=13183KB/s, iops=12 , runt=397707msec clat (msec): min=11 , max=2951 , avg=77.67, stdev=147.40 lat (msec): min=11 , max=2951 , avg=77.67, stdev=147.40 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 13], | 30.00th=[ 13], 40.00th=[ 14], 50.00th=[ 17], 60.00th=[ 21], | 70.00th=[ 32], 80.00th=[ 106], 90.00th=[ 314], 95.00th=[ 347], | 99.00th=[ 441], 99.50th=[ 619], 99.90th=[ 1795] bw (KB/s) : min= 347, max=33851, per=29.67%, avg=14144.82, stdev=6534.68 lat (msec) : 20=58.89%, 50=18.38%, 100=2.62%, 250=7.17%, 500=12.23% lat (msec) : 750=0.37%, 1000=0.14%, 2000=0.12%, >=2000=0.10% cpu : usr=0.01%, sys=0.13%, ctx=5220, majf=0, minf=282 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=5120/w=0/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): READ: io=20480MB, aggrb=47668KB/s, minb=12203KB/s, maxb=13753KB/s, mint=390342msec, maxt=439947msec Disk stats (read/write): sda: ios=59655/1473, merge=46382/960, ticks=9162485/989380, in_queue=10027085, util=100.00%
多重度4のシーケンシャルライト
[seqwrite4] readwrite=write blocksize=1m size=1g directory=./ direct=1 numjobs=4 loops=5
# fio seqwrite4.fio seqwrite4: (g=0): rw=write, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1 ... seqwrite4: (g=0): rw=write, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1 fio 2.0.5 Starting 4 processes seqwrite4: Laying out IO file(s) (1 file(s) / 1024MB) seqwrite4: Laying out IO file(s) (1 file(s) / 1024MB) seqwrite4: Laying out IO file(s) (1 file(s) / 1024MB) seqwrite4: Laying out IO file(s) (1 file(s) / 1024MB) Jobs: 1 (f=1): [W___] [100.0% done] [0K/42820K /s] [0 /40 iops] [eta 00m:00s] seqwrite4: (groupid=0, jobs=1): err= 0: pid=9029 write: io=5120.0MB, bw=9273.5KB/s, iops=9 , runt=565367msec clat (msec): min=14 , max=2243 , avg=110.33, stdev=141.81 lat (msec): min=14 , max=2243 , avg=110.42, stdev=141.81 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 17], 10.00th=[ 18], 20.00th=[ 20], | 30.00th=[ 27], 40.00th=[ 41], 50.00th=[ 61], 60.00th=[ 84], | 70.00th=[ 122], 80.00th=[ 180], 90.00th=[ 273], 95.00th=[ 343], | 99.00th=[ 627], 99.50th=[ 906], 99.90th=[ 1319] bw (KB/s) : min= 802, max=50906, per=26.56%, avg=9852.55, stdev=5032.29 lat (msec) : 20=20.68%, 50=23.42%, 100=21.37%, 250=22.73%, 500=10.04% lat (msec) : 750=1.05%, 1000=0.25%, 2000=0.43%, >=2000=0.02% cpu : usr=0.08%, sys=0.10%, ctx=5753, majf=0, minf=25 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=5120/d=0, short=r=0/w=0/d=0 seqwrite4: (groupid=0, jobs=1): err= 0: pid=9030 write: io=5120.0MB, bw=9315.6KB/s, iops=9 , runt=562809msec clat (msec): min=13 , max=4681 , avg=109.83, stdev=164.63 lat (msec): min=13 , max=4681 , avg=109.92, stdev=164.63 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 17], 10.00th=[ 18], 20.00th=[ 19], | 30.00th=[ 25], 40.00th=[ 39], 50.00th=[ 60], 60.00th=[ 81], | 70.00th=[ 116], 80.00th=[ 180], 90.00th=[ 269], 95.00th=[ 343], | 99.00th=[ 627], 99.50th=[ 881], 99.90th=[ 1860] bw (KB/s) : min= 344, max=34403, per=27.01%, avg=10020.61, stdev=4993.68 lat (msec) : 20=22.99%, 50=22.42%, 100=21.11%, 250=21.99%, 500=9.80% lat (msec) : 750=0.92%, 1000=0.35%, 2000=0.33%, >=2000=0.08% cpu : usr=0.08%, sys=0.09%, ctx=5741, majf=0, minf=25 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=5120/d=0, short=r=0/w=0/d=0 seqwrite4: (groupid=0, jobs=1): err= 0: pid=9031 write: io=5120.0MB, bw=9333.9KB/s, iops=9 , runt=561710msec clat (msec): min=13 , max=2428 , avg=109.62, stdev=145.08 lat (msec): min=13 , max=2429 , avg=109.70, stdev=145.08 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 17], 10.00th=[ 18], 20.00th=[ 19], | 30.00th=[ 26], 40.00th=[ 39], 50.00th=[ 60], 60.00th=[ 82], | 70.00th=[ 119], 80.00th=[ 180], 90.00th=[ 269], 95.00th=[ 347], | 99.00th=[ 619], 99.50th=[ 906], 99.90th=[ 1434] bw (KB/s) : min= 678, max=31507, per=26.78%, avg=9933.07, stdev=4731.98 lat (msec) : 20=22.29%, 50=23.09%, 100=20.55%, 250=22.38%, 500=9.92% lat (msec) : 750=1.05%, 1000=0.37%, 2000=0.31%, >=2000=0.04% cpu : usr=0.08%, sys=0.10%, ctx=5765, majf=0, minf=26 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=5120/d=0, short=r=0/w=0/d=0 seqwrite4: (groupid=0, jobs=1): err= 0: pid=9032 write: io=5120.0MB, bw=9325.5KB/s, iops=9 , runt=562214msec clat (msec): min=14 , max=4000 , avg=109.72, stdev=170.30 lat (msec): min=14 , max=4000 , avg=109.80, stdev=170.30 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 17], 10.00th=[ 18], 20.00th=[ 20], | 30.00th=[ 26], 40.00th=[ 39], 50.00th=[ 59], 60.00th=[ 81], | 70.00th=[ 116], 80.00th=[ 174], 90.00th=[ 265], 95.00th=[ 338], | 99.00th=[ 660], 99.50th=[ 938], 99.90th=[ 2147] bw (KB/s) : min= 508, max=35929, per=27.24%, avg=10105.39, stdev=4912.41 lat (msec) : 20=21.64%, 50=23.79%, 100=21.00%, 250=22.68%, 500=9.22% lat (msec) : 750=0.86%, 1000=0.39%, 2000=0.29%, >=2000=0.14% cpu : usr=0.08%, sys=0.09%, ctx=5807, majf=0, minf=25 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=5120/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): WRITE: io=20480MB, aggrb=37093KB/s, minb=9495KB/s, maxb=9557KB/s, mint=561710msec, maxt=565367msec Disk stats (read/write): sda: ios=0/72719, merge=0/44002, ticks=0/10909055, in_queue=10909124, util=99.98%
ランダムリード512k
[randread512k] readwrite=randread blocksize=512k size=1g directory=./ direct=1 loops=5
# fio randread512k.fio randread512k: (g=0): rw=randread, bs=512K-512K/512K-512K, ioengine=sync, iodepth=1 fio 2.0.5 Starting 1 process randread512k: Laying out IO file(s) (1 file(s) / 1024MB) Jobs: 1 (f=1): [r] [100.0% done] [10966K/0K /s] [20 /0 iops] [eta 00m:00s] randread512k: (groupid=0, jobs=1): err= 0: pid=9073 read : io=5120.0MB, bw=11334KB/s, iops=22 , runt=462585msec clat (msecin=5 , max=1213 , avg=45.17, stdev=35.23 lat (msec): min=5 , max=1213 , avg=45.17, stdev=35.23 clat perntiles (msec): | 1.00th=[ 6], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 18], | 30.00th=[ 28], 40.00th=[ 35], 50.00th=[ 42], 60.00th=[ 49], | 70.00th=[ 57], 80.00th=[ 67], 90.00th=[ 83], 95.00th=[ 102], | 99.00th=[ 151], 99.50th=[ 178], 99.90th=[ 285] bw (KB/s) : min= 640, max=20480, per=100.00%, avg=11392.76, stdev=2561.48 lat (msec) : 10=17.86%, 20=3.89%, 50=40.21%, 100=32.78%, 250=5.01% lat (msec) : 500=0.23%, 750=0.01%, 2000=0.01% cpu : usr=0.02%, sys=0.15%, ctx=10292, majf=0, minf=154 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=10240/w=0/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): READ: io=5120.0MB, aggrb=11333KB/s, minb=11605KB/s, maxb=11605KB/s, mint=462585msec, maxt=462585msec Disk stats (read/write): sda: ios=12996/1670, merge=12779/625, ticks=673643/430788, in_queue=1104454, util=99.92%
ランダムライト512k
[randwrite512k] readwrite=randwrite blocksize=512k size=1g directory=./ direct=1 loops=5
# fio randwrite512k.fio randwrite512k: (g=0): rw=randwrite, bs=512K-512K/512K-512K, ioengine=sync, iodepth=1 fio 2.0.5 Starting 1 process randwrite512k: Laying out IO file(s) (1 file(s) / 1024MB) Jobs: 1 (f=1): [w] [100.0% done] [0K/29765K /s] [0 /56 iops] [eta 00m:00s] randwrite512k: (groupid=0, jobs=1): err= 0: pid=22853 write: io=5120.0MB, bw=34276KB/s, iops=66 , runt=152959msec clat (msec): min=7 , max=1158 , avg=14.89, stdev=28.56 lat (msec): min=7 , max=1158 , avg=14.93, stdev=28.56 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 9], 10.00th=[ 9], 20.00th=[ 9], | 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 11], 80.00th=[ 14], 90.00th=[ 23], 95.00th=[ 35], | 99.00th=[ 90], 99.50th=[ 128], 99.90th=[ 416] bw (KB/s) : min= 873, max=56440, per=100.00%, avg=35549.34, stdev=12392.43 lat (msec) : 10=60.17%, 20=26.65%, 50=10.36%, 100=2.06%, 250=0.58% lat (msec) : 500=0.14%, 750=0.01%, 1000=0.02%, 2000=0.02% cpu : usr=0.27%, sys=1.67%, ctx=10951, majf=0, minf=25 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=10240/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): WRITE: io=5120.0MB, aggrb=34276KB/s, minb=35099KB/s, maxb=35099KB/s, mint=152959msec, maxt=152959msec Disk stats (read/write): sda: ios=0/23917, merge=0/260483, ticks=0/773513, in_queue=773513, util=98.40%
ランダムリード4k
[randread4k] readwrite=randread blocksize=4k size=100m directory=./ direct=1 loops=3
# fio randread4k.fio randread4k: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1 fio 2.0.5 Starting 1 process randread4k: Laying out IO file(s) (1 file(s) / 100MB) Jobs: 1 (f=1): [r] [100.0% done] [11704K/0K /s] [2857 /0 iops] [eta 00m:00s] randread4k: (groupid=0, jobs=1): err= 0: pid=22895 read : io=307200KB, bw=10796KB/s, iops=2699 , runt= 28454msec clat (usec): min=256 , max=128032 , avg=365.92, stdev=839.38 lat (usec): min=256 , max=128033 , avg=366.17, stdev=839.38 clat percentiles (usec): | 1.00th=[ 330], 5.00th=[ 334], 10.00th=[ 338], 20.00th=[ 338], | 30.00th=[ 342], 40.00th=[ 342], 50.00th=[ 346], 60.00th=[ 350], | 70.00th=[ 354], 80.00th=[ 362], 90.00th=[ 374], 95.00th=[ 382], | 99.00th=[ 402], 99.50th=[ 426], 99.90th=[ 1020] bw (KB/s) : min= 7024, max=11440, per=99.96%, avg=10791.29, stdev=955.92 lat (usec) : 500=99.79%, 750=0.07%, 1000=0.04% lat (msec) : 2=0.03%, 4=0.01%, 10=0.02%, 20=0.03%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=1.39%, sys=3.71%, ctx=76802, majf=0, minf=27 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=76800/w=0/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): READ: io=307200KB, aggrb=10796KB/s, minb=11055KB/s, maxb=11055KB/s, mint=28454msec, maxt=28454msec Disk stats (read/write): sda: ios=76084/241, merge=0/87, ticks=27026/15233, in_queue=42258, util=95.51%
ランダムライト4k
[randwrite4k] readwrite=randwrite blocksize=4k size=100m directory=./ direct=1 loops=3
# fio randwrite4k.fio randwrite4k: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1 fio 2.0.5 Starting 1 process randwrite4k: Laying out IO file(s) (1 file(s) / 100MB) Jobs: 1 (f=1): [w] [100.0% done] [0K/9893K /s] [0 /2415 iops] [eta 00m:00s] randwrite4k: (groupid=0, jobs=1): err= 0: pid=22907 write: io=307200KB, bw=8351.3KB/s, iops=2087 , runt= 36786msec clat (usec): min=323 , max=131784 , avg=474.03, stdev=1177.03 lat (usec): min=323 , max=131785 , avg=474.44, stdev=1177.03 clat percentiles (usec): | 1.00th=[ 350], 5.00th=[ 354], 10.00th=[ 358], 20.00th=[ 362], | 30.00th=[ 370], 40.00th=[ 374], 50.00th=[ 386], 60.00th=[ 398], | 70.00th=[ 502], 80.00th=[ 532], 90.00th=[ 588], 95.00th=[ 604], | 99.00th=[ 724], 99.50th=[ 884], 99.90th=[14272] bw (KB/s) : min= 296, max=10664, per=99.71%, avg=8327.08, stdev=2390.60 lat (usec) : 500=68.98%, 750=30.12%, 1000=0.50% lat (msec) : 2=0.20%, 4=0.04%, 10=0.04%, 20=0.05%, 50=0.05% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=1.30%, sys=4.31%, ctx=76814, majf=0, minf=25 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=76800/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): WRITE: io=307200KB, aggrb=8351KB/s, minb=8551KB/s, maxb=8551KB/s, mint=36786msec, maxt=36786msec Disk stats (read/write): sda: ios=0/76445, merge=0/392, ticks=0/47345, in_queue=47412, util=95.54%
多重度32のランダムライト4
[randread4k32] readwrite=randread blocksize=4k size=10m directory=./ direct=1 numjobs=32 loops=3
# fio randread4k32.fio randread4k32: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1 ... randread4k32: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1 fio 2.0.5 Starting 32 processes (略) randread4k32: (groupid=0, jobs=1): err= 0: pid=9275 read : io=30720KB, bw=564134 B/s, iops=137 , runt= 55762msec clat (usec): min=250 , max=206707 , avg=7254.77, stdev=14087.63 lat (usec): min=250 , max=206707 , avg=7255.13, stdev=14087.65 clat percentiles (usec): | 1.00th=[ 318], 5.00th=[ 334], 10.00th=[ 334], 20.00th=[ 342], | 30.00th=[ 346], 40.00th=[ 354], 50.00th=[ 374], 60.00th=[ 1304], | 70.00th=[ 4448], 80.00th=[11968], 90.00th=[25216], 95.00th=[36608], | 99.00th=[63744], 99.50th=[79360], 99.90th=[109056] bw (KB/s) : min= 87, max= 2216, per=3.09%, avg=528.11, stdev=462.15 lat (usec) : 500=56.29%, 750=1.65%, 1000=1.13% lat (msec) : 2=3.57%, 4=6.48%, 10=8.53%, 20=9.23%, 50=11.05% lat (msec) : 100=1.88%, 250=0.18% cpu : usr=0.10%, sys=0.25%, ctx=8095, majf=0, minf=27 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=7680/w=0/d=0, short=r=0/w=0/d=0 randread4k32: (groupid=0, jobs=1): err= 0: pid=9276 read : io=30720KB, bw=640117 B/s, iops=156 , runt= 49143msec clat (usec): min=267 , max=233232 , avg=6393.08, stdev=13452.57 lat (usec): min=267 , max=233232 , avg=6393.42, stdev=13452.63 clat percentiles (usec): | 1.00th=[ 322], 5.00th=[ 334], 10.00th=[ 338], 20.00th=[ 338], | 30.00th=[ 342], 40.00th=[ 350], 50.00th=[ 362], 60.00th=[ 386], | 70.00th=[ 2448], 80.00th=[ 9664], 90.00th=[22656], 95.00th=[34048], | 99.00th=[60672], 99.50th=[77312], 99.90th=[111104] bw (KB/s) : min= 62, max= 1801, per=3.66%, avg=625.30, stdev=460.94 lat (usec) : 500=63.33%, 750=0.64%, 1000=0.69% lat (msec) : 2=2.93%, 4=5.48%, 10=7.11%, 20=8.16%, 50=9.88% lat (msec) : 100=1.59%, 250=0.18% cpu : usr=0.12%, sys=0.29%, ctx=8059, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=7680/w=0/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): READ: io=983040KB, aggrb=17083KB/s, minb=546KB/s, maxb=807KB/s, mint=38957msec, maxt=57543msec Disk stats (read/write): sda: ios=245065/214, merge=0/123, ticks=1641829/5071, in_queue=1646895, util=99.88%
多重度32のランダムライト4k
[randwrite4k32] readwrite=randwrite blocksize=4k size=10m directory=/c direct=1 numjobs=32 loops=3
# fio randwrite4k32.fio randwrite4k32: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1 ... randwrite4k32: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1 fio 2.0.5 Starting 32 processes (略) randwrite4k32: (groupid=0, jobs=1): err= 0: pid=9386 write: io=30720KB, bw=261951 B/s, iops=63 , runt=120088msec clat (usec): min=327 , max=5199.8K, avg=15631.22, stdev=137044.57 lat (usec): min=328 , max=5199.8K, avg=15631.68, stdev=137044.60 clat percentiles (usec): | 1.00th=[ 350], 5.00th=[ 358], 10.00th=[ 362], 20.00th=[ 366], | 30.00th=[ 370], 40.00th=[ 378], 50.00th=[ 386], 60.00th=[ 398], | 70.00th=[ 410], 80.00th=[ 466], 90.00th=[ 716], 95.00th=[ 756], | 99.00th=[460800], 99.50th=[684032], 99.90th=[1810432] bw (KB/s) : min= 0, max= 2424, per=4.43%, avg=303.23, stdev=588.61 lat (usec) : 500=81.11%, 750=13.41%, 1000=2.02% lat (msec) : 2=0.12%, 4=0.17%, 10=0.03%, 20=0.05%, 50=0.13% lat (msec) : 100=0.14%, 250=0.89%, 500=1.07%, 750=0.44%, 1000=0.12% lat (msec) : 2000=0.22%, >=2000=0.09% cpu : usr=0.04%, sys=0.06%, ctx=7711, majf=0, minf=26 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=7680/d=0, short=r=0/w=0/d=0 randwrite4k32: (groupid=0, jobs=1): err= 0: pid=9387 write: io=30720KB, bw=340819 B/s, iops=83 , runt= 92299msec clat (usec): min=313 , max=5199.1K, avg=12012.91, stdev=122783.50 lat (usec): min=314 , max=5199.1K, avg=12013.35, stdev=122783.51 clat percentiles (usec): | 1.00th=[ 350], 5.00th=[ 354], 10.00th=[ 358], 20.00th=[ 366], | 30.00th=[ 374], 40.00th=[ 378], 50.00th=[ 386], 60.00th=[ 394], | 70.00th=[ 410], 80.00th=[ 438], 90.00th=[ 580], 95.00th=[ 716], | 99.00th=[350208], 99.50th=[593920], 99.90th=[1810432] bw (KB/s) : min= 0, max= 3847, per=6.04%, avg=414.00, stdev=771.07 lat (usec) : 500=86.95%, 750=9.61%, 1000=0.72% lat (msec) : 2=0.10%, 4=0.17%, 10=0.05%, 20=0.03%, 50=0.10% lat (msec) : 100=0.09%, 250=0.68%, 500=0.92%, 750=0.20%, 1000=0.10% lat (msec) : 2000=0.20%, >=2000=0.08% cpu : usr=0.05%, sys=0.11%, ctx=7721, majf=0, minf=26 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=7680/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): WRITE: io=983040KB, aggrb=6852KB/s, minb=219KB/s, maxb=643KB/s, mint=48878msec, maxt=143459msec Disk stats (read/write): sda: ios=0/245669, merge=0/380, ticks=0/3939189, in_queue=3939238, util=100.00%
上記の結果をまとめた物が下の表になる。多重化した時のIOPSがよく分からないので空白、たぶん省略した場所の数字が必要なのだろうけど、文字数制限を軽く超えるため割愛。
Type | Speed (MB/s) | IOPS |
---|---|---|
seq read | 66.2 | 64 |
seq write | 36.1 | 35 |
seq read*4 | 47.7 | |
seq write*4 | 37.1 | |
rand read 512k | 11.3 | 22 |
rand write 512k | 34.3 | 66 |
rand read 4k | 10.8 | 2699 |
rand write 4k | 8.4 | 2087 |
rand read 4k*32 | 17.0 | |
rand write 4k*32 | 6.9 |
ランダム4kの性能が非常に良いように見えるが、これはReadyNASのキャッシュが効いているためだと思われる。
逆にシーケンシャルの速度が見劣りするが、それがX-RAID2のせいなのかが気になる。
そのため、X-RAID2のベンチマークを取ってみる。NFSは後回し。