$ fio --name TEST --eta-newline=5s --filename=fio-tempfile.dat --rw=write --size=500m --io_size=10g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=60 --group_reporting
TEST: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
fio-3.12
Starting 1 process
TEST: Laying out IO file (1 file / 500MiB)
Jobs: 1 (f=1): [W(1)][11.7%][w=52.1MiB/s][w=52 IOPS][eta 00m:53s]
Jobs: 1 (f=1): [W(1)][21.7%][w=58.0MiB/s][w=58 IOPS][eta 00m:47s]
Jobs: 1 (f=1): [W(1)][31.7%][w=72.0MiB/s][w=72 IOPS][eta 00m:41s]
Jobs: 1 (f=1): [W(1)][41.7%][w=59.0MiB/s][w=59 IOPS][eta 00m:35s]
Jobs: 1 (f=1): [W(1)][51.7%][w=53.1MiB/s][w=53 IOPS][eta 00m:29s]
Jobs: 1 (f=1): [W(1)][61.7%][w=70.0MiB/s][w=70 IOPS][eta 00m:23s]
Jobs: 1 (f=1): [W(1)][71.7%][w=58.0MiB/s][w=58 IOPS][eta 00m:17s]
Jobs: 1 (f=1): [W(1)][81.7%][w=56.1MiB/s][w=56 IOPS][eta 00m:11s]
Jobs: 1 (f=1): [W(1)][91.7%][w=76.0MiB/s][w=76 IOPS][eta 00m:05s]
Jobs: 1 (f=0): [f(1)][100.0%][w=48.0MiB/s][w=48 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=2510: Mon Oct 14 11:41:44 2019
write: IOPS=55, BW=55.6MiB/s (58.3MB/s)(3370MiB/60585msec); 0 zone resets
slat (usec): min=343, max=502171, avg=17032.43, stdev=38500.24
clat (msec): min=33, max=1135, avg=555.86, stdev=142.77
lat (msec): min=35, max=1195, avg=572.90, stdev=144.77
clat percentiles (msec):
| 1.00th=[ 155], 5.00th=[ 275], 10.00th=[ 447], 20.00th=[ 498],
| 30.00th=[ 558], 40.00th=[ 558], 50.00th=[ 558], 60.00th=[ 558],
| 70.00th=[ 558], 80.00th=[ 584], 90.00th=[ 701], 95.00th=[ 844],
| 99.00th=[ 1003], 99.50th=[ 1020], 99.90th=[ 1070], 99.95th=[ 1133],
| 99.99th=[ 1133]
bw ( KiB/s): min= 2048, max=86016, per=99.21%, avg=56510.84, stdev=10313.10, samples=121
iops : min= 2, max= 84, avg=55.16, stdev=10.07, samples=121
lat (msec) : 50=0.06%, 100=0.59%, 250=3.53%, 500=17.54%, 750=70.21%
lat (msec) : 1000=6.77%
cpu : usr=1.46%, sys=2.02%, ctx=1184, majf=0, minf=24
IO depths : 1=0.2%, 2=0.4%, 4=0.8%, 8=1.7%, 16=3.3%, 32=93.6%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.2%, 64=0.0%, >=64=0.0%
issued rwts: total=0,3370,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: bw=55.6MiB/s (58.3MB/s), 55.6MiB/s-55.6MiB/s (58.3MB/s-58.3MB/s), io=3370MiB (3534MB), run=60585-60585msec
Disk stats (read/write):
sdb: ios=0/6764, merge=0/27, ticks=0/3077549, in_queue=1629960, util=96.38%
$ fio --name TEST --eta-newline=5s --filename=fio-tempfile.dat --rw=read --size=500m --io_size=10g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=60 --group_reporting
TEST: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [R(1)][11.7%][r=55.1MiB/s][r=55 IOPS][eta 00m:53s]
Jobs: 1 (f=1): [R(1)][21.7%][r=59.1MiB/s][r=59 IOPS][eta 00m:47s]
Jobs: 1 (f=1): [R(1)][31.7%][r=57.1MiB/s][r=57 IOPS][eta 00m:41s]
Jobs: 1 (f=1): [R(1)][42.6%][r=56.1MiB/s][r=56 IOPS][eta 00m:35s]
Jobs: 1 (f=1): [R(1)][52.5%][r=56.0MiB/s][r=56 IOPS][eta 00m:29s]
Jobs: 1 (f=1): [R(1)][62.3%][r=56.0MiB/s][r=56 IOPS][eta 00m:23s]
Jobs: 1 (f=1): [R(1)][72.1%][r=60.0MiB/s][r=60 IOPS][eta 00m:17s]
Jobs: 1 (f=1): [R(1)][82.0%][r=56.0MiB/s][r=56 IOPS][eta 00m:11s]
Jobs: 1 (f=1): [R(1)][91.8%][r=56.1MiB/s][r=56 IOPS][eta 00m:05s]
Jobs: 1 (f=1): [R(1)][100.0%][r=56.0MiB/s][r=56 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=2543: Mon Oct 14 11:43:05 2019
read: IOPS=56, BW=56.6MiB/s (59.3MB/s)(3422MiB/60488msec)
slat (usec): min=303, max=102262, avg=16671.52, stdev=26811.05
clat (msec): min=80, max=1038, avg=545.91, stdev=94.86
lat (msec): min=81, max=1047, avg=562.58, stdev=97.33
clat percentiles (msec):
| 1.00th=[ 182], 5.00th=[ 460], 10.00th=[ 493], 20.00th=[ 550],
| 30.00th=[ 558], 40.00th=[ 558], 50.00th=[ 558], 60.00th=[ 558],
| 70.00th=[ 558], 80.00th=[ 558], 90.00th=[ 558], 95.00th=[ 600],
| 99.00th=[ 927], 99.50th=[ 969], 99.90th=[ 1028], 99.95th=[ 1028],
| 99.99th=[ 1036]
bw ( KiB/s): min=10240, max=110592, per=99.87%, avg=57854.97, stdev=10690.12, samples=120
iops : min= 10, max= 108, avg=56.49, stdev=10.44, samples=120
lat (msec) : 100=0.35%, 250=1.69%, 500=15.98%, 750=78.90%, 1000=2.86%
cpu : usr=0.06%, sys=2.48%, ctx=1217, majf=0, minf=8214
IO depths : 1=0.2%, 2=0.4%, 4=0.8%, 8=1.6%, 16=3.3%, 32=93.7%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.2%, 64=0.0%, >=64=0.0%
issued rwts: total=3422,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=56.6MiB/s (59.3MB/s), 56.6MiB/s-56.6MiB/s (59.3MB/s-59.3MB/s), io=3422MiB (3588MB), run=60488-60488msec
Disk stats (read/write):
sdb: ios=6832/3, merge=0/1, ticks=3208754/1907, in_queue=1716130, util=99.36%
$ fio --name TEST --eta-newline=5s --filename=fio-tempfile.dat --rw=randread --size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 --iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting
TEST: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][11.7%][r=524KiB/s][r=131 IOPS][eta 00m:53s]
Jobs: 1 (f=1): [r(1)][23.0%][r=512KiB/s][r=128 IOPS][eta 00m:47s]
Jobs: 1 (f=1): [r(1)][32.8%][r=516KiB/s][r=129 IOPS][eta 00m:41s]
Jobs: 1 (f=1): [r(1)][42.6%][r=512KiB/s][r=128 IOPS][eta 00m:35s]
Jobs: 1 (f=1): [r(1)][52.5%][r=504KiB/s][r=126 IOPS][eta 00m:29s]
Jobs: 1 (f=1): [r(1)][62.3%][r=556KiB/s][r=139 IOPS][eta 00m:23s]
Jobs: 1 (f=1): [r(1)][72.1%][r=528KiB/s][r=132 IOPS][eta 00m:17s]
Jobs: 1 (f=1): [r(1)][82.0%][r=504KiB/s][r=126 IOPS][eta 00m:11s]
Jobs: 1 (f=1): [r(1)][91.8%][r=508KiB/s][r=127 IOPS][eta 00m:05s]
Jobs: 1 (f=1): [r(1)][100.0%][r=532KiB/s][r=133 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=2577: Mon Oct 14 12:04:19 2019
read: IOPS=130, BW=522KiB/s (534kB/s)(30.6MiB/60009msec)
slat (usec): min=65, max=296, avg=68.08, stdev= 6.24
clat (usec): min=165, max=16041, avg=7584.24, stdev=2743.02
lat (usec): min=236, max=16353, avg=7654.91, stdev=2743.03
clat percentiles (usec):
| 1.00th=[ 289], 5.00th=[ 3195], 10.00th=[ 4047], 20.00th=[ 5145],
| 30.00th=[ 5932], 40.00th=[ 6783], 50.00th=[ 7635], 60.00th=[ 8455],
| 70.00th=[ 9241], 80.00th=[10159], 90.00th=[11207], 95.00th=[11863],
| 99.00th=[12780], 99.50th=[13042], 99.90th=[14091], 99.95th=[14353],
| 99.99th=[16057]
bw ( KiB/s): min= 464, max= 584, per=100.00%, avg=521.53, stdev=21.21, samples=120
iops : min= 116, max= 146, avg=130.36, stdev= 5.32, samples=120
lat (usec) : 250=0.73%, 500=0.68%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.35%, 4=7.95%, 10=68.56%, 20=21.71%
cpu : usr=0.27%, sys=1.49%, ctx=7883, majf=0, minf=20
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=7825,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=522KiB/s (534kB/s), 522KiB/s-522KiB/s (534kB/s-534kB/s), io=30.6MiB (32.1MB), run=60009-60009msec
Disk stats (read/write):
sdb: ios=7810/0, merge=0/0, ticks=59294/0, in_queue=58790, util=98.05%
$ fio --name TEST --eta-newline=5s --filename=fio-tempfile.dat --rw=randrw --size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 --iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting
TEST: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [m(1)][11.7%][r=68KiB/s,w=52KiB/s][r=17,w=13 IOPS][eta 00m:53s]
Jobs: 1 (f=1): [m(1)][21.7%][r=28KiB/s,w=52KiB/s][r=7,w=13 IOPS][eta 00m:47s]
Jobs: 1 (f=1): [m(1)][31.7%][r=36KiB/s,w=52KiB/s][r=9,w=13 IOPS][eta 00m:41s]
Jobs: 1 (f=1): [m(1)][42.6%][r=28KiB/s,w=48KiB/s][r=7,w=12 IOPS][eta 00m:35s]
Jobs: 1 (f=1): [m(1)][52.5%][r=36KiB/s,w=56KiB/s][r=9,w=14 IOPS][eta 00m:29s]
Jobs: 1 (f=1): [m(1)][62.3%][r=32KiB/s,w=52KiB/s][r=8,w=13 IOPS][eta 00m:23s]
Jobs: 1 (f=1): [m(1)][72.1%][r=48KiB/s,w=52KiB/s][r=12,w=13 IOPS][eta 00m:17s]
Jobs: 1 (f=1): [m(1)][82.0%][r=72KiB/s,w=44KiB/s][r=18,w=11 IOPS][eta 00m:11s]
Jobs: 1 (f=1): [m(1)][91.8%][r=44KiB/s,w=48KiB/s][r=11,w=12 IOPS][eta 00m:05s]
Jobs: 1 (f=1): [m(1)][100.0%][r=72KiB/s,w=44KiB/s][r=18,w=11 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=2592: Mon Oct 14 12:08:43 2019
read: IOPS=12, BW=48.0KiB/s (49.2kB/s)(2884KiB/60049msec)
slat (usec): min=67, max=195, avg=86.00, stdev=16.25
clat (usec): min=175, max=18905, avg=12960.34, stdev=4113.45
lat (usec): min=259, max=19008, avg=13049.30, stdev=4120.38
clat percentiles (usec):
| 1.00th=[ 306], 5.00th=[ 4490], 10.00th=[ 6718], 20.00th=[10421],
| 30.00th=[11469], 40.00th=[12387], 50.00th=[13566], 60.00th=[14615],
| 70.00th=[15664], 80.00th=[16581], 90.00th=[17695], 95.00th=[18220],
| 99.00th=[18744], 99.50th=[18744], 99.90th=[19006], 99.95th=[19006],
| 99.99th=[19006]
bw ( KiB/s): min= 8, max= 152, per=100.00%, avg=48.44, stdev=22.59, samples=119
iops : min= 2, max= 38, avg=12.08, stdev= 5.64, samples=119
write: IOPS=12, BW=49.0KiB/s (51.2kB/s)(3000KiB/60049msec); 0 zone resets
slat (usec): min=94, max=268, avg=103.63, stdev=10.70
clat (usec): min=104, max=457, avg=234.82, stdev=17.52
lat (usec): min=318, max=571, avg=341.33, stdev=18.26
clat percentiles (usec):
| 1.00th=[ 215], 5.00th=[ 221], 10.00th=[ 223], 20.00th=[ 225],
| 30.00th=[ 227], 40.00th=[ 231], 50.00th=[ 233], 60.00th=[ 237],
| 70.00th=[ 241], 80.00th=[ 243], 90.00th=[ 247], 95.00th=[ 251],
| 99.00th=[ 289], 99.50th=[ 343], 99.90th=[ 457], 99.95th=[ 457],
| 99.99th=[ 457]
bw ( KiB/s): min= 32, max= 72, per=100.00%, avg=49.97, stdev= 6.89, samples=120
iops : min= 8, max= 18, avg=12.47, stdev= 1.72, samples=120
lat (usec) : 250=48.33%, 500=3.20%
lat (msec) : 2=0.14%, 4=1.36%, 10=7.41%, 20=39.56%
fsync/fdatasync/sync_file_range:
sync (nsec): min=1648, max=36963, avg=2001.28, stdev=930.47
sync percentiles (nsec):
| 1.00th=[ 1736], 5.00th=[ 1800], 10.00th=[ 1816], 20.00th=[ 1848],
| 30.00th=[ 1896], 40.00th=[ 1912], 50.00th=[ 1928], 60.00th=[ 1960],
| 70.00th=[ 2008], 80.00th=[ 2064], 90.00th=[ 2192], 95.00th=[ 2352],
| 99.00th=[ 2704], 99.50th=[ 2736], 99.90th=[ 2960], 99.95th=[37120],
| 99.99th=[37120]
cpu : usr=0.21%, sys=0.38%, ctx=3317, majf=0, minf=22
IO depths : 1=199.8%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=721,750,0,1468 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=48.0KiB/s (49.2kB/s), 48.0KiB/s-48.0KiB/s (49.2kB/s-49.2kB/s), io=2884KiB (2953kB), run=60049-60049msec
WRITE: bw=49.0KiB/s (51.2kB/s), 49.0KiB/s-49.0KiB/s (51.2kB/s-51.2kB/s), io=3000KiB (3072kB), run=60049-60049msec
Disk stats (read/write):
sdb: ios=717/3323, merge=0/1112, ticks=9304/50124, in_queue=9630, util=16.06%