<code>
$ fio --name TEST --eta-newline=5s --filename=fio-tempfile.dat --rw=write --size=500m --io_size=10g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=60 --group_reporting
</code>

<code>
TEST: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
fio-3.12
Starting 1 process
TEST: Laying out IO file (1 file / 500MiB)
Jobs: 1 (f=1): [W(1)][13.1%][w=60.0MiB/s][w=60 IOPS][eta 00m:53s]
Jobs: 1 (f=1): [W(1)][23.0%][w=53.1MiB/s][w=53 IOPS][eta 00m:47s]
Jobs: 1 (f=1): [W(1)][32.8%][w=55.1MiB/s][w=55 IOPS][eta 00m:41s]
Jobs: 1 (f=1): [W(1)][42.6%][w=50.0MiB/s][w=50 IOPS][eta 00m:35s]
Jobs: 1 (f=1): [W(1)][52.5%][w=53.1MiB/s][w=53 IOPS][eta 00m:29s]
Jobs: 1 (f=1): [W(1)][62.3%][w=52.0MiB/s][w=52 IOPS][eta 00m:23s]
Jobs: 1 (f=1): [W(1)][72.1%][w=53.0MiB/s][w=53 IOPS][eta 00m:17s]
Jobs: 1 (f=1): [W(1)][82.0%][w=56.0MiB/s][w=56 IOPS][eta 00m:11s]
Jobs: 1 (f=1): [W(1)][91.8%][w=53.0MiB/s][w=53 IOPS][eta 00m:05s]
Jobs: 1 (f=1): [W(1)][100.0%][w=52.1MiB/s][w=52 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=2458: Mon Oct 14 11:37:52 2019
  write: IOPS=52, BW=52.9MiB/s (55.5MB/s)(3203MiB/60528msec); 0 zone resets
    slat (usec): min=345, max=526452, avg=17862.23, stdev=40922.72
    clat (msec): min=35, max=1230, avg=584.41, stdev=161.19
     lat (msec): min=37, max=1230, avg=602.27, stdev=162.97
    clat percentiles (msec):
     |  1.00th=[  157],  5.00th=[  251], 10.00th=[  405], 20.00th=[  527],
     | 30.00th=[  575], 40.00th=[  584], 50.00th=[  592], 60.00th=[  592],
     | 70.00th=[  592], 80.00th=[  617], 90.00th=[  768], 95.00th=[  919],
     | 99.00th=[ 1045], 99.50th=[ 1070], 99.90th=[ 1167], 99.95th=[ 1234],
     | 99.99th=[ 1234]
   bw (  KiB/s): min= 2048, max=98304, per=99.07%, avg=53683.87, stdev=17800.57, samples=121
   iops        : min=    2, max=   96, avg=52.40, stdev=17.37, samples=121
  lat (msec)   : 50=0.06%, 100=0.78%, 250=4.18%, 500=9.12%, 750=74.96%
  lat (msec)   : 1000=8.15%
  cpu          : usr=1.32%, sys=2.03%, ctx=1145, majf=0, minf=24
  IO depths    : 1=0.2%, 2=0.4%, 4=0.9%, 8=1.7%, 16=3.5%, 32=93.2%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.2%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,3203,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=52.9MiB/s (55.5MB/s), 52.9MiB/s-52.9MiB/s (55.5MB/s-55.5MB/s), io=3203MiB (3359MB), run=60528-60528msec

Disk stats (read/write):
  sda: ios=0/6424, merge=0/25, ticks=0/3059936, in_queue=1649740, util=97.77%
</code>

<code>
$ fio --name TEST --eta-newline=5s --filename=fio-tempfile.dat --rw=read --size=500m --io_size=10g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=60 --group_reporting
</code>

<code>
TEST: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [R(1)][11.7%][r=52.1MiB/s][r=52 IOPS][eta 00m:53s]
Jobs: 1 (f=1): [R(1)][21.7%][r=56.1MiB/s][r=56 IOPS][eta 00m:47s]
Jobs: 1 (f=1): [R(1)][31.7%][r=52.1MiB/s][r=52 IOPS][eta 00m:41s]
Jobs: 1 (f=1): [R(1)][41.7%][r=56.1MiB/s][r=56 IOPS][eta 00m:35s]
Jobs: 1 (f=1): [R(1)][51.7%][r=52.1MiB/s][r=52 IOPS][eta 00m:29s]
Jobs: 1 (f=1): [R(1)][61.7%][r=56.1MiB/s][r=56 IOPS][eta 00m:23s]
Jobs: 1 (f=1): [R(1)][72.1%][r=56.0MiB/s][r=56 IOPS][eta 00m:17s]
Jobs: 1 (f=1): [R(1)][82.0%][r=56.0MiB/s][r=56 IOPS][eta 00m:11s]
Jobs: 1 (f=1): [R(1)][91.8%][r=37.0MiB/s][r=37 IOPS][eta 00m:05s]
Jobs: 1 (f=1): [R(1)][100.0%][r=52.1MiB/s][r=52 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=2504: Mon Oct 14 11:40:07 2019
  read: IOPS=54, BW=54.5MiB/s (57.2MB/s)(3300MiB/60530msec)
    slat (usec): min=302, max=96515, avg=17267.25, stdev=27835.97
    clat (msec): min=80, max=1072, avg=566.45, stdev=100.02
     lat (msec): min=81, max=1072, avg=583.73, stdev=102.58
    clat percentiles (msec):
     |  1.00th=[  178],  5.00th=[  451], 10.00th=[  510], 20.00th=[  575],
     | 30.00th=[  575], 40.00th=[  575], 50.00th=[  575], 60.00th=[  575],
     | 70.00th=[  584], 80.00th=[  584], 90.00th=[  584], 95.00th=[  617],
     | 99.00th=[  944], 99.50th=[ 1003], 99.90th=[ 1062], 99.95th=[ 1070],
     | 99.99th=[ 1070]
   bw (  KiB/s): min= 2048, max=110592, per=99.10%, avg=55326.35, stdev=12225.91, samples=121
   iops        : min=    2, max=  108, avg=54.00, stdev=11.95, samples=121
  lat (msec)   : 100=0.36%, 250=1.73%, 500=3.48%, 750=91.03%, 1000=2.48%
  cpu          : usr=0.08%, sys=2.39%, ctx=1176, majf=0, minf=8215
  IO depths    : 1=0.2%, 2=0.4%, 4=0.8%, 8=1.7%, 16=3.4%, 32=93.4%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.2%, 64=0.0%, >=64=0.0%
     issued rwts: total=3300,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=54.5MiB/s (57.2MB/s), 54.5MiB/s-54.5MiB/s (57.2MB/s-57.2MB/s), io=3300MiB (3460MB), run=60530-60530msec

Disk stats (read/write):
  sda: ios=6580/3, merge=0/1, ticks=3203220/2094, in_queue=1721700, util=99.75%
</code>

<code>
$ fio --name TEST --eta-newline=5s --filename=fio-tempfile.dat --rw=randread --size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 --iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting
</code>

<code>
TEST: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][11.7%][r=496KiB/s][r=124 IOPS][eta 00m:53s]
Jobs: 1 (f=1): [r(1)][21.7%][r=512KiB/s][r=128 IOPS][eta 00m:47s]
Jobs: 1 (f=1): [r(1)][31.7%][r=516KiB/s][r=129 IOPS][eta 00m:41s]
Jobs: 1 (f=1): [r(1)][41.7%][r=524KiB/s][r=131 IOPS][eta 00m:35s]
Jobs: 1 (f=1): [r(1)][51.7%][r=524KiB/s][r=131 IOPS][eta 00m:29s]
Jobs: 1 (f=1): [r(1)][61.7%][r=520KiB/s][r=130 IOPS][eta 00m:23s]
Jobs: 1 (f=1): [r(1)][72.1%][r=504KiB/s][r=126 IOPS][eta 00m:17s]
Jobs: 1 (f=1): [r(1)][82.0%][r=536KiB/s][r=134 IOPS][eta 00m:11s]
Jobs: 1 (f=1): [r(1)][91.8%][r=512KiB/s][r=128 IOPS][eta 00m:05s]
Jobs: 1 (f=1): [r(1)][100.0%][r=540KiB/s][r=135 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=2570: Mon Oct 14 12:02:13 2019
  read: IOPS=130, BW=521KiB/s (533kB/s)(30.5MiB/60009msec)
    slat (usec): min=65, max=219, avg=67.73, stdev= 4.16
    clat (usec): min=261, max=19685, avg=7594.69, stdev=2783.37
     lat (usec): min=331, max=19975, avg=7664.96, stdev=2783.49
    clat percentiles (usec):
     |  1.00th=[  297],  5.00th=[ 3523], 10.00th=[ 4080], 20.00th=[ 4948],
     | 30.00th=[ 5866], 40.00th=[ 6718], 50.00th=[ 7570], 60.00th=[ 8455],
     | 70.00th=[ 9372], 80.00th=[10159], 90.00th=[11076], 95.00th=[11600],
     | 99.00th=[14353], 99.50th=[15795], 99.90th=[17171], 99.95th=[17695],
     | 99.99th=[19792]
   bw (  KiB/s): min=  472, max=  584, per=100.00%, avg=520.88, stdev=23.68, samples=120
   iops        : min=  118, max=  146, avg=130.21, stdev= 5.92, samples=120
  lat (usec)   : 500=1.46%, 750=0.01%
  lat (msec)   : 2=0.03%, 4=7.68%, 10=68.57%, 20=22.25%
  cpu          : usr=0.24%, sys=1.45%, ctx=7825, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=7815,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=521KiB/s (533kB/s), 521KiB/s-521KiB/s (533kB/s-533kB/s), io=30.5MiB (32.0MB), run=60009-60009msec

Disk stats (read/write):
  sda: ios=7792/0, merge=0/0, ticks=59279/0, in_queue=57170, util=95.41%
</code>

<code>
$ fio --name TEST --eta-newline=5s --filename=fio-tempfile.dat --rw=randrw --size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 --iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting
</code>

<code>
TEST: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [m(1)][11.7%][r=52KiB/s,w=60KiB/s][r=13,w=15 IOPS][eta 00m:53s]
Jobs: 1 (f=1): [m(1)][21.7%][r=48KiB/s,w=60KiB/s][r=12,w=15 IOPS][eta 00m:47s]
Jobs: 1 (f=1): [m(1)][31.7%][r=40KiB/s,w=64KiB/s][r=10,w=16 IOPS][eta 00m:41s]
Jobs: 1 (f=1): [m(1)][42.6%][r=44KiB/s,w=60KiB/s][r=11,w=15 IOPS][eta 00m:35s]
Jobs: 1 (f=1): [m(1)][52.5%][r=40KiB/s,w=60KiB/s][r=10,w=15 IOPS][eta 00m:29s]
Jobs: 1 (f=1): [m(1)][62.3%][r=76KiB/s,w=56KiB/s][r=19,w=14 IOPS][eta 00m:23s]
Jobs: 1 (f=1): [m(1)][72.1%][r=72KiB/s,w=56KiB/s][r=18,w=14 IOPS][eta 00m:17s]
Jobs: 1 (f=1): [m(1)][82.0%][r=28KiB/s,w=72KiB/s][r=7,w=18 IOPS][eta 00m:11s]
Jobs: 1 (f=1): [m(1)][91.8%][r=44KiB/s,w=68KiB/s][r=11,w=17 IOPS][eta 00m:05s]
Jobs: 1 (f=1): [m(1)][100.0%][r=44KiB/s,w=64KiB/s][r=11,w=16 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=2585: Mon Oct 14 12:06:56 2019
  read: IOPS=13, BW=55.4KiB/s (56.8kB/s)(3328KiB/60030msec)
    slat (usec): min=67, max=197, avg=84.88, stdev=15.39
    clat (usec): min=245, max=27145, avg=14465.36, stdev=4927.55
     lat (usec): min=335, max=27248, avg=14553.08, stdev=4934.20
    clat percentiles (usec):
     |  1.00th=[  285],  5.00th=[ 4752], 10.00th=[ 7046], 20.00th=[10421],
     | 30.00th=[12780], 40.00th=[14091], 50.00th=[15401], 60.00th=[16581],
     | 70.00th=[17433], 80.00th=[18744], 90.00th=[20055], 95.00th=[20579],
     | 99.00th=[22676], 99.50th=[23987], 99.90th=[27132], 99.95th=[27132],
     | 99.99th=[27132]
   bw (  KiB/s): min=   16, max=  144, per=100.00%, avg=55.46, stdev=20.88, samples=120
   iops        : min=    4, max=   36, avg=13.86, stdev= 5.23, samples=120
  write: IOPS=14, BW=58.7KiB/s (60.1kB/s)(3524KiB/60030msec); 0 zone resets
    slat (usec): min=94, max=250, avg=102.18, stdev= 9.69
    clat (usec): min=132, max=598, avg=209.19, stdev=15.27
     lat (usec): min=297, max=708, avg=314.15, stdev=17.34
    clat percentiles (usec):
     |  1.00th=[  194],  5.00th=[  202], 10.00th=[  204], 20.00th=[  204],
     | 30.00th=[  206], 40.00th=[  206], 50.00th=[  208], 60.00th=[  208],
     | 70.00th=[  210], 80.00th=[  215], 90.00th=[  217], 95.00th=[  219],
     | 99.00th=[  233], 99.50th=[  245], 99.90th=[  603], 99.95th=[  603],
     | 99.99th=[  603]
   bw (  KiB/s): min=   32, max=   80, per=100.00%, avg=58.72, stdev= 8.57, samples=120
   iops        : min=    8, max=   20, avg=14.67, stdev= 2.13, samples=120
  lat (usec)   : 250=51.26%, 500=0.76%, 750=0.06%
  lat (msec)   : 4=1.23%, 10=7.06%, 20=34.38%, 50=5.25%
  fsync/fdatasync/sync_file_range:
    sync (nsec): min=1537, max=14055, avg=2013.36, stdev=342.65
    sync percentiles (nsec):
     |  1.00th=[ 1720],  5.00th=[ 1800], 10.00th=[ 1832], 20.00th=[ 1864],
     | 30.00th=[ 1912], 40.00th=[ 1928], 50.00th=[ 1960], 60.00th=[ 2008],
     | 70.00th=[ 2064], 80.00th=[ 2128], 90.00th=[ 2256], 95.00th=[ 2352],
     | 99.00th=[ 2640], 99.50th=[ 2768], 99.90th=[ 2960], 99.95th=[14016],
     | 99.99th=[14016]
  cpu          : usr=0.09%, sys=0.57%, ctx=3835, majf=0, minf=22
  IO depths    : 1=199.8%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=832,881,0,1710 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=55.4KiB/s (56.8kB/s), 55.4KiB/s-55.4KiB/s (56.8kB/s-56.8kB/s), io=3328KiB (3408kB), run=60030-60030msec
  WRITE: bw=58.7KiB/s (60.1kB/s), 58.7KiB/s-58.7KiB/s (60.1kB/s-60.1kB/s), io=3524KiB (3609kB), run=60030-60030msec

Disk stats (read/write):
  sda: ios=827/3885, merge=0/1303, ticks=11980/47326, in_queue=12100, util=20.19%
</code>