Professional Web Applications Themes

(LONG) ATA Benchmark: 5.x Reads Slower than Writes - FreeBSD

Hello, BACKGROUND I need to purchase a new system for our developers, for use as a Postgres database test server. Having a RAID, probably RAID1, is desirable for performance and reliability. I have recently set up a system with a gmirror-based software RAID1 on a pair of 250GB ATA drives. I would like to stick with gmirror, because: - We save money on extra hardware. - Since gmirror is part of FreeBSD, maintenance is a lot easier than with a hardware solution. SUMMARY But before I go balls out, I should see how well it compares to hardware RAID. So, ...

  1. #1

    Default (LONG) ATA Benchmark: 5.x Reads Slower than Writes

    Hello,

    BACKGROUND

    I need to purchase a new system for our developers, for use as a
    Postgres database test server. Having a RAID, probably RAID1, is
    desirable for performance and reliability. I have recently set up a
    system with a gmirror-based software RAID1 on a pair of 250GB ATA
    drives. I would like to stick with gmirror, because:

    - We save money on extra hardware.
    - Since gmirror is part of FreeBSD, maintenance is a lot easier than
    with a hardware solution.

    SUMMARY

    But before I go balls out, I should see how well it compares to hardware
    RAID. So, I do some benchmarks with bonnie++. Since this simulates
    create, write and read on thousands of random files, this sounds like a
    good approximation of what Postgres does. :)

    I don't have the time and hardware to do very scientific tests, but I
    have been able to run a series of benchmarks using bonnie++ on some
    systems I have available to me. The ATA-based gmirror performs
    extremely well, compared to a few Adaptec RAIDs that we have, EXCEPT
    that the sequential and random reads are MUCH SLOWER than the hardware
    solution, and even *slower than the preceding write operations*. This
    is counter-intuitive, especially since RAID1 implies slowed writes and
    faster reads. I tried the benchmark on my workstation (single 2.5" IDE
    in a laptop) and got comparable write-faster-than-read results.

    DATA

    I was able to make use of the following test systems. I ran tests in
    multi-user, but tried to favor times when there wasn't much background
    activity:

    mito: (lone 2.5" ATA)
    5.4-PRERELEASE
    CPU: Intel(R) Pentium(R) M processor 1.50GHz (1495.16-MHz 686-class CPU)
    atapci0: <Intel ICH4 UDMA100 controller>
    ata0: channel #0 on atapci0
    ata1: channel #1 on atapci0
    ad0: 28615MB <TOSHIBA MK3021GAS/GA129D> [58140/16/63] at ata0-master UDMA100

    amun: (gmirror RAID1 2 x WD250GB High Intensity)
    FreeBSD 5.3-RELEASE
    CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2799.22-MHz 686-class CPU)
    atapci0: <SiI 3112 SATA150 controller>
    ata2: channel #0 on atapci0
    ata3: channel #1 on atapci0
    ad4: 238475MB <WDC WD2500SD-01KCB0/08.02D08> [484521/16/63] at
    ata2-master SATA150
    ad6: 238475MB <WDC WD2500SD-01KCB0/08.02D08> [484521/16/63] at
    ata3-master SATA150
    atapci1: <SiI 3112 SATA150 controller>
    acd0: CDROM <CD-224E/1.9A> at ata1-master UDMA33

    janus: (Adaptec RAID1 2 x 72G 10,000 RPM SCSI)
    FreeBSD 4.10-STABLE
    CPU: Intel(R) Xeon(TM) CPU 2.40GHz (2399.33-MHz 686-class CPU) [DUAL]
    aac0: <Adaptec SCSI RAID 2120S> mem 0xf8000000-0xfbffffff irq 16 at
    device 1.0 on pci2
    aac0: i960RX 100MHz, 48MB cache memory, optional battery present
    aac0: Kernel 4.0-0, Build 6011, S/N baec64
    aac0: Supported
    Options=1f7e<CLUSTERS,WCACHE,DATA64,HOSTTIME,RAID5 0,WINDOW4GB,SOFTERR,NORECOND,SGMAP64,ALARM,NONDASD >
    aacd0: <RAID 1 (Mirror)> on aac0
    aacd0: 69998MB (143357184 sectors)

    db2: (Adaptec RAID10 4 x 36G 15,000 RPM SCSI)
    FreeBSD 4.8-STABLE
    CPU: Intel(R) Xeon(TM) CPU 3.06GHz (3065.81-MHz 686-class CPU) [DUAL]
    aac0: <Adaptec SCSI RAID 2120S> mem 0xf8000000-0xfbffffff irq 18 at
    device 2.0 on pci5
    aac0: i960RX 100MHz, 48MB cache memory, optional battery present
    aac0: Kernel 4.0-0, Build 6008, S/N b97ce8
    aac0: Supported
    Options=1f7e<CLUSTERS,WCACHE,DATA64,HOSTTIME,RAID5 0,WINDOW4GB,SOFTERR,NORECOND,SGMAP64,ALARM,NONDASD >
    aacp0: <SCSI Passthrough Bus> on aac0

    The raw data can be viewed at
    http://dannyman.toldme.com/scratch/benchmarks/

    YSIS

    Unfortunately, my hardware RAIDs are on FreeBSD 4, and gmirror is on 5.
    My hardware RAIDs are on dual CPU systems, with 2G RAM, and my gmirror
    is on a single hyperthreaded CPU with 512M. Yes, sorry, not especially
    scientific. Maybe the changes in FreeBSD make a big difference? Maybe
    RAM makes a big difference?

    The first results show a serious advantage for the gmirror setup:

    Sequential output (char)
    gmirror ATA RAID1: avg 320K/s
    Adaptec SCSI RAID1: avg 222K/s
    Adaptec SCSI RAID10: avg 202K/s

    Sequential input (char)
    gmirror ATA RAID1: avg 617K/s
    Adaptec SCSI RAID1: avg 345K/s
    Adaptec SCSI RAID10: avg 336K/s

    Sequential Output (block)
    gmirror ATA RAID1: avg 37893K/s
    Adaptec SCSI RAID1: avg 13829K/s
    Adaptec SCSI RAID10: avg 40440K/s

    The gmirror sees slightly poorer performance in random seeks:

    Rndom Seeks
    gmirror ATA RAID1: avg 4144/s
    Adaptec SCSI RAID1: avg 5428/s
    Adaptec SCSI RAID10: avg 13302/s

    That all sounds great if I was streaming video, but I want to run a
    database, opening and closing, reading, writing, and rewriting several
    small files. This is where things seem to go rotten.

    We see the ATA performance go to heck on the File Create tests:

    Sequential Create
    laptop 2.5" ATA: avg 101/s
    gmirror ATA RAID1: avg 365/s
    Adaptec SCSI RAID1: avg 160/s
    Adaptec SCSI RAID10: avg 412/s

    Sequential Read
    laptop 2.5" ATA: avg 76/s # SLOWER than write!
    gmirror ATA RAID1: avg 251/s # SLOWER than write!
    Adaptec SCSI RAID1: avg 7862/s
    Adaptec SCSI RAID10: avg 7618/s

    Random Create
    laptop 2.5" ATA: avg 124/s
    gmirror ATA RAID1: avg 354/s
    Adaptec SCSI RAID1: avg 155/s
    Adaptec SCSI RAID10: avg 504/s

    Random Read
    laptop 2.5" ATA: avg 57/s # SLOWER than write!
    gmirror ATA RAID1: avg 144/s # SLOWER than write!
    Adaptec SCSI RAID1: avg 7655/s
    Adaptec SCSI RAID10: avg 7413/s

    CONFUSION

    Now, I could explain poor read performance by:
    - Less RAM == Less buffer
    - Bigger Disks == Slower Seeks
    - Less CPU == ???

    I DO have a 4.8-STABLE with a single IDE disk, no Soft Updates, and
    faster read than write:

    Version 1.93c ------Sequential Create------ --------Random
    Create--------
    anubis. -Create-- --Read--- -Delete-- -Create-- --Read---
    -Delete--
    files:max /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
    /sec %CP
    10:104884:0/5 183 32 1739 97 502 16 176 32 1624 94
    368 13
    Latency 707ms 11487us 32824us 488ms 207ms
    117ms

    However, seeing read SLOWER than write ... I have to wonder if something
    fishy is going on. Suggestions? Ideas? I'm fresh out, at the moment.
    My suspicion is that something in 5.x is out-of-tune!?

    Thanks a lot.

    Sincerely,
    -danny

    --
    http://dannyman.toldme.com/

    Danny Guest

  2. #2

    Default Re: (LONG) ATA Benchmark: 5.x Reads Slower than Writes

    On April 8, 2005 04:41 pm, Danny Howard wrote: 

    May I suggest that you turn softupdates off and sync on for the filesystems
    you are testing. If you don't you are not really testing the hardware. You
    are testing the FreeBSD disk caching system in the kernel.
     

    --
    Ean Kingston

    E-Mail: ean AT hedron DOT org
    URL: http://www.hedron.org/
    Ean Guest

  3. #3

    Default Re: (LONG) ATA Benchmark: 5.x Reads Slower than Writes

    On Fri, 8 Apr 2005, Danny Howard wrote:
     
     

    Could you place the 'dmesg' output for each system in this directory?

    The output here is marginally useful since it shows the bonnie command
    line. However, 100MB as the test filesize is really small unless the
    systems have 64MB of RAM though -- otherwise you're testing how well
    FreeBSD manages memory (or how much crap the systems are running when you
    run this test).

    For recent I/O tests I was doing with iozone I was using 10GB filesizes.
    This s out the cache on just about everything.
     

    Yes, lots. Both 4.x vs. 5.x and RAM :)
     

    I'd be really careful here... that is # of files read per second after the
    create, and as pointed out before, SoftUpdates usually gets you a big win
    until it has to flush the directories out then things suffer during the
    actual flush op since the disk gets hammered. Lots of free memory to use
    for the directory cache helps. The disk cache on the RAID controller is
    buying you even more. From your results you were getting 9ms latency which
    is spot-on so I think you are simply misinterpreting your results here.

    File creation tests are usually more for filesystem-specific benchmarking
    than for throughput benchmarking. I'd suggest something more like iozone
    for throughput testing. If the volumes have nothing on them you care about
    then rawio can also be instructive.

    --
    Doug White | FreeBSD: The Power to Serve
    com | www.FreeBSD.org
    Doug Guest

  4. #4

    Default Re: (LONG) ATA Benchmark: 5.x Reads Slower than Writes

    Have you been certain to disable write caching on the drives? If not I
    bet that's what you're seeing (which also means you're begging for
    database corruption).
    --
    Jim C. Nasby, Database Consultant org
    Give your computer some brain candy! www.distributed.net Team #1828

    Windows: "Where do you want to go today?"
    Linux: "Where do you want to go tomorrow?"
    FreeBSD: "Are you guys coming, or what?"
    Jim Guest

Similar Threads

  1. IO::Pty - reads/writes fail *only* with system()
    By Alexis Huxley in forum PERL Modules
    Replies: 3
    Last Post: January 20th, 02:24 AM
  2. Replies: 5
    Last Post: July 19th, 07:08 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139