Professional Web Applications Themes

multi-second hang in getdents (Linux 2.4, x86, ext3fs) - Linux / Unix Administration

I'm seeing a strange performance problem on one of our systems. When running a find on one filesystem, there are long hangs (up to 30 seconds plus) during the find. Strace reveals that some of the getdents calls are taking that much time! From 'strace -T': getdents64(0x4, 0x805ed28, 0x1000, 0x805ed28) = 48 <35.798045> Oddly enough, not all of them are taking so long: getdents64(0x4, 0x805ed28, 0x1000, 0x805ed28) = 48 <0.000016> getdents64(0x4, 0x8059cd0, 0x1000, 0x8059cd0) = 240 <0.000017> The system is otherwise almost completely idle during the find. Nothing else is putting much of a demand on the filesystem. As you ...

  1. #1

    Default multi-second hang in getdents (Linux 2.4, x86, ext3fs)

    I'm seeing a strange performance problem on one of our systems.
    When running a find on one filesystem, there are long hangs (up to 30
    seconds plus) during the find. Strace reveals that some of the getdents
    calls are taking that much time!

    From 'strace -T':
    getdents64(0x4, 0x805ed28, 0x1000, 0x805ed28) = 48 <35.798045>

    Oddly enough, not all of them are taking so long:
    getdents64(0x4, 0x805ed28, 0x1000, 0x805ed28) = 48 <0.000016>
    getdents64(0x4, 0x8059cd0, 0x1000, 0x8059cd0) = 240 <0.000017>

    The system is otherwise almost completely idle during the find.
    Nothing else is putting much of a demand on the filesystem. As you can
    see, the time the getdents takes does not seem to be a function of how
    much it returns. None of the directories the find traverses have very
    many entries - none have more than 100, even. During the hang, vmstat
    shows a blocked process, but no time spent in iowait. Any attempt to
    access the same filesystem during a hang also hangs, so it's not just
    the one process. The hangs are not repeatable at the same directory -
    if I try just running a find at the point where a find hung before, it's
    more likely than not to proceed just fine.

    The filesystem is ext3, on a mirrored pair of Maxtor 180 GB SATA
    drives attached to a 3Ware 8500-4. The 3Ware software does not show any
    problems with the mirror. These are the only drives in the system, so I
    can't directly compare them against anything else. Nothing out of the
    ordinary appears in dmesg or /var/log/messages. I'm pretty out of
    ideas...the length of time feels like an ATA timeout to me, but I don't
    have any real evidence for that. Any suggestions would be much
    appreciated.

    JDW

    Jeremiah Guest

  2. #2

    Default Re: multi-second hang in getdents (Linux 2.4, x86, ext3fs)

    > I'm seeing a strange performance problem on one of our systems. 

    It is exceedingly probable that the delay is caused by writeback
    to the disk of the updated access time for all of the directories
    that have been visited by the 'find'. This saturates the disk
    queues for a while.

    The workaround is easy: add "nodiratime" to the mount options in
    /etc/fstab. For example:
    /dev/hda3 / ext3 defaults,nodiratime 1 1
    Of course, this means that the recorded time of last access on
    directories won't be "correct", but almost nobody uses it anyway,
    because it is _not_ updated just because a file within it gets
    changed. Thus access time on a directory cannot be used as a hint
    of whether a backup program needs to descend into the directory
    in order to see all files that were modified after a given date.

    --
    John Guest

  3. #3

    Default Re: multi-second hang in getdents (Linux 2.4, x86, ext3fs)

    John Reiser <com> wrote: 

    This looks like it does the trick. Thanks very much!

    JDW

    Jeremiah Guest

Similar Threads

  1. Implications for a multi lingual, multi curreny e commerce site ??
    By James Noon in forum Dreamweaver AppDev
    Replies: 1
    Last Post: April 17th, 09:27 PM
  2. Multi-demensional, multi-key hash : Help :(
    By Jason Normandin in forum PERL Beginners
    Replies: 4
    Last Post: November 15th, 09:00 AM
  3. Form validation for multi-rows and multi-columns
    By Jean in forum ASP Database
    Replies: 12
    Last Post: November 4th, 02:02 PM
  4. multi file multi colomn
    By Sven in forum PHP Development
    Replies: 0
    Last Post: July 16th, 10:41 AM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139