Professional Web Applications Themes

Illogical usage of swap - FreeBSD

Perhaps this resonates with someone, I sure hope so. We bought and deployed a new server recently. Owing to various combinations of nics/raid controllers lack of support, instead of installing 5.3 we installed 4.11. The motherboard is a tyan tiger i7320s5350, dual Xeon 3.0GHz w/1MB cache and an moderately impressive 4GB of RAM. Hyperthreading *is* enabled, here's a sysctl hw snapshot: gonzo# sysctl -a |grep hw hw.machine: i386 hw.model: Intel(R) Xeon(TM) CPU 3.00GHz hw.ncpu: 4 #other stuff that may be meaningful to someone: hw.physmem: 3618050048 hw.usermem: 3261022208 hw.pagesize: 4096 hw.floatingpoint: 1 hw.machine_arch: i386 hw.aac.iosize_max: 65536 hw.an.an_dump: off #and so on..... ...

  1. #1

    Default Illogical usage of swap

    Perhaps this resonates with someone, I sure hope so.

    We bought and deployed a new server recently. Owing to various
    combinations of nics/raid controllers lack of support, instead of
    installing 5.3 we installed 4.11.

    The motherboard is a tyan tiger i7320s5350, dual Xeon 3.0GHz w/1MB cache
    and an moderately impressive 4GB of RAM. Hyperthreading *is* enabled,
    here's a sysctl hw snapshot:
    gonzo# sysctl -a |grep hw
    hw.machine: i386
    hw.model: Intel(R) Xeon(TM) CPU 3.00GHz
    hw.ncpu: 4
    #other stuff that may be meaningful to someone:
    hw.physmem: 3618050048
    hw.usermem: 3261022208
    hw.pagesize: 4096
    hw.floatingpoint: 1
    hw.machine_arch: i386
    hw.aac.iosize_max: 65536
    hw.an.an_dump: off
    #and so on.....


    Hard to believe, but this beast is swapping. Here's a typical top
    snapshot:

    load averages: 0.75, 0.36, 0.23
    up 14+12:39:04 12:40:02
    128 processes: 1 running, 125 sleeping, 2 zombie
    CPU states:
    7.5% user, 0.0% nice, 1.3% system, 0.0% interrupt, 91.2% idle
    Mem:
    479M Active, 2470M Inact, 337M Wired, 101M Cache, 199M Buf, 5468K Free
    Swap: 8192M Total, 116K Used, 8192M Free

    The box is replacing a machine with dual AMD athlon MP CPU's and 1GB of
    RAM. The old box never swapped once, though then - as now - free memory
    dipped down into numbers we weren't comfortable with occasionally. The
    old box *was* quite heavily loaded at times so this wasn't so
    surprising. The new "superbox" however doesn't even have some stuff
    running on it that the old one had! So under no cirstances could the
    workload on the new box be described as heavy.

    Can anyone suggest *anything* that might cause this uttely illogical
    situation to exist? We're frankly baffled.

    Regards & TIA,
    -Colin
    --
    Colin J. Raven
    Wed Mar 16 12:54:00 CET 2005
    Colin Guest

  2. #2

    Default Re: Illogical usage of swap

    "Colin J. Raven" <kozy-kabin.nl> writes:
     

    Not a problem.

    http://www.freebsd.org/doc/en_US.ISO8859-1/books/faq/misc.html#TOP-FREEMEM
    Lowell Guest

  3. #3

    Default Re: Illogical usage of swap

    On Mar 16 at 07:29, Lowell Gilbert launched this into the bitstream:
     
    >
    > Not a problem.
    >
    > http://www.freebsd.org/doc/en_US.ISO8859-1/books/faq/misc.html#TOP-FREEMEM[/ref]

    Hmm, yeah I should have included what I've actually read in the handbook
    and elsewhere. In fact I did read that exerpt you quoted the URL
    for above sometime previously, when it was quoted in another thread
    on the list.

    Free memory (or the lack thereof) isn't the issue though.

    The issue is this:
    Swap: 8192M Total, 116K Used, 8192M Free
    and that's the piece of the puzzle that has us all utterly baffled.

    No way in creation this box should be swapping.

    Thanks for your response Lowell.

    Regards,
    -Colin
    --
    Colin J. Raven
    Wed Mar 16 13:36:00 CET 2005
    Colin Guest

  4. #4

    Default Re: Illogical usage of swap

    On 2005-03-16 13:36, "Colin J. Raven" <kozy-kabin.nl> wrote: 
    >>
    >> Not a problem.
    >>
    >> http://www.freebsd.org/doc/en_US.ISO8859-1/books/faq/misc.html#TOP-FREEMEM[/ref]
    >
    > Hmm, yeah I should have included what I've actually read in the
    > handbook and elsewhere. In fact I did read that exerpt you quoted the
    > URL for above sometime previously, when it was quoted in another
    > thread on the list.
    >
    > Free memory (or the lack thereof) isn't the issue though.
    >
    > The issue is this:
    > Swap: 8192M Total, 116K Used, 8192M Free
    > and that's the piece of the puzzle that has us all utterly baffled.
    >
    > No way in creation this box should be swapping.[/ref]

    Do you, by any chance, have vm.swap_idle_enabled set to 1?

    Giorgos Guest

  5. #5

    Default Re: Illogical usage of swap

    On Mar 16 at 14:54, Giorgos Keramidas asked:
     
    >>
    >> Hmm, yeah I should have included what I've actually read in the
    >> handbook and elsewhere. In fact I did read that exerpt you quoted the
    >> URL for above sometime previously, when it was quoted in another
    >> thread on the list.
    >>
    >> Free memory (or the lack thereof) isn't the issue though.
    >>
    >> The issue is this:
    >> Swap: 8192M Total, 116K Used, 8192M Free
    >> and that's the piece of the puzzle that has us all utterly baffled.
    >>
    >> No way in creation this box should be swapping.[/ref]
    >
    > Do you, by any chance, have vm.swap_idle_enabled set to 1?[/ref]

    No, it isn't:
    gonzo# sysctl -a |grep vm.swap_idle_enabled
    vm.swap_idle_enabled: 0

    Regards,
    -Colin
    --
    Colin J. Raven
    Wed Mar 16 14:01:00 CET 2005
    Colin Guest

  6. #6

    Default Re: Illogical usage of swap

    On 2005-03-16 14:02, "Colin J. Raven" <kozy-kabin.nl> wrote: 
    >>
    >>Do you, by any chance, have vm.swap_idle_enabled set to 1?[/ref]
    >
    > No, it isn't:
    > gonzo# sysctl -a |grep vm.swap_idle_enabled
    > vm.swap_idle_enabled: 0[/ref]

    Strange. AFAIK, processes are not swapped out to disk unless there is a
    severe memory shortage. The usual case is to have them "paged" out,
    which is a bit different.

    Giorgos Guest

  7. #7

    Default Re: Illogical usage of swap

    On Mar 16 at 15:07, Giorgos Keramidas said:
     
    >>
    >> No, it isn't:
    >> gonzo# sysctl -a |grep vm.swap_idle_enabled
    >> vm.swap_idle_enabled: 0[/ref]
    >
    > Strange. AFAIK, processes are not swapped out to disk unless there is a
    > severe memory shortage. The usual case is to have them "paged" out,
    > which is a bit different.[/ref]

    That was my understanding also, and that's what's baffling us

    Regards,
    -Colin
    --
    Colin J. Raven
    Wed Mar 16 14:17:00 CET 2005
    Colin Guest

  8. #8

    Default Re: Illogical usage of swap

    On 2005-03-16 14:17, "Colin J. Raven" <kozy-kabin.nl> wrote: 
    >>
    >>Strange. AFAIK, processes are not swapped out to disk unless there is a
    >>severe memory shortage. The usual case is to have them "paged" out,
    >>which is a bit different.[/ref]
    >
    > That was my understanding also, and that's what's baffling us[/ref]

    Unless, of course, there _was_ a short period of memory shortage, some
    processes were swapped out (a pair of idle getty instances, for example)
    and they were never swapped back in because they are still idle.

    You can probably track this down to the specific process or processes
    that have the PS_INMEM bit turned off in their proc->p_sflag. But 116K
    seems too small to be a complete process.

    Giorgos Guest

  9. #9

    Default Re: Illogical usage of swap

    On Mar 16 at 15:34, Giorgos Keramidas suggested:
     
    >>
    >> That was my understanding also, and that's what's baffling us[/ref]
    >
    > Unless, of course, there _was_ a short period of memory shortage, some
    > processes were swapped out (a pair of idle getty instances, for example)
    > and they were never swapped back in because they are still idle.
    >
    > You can probably track this down to the specific process or processes
    > that have the PS_INMEM bit turned off in their proc->p_sflag. But 116K
    > seems too small to be a complete process.[/ref]

    This is going way above my head now, but great as a learning process
    anyway!

    May I ask how we would go about tracking processes with the
    PS-INMEM bit turned off? That sounds as if - to me, still a journeyman
    on the road to knowledge - grepping multiple source trees might be
    necessary, which frankly is a daunting process (to say the very least).
    Is there one meta-search string that would show this? and if so, where
    should this search be applied?

    Regards,
    -Colin
    --
    Colin J. Raven
    Wed Mar 16 14:43:00 CET 2005


    Colin Guest

  10. #10

    Default Re: Illogical usage of swap

    "Colin J. Raven" <kozy-kabin.nl> writes:
     
     
    >
    > Hmm, yeah I should have included what I've actually read in the
    > handbook and elsewhere. In fact I did read that exerpt you quoted the
    > URL for above sometime previously, when it was quoted in another
    > thread on the list.
    >
    > Free memory (or the lack thereof) isn't the issue though.
    >
    > The issue is this:
    > Swap: 8192M Total, 116K Used, 8192M Free
    > and that's the piece of the puzzle that has us all utterly baffled.[/ref]

    The same logic applies to some extent, though.
     

    I have noticed that FreeBSD 5.x is swapping out small amounts of
    memory in situations (as near I can tell; it is quite hard to be sure
    that the situation is really identical) where 4.x was not. I haven't
    really tried to track down exactly what's going on, but it's always
    been less than the text segment of any task in the system, so I was
    pretty sure I wasn't seeing a problem.
    Lowell Guest

  11. #11

    Default Re: Illogical usage of swap

    On Mar 16 at 09:24, Lowell Gilbert then said:
     

    >>
    >> Hmm, yeah I should have included what I've actually read in the
    >> handbook and elsewhere. In fact I did read that exerpt you quoted the
    >> URL for above sometime previously, when it was quoted in another
    >> thread on the list.
    >>
    >> Free memory (or the lack thereof) isn't the issue though.
    >>
    >> The issue is this:
    >> Swap: 8192M Total, 116K Used, 8192M Free
    >> and that's the piece of the puzzle that has us all utterly baffled.[/ref]
    >
    > The same logic applies to some extent, though.

    >
    > I have noticed that FreeBSD 5.x is swapping out small amounts of
    > memory in situations (as near I can tell; it is quite hard to be sure
    > that the situation is really identical) where 4.x was not. I haven't
    > really tried to track down exactly what's going on, but it's always
    > been less than the text segment of any task in the system, so I was
    > pretty sure I wasn't seeing a problem.[/ref]

    Except (if I'm following you sequentially, and if I'm not, then
    apologies!) this is FreeBSD 4.11.

    Regards,
    -Colin
    --
    Colin J. Raven
    Wed Mar 16 15:29:00 CET 2005

    Colin Guest

  12. #12

    Default Re: Illogical usage of swap



    > >
    > > Hmm, yeah I should have included what I've actually read in the
    > > handbook and elsewhere. In fact I did read that exerpt you quoted the
    > > URL for above sometime previously, when it was quoted in another
    > > thread on the list.
    > >
    > > Free memory (or the lack thereof) isn't the issue though.
    > >
    > > The issue is this:
    > > Swap: 8192M Total, 116K Used, 8192M Free
    > > and that's the piece of the puzzle that has us all utterly baffled.[/ref]
    >
    > The same logic applies to some extent, though.

    >
    > I have noticed that FreeBSD 5.x is swapping out small amounts of
    > memory in situations (as near I can tell; it is quite hard to be sure
    > that the situation is really identical) where 4.x was not. I haven't
    > really tried to track down exactly what's going on, but it's always
    > been less than the text segment of any task in the system, so I was
    > pretty sure I wasn't seeing a problem.[/ref]

    Swap space is also used for paging. If you are running through some
    big file or have processes that are live but not active, they can
    use up some page space. That space just stays there until it is
    needed otherwise. At least that is my understanding.

    ////jerry

    Jerry Guest

  13. #13

    Default Re: Illogical usage of swap

    On Mar 16 at 09:47, Jerry McAllister commented:
     
    >> 
    >>
    >> The same logic applies to some extent, though.
    >> 
    >>
    >> I have noticed that FreeBSD 5.x is swapping out small amounts of
    >> memory in situations (as near I can tell; it is quite hard to be sure
    >> that the situation is really identical) where 4.x was not. I haven't
    >> really tried to track down exactly what's going on, but it's always
    >> been less than the text segment of any task in the system, so I was
    >> pretty sure I wasn't seeing a problem.[/ref]
    >
    > Swap space is also used for paging. If you are running through some
    > big file or have processes that are live but not active, they can
    > use up some page space. That space just stays there until it is
    > needed otherwise. At least that is my understanding.[/ref]

    With 4GB of RAM to chew on, I must say this surprises me more than a
    bit. Recalling earlier in this thread, I said the box that this one
    replaced had dual Athlon MP's (forget what speed) and 1GB of RAM. It
    *never* swapped. It ran 4.10, then later upgraded to 5.3.

    We couldn't do 5.3 on this box because some controllers on the mobo
    weren't (yet) supported, so we elected to install 4.11. Then we took
    some jobs away from this server and reassigned 'em elsewhere (only
    because of the delay in sourcing the replacement) . So - purely
    theoretically - this server *should* have been sitting around less than
    fully tasked. It is way more powerful (by significant orders of
    magnitude) CPU-wise, has four times as much - and faster - memory, a
    faster FSB, SATA instead of ATA drives, 4.11-and-not-the-bleeding-edge-choice-of-OS.

    What I'm saying is that (almost) any of these factors taken individually
    should show a fairly relaxed, under-tasked errr..."happy" box. Take the
    ulative effect of the rather healthy upgrade and this box should be
    sitting around picking its nose so to speak, and watching TV because it
    has so little to do...hell we took work *away* from it.

    Except it swaps...sometimes as much as 2MB, mostly around 100 - 350K.

    I bring this to the list only because of the utterly insane nature of
    the situation. It's as if you replaced your 333MHZ 256MB/memory Dell
    Dimension with a dual 3.0 Xeon/4GB HP Kayak (or equivalent) workstation,
    then in the middle of reading email you suddenly hear it writing to
    disk, check your taskmanager and discover to your absolute horror that
    the thing is swapping......

    We have no custom apps running here, only stuff that comes right out of
    the ports tree. We're running email, DNS (BIND9)+PowerDNS, various A/V
    and antispam packages, screen, pine, mc, apache, perl, php, mysql,
    Courier IMAP, Postfix....absolutely nothing unusual that isn't run every
    day in hundreds of thousands of production environments. The custom
    stuff that *does* have the potential (but never did so on the old lower
    powered box) to use up memory has been moved off to another server or
    two.

    So there we have it....the mystery continues :) Thanks for your
    ongoing interest and curiosity.

    Regards,
    -Colin
    --
    Colin J. Raven
    Wed Mar 16 16:10:00 CET 2005



    Colin Guest

  14. #14

    Default Re: Illogical usage of swap


    > >
    > > Swap space is also used for paging. If you are running through some
    > > big file or have processes that are live but not active, they can
    > > use up some page space. That space just stays there until it is
    > > needed otherwise. At least that is my understanding.[/ref]
    >
    > With 4GB of RAM to chew on, I must say this surprises me more than a
    > bit. Recalling earlier in this thread, I said the box that this one
    > replaced had dual Athlon MP's (forget what speed) and 1GB of RAM. It
    > *never* swapped. It ran 4.10, then later upgraded to 5.3.
    >
    > We couldn't do 5.3 on this box because some controllers on the mobo
    > weren't (yet) supported, so we elected to install 4.11. Then we took
    > some jobs away from this server and reassigned 'em elsewhere (only
    > because of the delay in sourcing the replacement) . So - purely
    > theoretically - this server *should* have been sitting around less than
    > fully tasked. It is way more powerful (by significant orders of
    > magnitude) CPU-wise, has four times as much - and faster - memory, a
    > faster FSB, SATA instead of ATA drives, 4.11-and-not-the-bleeding-edge-choice-of-OS.
    >
    > What I'm saying is that (almost) any of these factors taken individually
    > should show a fairly relaxed, under-tasked errr..."happy" box. Take the
    > ulative effect of the rather healthy upgrade and this box should be
    > sitting around picking its nose so to speak, and watching TV because it
    > has so little to do...hell we took work *away* from it.
    >
    > Except it swaps...sometimes as much as 2MB, mostly around 100 - 350K.[/ref]

    That it uses some swap space does not mean it is not still
    very "undertasked" nor that it is getting close to being overloaded.

    It just uses that space in the course of its normal managing of
    processes. Someone else can much better tell you how that
    algorithm works. I'm just don't sweat it. It is normal and reasonable.

    ////jerry

     

    Jerry Guest

  15. #15

    Default Re: Illogical usage of swap

    Colin J. Raven wrote:
     
    That only tells you how much swap space you have available (8.2GB), it
    tells
    you absolutely nothing about why the system might be using some of that
    space.

    The previous line
     

    says that out of roughly 3.6 GB of memory, only 5.5 MB of it is not in
    use.
    Something ate the memory. The next clue is that 2.5 GB of that is
    "Inactive"
    which means that whatever is using it hasn't accessed it for a while
    (but it is
    still claiming to own it). Perhaps looking at the "RES" column of top will
    suggest a culprit without requiring you to become an expert on VM stats.


    Memory pages in the "inactive" queue have been removed from the page
    table for the owning processes, but are still intact and a page can be
    easily reassigned to the process that owns its contents if the process
    tries to access it. If it stays "inactive" long enough, and the "cache"
    queue starts to run low because processes are asking for memory,
    a page may be moved to the "cache" queue. This means that whatever
    action is necessary to make the memory available to other processes
    has been taken, which in some cases (but not all) will mean that the
    contents of the memory have been written out to the backing store
    (swap). Things that can be recovered by other means (e.g. out of the
    disk file that they were originally read from) need not be swapped,
    they can simply be discarded, but I don't know how smart the VM
    system is about that ("pretty smart" seems to be first-order answer).

    It looks to me like something small had not been used in a long time, so it
    got written out to the backing store (swap) when you started to get low on
    memory. Note that something in the "cache" queue may still reside
    in memory but also reside in swap. That means the memory it uses is
    immediately available for some other use if necessary (because it is
    already
    in swap), but is also immediately available for use by the process that
    owns
    the object in that memory space (because it still resides in memory).
    Thus,
    the mere fact that something has appeared in swap does not mean that
    any inefficiency has occured. It just means you were getting low on
    memory and the VM system started planning ahead by making more
    memory available. As long as the process that owns the memory object
    in swap doesn't try to access it, it will probably just sit there, doing no
    harm.

    Pages that don't contain anything owned by any process are in the
    "free" queue. If too much "free" memory aculates, the VM
    system will attempt to find something useful for it to do, perhaps
    assigning it to i/o buffers (the "buf" queue). I suppose that it could
    even move objects out of swap and back in to the "cache" queue, but I
    don't know if that actually happens, and even if it does, there is no
    reason to erase an object from swap until the owning process actually
    accesses it and makes it active again (or if you run low on swap).

    At least, that's how I understand it, but I may have some of it wrong.
    Read
    http://www.freebsd.org/doc/en_US.ISO8859-1/books/arch-handbook/vm.html
    and see what you think.

    - Bob







    Bob Guest

Similar Threads

  1. Swap image also swap link
    By Timothy Robertson in forum Dreamweaver AppDev
    Replies: 2
    Last Post: May 26th, 05:29 PM
  2. Print Outline Box Illogical Problem
    By Jeff_Ngan@adobeforums.com in forum Adobe Illustrator Windows
    Replies: 3
    Last Post: April 28th, 04:19 PM
  3. Replies: 6
    Last Post: November 29th, 07:16 PM
  4. USE UP ALL OF THE RAM WITHOUT ANY USAGE OF SWAP AND FINALLY HANG UP.
    By jenniferyiu in forum Linux Setup, Configuration & Administration
    Replies: 3
    Last Post: September 9th, 07:55 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139