Professional Web Applications Themes

select() max number of file descriptors - UNIX Programming

It seems that my system's select call has a limitation of 1024 open file descriptors. I need to learn about this restriction as I am programming a multiplexed I/O server. Is the 1024 open file descriptor limitation a per process limitation or a system wide limitation? How can I find out how many open files my system supports? Where do I look to find out if my C / C++ implementation has similar such file limits? I have done some googling and have a general idea of the implications, but I need details for my project. What are some solutions ...

  1. #1

    Default select() max number of file descriptors

    It seems that my system's select call has a limitation of 1024 open
    file descriptors.

    I need to learn about this restriction as I am programming a
    multiplexed I/O server.

    Is the 1024 open file descriptor limitation a per process limitation
    or a system wide limitation?

    How can I find out how many open files my system supports?

    Where do I look to find out if my C / C++ implementation has similar
    such file limits?

    I have done some googling and have a general idea of the implications,
    but I need details for my project.

    What are some solutions for handling a large number ( 5000+ ) of
    simultaneously open sockets?

    Thank you for reading,
    Charles Wilkins

    Charles Wilkins Guest

  2. #2

    Default Re: select() max number of file descriptors

    Charles Wilkins wrote :
    > Is the 1024 open file descriptor limitation a per process limitation
    > or a system wide limitation?
    >
    > How can I find out how many open files my system supports?
    sysconf(_SC_OPEN_MAX)
    > What are some solutions for handling a large number ( 5000+ ) of
    > simultaneously open sockets?
    could you use poll() ?
    But I am wondering if there is no such limit with poll()

    --
    DINH V. Hoa,

    "s/^\(\(\\.\|[^\[]\|\[\(\^.\|[^^]\)[^]]*\]\)*\)\(\\[^\*[]\)/\1\\\4/"
    -- Stéphane CHAZELAS

    DINH Viêt Hoà Guest

  3. #3

    Default Re: select() max number of file descriptors

    DINH =?iso-8859-1?Q?Vi=EAt_Ho=E0?= <dinh.viet.hoafree.fr> writes:
    >Charles Wilkins wrote :
    >> Is the 1024 open file descriptor limitation a per process limitation
    >> or a system wide limitation?
    >>
    >> How can I find out how many open files my system supports?
    >sysconf(_SC_OPEN_MAX)
    Might just return your current soft limit and ntot the real system limit.

    Generally, there are three limits:

    - max numbers of fds in a single process (could be as low as 1024)
    - process hard limit (equal or less than the first)
    - process soft limit.
    >> What are some solutions for handling a large number ( 5000+ ) of
    >> simultaneously open sockets?
    >could you use poll() ?
    >But I am wondering if there is no such limit with poll()
    poll has no restriction on the size of the pollset but your process
    may stil be limited.

    Solaris does not have a kernel enforced hard limit but a per-proces
    hard limit of 1024 which can be raised or lowered.


    Using larger fd sets generally requires a recompile with a certain
    flags (typeically -DFD_SETSIZE=value)

    Casper
    --
    Expressed in this posting are my opinions. They are in no way related
    to opinions held by my employer, Sun Microsystems.
    Statements on Sun products included here are not gospel and may
    be fiction rather than truth.
    Casper H.S. Dik Guest

  4. #4

    Default Re: select() max number of file descriptors

    :::Snip:::
    > What are some solutions for handling a large number ( 5000+ ) of
    > simultaneously open sockets?
    It depends on what type of processing will be needed to be done on the
    messages received by the clients. For instance, I've got servers at work
    that handle very little load, and then some that need to process 400-1000+
    blocked messages a second, and do alot with each message in the block.
    If you want to have 5000+ connections, even if it's the simplest, I
    would probably suggest multithreading the application with pthreads. You
    could for instance have a thread accepting new connections and then either
    pass the connection off to an existing thread (i.e. have 20 connections
    per), or create a new thread if that limit has been reached. I've done this
    and it allows me to change the connections per thread, and maximum threads
    via a config file.


    Brian C Guest

  5. #5

    Default Re: select() max number of file descriptors

    On Thu, 7 Aug 2003 11:59:42 +0200 DINH Vi?t Ho? <fr> wrote:
    | Charles Wilkins wrote :
    |
    |> Is the 1024 open file descriptor limitation a per process limitation
    |> or a system wide limitation?
    |>
    |> How can I find out how many open files my system supports?
    |
    | sysconf(_SC_OPEN_MAX)
    |
    |> What are some solutions for handling a large number ( 5000+ ) of
    |> simultaneously open sockets?
    |
    | could you use poll() ?
    | But I am wondering if there is no such limit with poll()

    There would not need to be a compile time limit, other than an advisory one.
    Your limit would be how many descriptors you can actually get before you end
    up with errors like EMFILE or ENFILE. If you actually have the descriptors,
    I'm sure poll() will let you wait on them. Now the issue will be finding
    enough virtual memory to hold the array of struct pollfd in one big chunk.

    --
    -----------------------------------------------------------------------------
    | Phil Howard KA9WGN | http://linuxhomepage.com/ http://ham.org/ |
    | (first name) at ipal.net | http://phil.ipal.org/ http://ka9wgn.ham.org/ |
    -----------------------------------------------------------------------------
    phil-news-nospam@ipal.net Guest

  6. #6

    Default Re: select() max number of file descriptors

    >>> Charles Wilkins wrote:

    CW> It seems that my system's select call has a limitation of 1024 open
    CW> file descriptors.
    CW> I need to learn about this restriction as I am programming a
    CW> multiplexed I/O server.

    Restriction can be on user level or on kernel level. On all unixes I seen,
    fd_set is a bit set in which one bit corresponds to one descriptor.
    Even if your system (differently from BSD) doesn't allow to redefine
    FD_SETSIZE to a larger value, but if kernel has no such limit, you can
    allocate aribtrary-sized chunks from heap and use them as fd_set.
    But, this is too tricky. Try to use poll(), /dev/poll, kqueue, etc. - whatever
    is accessible.

    CW> Is the 1024 open file descriptor limitation a per process limitation
    CW> or a system wide limitation?

    It can be said unless you name your system.

    CW> How can I find out how many open files my system supports?

    There can be system-wide limit, per-user limit, per-process limit, etc.
    sysconf(_SC_OPEN_MAX); getrlimit(RLIMIT_NOFILE) and so on.

    CW> What are some solutions for handling a large number ( 5000+ ) of
    CW> simultaneously open sockets?

    http://www.kegel.com/c10k.html


    -netch-
    Valentin Guest

Similar Threads

  1. File Descriptors
    By Jim Bo in forum Linux / Unix Administration
    Replies: 1
    Last Post: August 18th, 04:16 PM
  2. Displaying Open File Descriptors for a Process
    By Peter von Zirpolo in forum Linux / Unix Administration
    Replies: 5
    Last Post: January 25th, 09:54 PM
  3. Saving file descriptors
    By Peteris Krumins in forum UNIX Programming
    Replies: 4
    Last Post: July 24th, 12:51 PM
  4. increasing file descriptors (Solaris 8) on the fly
    By Oscar del Rio in forum Sun Solaris
    Replies: 0
    Last Post: June 24th, 04:44 PM
  5. select only number from varchar2
    By in forum Oracle Server
    Replies: 1
    Last Post: January 5th, 04:30 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139