Professional Web Applications Themes

Max files in unix folder from PIL process - FreeBSD

Hi. I am creating a python application that uses PIL to generate thumbnails and sized images. It is beginning to look the volume of images will be large. This has got me to thinking. Is there a number that Unix can handle in a single directory. I am using FreeBSD4.x at the moment. I am thinking the number could be as high 500,000 images in a single directory but more likely in the range of 6,000 to 30,000 for most. I did not want to store these in Postgres. I will most likely to break these into directories by size ...

  1. #1

    Default Re: Max files in unix folder from PIL process

    Hi. I am creating a python application that uses PIL to generate
    thumbnails and sized images. It is beginning to look the volume of
    images will be large. This has got me to thinking. Is there a number
    that Unix can handle in a single directory. I am using FreeBSD4.x at
    the moment. I am thinking the number could be as high 500,000 images in
    a single directory but more likely in the range of 6,000 to 30,000 for
    most. I did not want to store these in Postgres. I will most likely to
    break these into directories by size ie. thumbnail, small, medium,
    large, etc. . That will at least take it down by a factor of the
    number of sizes used but still the possibility of a very large number
    (maximum to perhaps 100,000 or more) There is really no other way that
    I can think of to categorize these at the moment. Should this pose a
    problem on the filesystem? How will it affect the use of Unix tools?
    Will there be access problems that affect speed? This is unchartered
    territory for me so hope someone who has been there, done that can
    provide some of what they learned from experience. Many thanks.

    Regards,
    David

    David Guest

  2. #2

    Default Re: Max files in unix folder from PIL process

    On Mon, 28 Mar 2005 21:25:28 -0400, David Pratt <ca> wrote: 


    I've had apps that had over 800k subdirectories and files. As long as
    you know the filename, performance will not be an issue. However, if
    you don't know the full path, then you may want to either redesign the
    app or consider storing everything in the DB. The apps that I've seen
    had the full path stored in the database, and the image on a file
    server.

    I've also had apps that stored everything in the database, and that
    turned out to be a nightmare. Eventually, we moved everything from the
    DB to the filesystem.

    Corey
    Corey Guest

  3. #3

    Default Re: Max files in unix folder from PIL process

    Hi Corey. Thank you for your reply. 800K is pretty significant. Yes,
    the reason I want to use the filesystem is to avoid the speed problems
    that come from storing images in the database. I can see there really
    being no limit when it comes to spreading the numbers thinner but most
    concerned about so many in a single directory. References to the
    images will come from the database. Do you don't think I should have
    any problem with 150 - 175 K in a single directory - no subdirectories
    - (just 175K jpg image files of a particular size)? I guess you
    definitely don't want to do an ls. Would that crap out the server?

    Regards,
    David

    On Monday, March 28, 2005, at 09:44 PM, Corey Brune wrote:
     
    David Guest

  4. #4

    Default Re: Max files in unix folder from PIL process

    On Tuesday 29 March 2005 03:25, David Pratt wrote: 

    I'd say hardcode a split-up at about ~5000 images per dir and renice if
    needed. Test with python whether reading one huge dir or reading and
    concatting several smaller (say 5000 files) dirs works better. Apparantly you
    got the pr0n, err, data, .. so you're in a position to test just what you're
    wondering about. If you want speed, stay away from databases/sql I think.

    I only have experience with my own pr0n, err, data, and its far less than what
    you want to deal with (pil works great though).

    My EUR 0.02

    Dan
    Danny Guest

Similar Threads

  1. How do I batch process a whole folder?
    By Karo_Karo@adobeforums.com in forum Adobe Photoshop Mac CS, CS2 & CS3
    Replies: 6
    Last Post: May 9th, 11:02 PM
  2. process id 0 in unix
    By manohar in forum UNIX Programming
    Replies: 4
    Last Post: November 25th, 07:03 PM
  3. Replies: 2
    Last Post: October 10th, 07:20 AM
  4. Manipulating a Unix mailbox folder.
    By Steffen Müller in forum PERL Modules
    Replies: 0
    Last Post: August 18th, 08:59 AM
  5. Reading UNIX files on PC
    By kevin bailey in forum SCO
    Replies: 1
    Last Post: July 29th, 04:39 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139