Professional Web Applications Themes

Out of Memory! error on large tied hash - PERL Beginners

I've been using the DB_File module lately and I've run into a problem. I can create a tied hash and populate it with 23 million records. When I use another script to open the same file and tie a hash to it I get the "Out of Memory!" error. Did I do something wrong or is this a limitation of perl or the DB_File module? This is perl, v5.8.0 built for aix-thread-multi AIX 5.2...

  1. #1

    Default Out of Memory! error on large tied hash

    I've been using the DB_File module lately and I've run into a problem.
    I can create a tied hash and populate it with 23 million records. When
    I use another script to open the same file and tie a hash to it I get
    the "Out of Memory!" error. Did I do something wrong or is this a
    limitation of perl or the DB_File module?

    This is perl, v5.8.0 built for aix-thread-multi
    AIX 5.2
    Rob Guest

  2. Moderated Post

    Default Re: Out of Memory! error on large tied hash

    Removed by Administrator
    Chris Guest
    Moderated Post

  3. Moderated Post

    Default Re: Out of Memory! error on large tied hash

    Removed by Administrator
    Rob Guest
    Moderated Post

  4. #4

    Default Re: Out of Memory! error on large tied hash

    On Mon, 27 Sep 2004, Rob Benton wrote:
     

    Times 23 million records... 782,000,000,000 bytes / 728 gigabytes.

    Not counting overhead for managing each record in the file.

    This is nearly a dozen times larger than your physical memory.

    This process will be thrashing like crazy -- no wonder it explodes.

    Is it possible to operate on part of the data at a time, iterating over
    the results and only loading part of it into memory at once? Maybe not,
    I'm not sure, but it seems like this is the sort of thing to try.

    Alternatively, break it into chunks and try to infer totals (or whatever
    data ysis you're trying to do on the whole set) based on ysis of
    fragments of the data set that can be handled reliably.

    Either way (these ideas are of course variants of each other), you
    probably need to benchmark how much data you can insert before things go
    wrong, so that you know what your ceilings are.

    This may be the sort of problem that would best be handled by a proper
    database server. The folks writing RDBMSes have been tackling problems
    like this for decades now, and have some useful techniques; rather than
    reinvent the wheel, you can just leverage their efforts by putting
    everything into (say) MySQL or PostgreSQL. (Of course, that makes things
    more complicated too, but should at least get you past this problem.)



    --
    Chris Devers
    Chris Guest

  5. #5

    Default RE: Out of Memory! error on large tied hash

    Rob Benton wrote:
    .... 

    Don't do that. keys() needs to build an internal list to iterate over. Use
    scalar form of each() instead:

    while (my $key = each %tied_hash) { ... }
    Bob Guest

  6. #6

    Default Re: Out of Memory! error on large tied hash

    Chris Devers wrote:
     

    Yes this little job is getting too big for a dbm file. This was the
    first time I've tried it on anything > 8 million.
    Rob Guest

Similar Threads

  1. Hash aggregates ing out memory
    By Mike in forum PostgreSQL / PGSQL
    Replies: 5
    Last Post: February 26th, 03:12 PM
  2. Replies: 3
    Last Post: January 14th, 12:11 PM
  3. DB_File, large hash, memory.
    By Paul Marquess in forum PERL Miscellaneous
    Replies: 0
    Last Post: June 26th, 10:11 AM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139