Professional Web Applications Themes

Marshal efficiency - Ruby

Folks, As an intermediate step in a small software system that performs a large amount of data gathering, I am using Marshal to store processed results, so that other programs can use this information. At first, this was an excellent solution: couldn't be easier, and it demonstrated good performance. However, as I am collecting more and more data, the period of time required to serialise the data to disk is balooning - it seems quadratically. The current size of my data is: an array with 10000 moderately-sized objects. The size on disk is only around 900K, and it took at ...

  1. #1

    Default Marshal efficiency

    Folks,

    As an intermediate step in a small software system that performs a large
    amount of data gathering, I am using Marshal to store processed results,
    so that other programs can use this information.

    At first, this was an excellent solution: couldn't be easier, and it
    demonstrated good performance. However, as I am collecting more and more
    data, the period of time required to serialise the data to disk is
    balooning - it seems quadratically. The current size of my data is: an
    array with 10000 moderately-sized objects. The size on disk is only
    around 900K, and it took at least 10 minutes to write it.

    Part of the problem is my approach:
    - read the marshalled data
    - gather some more data into the array
    - write the marshalled data

    Each iteration produces a few thousand more data elements, but obviously
    has to write _all_ of the data back to disk.

    Can anyone suggest a better approach to storing the data? I am open to
    all suggestions, but I am hoping for a very easy solution. At work, where
    the use of Ruby is not admired, I don't want to expand the software
    dependencies if I can avoid it. I don't have much time, either, so I need
    a simple solution.

    This is using ruby 1.6.5 (I know I'm bad...) on Cygwin.

    Thanks,
    Gavin






    Gavin Sinclair Guest

  2. #2

    Default Re: Marshal efficiency [precompiled 1.8 for cygwin?]

    > Folks,
    >
    I wrote:
    > As an intermediate step in a small software system that performs a large
    > amount of data gathering, I am using Marshal to store processed results,
    > so that other programs can use this information.
    >
    > At first, this was an excellent solution: couldn't be easier, and it
    > demonstrated good performance. However, as I am collecting more and
    > more data, the period of time required to serialise the data to disk is
    > balooning - it seems quadratically. [...]

    Soon after I sent this, I decided to search the mailing list for existing
    information (yes, I often get things backwards...). It turns out that
    Marshal is known to be much faster in 1.7/1.8 than in 1.6.

    The mailing list does not, however, lead me to a precompiled 1.8 for
    Cygwin. So for future posterity, here it is, courtesy of Google:

    [url]http://mirrors.sunsite.dk/ruby/binaries/cygwin/1.8[/url]

    Gavin




    Gavin Sinclair Guest

  3. #3

    Default Re: Marshal efficiency

    Joel VanderWerf <vjoelPATH.Berkeley.EDU> wrote in message news:<3F17759B.2030100path.berkeley.edu>...
    > Gavin Sinclair wrote:
    > > Folks,
    > >
    > > As an intermediate step in a small software system that performs a large
    ....
    > > a simple solution.
    Try using ruby 1.8 and tell us the results!
    Dr AK Guest

Similar Threads

  1. Efficiency question
    By PlastechFish in forum Coldfusion Database Access
    Replies: 3
    Last Post: June 4th, 12:00 PM
  2. Marshal.GetActiveObject throws an exception in a C# webservice but
    By OfurGørn in forum ASP.NET Web Services
    Replies: 30
    Last Post: February 17th, 02:47 AM
  3. Database Efficiency
    By McKirahan in forum ASP Database
    Replies: 1
    Last Post: July 31st, 03:50 PM
  4. Ruby 1.8 and Marshal.load/Marshal.dump
    By Michael Davis in forum Ruby
    Replies: 0
    Last Post: October 10th, 10:15 PM
  5. Can't marshal MatchData?
    By Gavin Sinclair in forum Ruby
    Replies: 3
    Last Post: August 6th, 12:20 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139