file-backed array of classes?

[Rasputin]

I was looking at prototyping a write once file service, a bit like
Plan9s Venti system. I delete files a lot :)
As usual, my grand designs are thwarted by the basics...

I'm at the 'get the unit tests of the simplest possible implementation to
pass' stage, and need something like an append-only file of fixed
size records (data blocks). Have we got classes or methods to do
record-based file IO? I dont mind reinventing a wheel otherwise.

Next problem (I learned to never look more than 1 problem ahead or I
never start anything). I need an index that gets dynamically updated as
the record file is written to.

The ideal solution would be a hash that was backed by a file somehow-
then I realised I could implement the data blocks themselves
the same way.

I looked through pickAxe and RAA for these two and found a lot of SQL
persistence APIs, but that feels like overkill....

Does anyone know of a more lightweight solution?
--
I don't care who does the electing as long as I get to do the
nominating.
-- Boss Tweed
Rasputin :: Jack of All Trades - Master of Nuns

[Robert Klemme]

"Rasputin" <rasputin@idoru.mine.nu> schrieb im Newsbeitrag
news:20031114115944.GA18331@lb.tenfour...

>
> I was looking at prototyping a write once file service, a bit like
> Plan9s Venti system. I delete files a lot :)
> As usual, my grand designs are thwarted by the basics...
>
> I'm at the 'get the unit tests of the simplest possible implementation

to

> pass' stage, and need something like an append-only file of fixed
> size records (data blocks). Have we got classes or methods to do
> record-based file IO? I dont mind reinventing a wheel otherwise.

A typical solution might involve IO with binary mode with Array#pack and
String.unpack. Then you'll have to do the byte layout and record size
fixing yourself.

Or you can use Marshalling for this: define a class and marshal instances
one at a time into the file:

st = Struct.new( "FooRecord", :name, :age )
records = (1..10).map{|i| st.new("hello", i)}

File.open("foo.bin", "wb") do |io|
records.each do |rec|
Marshal.dump(rec, io)
end
end

File.open("foo.bin", "rb") do |io|
until io.eof?
obj = Marshal.load(io)
puts obj
end
end

Of courese you can marshal a complete array, too. But then, all instances
are loaded at once and with the approach above you can read one record at
a time, thus navigating (not very efficient though).

> Next problem (I learned to never look more than 1 problem ahead or I
> never start anything). I need an index that gets dynamically updated as
> the record file is written to.

If the dataset will grow not too big, you could use a Hash and Marshal
that in a file...

> The ideal solution would be a hash that was backed by a file somehow-
> then I realised I could implement the data blocks themselves
> the same way.
>
> I looked through pickAxe and RAA for these two and found a lot of SQL
> persistence APIs, but that feels like overkill....
>
> Does anyone know of a more lightweight solution?

Not perfectly what you are looking for but maybe some ideas to play
with...

robert

[Neil Mc Laughlin]

Have you considered PStore in the standard library?


----- Original Message -----
From: "Rasputin" <rasputin@idoru.mine.nu>
To: "ruby-talk ML" <ruby-talk@ruby-lang.org>
Sent: Friday, November 14, 2003 11:59 AM
Subject: file-backed array of classes?

>
> I was looking at prototyping a write once file service, a bit like
> Plan9s Venti system. I delete files a lot :)
> As usual, my grand designs are thwarted by the basics...
>
> I'm at the 'get the unit tests of the simplest possible implementation to
> pass' stage, and need something like an append-only file of fixed
> size records (data blocks). Have we got classes or methods to do
> record-based file IO? I dont mind reinventing a wheel otherwise.
>
> Next problem (I learned to never look more than 1 problem ahead or I
> never start anything). I need an index that gets dynamically updated as
> the record file is written to.
>
> The ideal solution would be a hash that was backed by a file somehow-
> then I realised I could implement the data blocks themselves
> the same way.
>
> I looked through pickAxe and RAA for these two and found a lot of SQL
> persistence APIs, but that feels like overkill....
>
> Does anyone know of a more lightweight solution?
> --
> I don't care who does the electing as long as I get to do the
> nominating.
> -- Boss Tweed
> Rasputin :: Jack of All Trades - Master of Nuns
>

[Simon Strandgaard]

On Fri, 14 Nov 2003 20:59:47 +0900, Rasputin wrote:

> I'm at the 'get the unit tests of the simplest possible implementation to
> pass' stage, and need something like an append-only file of fixed
> size records (data blocks). Have we got classes or methods to do
> record-based file IO? I dont mind reinventing a wheel otherwise.

record-based file IO, can you explain further?

> Next problem (I learned to never look more than 1 problem ahead or I
> never start anything). I need an index that gets dynamically updated as
> the record file is written to.

I have written a robust-iterator primitive, where iterators gets notified
when somebody insert/delete data, so they iterator keeps pointing at the
same position.

[url]http://rubyforge.org/cgi-bin/viewcvs/cgi/viewcvs.cgi/projects/experimental/iterator2/?cvsroot=aeditor[/url]

--
Simon Strandgaard