Is this IPC scheme brain-dead?

[Richard]

I'm designing an IPC scheme that I've never used before, and I need
to vet it. I don't have any local resources to run it by, and would
appreciate it if someone would point out if I'm about to do something
really stupid due to possible personal ignorance of mine.

I have three processes. Inputs to the process #1 consist of a steady
stream of data, maxing at about 8MB/sec but averaging not much lower.
The first process just tries to keep up with the input, and adds a
little metadata to it before sending it to process #2 via shared
memory. Process #1 queues this data for about 200 ms before sending
it, so that it sends a big block rather than a bunch of little
records. The block size will naturally vary from

Process #2 takes its inputs from shared memory and does some
filtering and restructing (summarizing) of the data before sending it
to process #3 via shared memory.

The (possibly) unusual part is that I want to use FIFOs between
processes to coordinate the update/read status of shared memory
segments. My initial idea was to logically structure the shared
memory as an array of fixed-length records, and use semaphores to
implement a record-locking scheme for updates and reads. (I can
provide more detail on this, but it's hopefully pretty obvious what I
had in mind.)

The problem with fixed-length records, of course, is sizing them
correctly to avoid wasting space--or spending a lot of time dealing
with the individual data records that are bigger than the fixed
record size and winding up with a lot of overhead to manage the
overflow mappings.

To get around this, I figured I could write large blocks of varying
size to shared memory--and just use a FIFO from proc #1 that gives
the OFFSET and BYTE_LEN of the data written to shared memory. Proc #
2 picks this up, reads the block of data, and then uses a FIFO back
to proc #1 with OFFSET and BYTE_LEN to advise proc #1 that it's read
that block.

The writer to shared memory would just write along the shared memory
size until it wrapped around back to the top. The FIFOs would be
used mainly to advise the shared memory writer that a given
OFFSET/BYTE_SIZE message had been received by the reader and the data
picked up.

I do a similar thing between procs #2 and #3.

Now, it doesn't work too well if the writer fills up shared memory
faster than the reader reads it, but that's true anyway no matter
what access scheme I use.

Anyway, like I've said, if this is a notoriously stupid idea that I
just haven't bumped into yet...would somebody mind pointing that out?

(I know I'm asking a lot to ask programmers to point out what's wrong
with some other programmer's approach... ;)

--
For email, put NOT SPAM in Subject or I'll probably miss it.
<><

# rm /bin/laden
# /bin/laden: Not found

[Marc Rochkind]

On Sun, 27 Jul 2003 21:06:01 GMT, Richard <rh310@hotmail.com> wrote:

> I'm designing an IPC scheme that I've never used before, and I need to
> vet it. I don't have any local resources to run it by, and would
> appreciate it if someone would point out if I'm about to do something
> really stupid due to possible personal ignorance of mine.
>

[snipped .. see original msg]

This sounds like a promising scheme to me. With shared memory, you always
need one other IPC mechanism for synchronization. Using FIFOs or messages
also allows you to include some data in the synchronization message, which
is what you are doing here.

Design the thing so that you can play with various sizing parameters to
discover what works best.

An alternative to FIFOs would be SysV or POSIX messages. They have some
advantages over FIFOs, in that they are inherently messages, whereas with
FIFOs you have to superimpose the message structures on the stream-oriented
FIFO.

Staying away from semaphores is a good idea -- avoids lots of complex
analysis to ensure that you are deadlock and starvation free.

--Marc

>

[Larry Blanchard]

In article <oprszt5rolojfyi9@den.news.speakeasy.net>,
[email]rochkind@basepath.com[/email] says...

> An alternative to FIFOs would be SysV or POSIX messages. They have some
> advantages over FIFOs, in that they are inherently messages, whereas with
> FIFOs you have to superimpose the message structures on the stream-oriented
> FIFO.
>

Back in the dim days of slow 486's, I sent, IIRC, at least 10,000 small
messages a second around a loop of 3 or 4 processes. It might be
interesting to see what the throughput would be with todays hardware.
You might not even need shared memory :-).

BTW, that was with SYSV IPC on SCO's Xenix. I did have to play with the
kernel message queue parameters.

--
Where ARE those Iraqi WMDs?

[Marc Rochkind]

On Mon, 28 Jul 2003 09:48:39 -0700, Larry Blanchard <lblanch@fastmail.fm>
wrote:

[snip]

>>

> Back in the dim days of slow 486's, I sent, IIRC, at least 10,000 small
> messages a second around a loop of 3 or 4 processes. It might be
> interesting to see what the throughput would be with todays hardware.
> You might not even need shared memory :-).
>
> BTW, that was with SYSV IPC on SCO's Xenix. I did have to play with the
> kernel message queue parameters.
>

Depends on the message size. For example, for some tests I ran with
messages sizes of 100 bytes, 2000, 20000, and 100000,
the time to pass the data between processes using shared memory with
semaphores was flat -- around 1.3 units. Times for SysV messages were .9
for 100 byte messages and 1.82 for 2000 byte messages. The two larger sizes
were too big. For AF_UNIX sockets, times were 1.84, 2.15, 10.15, and 44.13.
(All times were normalized by running the test for FIFOs with 100-byte
messges and setting that time to 1.00.) I ran the test with Solaris, Linux,
FreeBSD, and Darwin (Mac OS X) with very similar results.

Conclusion: Shared memory is only valuable if (1) there is some reason why
the memory has to be shared, or (2) the message size is more than a few
hundred bytes.

--Marc

[Richard]

[email]apm35@student.open.ac.uk[/email] wrote...

> Marc Rochkind <rochkind@basepath.com> wrote in message news:<oprszt5rolojfyi9@den.news.speakeasy.net>...

> > On Sun, 27 Jul 2003 21:06:01 GMT, Richard <rh310@hotmail.com> wrote:
> > This sounds like a promising scheme to me.

>
> Me too. Will you GPL it when it's finished?

I'd be happy to. I guess I should make sure some leech doesn't have
a patent on it first. Thanks for the suggestion.

> > With shared memory, you always
> > need one other IPC mechanism for synchronization. Using FIFOs or messages
> > also allows you to include some data in the synchronization message, which
> > is what you are doing here.

>
> I think that what he is after is actually a simulation of FIFOs but
> without the FIFO limiation that there is a relatively small maximum
> message size. Obviously, a msg cannot be larger than the shared memory
> file but FIFOs have a msg limit imposed by the kernel and IMO it is
> quite small. It can lead to queueing problems when there is latency.
>

> > An alternative to FIFOs would be SysV or POSIX messages.

>
> Please don't use SysV msg queues. See Stevens for why these are
> inferior (despite advantages such as builtin demultiplexing).

This is the first time I've used POSIX shared memory, and so far I've
been very pleasantly surprised at how flexible and easy it is to use.
For some reason, I was expecting it to be more difficult than it
turned out to be--it took me a few hours just to even recognize its
simplicity.

I'd prefer to use messages over FIFOs, and will look that doing that
today. FIFOs are simple and it was easy to start there, but I do
have some concerns about the 4K limit, even though my "records" into
the FIFO are only 10 bytes each.

--
For email, put NOT SPAM in Subject or I'll probably miss it.
<><