strange multithreaded read() behavior

[John Galt]

I have a multithreaded program that behaves very strangely when
read()ing and write()ing to different sockets. Any real
thread/socket/select gurus out there?

The program consists of a main thread that does a select() like this:

int client_sockfds[10]; /* there are 10 clients */
FD_SET(listening_sockfd, &clients);
for(i=0; i < 10; i++)
FD_SET(client_sockfds[i], &clients);

select(12, &clients, (fd_set *)0, (fd_set *)0, (struct timeval *)0);

for(i=0; i < 10; i++)
{
if (FD_ISSET(client_sockfds[i], &clients))
/* mq_send() a message to the threadpool with the socket
descriptor */
}

And worker threads basically do this:

for(;;)
{
mq_receive(...); /* blocks until a message arrives from the main
thread
* this is synchronized; only one thread will
awake.
* the message is the socket descriptor of a
client
* that, according to select(), is ready for
reading
*/
len = read(sockfd, buf, BUFSIZ);
write(sockfd, buf, len);
/* go back to waiting for a message */
}

I am seeing this behavior:

- The main sends two messages, each with a different client socket
descriptor.
- Two different threads get one socket descriptor each
- The first thread to complete a read() basically seems to take over,
i.e.
the other thread's read()s are blocked until the first client
disconnects!

I am at a loss as to why read()s from other threads block in this
manner until the first client disconnects. My original intent was for
threads to complete one read()/write() (i.e. echo) and then go back to
waiting.

I've been told to put them into nonblocking mode, but am not sure
_why_ that will work. (BTW, these are plain vanilla SOCK_STREAM
sockets on a SPARC Blade 1000 running Solaris 8, using gcc, compiled
with -lpthread, -lsocket -lnsl).

I welcome any advice/comment/suggestions. Thanks in advance!

regards,
John Galt.

[John Galt]

I have a multithreaded program that behaves very strangely when
read()ing and write()ing to different sockets. Any real
thread/socket/select gurus out there?

The program consists of a main thread that does a select() like this:

int client_sockfds[10]; /* there are 10 clients */
FD_SET(listening_sockfd, &clients);
for(i=0; i < 10; i++)
FD_SET(client_sockfds[i], &clients);

select(12, &clients, (fd_set *)0, (fd_set *)0, (struct timeval *)0);

for(i=0; i < 10; i++)
{
if (FD_ISSET(client_sockfds[i], &clients))
/* mq_send() a message to the threadpool with the socket
descriptor */
}

And worker threads basically do this:

for(;;)
{
mq_receive(...); /* blocks until a message arrives from the main
thread
* this is synchronized; only one thread will
awake.
* the message is the socket descriptor of a
client
* that, according to select(), is ready for
reading
*/
len = read(sockfd, buf, BUFSIZ);
write(sockfd, buf, len);
/* go back to waiting for a message */
}

I am seeing this behavior:

- The main sends two messages, each with a different client socket
descriptor.
- Two different threads get one socket descriptor each
- The first thread to complete a read() basically seems to take over,
i.e.
the other thread's read()s are blocked until the first client
disconnects!

I am at a loss as to why read()s from other threads block in this
manner until the first client disconnects. My original intent was for
threads to complete one read()/write() (i.e. echo) and then go back to
waiting.

I've been told to put them into nonblocking mode, but am not sure
_why_ that will work. (BTW, these are plain vanilla SOCK_STREAM
sockets on a SPARC Blade 1000 running Solaris 8, using gcc, compiled
with -lpthread, -lsocket -lnsl).

I welcome any advice/comment/suggestions. Thanks in advance!

regards,
John Galt.

[David Schwartz]

"John Galt" <johngalt__@hotmail.com> wrote in message
news:216f064.0307082208.7aaab8f0@posting.google.co m...

> I have a multithreaded program that behaves very strangely when
> read()ing and write()ing to different sockets. Any real
> thread/socket/select gurus out there?
>
> The program consists of a main thread that does a select() like this:
>
> int client_sockfds[10]; /* there are 10 clients */
> FD_SET(listening_sockfd, &clients);
> for(i=0; i < 10; i++)
> FD_SET(client_sockfds[i], &clients);
>
> select(12, &clients, (fd_set *)0, (fd_set *)0, (struct timeval *)0);
>
> for(i=0; i < 10; i++)
> {
> if (FD_ISSET(client_sockfds[i], &clients))
> /* mq_send() a message to the threadpool with the socket
> descriptor */
> }

Ouch. If it loops, it will send bazillions of messages because the
socket is still ready for reading!

> And worker threads basically do this:
>
> for(;;)
> {
> mq_receive(...); /* blocks until a message arrives from the main
> thread
> * this is synchronized; only one thread will
> awake.
> * the message is the socket descriptor of a
> client
> * that, according to select(), is ready for
> reading
> */
> len = read(sockfd, buf, BUFSIZ);
> write(sockfd, buf, len);
> /* go back to waiting for a message */
> }

> I am seeing this behavior:
>
> - The main sends two messages, each with a different client socket
> descriptor.
> - Two different threads get one socket descriptor each
> - The first thread to complete a read() basically seems to take over,
> i.e.
> the other thread's read()s are blocked until the first client
> disconnects!

Probably because *all* the threads are blocked on the other decriptor.

> I've been told to put them into nonblocking mode, but am not sure
> _why_ that will work. (BTW, these are plain vanilla SOCK_STREAM
> sockets on a SPARC Blade 1000 running Solaris 8, using gcc, compiled
> with -lpthread, -lsocket -lnsl).

Because you don't want to block and you can't assure that you don't
block without non-blocking sockets. However, you seem to want to do blocking
writes, which is kind of odd.

DS

[David Schwartz]

"John Galt" <johngalt__@hotmail.com> wrote in message
news:216f064.0307082208.7aaab8f0@posting.google.co m...

> I have a multithreaded program that behaves very strangely when
> read()ing and write()ing to different sockets. Any real
> thread/socket/select gurus out there?
>
> The program consists of a main thread that does a select() like this:
>
> int client_sockfds[10]; /* there are 10 clients */
> FD_SET(listening_sockfd, &clients);
> for(i=0; i < 10; i++)
> FD_SET(client_sockfds[i], &clients);
>
> select(12, &clients, (fd_set *)0, (fd_set *)0, (struct timeval *)0);
>
> for(i=0; i < 10; i++)
> {
> if (FD_ISSET(client_sockfds[i], &clients))
> /* mq_send() a message to the threadpool with the socket
> descriptor */
> }

Ouch. If it loops, it will send bazillions of messages because the
socket is still ready for reading!

> And worker threads basically do this:
>
> for(;;)
> {
> mq_receive(...); /* blocks until a message arrives from the main
> thread
> * this is synchronized; only one thread will
> awake.
> * the message is the socket descriptor of a
> client
> * that, according to select(), is ready for
> reading
> */
> len = read(sockfd, buf, BUFSIZ);
> write(sockfd, buf, len);
> /* go back to waiting for a message */
> }

> I am seeing this behavior:
>
> - The main sends two messages, each with a different client socket
> descriptor.
> - Two different threads get one socket descriptor each
> - The first thread to complete a read() basically seems to take over,
> i.e.
> the other thread's read()s are blocked until the first client
> disconnects!

Probably because *all* the threads are blocked on the other decriptor.

> I've been told to put them into nonblocking mode, but am not sure
> _why_ that will work. (BTW, these are plain vanilla SOCK_STREAM
> sockets on a SPARC Blade 1000 running Solaris 8, using gcc, compiled
> with -lpthread, -lsocket -lnsl).

Because you don't want to block and you can't assure that you don't
block without non-blocking sockets. However, you seem to want to do blocking
writes, which is kind of odd.

DS

[John Galt]

> > int client_sockfds[10]; /* there are 10 clients */

> > FD_SET(listening_sockfd, &clients);
> > for(i=0; i < 10; i++)
> > FD_SET(client_sockfds[i], &clients);
> >
> > select(12, &clients, (fd_set *)0, (fd_set *)0, (struct timeval *)0);
> >
> > for(i=0; i < 10; i++)
> > {
> > if (FD_ISSET(client_sockfds[i], &clients))
> > /* mq_send() a message to the threadpool with the socket
> > descriptor */
> > }

>
> Ouch. If it loops, it will send bazillions of messages because the
> socket is still ready for reading!

Yes! Man David, you're good... What's a good way around this? Can I
put a timeout in select()? (I am guessing not; does the timeout
indicate how long to wait _until_ something is ready?)

Also, will a client socket test true for reading all the time? Or will
it test true only if the client actually sent something. To get more
precise, suppose the client sends the string "hello\r\n", will the
socket test true for reading 7 times or just one time?

>

> > And worker threads basically do this:
> >
> > for(;;)
> > {
> > mq_receive(...); /* blocks until a message arrives from the main
> > thread
> > * this is synchronized; only one thread will
> > awake.
> > * the message is the socket descriptor of a
> > client
> > * that, according to select(), is ready for
> > reading
> > */
> > len = read(sockfd, buf, BUFSIZ);
> > write(sockfd, buf, len);
> > /* go back to waiting for a message */
> > }

>

> > I am seeing this behavior:
> >
> > - The main sends two messages, each with a different client socket
> > descriptor.
> > - Two different threads get one socket descriptor each
> > - The first thread to complete a read() basically seems to take over,
> > i.e.
> > the other thread's read()s are blocked until the first client
> > disconnects!

>
> Probably because *all* the threads are blocked on the other decriptor.

Not sure what you mean here. Only one thread will get the message and
come out of the mq_receive(). The other threads will remain blocked in
mq_receive(). (I don't think that more than one thread is blocked on
the same socket descriptor because the first client does get a full
response, inspite of the bazillions of messages.)

>

> > I've been told to put them into nonblocking mode, but am not sure
> > _why_ that will work. (BTW, these are plain vanilla SOCK_STREAM
> > sockets on a SPARC Blade 1000 running Solaris 8, using gcc, compiled
> > with -lpthread, -lsocket -lnsl).

>
> Because you don't want to block and you can't assure that you don't
> block without non-blocking sockets. However, you seem to want to do blocking
> writes, which is kind of odd.

Hmm. If a socket tests true for reading after a select(), then why
would a subsequent read() block? I do take your point on the write()
part. In fact, from the debugging messages in my code, I note that
there are a helluva lot of zero byte read()s.


John

>
> DS

[John Galt]

> > int client_sockfds[10]; /* there are 10 clients */

> > FD_SET(listening_sockfd, &clients);
> > for(i=0; i < 10; i++)
> > FD_SET(client_sockfds[i], &clients);
> >
> > select(12, &clients, (fd_set *)0, (fd_set *)0, (struct timeval *)0);
> >
> > for(i=0; i < 10; i++)
> > {
> > if (FD_ISSET(client_sockfds[i], &clients))
> > /* mq_send() a message to the threadpool with the socket
> > descriptor */
> > }

>
> Ouch. If it loops, it will send bazillions of messages because the
> socket is still ready for reading!

Yes! Man David, you're good... What's a good way around this? Can I
put a timeout in select()? (I am guessing not; does the timeout
indicate how long to wait _until_ something is ready?)

Also, will a client socket test true for reading all the time? Or will
it test true only if the client actually sent something. To get more
precise, suppose the client sends the string "hello\r\n", will the
socket test true for reading 7 times or just one time?

>

> > And worker threads basically do this:
> >
> > for(;;)
> > {
> > mq_receive(...); /* blocks until a message arrives from the main
> > thread
> > * this is synchronized; only one thread will
> > awake.
> > * the message is the socket descriptor of a
> > client
> > * that, according to select(), is ready for
> > reading
> > */
> > len = read(sockfd, buf, BUFSIZ);
> > write(sockfd, buf, len);
> > /* go back to waiting for a message */
> > }

>

> > I am seeing this behavior:
> >
> > - The main sends two messages, each with a different client socket
> > descriptor.
> > - Two different threads get one socket descriptor each
> > - The first thread to complete a read() basically seems to take over,
> > i.e.
> > the other thread's read()s are blocked until the first client
> > disconnects!

>
> Probably because *all* the threads are blocked on the other decriptor.

Not sure what you mean here. Only one thread will get the message and
come out of the mq_receive(). The other threads will remain blocked in
mq_receive(). (I don't think that more than one thread is blocked on
the same socket descriptor because the first client does get a full
response, inspite of the bazillions of messages.)

>

> > I've been told to put them into nonblocking mode, but am not sure
> > _why_ that will work. (BTW, these are plain vanilla SOCK_STREAM
> > sockets on a SPARC Blade 1000 running Solaris 8, using gcc, compiled
> > with -lpthread, -lsocket -lnsl).

>
> Because you don't want to block and you can't assure that you don't
> block without non-blocking sockets. However, you seem to want to do blocking
> writes, which is kind of odd.

Hmm. If a socket tests true for reading after a select(), then why
would a subsequent read() block? I do take your point on the write()
part. In fact, from the debugging messages in my code, I note that
there are a helluva lot of zero byte read()s.


John

>
> DS

[David Schwartz]

"John Galt" <johngalt__@hotmail.com> wrote in message
news:216f064.0307091258.2804f210@posting.google.co m...

> > Ouch. If it loops, it will send bazillions of messages because the
> > socket is still ready for reading!

> Yes! Man David, you're good... What's a good way around this? Can I
> put a timeout in select()? (I am guessing not; does the timeout
> indicate how long to wait _until_ something is ready?)

No, you can't. There are several ways you can do this, here are a few
good ones:

1) Don't call select until all the worker threads have finished doing
their jobs. The problem with this is that if a job might block for a long
time, then other jobs won't be able to do new I/O.

2) Remove the socket from the select set until the worker thread has
finished with it. The problem with this is that the main select thread may
still be in select when the worker finishes and the socket won't be selected
on until a new select is started.

There are variant approaches to both of these main approaches. For
example, a variant of the first approach allows a worker thread to signal
when it's done the I/O (but not necessarily finished all the processing it
might do), at this point, it's safe to put the socket back in the select set
but may not be safe to dispatch a worker thread to service it. A variant of
the second approach uses a pipe that's also in the select set -- workers
send a dummy byte on the pipe to force the main thread to start a new
'select' when they're done.

What works best depends upon your particular design constraints and what
code you've already got to work with. You should probably keep at least two
copies of each select set -- one is the one you pass to select directly and
is only touched by the thread that calls select, the other is the master
copy that file descriptors are added to and removed from, and that can be
protected by a lock so other threads can modify it. The select thread then
locks the master sets, copies them, unlocks the master sets, and passes the
copies to select.

> Also, will a client socket test true for reading all the time? Or will
> it test true only if the client actually sent something. To get more
> precise, suppose the client sends the string "hello\r\n", will the
> socket test true for reading 7 times or just one time?

It will test true for reading so long as there is something to read. It
will continue to test ready for reading until all seven bytes are read.
Detecting a hit for read has no effect on the state of the socket and its
readability. There are no events to be consume, just conditions to be
detected so long as the conditions apply.

> > Probably because *all* the threads are blocked on the other

decriptor.

> Not sure what you mean here. Only one thread will get the message and
> come out of the mq_receive(). The other threads will remain blocked in
> mq_receive(). (I don't think that more than one thread is blocked on
> the same socket descriptor because the first client does get a full
> response, inspite of the bazillions of messages.)

Only one thread will get *each* message, but there's no limit to the
number of messages per socket the main thread can send (can you show me any
such limit?). So at some point it's entirely possible that every thread is
blocked on one descriptor even though the very first thread got all the
data.

> > I've been told to put them into nonblocking mode, but am not sure

> > > _why_ that will work. (BTW, these are plain vanilla SOCK_STREAM
> > > sockets on a SPARC Blade 1000 running Solaris 8, using gcc, compiled
> > > with -lpthread, -lsocket -lnsl).

> > Because you don't want to block and you can't assure that you don't
> > block without non-blocking sockets. However, you seem to want to do

blocking

> > writes, which is kind of odd.

> Hmm. If a socket tests true for reading after a select(), then why
> would a subsequent read() block?

Because another thread already did the read inbetween when the main
thread saw a read hit on select and that thread managed to get the event and
call read. Consider:

1) Master thread calls select, sees socket 1 is ready for reading.
2) Master thread send message.
3) Master thread calls select, sees socket 1 is ready for reading.
4) Master thread sends message.
5) Slave thread A gets the first message.
6) Slave thread B gets the second message.
7) Slave thread A reads all the data.
8) Slave thread B blocks in read, it's lost to the pool until data is sent
on socket 1.

See? If you have 10 threads in the pool, it's possible all 10 will wind
up blocked on one socket this way.

> I do take your point on the write()
> part. In fact, from the debugging messages in my code, I note that
> there are a helluva lot of zero byte read()s.

Hopefully, the above will clarify for you why that is.

DS

[David Schwartz]

"John Galt" <johngalt__@hotmail.com> wrote in message
news:216f064.0307091258.2804f210@posting.google.co m...

> > Ouch. If it loops, it will send bazillions of messages because the
> > socket is still ready for reading!

> Yes! Man David, you're good... What's a good way around this? Can I
> put a timeout in select()? (I am guessing not; does the timeout
> indicate how long to wait _until_ something is ready?)

No, you can't. There are several ways you can do this, here are a few
good ones:

1) Don't call select until all the worker threads have finished doing
their jobs. The problem with this is that if a job might block for a long
time, then other jobs won't be able to do new I/O.

2) Remove the socket from the select set until the worker thread has
finished with it. The problem with this is that the main select thread may
still be in select when the worker finishes and the socket won't be selected
on until a new select is started.

There are variant approaches to both of these main approaches. For
example, a variant of the first approach allows a worker thread to signal
when it's done the I/O (but not necessarily finished all the processing it
might do), at this point, it's safe to put the socket back in the select set
but may not be safe to dispatch a worker thread to service it. A variant of
the second approach uses a pipe that's also in the select set -- workers
send a dummy byte on the pipe to force the main thread to start a new
'select' when they're done.

What works best depends upon your particular design constraints and what
code you've already got to work with. You should probably keep at least two
copies of each select set -- one is the one you pass to select directly and
is only touched by the thread that calls select, the other is the master
copy that file descriptors are added to and removed from, and that can be
protected by a lock so other threads can modify it. The select thread then
locks the master sets, copies them, unlocks the master sets, and passes the
copies to select.

> Also, will a client socket test true for reading all the time? Or will
> it test true only if the client actually sent something. To get more
> precise, suppose the client sends the string "hello\r\n", will the
> socket test true for reading 7 times or just one time?

It will test true for reading so long as there is something to read. It
will continue to test ready for reading until all seven bytes are read.
Detecting a hit for read has no effect on the state of the socket and its
readability. There are no events to be consume, just conditions to be
detected so long as the conditions apply.

> > Probably because *all* the threads are blocked on the other

decriptor.

> Not sure what you mean here. Only one thread will get the message and
> come out of the mq_receive(). The other threads will remain blocked in
> mq_receive(). (I don't think that more than one thread is blocked on
> the same socket descriptor because the first client does get a full
> response, inspite of the bazillions of messages.)

Only one thread will get *each* message, but there's no limit to the
number of messages per socket the main thread can send (can you show me any
such limit?). So at some point it's entirely possible that every thread is
blocked on one descriptor even though the very first thread got all the
data.

> > I've been told to put them into nonblocking mode, but am not sure

> > > _why_ that will work. (BTW, these are plain vanilla SOCK_STREAM
> > > sockets on a SPARC Blade 1000 running Solaris 8, using gcc, compiled
> > > with -lpthread, -lsocket -lnsl).

> > Because you don't want to block and you can't assure that you don't
> > block without non-blocking sockets. However, you seem to want to do

blocking

> > writes, which is kind of odd.

> Hmm. If a socket tests true for reading after a select(), then why
> would a subsequent read() block?

Because another thread already did the read inbetween when the main
thread saw a read hit on select and that thread managed to get the event and
call read. Consider:

1) Master thread calls select, sees socket 1 is ready for reading.
2) Master thread send message.
3) Master thread calls select, sees socket 1 is ready for reading.
4) Master thread sends message.
5) Slave thread A gets the first message.
6) Slave thread B gets the second message.
7) Slave thread A reads all the data.
8) Slave thread B blocks in read, it's lost to the pool until data is sent
on socket 1.

See? If you have 10 threads in the pool, it's possible all 10 will wind
up blocked on one socket this way.

> I do take your point on the write()
> part. In fact, from the debugging messages in my code, I note that
> there are a helluva lot of zero byte read()s.

Hopefully, the above will clarify for you why that is.

DS

[Casper H.S. Dik]

[email]johngalt__@hotmail.com[/email] (John Galt) writes:

>Also, will a client socket test true for reading all the time? Or will
>it test true only if the client actually sent something. To get more
>precise, suppose the client sends the string "hello\r\n", will the
>socket test true for reading 7 times or just one time?

It tests true until the information was read from the socket;
so it can test true anywhere between 1 and a gazillion times.

You cannot reliably use select in one thread and have another
thread act on that information unless you do a handoff of the
filedescriptor and have the thread give it back to you once it's
done. (E.g., by setting a busy flag on the fd and then skipping
it when you're building the select() list).

>Not sure what you mean here. Only one thread will get the message and
>come out of the mq_receive(). The other threads will remain blocked in
>mq_receive(). (I don't think that more than one thread is blocked on
>the same socket descriptor because the first client does get a full
>response, inspite of the bazillions of messages.)

What you'd really want to do, I think, is create a work queue and
use condition variables to signify that work is available.


Casper
--
Expressed in this posting are my opinions. They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

[Casper H.S. Dik]

[email]johngalt__@hotmail.com[/email] (John Galt) writes:

>Also, will a client socket test true for reading all the time? Or will
>it test true only if the client actually sent something. To get more
>precise, suppose the client sends the string "hello\r\n", will the
>socket test true for reading 7 times or just one time?

It tests true until the information was read from the socket;
so it can test true anywhere between 1 and a gazillion times.

You cannot reliably use select in one thread and have another
thread act on that information unless you do a handoff of the
filedescriptor and have the thread give it back to you once it's
done. (E.g., by setting a busy flag on the fd and then skipping
it when you're building the select() list).

>Not sure what you mean here. Only one thread will get the message and
>come out of the mq_receive(). The other threads will remain blocked in
>mq_receive(). (I don't think that more than one thread is blocked on
>the same socket descriptor because the first client does get a full
>response, inspite of the bazillions of messages.)

What you'd really want to do, I think, is create a work queue and
use condition variables to signify that work is available.


Casper
--
Expressed in this posting are my opinions. They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.