signals, threads, SIGPIPE, sigpending (fun!)

[Loic Domaigne]

Hi Frank,

Pardon me for butting in a thread that I didn't follow from the
beginning... But

> > For the sake of discussion, let's say that SIGPIPE is delivered to the
> > same thread. I'll test that in a few days and report back.

AFAIK, SUSv3 states (in the rationale volume in believe) that SIGPIPE
_MAY_ be delivered to the same thread...

OTOH from SUSv3:

| The write() and [XSI] pwrite() functions shall fail if:
|
| [EPIPE]
| An attempt is made to write to a pipe or FIFO that is not open
| for reading by any process, or that only has one end open.
| A SIGPIPE signal shall also be sent to the thread.

So like you, I understand that in this case SIGPIPE should be
delivered to the thread that calls send().

> So a few days has become a few weeks. But I have results. And they
> are not good.
>
> glibc-2.1.3: SIGPIPE always goes to thread generating it
> glibc-2.3.2: depends on which threads are blocking SIGPIPE

Is it on Linux?

I have unfortunately not the time to watch at your code in details.
But two things that I noticed:

- When SIGPIPE is delivered as a process-oriented signal (i.e. the
signal is sent to the process, not to the offending thread), then I
believe this code is broken. Indeed, even with PIPESRV enabled the
signal could be delivered to the initial thread or the client as well
(I believe, you are seing this result).
[ Ok, I agree that this shouldn't be the case, if the implementation
conforms to SUS. ]

- There is a time windows between the sigpending() and sigwait()
(might not be relevant here).


Regards,
Loic.

[David Butenhof]

Loic Domaigne wrote:

>> > For the sake of discussion, let's say that SIGPIPE is delivered to the
>> > same thread. I'll test that in a few days and report back.

>
> AFAIK, SUSv3 states (in the rationale volume in believe) that SIGPIPE
> _MAY_ be delivered to the same thread...

I'm not going to search the rationale without a specific reference; but in
any case remember that POSIX/SUS rationale is not part of the standard. In
particular, an incorrect statement in rationale places no obligation or
restrictions on implementations or applications.

> OTOH from SUSv3:
>
> | The write() and [XSI] pwrite() functions shall fail if:
> |
> | [EPIPE]
> | An attempt is made to write to a pipe or FIFO that is not open
> | for reading by any process, or that only has one end open.
> | A SIGPIPE signal shall also be sent to the thread.
>
> So like you, I understand that in this case SIGPIPE should be
> delivered to the thread that calls send().

SIGPIPE is a "thread directed" signal, and therefore SHALL (must) be
delivered to the signal that initiated the action -- e.g., calling send()
on a broken pipe.

This was always the intent, but the wording was incorrect until relatively
recently. POSIX 1003.1-1996 says "sent to the process" (incorrect) while
-2001 says "sent to the thread" (correct). (We're still finding occurrences
of the word "process" that ought to have been changed long ago to
"thread".) I'm not sure right now what SUSv2 said -- but it explicitly
defers to 1996 in case of contradiction so, technically speaking, it can't
really have fixed this.

There are currently no systems carrying the UNIX 03 brand, or claiming
conformance to the (fixed) 2001 or 2003 editions of POSIX 1003.1. So while
some existing implementations did it correctly anyway, you can't (portably)
depend on correct behavior. But of course you should complain to vendors
who did it wrong, and warn them that they'll need to fix it. ;-)

--
/--------------------[ [email]David.Butenhof@hp.com[/email] ]--------------------\
| Hewlett-Packard Company Tru64 UNIX & VMS Thread Architect |
| My book: [url]http://www.awl.com/cseng/titles/0-201-63392-2/[/url] |
\----[ [url]http://homepage.mac.com/dbutenhof/Threads/Threads.html[/url] ]---/

[Loic Domaigne]

Hi Frank!

> Lest readers think that the existing replies to this thread have ended
> it (ie, satisfied my questions), I want to stress that I'm still very
> interested in some opinion on the behavior below.

> Yes. 2.2.19 for glibc-2.1.3 and 2.4.20 for glibc-2.3.2. I suppose
> the kernel thread support might make a difference (whether a userland
> thread lib decides where to send the signal v. the kernel deciding).
> Any ideas on if 2.2 v. 2.4 might be the real distinguisher here?

I think I have some ideas regarding that behaviour. But one question
first: are you by chance using RedHat v9.0 for the kernel 2.4.20 with
glibc-2.3.2?

> > With Solaris it's really strange. The client thread ALWAYS gets SIGPIPE.
> > But sometimes, another thread does as well, as best as I can tell:
> >
> > [root@brak]# gcc -o block -lpthread -lrt -lsocket -lnsl block.c
> > [root@brak]# for f in 0 1 2 3 4 5 6 7 8 9 ; do ./block ; done
> > SIGPIPE delivered to client
> > SIGPIPE delivered to client
> > SIGPIPE delivered to client
> > Broken Pipe
> > SIGPIPE delivered to client
> > SIGPIPE delivered to client
> > SIGPIPE delivered to client
> > SIGPIPE delivered to client
> > SIGPIPE delivered to client
> > SIGPIPE delivered to client
> > SIGPIPE delivered to client
> >
> > This seems really really wrong.

No, I don't think this is sooo wrong. I guess, the SIGPIPE gets
delivered to the process. As a result, any eligible thread might
receive the signal, unless you blocked it explicitely.

I believe, that's what you are seeing (e.g. if SIGPIPE is delivered to
the initial thread, then it results a "Broken Pipe").

> > Another note, without any #defines (like above), if I uncomment the
> > call to nanosleep(), SIGPIPE is always delivered twice! ie, I always
> > get both messages
> >
> > SIGPIPE delivered to client
> > Broken Pipe

This one looks strange. But, I would first fix the signal handling and
make it POSIX compliant. And then re-run your experiments. Some of
your oddities might disappear.

> > With -DPIPEMAIN, I get only the "delivered" message and a 1s pause, as
> > I would expect.

Which is IMHO logic, since with -DPIPEMAIN you are more "Posixly
correct" concerning the handling of SIGPIPE. Well, strictly speaking,
you don't need to block SIGPIPE again in the client, since the sigmask
is inherited from the initial thread. But that shouldn't hurt.

> > Doesn't this seem like a Solaris bug? Here's the code in its entirety:

Well, we shall see! But *please* fix first your code and check again!


Regards,
Loic.

[David Butenhof]

Loic Domaigne wrote:

>> > This seems really really wrong.

>
> No, I don't think this is sooo wrong. I guess, the SIGPIPE gets
> delivered to the process. As a result, any eligible thread might
> receive the signal, unless you blocked it explicitely.

Well, it's certainly wrong for SIGPIPE to be delivered to the process. I do
vaguely recall someone once saying that Solaris would sometimes treat
SIGPIPE as a "process directed" signal. I thought that was long before
Solaris 9 -- but then it's hard to keep track of releases. (Even of my own
OS, much less others!)

>> > Another note, without any #defines (like above), if I uncomment the
>> > call to nanosleep(), SIGPIPE is always delivered twice! ie, I always
>> > get both messages
>> >
>> > SIGPIPE delivered to client
>> > Broken Pipe

>
> This one looks strange. But, I would first fix the signal handling and
> make it POSIX compliant. And then re-run your experiments. Some of
> your oddities might disappear.

The code isn't strictly POSIX conforming -- but it is UNIX 98, and Solaris 9
is UNIX 98 branded. That is, specifically, POSIX lacks the sigpending()
function; but it's in the Single UNIX Specification.

>> > With -DPIPEMAIN, I get only the "delivered" message and a 1s pause, as
>> > I would expect.

>
> Which is IMHO logic, since with -DPIPEMAIN you are more "Posixly
> correct" concerning the handling of SIGPIPE. Well, strictly speaking,
> you don't need to block SIGPIPE again in the client, since the sigmask
> is inherited from the initial thread. But that shouldn't hurt.

"More POSIXly correct"? Well, that depends on how you look at it. When
dealing with a process directed signal, say SIGINT, you absolutely need at
least parts of the PIPEMAIN conditional. You should block the signal in
main before creating any threads so that all inherit it. At least, you need
to have the signal blocked in all threads before it might be generated.

However, none of that is relevant here because SIGPIPE isn't a process
directed signal. It should be quite sufficient that client_thread blocks
SIGPIPE before issuing the write() to the broken pipe. The signal must be
delivered to that thread, and should be held pending until sigwait() is
called.

The code IS correct.

At least, that is, for the intent of POSIX. As I said, the actual text of
the standard doesn't quite follow the intent in several accidental
respects, and hadn't been fixed in the version to which Solaris claims
conformance. While the behavior you see is certainly unfortunate, it may be
deliberate and isn't necessarily "illegal" or even "nonconforming".

>> > Doesn't this seem like a Solaris bug? Here's the code in its entirety:

>
> Well, we shall see! But *please* fix first your code and check again!

Granted, that if one accepts that Solaris incorrectly treats SIGPIPE as a
process directed signal, then the posted code isn't correct for Solaris. If
so, then the fact that the code follows the intent (and future wording) of
the standard is more or less irrelevant.

It's hard for an application to actually DO anything with a process directed
SIGPIPE (which is why it needs to be thread directed); but then most people
prefer to avoid SIGPIPE in favor of EPIPE return statuses anyway, and
there's a good reason for that. ;-)

--
/--------------------[ [email]David.Butenhof@hp.com[/email] ]--------------------\
| Hewlett-Packard Company Tru64 UNIX & VMS Thread Architect |
| My book: [url]http://www.awl.com/cseng/titles/0-201-63392-2/[/url] |
\----[ [url]http://homepage.mac.com/dbutenhof/Threads/Threads.html[/url] ]---/

[Frank Cusack]

Thanks Dave, Loic. Based on your feedback I'll bring this up with Sun.

Some addt'l comments inline.

On 6 Aug 2003 08:18:01 -0700 [email]loic-dev@gmx.net[/email] (Loic Domaigne) wrote:

> I think I have some ideas regarding that behaviour. But one question
> first: are you by chance using RedHat v9.0 for the kernel 2.4.20 with
> glibc-2.3.2?

Yes.

>> > With Solaris it's really strange. The client thread ALWAYS gets SIGPIPE.
>> > But sometimes, another thread does as well, as best as I can tell:
>> >
>> > [root@brak]# gcc -o block -lpthread -lrt -lsocket -lnsl block.c
>> > [root@brak]# for f in 0 1 2 3 4 5 6 7 8 9 ; do ./block ; done
>> > SIGPIPE delivered to client [0]
>> > SIGPIPE delivered to client [1]
>> > SIGPIPE delivered to client [2]
>> > Broken Pipe [2] !!!
>> > SIGPIPE delivered to client [3]
>> > SIGPIPE delivered to client [4]
>> > SIGPIPE delivered to client [5]
>> > SIGPIPE delivered to client [6]
>> > SIGPIPE delivered to client [7]
>> > SIGPIPE delivered to client [8]
>> > SIGPIPE delivered to client [9]
>> >
>> > This seems really really wrong.

>
> No, I don't think this is sooo wrong. I guess, the SIGPIPE gets
> delivered to the process. As a result, any eligible thread might
> receive the signal, unless you blocked it explicitely.
>
> I believe, that's what you are seeing (e.g. if SIGPIPE is delivered to
> the initial thread, then it results a "Broken Pipe").

Sorry, I wasn't clear about what was going on. I've added numbering
to the quote above. SIGPIPE is delivered to the client thread (first
"2" marker) *and* the initial thread ("Broken Pipe" report) ...

>> > Another note, without any #defines (like above), if I uncomment the
>> > call to nanosleep(), SIGPIPE is always delivered twice! ie, I always
>> > get both messages
>> >
>> > SIGPIPE delivered to client
>> > Broken Pipe

.... just like it does here. Above, it's intermittent. Here (with
nanosleep()) it's consistent. A misdirected signal, I can understand.
Receiving the signal twice ... I can't.

> This one looks strange. But, I would first fix the signal handling and
> make it POSIX compliant. And then re-run your experiments. Some of
> your oddities might disappear.

See Dave's comment, I've included it below.

>> > With -DPIPEMAIN, I get only the "delivered" message and a 1s pause, as
>> > I would expect.

>
> Which is IMHO logic, since with -DPIPEMAIN you are more "Posixly
> correct" concerning the handling of SIGPIPE. Well, strictly speaking,
> you don't need to block SIGPIPE again in the client, since the sigmask
> is inherited from the initial thread.

ahh .. I missed that. Thanks.

On Wed, 06 Aug 2003 12:47:58 -0400 David Butenhof <David.Butenhof@hp.com> wrote:

> The code isn't strictly POSIX conforming -- but it is UNIX 98, and Solaris 9
> is UNIX 98 branded. That is, specifically, POSIX lacks the sigpending()
> function; but it's in the Single UNIX Specification.

So many standards to choose from! argh! Is it worthwhile to care about
POSIX these days (ie, just shoot for SUS wrt portable programs)?

> The code IS correct.

Cool. That's reassuring. (and a large part of what I was looking for --
verification that it's not just me).

/fc

[Roger Faulkner]

David Butenhof <David.Butenhof@hp.com> wrote in message news:<3f2fb931@usenet01.boi.hp.com>...

>

> > OTOH from SUSv3:
> >
> > | The write() and [XSI] pwrite() functions shall fail if:
> > |
> > | [EPIPE]
> > | An attempt is made to write to a pipe or FIFO that is not open
> > | for reading by any process, or that only has one end open.
> > | A SIGPIPE signal shall also be sent to the thread.
> >
> > So like you, I understand that in this case SIGPIPE should be
> > delivered to the thread that calls send().

>
> SIGPIPE is a "thread directed" signal, and therefore SHALL (must) be
> delivered to the signal that initiated the action -- e.g., calling send()
> on a broken pipe.

David:

Perhaps this just another oversight in the SUSV3 spec, but it gives
two reasons for SIGPIPE being generated on write():

The write() and [XSI] pwrite() functions shall fail if:

[EPIPE] An attempt is made to write to a pipe or FIFO that is not
open for reading by any process, or that only has one end
open. A SIGPIPE signal shall also be sent to the thread.

The write() function shall fail if:

[EPIPE] A write was attempted on a socket that is shut down for
writing, or is no longer connected. In the latter case,
if the socket is of type SOCK_STREAM, the SIGPIPE signal
is generated to the calling process.

So, do I believe the last statement or do I assume it really meant
"to the calling thread"?

Roger Faulkner
[email]roger.faulkner@sun.com[/email]

[Loic Domaigne]

Hi Roger,

> > SIGPIPE is a "thread directed" signal, and therefore SHALL (must) be
> > delivered to the signal that initiated the action -- e.g., calling send()
> > on a broken pipe.

>
> David:
>
> Perhaps this just another oversight in the SUSV3 spec, but it gives
> two reasons for SIGPIPE being generated on write():
>
> The write() and [XSI] pwrite() functions shall fail if:
>
> [EPIPE] An attempt is made to write to a pipe or FIFO that is not
> open for reading by any process, or that only has one end
> open. A SIGPIPE signal shall also be sent to the thread.
>
> The write() function shall fail if:
>
> [EPIPE] A write was attempted on a socket that is shut down for
> writing, or is no longer connected. In the latter case,
> if the socket is of type SOCK_STREAM, the SIGPIPE signal
> is generated to the calling process.
>
> So, do I believe the last statement or do I assume it really meant
> "to the calling thread"?

Good point...

I checked out in TC1: this is exactly the same description... I
couldn't find any update ongoing regarding this statement.

Hum, looks like a bug to me (?)


Regards,
Loic.

[Frank Cusack]

On 6 Aug 2003 17:08:12 -0700 [email]loic-dev@gmx.net[/email] (Loic Domaigne) wrote:

> FC> Sorry, I wasn't clear about what was going on. I've added
> FC> numbering to the quote above. SIGPIPE is delivered to the
> FC> client thread (first "2" marker) *and* the initial thread
> FC> ("Broken Pipe" report) ...
>
> No, no... that's was perfectly clear. What I meant was:
>
> your experiment seems to indicate that SIGPIPE is sent to the process,
> and not to the thread that issued the send().

Yeah, I see what's going on here now.

I report a thread as having received a signal after calling
sigpending(), which isn't correct; the signal has not been delivered
yet. Loic, you did point this out but I didn't catch on.

So on Solaris, SIGPIPE is always being delivered to the calling thread
(in my limited test case, which is not thorough enough to be definitive)
but then that thread is sometimes scheduled out (or, blocks when I
call nanosleep()) and since Solaris isn't treating the signal as thread-
directed, it "moves" it to the to-be-scheduled thread. Which makes
sense since the signal should be delivered asap (% synchronous signals).

Linux/NPTL basically has the same bug, it's not treating SIGPIPE as
thread-directed. I'll send the maintainers a note.

Robert, can you open a bug for Solaris, or should I pursue it myself?

thanks all
/fc

[Frank Cusack]

On Thu, 07 Aug 2003 01:13:58 -0700 Frank Cusack <fcusack@fcusack.com> wrote:

> Robert, can you open a bug for Solaris, or should I pursue it myself?

err, sorry. I meant Roger.

/fc

[David Butenhof]

Roger Faulkner wrote:

> Perhaps this just another oversight in the SUSV3 spec, but it gives
> two reasons for SIGPIPE being generated on write():
>
> The write() and [XSI] pwrite() functions shall fail if:
>
> [EPIPE] An attempt is made to write to a pipe or FIFO that is not
> open for reading by any process, or that only has one end
> open. A SIGPIPE signal shall also be sent to the thread.
>
> The write() function shall fail if:
>
> [EPIPE] A write was attempted on a socket that is shut down for
> writing, or is no longer connected. In the latter case,
> if the socket is of type SOCK_STREAM, the SIGPIPE signal
> is generated to the calling process.
>
> So, do I believe the last statement or do I assume it really meant
> "to the calling thread"?

As I think I commented earlier, we're STILL finding incorrect uses of
process.

The intent of the working group is clear: any signal caused by the direct
synchronous action of a thread must be delivered to that thread, or held
pending against the thread if the signal is blocked. A signal that cannot
be attributed to the direct synchronous action of a thread must be
delivered to one and only one thread in the process, or held pending
against the process if all threads have it blocked.

SIGPIPE is generated by a write to a broken pipe; a synchronous action
clearly attributed to the direct action of a thread. It's therefore thread
directed.

BUT, "the standard says what it says", even when it's wrong. The standard
requires that a write() to a socket that is disconnected or shut down must
generate SIGPIPE to the "calling process". (Which is a self-evidently
stupid and meaningless statement since the process didn't make the write
call. ;-) ) Clearly, when someone fixed [EPIPE] to say "thread", they
missed the SECOND [EPIPE].

However, the fact that it's wrong means that it can and should change in a
future edition. Implementations should persue the winds of change and
implement correctly now. Applications of course may be in a tougher
position, if they really care... basically, the official answer would have
to be that until the standard is corrected EITHER the correct behavior or
the specified incorrect behavior is acceptable.

If only standards (or implementations, applications, or anything else) could
be perfect. ;-)

--
/--------------------[ [email]David.Butenhof@hp.com[/email] ]--------------------\
| Hewlett-Packard Company Tru64 UNIX & VMS Thread Architect |
| My book: [url]http://www.awl.com/cseng/titles/0-201-63392-2/[/url] |
\----[ [url]http://homepage.mac.com/dbutenhof/Threads/Threads.html[/url] ]---/

[David Butenhof]

Loic Domaigne wrote:

> I should really take care about my wording, espcially when - let's see
> what the backcover says, - "a recognized Pthreads authority" hangs on
> the newsgroup ;-)

On the other hand, how many of you would actually "recognize" me if you saw
me on the street? Shouldn't anyone printing a statement like that be
required to include a photograph to ensure the statement is true? ;-)

> DB> The code IS correct.
>
> Agree that the code is correct.
>
> Unfortunately, the theoritical and real world doesn't always match :(

One of my favorite quotes: "In theory, there's no difference between theory
and practice. In Practice, there's no similarity."

> FC> So many standards to choose from! argh! Is it worthwhile
> FC> to care about POSIX these days (ie, just shoot for SUS
> FC> wrt portable programs)?
>
> Actually, I ever heard that SUS is a superset of POSIX, and that
> actually any serious Un*x should in this respect follow SUS.
>
> But I guess, it makes still sense to follow POSIX these days. Indeed,
> POSIX is currently being aligned with SUS. So the next POSIX release
> will have a lot of cool new features :)

Actually, the 2001 edition of POSIX 1003.1 is the same document as SUSv3. So
these days "POSIX" (meaning, as is often the case, POSIX.1) is the same as
SUS in nearly all respects. (Except where SUSv3 is a superset.)

Implementations should strive to implement the Single UNIX Specification,
because by this strategy we can approach the ideal of, well, a "single
UNIX" interface and portable code. Applications, similarly, should be
designed to work with implementations that conform to SUS. Unfortunately,
they'll always still need to make those critical decisions about whether to
work around implementation (or even specification) bugs.

None of this, however, is the same as saying that POSIX is irrelevant. POSIX
is the basis and core of SUS.

--
/--------------------[ [email]David.Butenhof@hp.com[/email] ]--------------------\
| Hewlett-Packard Company Tru64 UNIX & VMS Thread Architect |
| My book: [url]http://www.awl.com/cseng/titles/0-201-63392-2/[/url] |
\----[ [url]http://homepage.mac.com/dbutenhof/Threads/Threads.html[/url] ]---/