Deadlock in Solaris 8

[Indira]

Hi,
We have developed a "multi-threaded" application in Solaris 8 running
in E-255 with 2 processor.The application hangs after running for
sometime. I'm attaching pstack output. We are using mutex for
locking.(wondering if it is causing the deadlock)
Any help/pointers to resolve the problem would be great.

Pstack:
----------------- lwp# 46 / thread# 3 --------------------
ffffffff7d2a3be8 lwp_sema_wait (ffffffff7c20dd18)
ffffffff7c90c834 _park (ffffffff7c20dd18, ffffffff7ca22000, 0,
ffffffff7c20dc10, 1, ffffffff7ca23cc4) + 114
ffffffff7c90c500 _swtch (ffffffff7c20dc10, ffffffff7ca28150,
ffffffff7ca22000, 0, 5, 1000) + 424
ffffffff7c910c68 _reap_wait (ffffffff7ca28150, ffffffff7ca2d6e0, 0,
0, 0, 0) + 38
ffffffff7c9109b8 _reaper (ffffffff7ca23d68, ffffffff7ca28150, 0,
ffffffff7ca23d40, 0, 1000) + 38
ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
----------------- lwp# 47 / thread# 47 --------------------
ffffffff7d29c1fc msgsys (2, f02, ffffffff74e05910, 208, 2f, 0)
ffffffff7c91e800 msgrcv (0, ffffffff70a05361, 1, ffffffff7ca23c80, 0,
ffffffff70a05c10) + 58
ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
----------------- lwp# 48 / thread# 48 --------------------
ffffffff7d29c1fc msgsys (2, f02, ffffffff74c03910, 208, 30, 0)
ffffffff7c91e800 msgrcv (0, ffffffff70905361, 1, ffffffff7ca23c80, 0,
ffffffff70905c10) + 58
ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
----------------- lwp# 49 / thread# 49 --------------------
ffffffff7d29c1fc msgsys (2, f02, ffffffff74a01910, 208, 31, 0)
ffffffff7c91e800 msgrcv (0, ffffffff70805361, 1, ffffffff7ca23c80, 0,
ffffffff70805c10) + 58
ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
----------------- lwp# 50 / thread# 50 --------------------
ffffffff7d29c1fc msgsys (2, f02, ffffffff7440f910, 208, 32, 0)
ffffffff7c91e800 msgrcv (0, ffffffff70705361, 1, ffffffff7ca23c80, 0,
ffffffff70705c10) + 58
ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
----------------- lwp# 51 / thread# 51 --------------------
ffffffff7d29c1fc msgsys (2, f02, ffffffff7420d910, 208, 33, 0)
ffffffff7c91e800 msgrcv (0, ffffffff70605361, 1, ffffffff7ca23c80, 0,
ffffffff70605c10) + 58
ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
----------------- lwp# 52 / thread# 52 --------------------
ffffffff7d29c1fc msgsys (2, f02, ffffffff7400b910, 208, 34, 0)
ffffffff7c91e800 msgrcv (0, ffffffff70505361, 1, ffffffff7ca23c80, 0,
ffffffff70505c10) + 58
ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
----------------- lwp# 53 / thread# 53 --------------------
ffffffff7d29c1fc msgsys (2, f02, ffffffff73e09910, 208, 35, 0)
ffffffff7c91e800 msgrcv (0, ffffffff70405361, 1, ffffffff7ca23c80, 0,
ffffffff70405c10) + 58
ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
----------------- lwp# 54 / thread# 54 --------------------
ffffffff7d29c1fc msgsys (2, f02, ffffffff73c07910, 208, 36, 0)
ffffffff7c91e800 msgrcv (0, ffffffff70305361, 1, ffffffff7ca23c80, 0,
ffffffff70305c10) + 58
ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
----------------- lwp# 55 / thread# 55 --------------------
ffffffff7d29c1fc msgsys (2, f02, ffffffff73a05910, 208, 37, 0)
ffffffff7c91e800 msgrcv (0, ffffffff70205361, 1, ffffffff7ca23c80, 0,
ffffffff70205c10) + 58
ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
----------------- lwp# 56 / thread# 56 --------------------
ffffffff7d29c1fc msgsys (2, f02, ffffffff73803910, 208, 38, 0)
ffffffff7c91e800 msgrcv (0, ffffffff70105361, 1, ffffffff7ca23c80, 0,
ffffffff70105c10) + 58
ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
----------------- lwp# 57 / thread# 57 --------------------
ffffffff7d29c1fc msgsys (2, f02, ffffffff73601910, 208, 39, 0)
ffffffff7c91e800 msgrcv (0, ffffffff70005361, 1, ffffffff7ca23c80, 0,
ffffffff70005c10) + 58
ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
----------------- lwp# 58 / thread# 58 --------------------
ffffffff7d29c1fc msgsys (2, f02, ffffffff7300f940, 208, 3a, 0)
ffffffff7c91e800 msgrcv (0, ffffffff6ff05361, 1, ffffffff7ca23c80, 0,
ffffffff6ff05c10) + 58
ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
----------------- lwp# 59 / thread# 59 --------------------
ffffffff7d29c1fc msgsys (2, f02, ffffffff72e0d940, 208, 3b, 0)
ffffffff7c91e800 msgrcv (0, ffffffff6fe05361, 1, ffffffff7ca23c80, 0,
ffffffff6fe05c10) + 58
ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
----------------- lwp# 60 / thread# 60 --------------------
ffffffff7d29c1fc msgsys (2, f02, ffffffff72c0b940, 208, 3c, 0)
ffffffff7c91e800 msgrcv (0, ffffffff6fd05361, 1, ffffffff7ca23c80, 0,
ffffffff6fd05c10) + 58
ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
----------------- lwp# 61 / thread# 61 --------------------
ffffffff7d29c1fc msgsys (2, f02, ffffffff72a09940, 208, 3d, 0)
ffffffff7c91e800 msgrcv (0, ffffffff6fc05361, 1, ffffffff7ca23c80, 0,
ffffffff6fc05c10) + 58
ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40

Thanks
Indira

[Dug Birdzell]

Hi,

Here's what I'd do.. Hope it helps!

Run the application through gdb, dbx or some such, and set
a break point at some of the addresses in the pstack output.
Calls to lwp_sema_wait and _thread_start might be good.
Make sure you have built the application with debug information
so you can display source. This should give you an idea
of where the different threads are when deadlock occurs.

Other thoughts...

Are you using stl strings? These can be subtlely problematic.
I have seen cases where a function taking an stl string as an
arg was passed a character constant. The stl string's type-conversion
operator created a new string instance from the constant data.
Unfortunately the function call was inside a loop that was
hit pretty hard by a number of threads. All that unnecessary string
instantiation was causing the system heap manager to work very hard,
which killing performance due to the multiprocessor environment.
The app actually ran slower when given more processors to run on.
When the code was rewritten to avoid the string instantiation via the
function call, the performance problem went away.

Put another way, frequent heap allocations should be avoided
in a multithreaded application running on a multiprocessor system.

HTH
--Dug


In comp.programming.threads Indira <indira1005@yahoo.com> wrote:

> Hi,
> We have developed a "multi-threaded" application in Solaris 8 running
> in E-255 with 2 processor.The application hangs after running for
> sometime. I'm attaching pstack output. We are using mutex for
> locking.(wondering if it is causing the deadlock)
> Any help/pointers to resolve the problem would be great.

> Pstack:
> ----------------- lwp# 46 / thread# 3 --------------------
> ffffffff7d2a3be8 lwp_sema_wait (ffffffff7c20dd18)
> ffffffff7c90c834 _park (ffffffff7c20dd18, ffffffff7ca22000, 0,
> ffffffff7c20dc10, 1, ffffffff7ca23cc4) + 114
> ffffffff7c90c500 _swtch (ffffffff7c20dc10, ffffffff7ca28150,
> ffffffff7ca22000, 0, 5, 1000) + 424
> ffffffff7c910c68 _reap_wait (ffffffff7ca28150, ffffffff7ca2d6e0, 0,
> 0, 0, 0) + 38
> ffffffff7c9109b8 _reaper (ffffffff7ca23d68, ffffffff7ca28150, 0,
> ffffffff7ca23d40, 0, 1000) + 38
> ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
> ----------------- lwp# 47 / thread# 47 --------------------
> ffffffff7d29c1fc msgsys (2, f02, ffffffff74e05910, 208, 2f, 0)
> ffffffff7c91e800 msgrcv (0, ffffffff70a05361, 1, ffffffff7ca23c80, 0,
> ffffffff70a05c10) + 58
> ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
> ----------------- lwp# 48 / thread# 48 --------------------
> ffffffff7d29c1fc msgsys (2, f02, ffffffff74c03910, 208, 30, 0)
> ffffffff7c91e800 msgrcv (0, ffffffff70905361, 1, ffffffff7ca23c80, 0,
> ffffffff70905c10) + 58
> ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
> ----------------- lwp# 49 / thread# 49 --------------------
> ffffffff7d29c1fc msgsys (2, f02, ffffffff74a01910, 208, 31, 0)
> ffffffff7c91e800 msgrcv (0, ffffffff70805361, 1, ffffffff7ca23c80, 0,
> ffffffff70805c10) + 58
> ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
> ----------------- lwp# 50 / thread# 50 --------------------
> ffffffff7d29c1fc msgsys (2, f02, ffffffff7440f910, 208, 32, 0)
> ffffffff7c91e800 msgrcv (0, ffffffff70705361, 1, ffffffff7ca23c80, 0,
> ffffffff70705c10) + 58
> ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
> ----------------- lwp# 51 / thread# 51 --------------------
> ffffffff7d29c1fc msgsys (2, f02, ffffffff7420d910, 208, 33, 0)
> ffffffff7c91e800 msgrcv (0, ffffffff70605361, 1, ffffffff7ca23c80, 0,
> ffffffff70605c10) + 58
> ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
> ----------------- lwp# 52 / thread# 52 --------------------
> ffffffff7d29c1fc msgsys (2, f02, ffffffff7400b910, 208, 34, 0)
> ffffffff7c91e800 msgrcv (0, ffffffff70505361, 1, ffffffff7ca23c80, 0,
> ffffffff70505c10) + 58
> ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
> ----------------- lwp# 53 / thread# 53 --------------------
> ffffffff7d29c1fc msgsys (2, f02, ffffffff73e09910, 208, 35, 0)
> ffffffff7c91e800 msgrcv (0, ffffffff70405361, 1, ffffffff7ca23c80, 0,
> ffffffff70405c10) + 58
> ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
> ----------------- lwp# 54 / thread# 54 --------------------
> ffffffff7d29c1fc msgsys (2, f02, ffffffff73c07910, 208, 36, 0)
> ffffffff7c91e800 msgrcv (0, ffffffff70305361, 1, ffffffff7ca23c80, 0,
> ffffffff70305c10) + 58
> ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
> ----------------- lwp# 55 / thread# 55 --------------------
> ffffffff7d29c1fc msgsys (2, f02, ffffffff73a05910, 208, 37, 0)
> ffffffff7c91e800 msgrcv (0, ffffffff70205361, 1, ffffffff7ca23c80, 0,
> ffffffff70205c10) + 58
> ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
> ----------------- lwp# 56 / thread# 56 --------------------
> ffffffff7d29c1fc msgsys (2, f02, ffffffff73803910, 208, 38, 0)
> ffffffff7c91e800 msgrcv (0, ffffffff70105361, 1, ffffffff7ca23c80, 0,
> ffffffff70105c10) + 58
> ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
> ----------------- lwp# 57 / thread# 57 --------------------
> ffffffff7d29c1fc msgsys (2, f02, ffffffff73601910, 208, 39, 0)
> ffffffff7c91e800 msgrcv (0, ffffffff70005361, 1, ffffffff7ca23c80, 0,
> ffffffff70005c10) + 58
> ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
> ----------------- lwp# 58 / thread# 58 --------------------
> ffffffff7d29c1fc msgsys (2, f02, ffffffff7300f940, 208, 3a, 0)
> ffffffff7c91e800 msgrcv (0, ffffffff6ff05361, 1, ffffffff7ca23c80, 0,
> ffffffff6ff05c10) + 58
> ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
> ----------------- lwp# 59 / thread# 59 --------------------
> ffffffff7d29c1fc msgsys (2, f02, ffffffff72e0d940, 208, 3b, 0)
> ffffffff7c91e800 msgrcv (0, ffffffff6fe05361, 1, ffffffff7ca23c80, 0,
> ffffffff6fe05c10) + 58
> ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
> ----------------- lwp# 60 / thread# 60 --------------------
> ffffffff7d29c1fc msgsys (2, f02, ffffffff72c0b940, 208, 3c, 0)
> ffffffff7c91e800 msgrcv (0, ffffffff6fd05361, 1, ffffffff7ca23c80, 0,
> ffffffff6fd05c10) + 58
> ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40
> ----------------- lwp# 61 / thread# 61 --------------------
> ffffffff7d29c1fc msgsys (2, f02, ffffffff72a09940, 208, 3d, 0)
> ffffffff7c91e800 msgrcv (0, ffffffff6fc05361, 1, ffffffff7ca23c80, 0,
> ffffffff6fc05c10) + 58
> ffffffff7c91f364 _thread_start (0, 0, 0, 0, 0, 0) + 40

> Thanks
> Indira