[Denis Kukharev]

I have a problem when catching a plenty of exceptions simultaneously in different threads: the Load Average indicator exceeds 100.0 and the process seems to "hang up".
I managed to simulate this situation having created a test program with a number of threads about 200 each containing a FINITE loop with throwing and catching an exception. The exceptions don't leave thread function. The test starts to work and at some moment blocks. Under some other conditions the test may finish (when number of threads is lower or when intervals between throwing exceptions are bigger). It can pass even with the same conditions, so it's behavior is unpredictable.
I use boost::thread over libpthread and g++ 4.2.1 on 2-core amd64 with FreeBSD7.0. It's interesting that under Linux this test runs smoothly.

when the test application hangs, I kill it with -11 signal and that is what I see in gdb:
(gdb) where
#0 0x0000000801554636 in __fixunssfti () from /lib/libgcc_s.so.1
]#1 0x00000008005c6343 in dl_iterate_phdr () from /libexec/ld-elf.so.1
#2 0x0000000801554bdf in _Unwind_Find_FDE () from /lib/libgcc_s.so.1
#3 0x0000000801551525 in _Unwind_GetIPInfo () from /lib/libgcc_s.so.1
#4 0x000000080155296e in _Unwind_RaiseException () from /lib/libgcc_s.so.1
#5 0x00000008012a2abd in __cxa_throw () from /usr/lib/libstdc++.so.6
.....

The same is in my real application when it hangs on exceptions.

[weaknessforcats]

Probably this is a deadlock where more then one thread is waiting on a resource owned by the other thread.

How are you handling locking where common resources are used?

[Denis Kukharev]

Quote:

Originally Posted by weaknessforcats

Probably this is a deadlock where more then one thread is waiting on a resource owned by the other thread.

How are you handling locking where common resources are used?

well, test code doesn't seem to utilize any shared resource. Here is test thread routine:

Expand|Select|Wrap|Line Numbers

1. void ThreadProc()
2. {
3.         cerr << "I'm working!!!" << endl;
4.  
5.         srand(time(NULL));
6.  
7.         //Throw and catch 10 exceptions
8.         for (int i = 0;i < 10;i++)
9.         {
10.             unsigned int seedTmp;
11.             try
12.             {
13.                 //random waiting between throws
14.                 int rand = (int) (500000 * ( (double) rand_r (&seedTmp) / RAND_MAX));
15.                 usleep(rand);
16.  
17.                 //throw my exception
18.                 throw ExceptionBase ("debug!");
19.             }
20.             catch (ExceptionBase &e)
21.             {
22.                 cerr << "Exception "<<i<<" has been caught: " << e.Message() << endl;
23.             }
24.         }
25. }
26.  

[Denis Kukharev]

Hadn't nobody faced the problem? Hey, gurus, try this test case, it's not so hard to do! :) You can replace the ExceptionBase with std::exception, the result was similar for me. If there is a problem in libraries - it should be uncovered, or may be I do something incorrect? Please, help me, for the problem disturbs working of production system for the long period!

[JosAH]

Quote:

Originally Posted by Denis Kukharev

Hadn't nobody faced the problem? Hey, gurus, try this test case, it's not so hard to do! :) You can replace the ExceptionBase with std::exception, the result was similar for me. If there is a problem in libraries - it should be uncovered, or may be I do something incorrect? Please, help me, for the problem disturbs working of production system for the long period!

This is just a guess; I don't have any (free)BSD installations here; btw, it ran fine
on my Linux box. My guess is that the stack unwinding goofs somewhere when
multiple threads are involved. Can you try to throw and catch a pointer to your
exception instead? (and delete the exception in your catch clause of course).

It would be a clumsy workaround but maintaining a single pointer from one stack
context is a bit easier than maintaining an entire local exception object.

As I wrote, this is just a guess ...

kind regards,

Jos

[Denis Kukharev]

Quote:

Originally Posted by JosAH

This is just a guess; I don't have any (free)BSD installations here; btw, it ran fine
on my Linux box. My guess is that the stack unwinding goofs somewhere when
multiple threads are involved. Can you try to throw and catch a pointer to your
exception instead? (and delete the exception in your catch clause of course).

It would be a clumsy workaround but maintaining a single pointer from one stack
context is a bit easier than maintaining an entire local exception object.

As I wrote, this is just a guess ...

kind regards,

Jos

Thank You for urgent attention! I'll try Your suggestion.
PS (As I mentioned in the initial message, on my Linux it works fine too... and that's the miracle! :) )

[JosAH]

Quote:

Originally Posted by Denis Kukharev

Thank You for urgent attention! I'll try Your suggestion.
PS (As I mentioned in the initial message, on my Linux it works fine too... and that's the miracle! :) )

Any (positive) results already?

kind regards,

Jos

[Denis Kukharev]

Quote:

Originally Posted by JosAH

Any (positive) results already?

kind regards,

Jos

Unfortunately, no... The result is the same.

Expand|Select|Wrap|Line Numbers

1. throw new std::exception();
2. ...
3. catch (std::exception *e)
4. {
5. ...
6. delete e;
7. }
8.  

[JosAH]

Does any of the versions work properly in one single thread? If so I guess your
standard libraries are a bit goofy. Any alternative versions available? Any known
bug-report on this particular issue?

kind regards,

Jos

[Denis Kukharev]

Quote:

Originally Posted by JosAH

Does any of the versions work properly in one single thread? If so I guess your
standard libraries are a bit goofy. Any alternative versions available? Any known
bug-report on this particular issue?

kind regards,

Jos

Yeah, 1-thread version works fine. I'll try to use other libs, but I tried the test on a number of systems under FreeBSD.
As for known bug reports - I've failed to find any...

Well, waiting for another experiments with this issue!

[JosAH]

Quote:

Originally Posted by Denis Kukharev

Yeah, 1-thread version works fine. I'll try to use other libs, but I tried the test on a number of systems under FreeBSD.
As for known bug reports - I've failed to find any...

Well, waiting for another experiments with this issue!

Sorry that I can't help you any further (I don't even run a BSD version here). Is
there a proprietary compiler flag or a special order of include files that need to
be obeyed? I remember from an old version of the pthread library that it had to
be the first one to be linked ... (if I remember well)

kind regards,

Jos

[Denis Kukharev]

Quote:

Originally Posted by JosAH

Sorry that I can't help you any further (I don't even run a BSD version here). Is
there a proprietary compiler flag or a special order of include files that need to
be obeyed? I remember from an old version of the pthread library that it had to
be the first one to be linked ... (if I remember well)

kind regards,

Jos

OK, I'll seek in that directions...
Thanx!

[JosAH]

Quote:

Originally Posted by Denis Kukharev

OK, I'll seek in that directions...
Thanx!

Best of luck; there's one more thing I'd like to mention: on my Linux box the
stack levels just before the throw and in the catch block are equal as they
should be. I wonder if that is true in your BSD installation ...

kind regards,

Jos

[Denis Kukharev]

Quote:

Originally Posted by JosAH

Best of luck; there's one more thing I'd like to mention: on my Linux box the
stack levels just before the throw and in the catch block are equal as they
should be. I wonder if that is true in your BSD installation ...

kind regards,

Jos

Kindly explain me how to display the stack levels?

[JosAH]

Quote:

Originally Posted by Denis Kukharev

Kindly explain me how to display the stack levels?

Type 'where' without the quotes and pay special attention to the first column.

kind regards,

Jos

[Denis Kukharev]

Quote:

Originally Posted by JosAH

Type 'where' without the quotes and pay special attention to the first column.

kind regards,

Jos

And gdb will display call stack (Like the one demonstrated in the initial post). But I didn't catch how to see stack levels before throw and in catch block... Does this require manipulations with breakpoints and step-by-step executing? I'm not quite familiar with gdb yet, You know. And the idea of tracking this levels is not perfectly clear so far...

[weaknessforcats]

I believe that rand uses a static variable so it can keep track of the previous random number. I expect all rand calls use the same static variable. That would be a shared resource that could cause a race condition.

It's the same reason why you can't use strok in a multi-threaded environment.

[Denis Kukharev]

Quote:

Originally Posted by weaknessforcats

I believe that rand uses a static variable so it can keep track of the previous random number. I expect all rand calls use the same static variable. That would be a shared resource that could cause a race condition.

It's the same reason why you can't use strok in a multi-threaded environment.

I use rand_r in test with local variable to avoid static buffers. Anyway, fragments in real program where exceptions "hang" don't have static or lockable resources. But all of them produce the same stack after the application being killed with sig -11 upon hanging.
And even if You were right, why on earth does this work fine under Linux?

[weaknessforcats]

I can't say. I was just oberving that a remembered random value is a shared resource. There has to be a place where the seed value is kept. All threads will use that same place unless the thread is spawned so that it has private memory for it's heap, stack and static mambers.

[Denis Kukharev]

Quote:

Originally Posted by weaknessforcats

I can't say. I was just oberving that a remembered random value is a shared resource. There has to be a place where the seed value is kept. All threads will use that same place unless the thread is spawned so that it has private memory for it's heap, stack and static mambers.

Well, explain Your guess about deadlock please. Is this the stack trace that provoked such a thought direction? Or accidental hanging of the application? Did You reproduce the test?

[JosAH]

Quote:

Originally Posted by Denis Kukharev

Well, explain Your guess about deadlock please. Is this the stack trace that provoked such a thought direction? Or accidental hanging of the application? Did You reproduce the test?

That isn't the problem; of course writing to a single static location from two
different threads yield unpredictable results w.r.t. the value stored at that
location but it can never make a thread hang up.

Just to be sure, you could skip the random value, use a fixed value and see
what happens on your BSD box ...

kind regards,

Jos

[Denis Kukharev]

Quote:

Originally Posted by JosAH

That isn't the problem; of course writing to a single static location from two
different threads yield unpredictable results w.r.t. the value stored at that
location but it can never make a thread hang up.

Just to be sure, you could skip the random value, use a fixed value and see
what happens on your BSD box ...

kind regards,

Jos

To clear it up, I've tried to get rid of random usleep() and random numbers at all. The problem remained. And I tried to leave randoms and usleep() intact having commented throwign the exception. The test ran just fine. So, the problem is likely about exceptions....

[Denis Kukharev]

Quote:

Originally Posted by Denis Kukharev

To clear it up, I've tried to get rid of random usleep() and random numbers at all. The problem remained. And I tried to leave randoms and usleep() intact having commented throwign the exception. The test ran just fine. So, the problem is likely about exceptions....

Despite my failure to find it as a known bug, there is a bugreport on this issue:

http://www.freebsd.org/cgi/query-pr.cgi?pr=123062

I' ve been pointed to it on another forum.

Will try to patch!
Thanks a lot! :)

[JosAH]

Quote:

Originally Posted by Denis Kukharev

Despite my failure to find it as a known bug, there is a bugreport on this issue:

http://www.freebsd.org/cgi/query-pr.cgi?pr=123062

I' ve been pointed to it on another forum.

Will try to patch!
Thanks a lot! :)

Be sure to apply the correct patch because they don't seem to agree on the
correctness of the patches themselves as well ;-)

kind regards,

Jos

[Denis Kukharev]

Quote:

Originally Posted by JosAH

Be sure to apply the correct patch because they don't seem to agree on the
correctness of the patches themselves as well ;-)

kind regards,

Jos

The problem has vanished after applying the patch!

[JosAH]

Quote:

Originally Posted by Denis Kukharev

The problem has vanished after applying the patch!

Congrats; I sincerely hate it when the bug is to be found in the tools or libs.

kind regards,

Jos