 | 
August 18th, 2008, 07:34 AM
| | Newbie | | Join Date: Aug 2008
Posts: 16
| |  C++ thread exceptions problem
I have a problem when catching a plenty of exceptions simultaneously in different threads: the Load Average indicator exceeds 100.0 and the process seems to "hang up".
I managed to simulate this situation having created a test program with a number of threads about 200 each containing a FINITE loop with throwing and catching an exception. The exceptions don't leave thread function. The test starts to work and at some moment blocks. Under some other conditions the test may finish (when number of threads is lower or when intervals between throwing exceptions are bigger). It can pass even with the same conditions, so it's behavior is unpredictable.
I use boost::thread over libpthread and g++ 4.2.1 on 2-core amd64 with FreeBSD7.0. It's interesting that under Linux this test runs smoothly.
when the test application hangs, I kill it with -11 signal and that is what I see in gdb:
(gdb) where
#0 0x0000000801554636 in __fixunssfti () from /lib/libgcc_s.so.1
]#1 0x00000008005c6343 in dl_iterate_phdr () from /libexec/ld-elf.so.1
#2 0x0000000801554bdf in _Unwind_Find_FDE () from /lib/libgcc_s.so.1
#3 0x0000000801551525 in _Unwind_GetIPInfo () from /lib/libgcc_s.so.1
#4 0x000000080155296e in _Unwind_RaiseException () from /lib/libgcc_s.so.1
#5 0x00000008012a2abd in __cxa_throw () from /usr/lib/libstdc++.so.6
.....
The same is in my real application when it hangs on exceptions.
 |
August 18th, 2008, 03:49 PM
| | Moderator | | Join Date: Mar 2007 Location: North Bend Washington USA Age: 68
Posts: 4,872
| |
Probably this is a deadlock where more then one thread is waiting on a resource owned by the other thread.
How are you handling locking where common resources are used?
|
August 19th, 2008, 07:24 AM
| | Newbie | | Join Date: Aug 2008
Posts: 16
| |
Quote: |
Originally Posted by weaknessforcats Probably this is a deadlock where more then one thread is waiting on a resource owned by the other thread.
How are you handling locking where common resources are used? | well, test code doesn't seem to utilize any shared resource. Here is test thread routine: -
void ThreadProc()
-
{
-
cerr << "I'm working!!!" << endl;
-
-
srand(time(NULL));
-
-
//Throw and catch 10 exceptions
-
for (int i = 0;i < 10;i++)
-
{
-
unsigned int seedTmp;
-
try
-
{
-
//random waiting between throws
-
int rand = (int) (500000 * ( (double) rand_r (&seedTmp) / RAND_MAX));
-
usleep(rand);
-
-
//throw my exception
-
throw ExceptionBase ("debug!");
-
}
-
catch (ExceptionBase &e)
-
{
-
cerr << "Exception "<<i<<" has been caught: " << e.Message() << endl;
-
}
-
}
-
}
-
|
August 26th, 2008, 08:00 AM
| | Newbie | | Join Date: Aug 2008
Posts: 16
| |
Hadn't nobody faced the problem? Hey, gurus, try this test case, it's not so hard to do! :) You can replace the ExceptionBase with std::exception, the result was similar for me. If there is a problem in libraries - it should be uncovered, or may be I do something incorrect? Please, help me, for the problem disturbs working of production system for the long period!
|
August 26th, 2008, 08:40 AM
| | Moderator | | Join Date: Mar 2007 Location: Voorschoten, the Netherlands Age: 52
Posts: 8,455
| |
Quote: |
Originally Posted by Denis Kukharev Hadn't nobody faced the problem? Hey, gurus, try this test case, it's not so hard to do! :) You can replace the ExceptionBase with std::exception, the result was similar for me. If there is a problem in libraries - it should be uncovered, or may be I do something incorrect? Please, help me, for the problem disturbs working of production system for the long period! | This is just a guess; I don't have any (free)BSD installations here; btw, it ran fine
on my Linux box. My guess is that the stack unwinding goofs somewhere when
multiple threads are involved. Can you try to throw and catch a pointer to your
exception instead? (and delete the exception in your catch clause of course).
It would be a clumsy workaround but maintaining a single pointer from one stack
context is a bit easier than maintaining an entire local exception object.
As I wrote, this is just a guess ...
kind regards,
Jos
|
August 26th, 2008, 09:25 AM
| | Newbie | | Join Date: Aug 2008
Posts: 16
| |
Quote: |
Originally Posted by JosAH This is just a guess; I don't have any (free)BSD installations here; btw, it ran fine
on my Linux box. My guess is that the stack unwinding goofs somewhere when
multiple threads are involved. Can you try to throw and catch a pointer to your
exception instead? (and delete the exception in your catch clause of course).
It would be a clumsy workaround but maintaining a single pointer from one stack
context is a bit easier than maintaining an entire local exception object.
As I wrote, this is just a guess ...
kind regards,
Jos | Thank You for urgent attention! I'll try Your suggestion.
PS (As I mentioned in the initial message, on my Linux it works fine too... and that's the miracle! :) )
|
August 26th, 2008, 10:24 AM
| | Moderator | | Join Date: Mar 2007 Location: Voorschoten, the Netherlands Age: 52
Posts: 8,455
| |
Quote: |
Originally Posted by Denis Kukharev Thank You for urgent attention! I'll try Your suggestion.
PS (As I mentioned in the initial message, on my Linux it works fine too... and that's the miracle! :) ) | Any (positive) results already?
kind regards,
Jos
|
August 26th, 2008, 12:17 PM
| | Newbie | | Join Date: Aug 2008
Posts: 16
| |
Quote: |
Originally Posted by JosAH Any (positive) results already?
kind regards,
Jos | Unfortunately, no... The result is the same. -
throw new std::exception();
-
...
-
catch (std::exception *e)
-
{
-
...
-
delete e;
-
}
-
|
August 26th, 2008, 12:29 PM
| | Moderator | | Join Date: Mar 2007 Location: Voorschoten, the Netherlands Age: 52
Posts: 8,455
| |
Does any of the versions work properly in one single thread? If so I guess your
standard libraries are a bit goofy. Any alternative versions available? Any known
bug-report on this particular issue?
kind regards,
Jos
|
August 26th, 2008, 01:03 PM
| | Newbie | | Join Date: Aug 2008
Posts: 16
| |
Quote: |
Originally Posted by JosAH Does any of the versions work properly in one single thread? If so I guess your
standard libraries are a bit goofy. Any alternative versions available? Any known
bug-report on this particular issue?
kind regards,
Jos | Yeah, 1-thread version works fine. I'll try to use other libs, but I tried the test on a number of systems under FreeBSD.
As for known bug reports - I've failed to find any...
Well, waiting for another experiments with this issue!
|
August 26th, 2008, 01:30 PM
| | Moderator | | Join Date: Mar 2007 Location: Voorschoten, the Netherlands Age: 52
Posts: 8,455
| |
Quote: |
Originally Posted by Denis Kukharev Yeah, 1-thread version works fine. I'll try to use other libs, but I tried the test on a number of systems under FreeBSD.
As for known bug reports - I've failed to find any...
Well, waiting for another experiments with this issue! | Sorry that I can't help you any further (I don't even run a BSD version here). Is
there a proprietary compiler flag or a special order of include files that need to
be obeyed? I remember from an old version of the pthread library that it had to
be the first one to be linked ... (if I remember well)
kind regards,
Jos
|
August 26th, 2008, 01:37 PM
| | Newbie | | Join Date: Aug 2008
Posts: 16
| |
Quote: |
Originally Posted by JosAH Sorry that I can't help you any further (I don't even run a BSD version here). Is
there a proprietary compiler flag or a special order of include files that need to
be obeyed? I remember from an old version of the pthread library that it had to
be the first one to be linked ... (if I remember well)
kind regards,
Jos | OK, I'll seek in that directions...
Thanx!
|
August 26th, 2008, 01:50 PM
| | Moderator | | Join Date: Mar 2007 Location: Voorschoten, the Netherlands Age: 52
Posts: 8,455
| |
Quote: |
Originally Posted by Denis Kukharev OK, I'll seek in that directions...
Thanx! | Best of luck; there's one more thing I'd like to mention: on my Linux box the
stack levels just before the throw and in the catch block are equal as they
should be. I wonder if that is true in your BSD installation ...
kind regards,
Jos
|
August 26th, 2008, 01:53 PM
| | Newbie | | Join Date: Aug 2008
Posts: 16
| |
Quote: |
Originally Posted by JosAH Best of luck; there's one more thing I'd like to mention: on my Linux box the
stack levels just before the throw and in the catch block are equal as they
should be. I wonder if that is true in your BSD installation ...
kind regards,
Jos | Kindly explain me how to display the stack levels?
|
August 26th, 2008, 02:35 PM
| | Moderator | | Join Date: Mar 2007 Location: Voorschoten, the Netherlands Age: 52
Posts: 8,455
| |
Quote: |
Originally Posted by Denis Kukharev Kindly explain me how to display the stack levels? | Type 'where' without the quotes and pay special attention to the first column.
kind regards,
Jos
|
August 26th, 2008, 06:18 PM
| | Newbie | | Join Date: Aug 2008
Posts: 16
| |
Quote: |
Originally Posted by JosAH Type 'where' without the quotes and pay special attention to the first column.
kind regards,
Jos | And gdb will display call stack (Like the one demonstrated in the initial post). But I didn't catch how to see stack levels before throw and in catch block... Does this require manipulations with breakpoints and step-by-step executing? I'm not quite familiar with gdb yet, You know. And the idea of tracking this levels is not perfectly clear so far...
|
August 26th, 2008, 07:11 PM
| | Moderator | | Join Date: Mar 2007 Location: North Bend Washington USA Age: 68
Posts: 4,872
| |
I believe that rand uses a static variable so it can keep track of the previous random number. I expect all rand calls use the same static variable. That would be a shared resource that could cause a race condition.
It's the same reason why you can't use strok in a multi-threaded environment.
|
August 26th, 2008, 07:23 PM
| | Newbie | | Join Date: Aug 2008
Posts: 16
| |
Quote: |
Originally Posted by weaknessforcats I believe that rand uses a static variable so it can keep track of the previous random number. I expect all rand calls use the same static variable. That would be a shared resource that could cause a race condition.
It's the same reason why you can't use strok in a multi-threaded environment. | I use rand_r in test with local variable to avoid static buffers. Anyway, fragments in real program where exceptions "hang" don't have static or lockable resources. But all of them produce the same stack after the application being killed with sig -11 upon hanging.
And even if You were right, why on earth does this work fine under Linux?
|
August 27th, 2008, 12:54 AM
| | Moderator | | Join Date: Mar 2007 Location: North Bend Washington USA Age: 68
Posts: 4,872
| |
I can't say. I was just oberving that a remembered random value is a shared resource. There has to be a place where the seed value is kept. All threads will use that same place unless the thread is spawned so that it has private memory for it's heap, stack and static mambers.
|
August 27th, 2008, 11:13 AM
| | Newbie | | Join Date: Aug 2008
Posts: 16
| |
Quote: |
Originally Posted by weaknessforcats I can't say. I was just oberving that a remembered random value is a shared resource. There has to be a place where the seed value is kept. All threads will use that same place unless the thread is spawned so that it has private memory for it's heap, stack and static mambers. | Well, explain Your guess about deadlock please. Is this the stack trace that provoked such a thought direction? Or accidental hanging of the application? Did You reproduce the test?
|
August 27th, 2008, 11:21 AM
| | Moderator | | Join Date: Mar 2007 Location: Voorschoten, the Netherlands Age: 52
Posts: 8,455
| |
Quote: |
Originally Posted by Denis Kukharev Well, explain Your guess about deadlock please. Is this the stack trace that provoked such a thought direction? Or accidental hanging of the application? Did You reproduce the test? | That isn't the problem; of course writing to a single static location from two
different threads yield unpredictable results w.r.t. the value stored at that
location but it can never make a thread hang up.
Just to be sure, you could skip the random value, use a fixed value and see
what happens on your BSD box ...
kind regards,
Jos
|
August 27th, 2008, 12:05 PM
| | Newbie | | Join Date: Aug 2008
Posts: 16
| |
Quote: |
Originally Posted by JosAH That isn't the problem; of course writing to a single static location from two
different threads yield unpredictable results w.r.t. the value stored at that
location but it can never make a thread hang up.
Just to be sure, you could skip the random value, use a fixed value and see
what happens on your BSD box ...
kind regards,
Jos | To clear it up, I've tried to get rid of random usleep() and random numbers at all. The problem remained. And I tried to leave randoms and usleep() intact having commented throwign the exception. The test ran just fine. So, the problem is likely about exceptions....
|
August 27th, 2008, 12:35 PM
| | Newbie | | Join Date: Aug 2008
Posts: 16
| |
Quote: |
Originally Posted by Denis Kukharev To clear it up, I've tried to get rid of random usleep() and random numbers at all. The problem remained. And I tried to leave randoms and usleep() intact having commented throwign the exception. The test ran just fine. So, the problem is likely about exceptions.... | Despite my failure to find it as a known bug, there is a bugreport on this issue: http://www.freebsd.org/cgi/query-pr.cgi?pr=123062
I' ve been pointed to it on another forum.
Will try to patch!
Thanks a lot! :)
|
August 27th, 2008, 05:26 PM
| | Moderator | | Join Date: Mar 2007 Location: Voorschoten, the Netherlands Age: 52
Posts: 8,455
| |
Quote: |
Originally Posted by Denis Kukharev | Be sure to apply the correct patch because they don't seem to agree on the
correctness of the patches themselves as well ;-)
kind regards,
Jos
|
September 1st, 2008, 11:46 AM
| | Newbie | | Join Date: Aug 2008
Posts: 16
| |
Quote: |
Originally Posted by JosAH Be sure to apply the correct patch because they don't seem to agree on the
correctness of the patches themselves as well ;-)
kind regards,
Jos | The problem has vanished after applying the patch!
|
September 1st, 2008, 12:31 PM
| | Moderator | | Join Date: Mar 2007 Location: Voorschoten, the Netherlands Age: 52
Posts: 8,455
| |
Quote: |
Originally Posted by Denis Kukharev The problem has vanished after applying the patch! | Congrats; I sincerely hate it when the bug is to be found in the tools or libs.
kind regards,
Jos
| |
Posting Rules
| You may not post new threads You may not post replies You may not post attachments You may not edit your posts
HTML code is Off | | |
| | | What is Bytes?
We are a network of experts and professionals in IT and software development that help one another with answers to tough questions and share insights.
Get the best answers to your questions from over network members.
|
