How do you kill a completly locked up thread?

Here is an article with an approach that allows to make any method call
"time-outable":
http://www.eggheadcafe.com/tutorials...-method-c.aspx
-- Peter
Site: http://www.eggheadcafe.com
UnBlog: http://petesbloggerama.blogspot.com
MetaFinder: http://www.blogmetafinder.com
"TheSilverHamme r" wrote:

Because C# has no native SSH class, I am using SharpSSH. Sometimes, for
reasons I do not know, a Connect call will totally lock up the thread and
never return. I am sure it has something to do with weirdness going on with
the server I am talking to. Anyhow, this locked up state happens once in a
while (maybe once per day) and I can't figure out how to deal with the locked
up thread.

If I issue a Thread.Abort() the exception never gets thrown in the thread
because it is locked up. This seems to be the only C# method I know of to
kill a thread. Is there some other way to kill off a thread?

A way you can simulate this yourself, is create any thread that connects to
a server where the connection takes some time, like 10 to 20 seconds. When
the thread is doing this connect (it will happen with even a simple TCP/IP
socket connect) issue a Thread.Abort() from another thread (the one that made
the Thread Object) and you will see that the ThreadAbortExce ption will NOT be
thrown until the Connect call returns.

Another way you can do this is after the connect call is finished and you
start to talk to a server, if you are on a recive data call and the server
stops sending data but never closes the connection, it will block forever.
You will once again not be able to get Thread.Abort() to kill the locked up
thread.

Is there anyone, especially a MSVP who can answer this?

"TheSilverHamme r" <Th************ *@discussions.m icrosoft.comwro te in
message news:7D******** *************** ***********@mic rosoft.com...

Because C# has no native SSH class, I am using SharpSSH. Sometimes, for
reasons I do not know, a Connect call will totally lock up the thread and
never return. I am sure it has something to do with weirdness going on
with
the server I am talking to. Anyhow, this locked up state happens once in
a
while (maybe once per day) and I can't figure out how to deal with the
locked
up thread.

If I issue a Thread.Abort() the exception never gets thrown in the thread
because it is locked up. This seems to be the only C# method I know of
to
kill a thread. Is there some other way to kill off a thread?

If you need to terminate a thread while it's running native code, especially
inside a kernel call, you have no way of knowing what state it is modifying
and keeping it coherent. You have to assume the whole process is corrupted.

The only safe way to forcibly end a failed thread like you have is to end
the process containing it.

Do you have access to the socket handle for the connection? If you shutdown
(non-gracefully by setting SO_DONTLINGER) the socket from a different thread
then that will probably cause the stuck operation to complete immediately.

Wow, these are some fast replies. Normally I can go several days without
one.

Anyway, I am using the SharpSSH class which I did not write, however, I do
have the source code and I suppose I could dig through it to find the socket
calls.

However I am not sure where all the deadlocks are happening in it so it
would be very hard to catch all the problems. In some cases I think the
connect succeeds and a lockup may occur in a receive data callback which runs
in the main thread of that instance of my object (opposed to the SSH object).
Ill look at the egg-head café solution, but I am not sure how applicable it
can be to all the instances of a lockup. For example, if an event handler
in your class has been called by another object (IE: SharpSSH asynch callback
for on data received) you can't wrap that in a method call that can time out
can you?

"Ben Voigt [C++ MVP]" wrote:

>
"TheSilverHamme r" <Th************ *@discussions.m icrosoft.comwro te in
message news:7D******** *************** ***********@mic rosoft.com...

Because C# has no native SSH class, I am using SharpSSH. Sometimes, for
reasons I do not know, a Connect call will totally lock up the thread and
never return. I am sure it has something to do with weirdness going on
with
the server I am talking to. Anyhow, this locked up state happens once in
a
while (maybe once per day) and I can't figure out how to deal with the
locked
up thread.

If I issue a Thread.Abort() the exception never gets thrown in the thread
because it is locked up. This seems to be the only C# method I know of
to
kill a thread. Is there some other way to kill off a thread?

If you need to terminate a thread while it's running native code, especially
inside a kernel call, you have no way of knowing what state it is modifying
and keeping it coherent. You have to assume the whole process is corrupted.

The only safe way to forcibly end a failed thread like you have is to end
the process containing it.

Do you have access to the socket handle for the connection? If you shutdown
(non-gracefully by setting SO_DONTLINGER) the socket from a different thread
then that will probably cause the stuck operation to complete immediately.

Here is a code snippit from an asynch callback that I am sure is one of the
causes of my thread being locked up when the SharpSSH shell object dies.

ReadDataCallbac k = new AsyncCallback(O nReadData);
shell.IO.BeginR ead(RecvBuff, 0, RecvBuff.Length ,
ReadDataCallbac k, null);

while (true == shell.ShellOpen ed)
{
// See if we have data to send
lock (SendBuffer)
{
if (0 != SendBuffer.Leng th)
{
shell.Write(Sen dBuffer);
SendBuffer = string.Empty;
}
}

Thread.Sleep(50 );
}

I am not sure how to set the ReadDataCallbac k up so that it can recover from
a hard lockup from the shell object. The method on the egghead cafe page
doesn't seem to fit this very well.

"Peter Bromberg [C# MVP]" wrote:

Here is an article with an approach that allows to make any method call
"time-outable":
http://www.eggheadcafe.com/tutorials...-method-c.aspx
-- Peter
Site: http://www.eggheadcafe.com
UnBlog: http://petesbloggerama.blogspot.com
MetaFinder: http://www.blogmetafinder.com
"TheSilverHamme r" wrote:

Because C# has no native SSH class, I am using SharpSSH. Sometimes, for
reasons I do not know, a Connect call will totally lock up the thread and
never return. I am sure it has something to do with weirdness going on with
the server I am talking to. Anyhow, this locked up state happens once in a
while (maybe once per day) and I can't figure out how to deal with the locked
up thread.

If I issue a Thread.Abort() the exception never gets thrown in the thread
because it is locked up. This seems to be the only C# method I know of to
kill a thread. Is there some other way to kill off a thread?

A way you can simulate this yourself, is create any thread that connects to
a server where the connection takes some time, like 10 to 20 seconds. When
the thread is doing this connect (it will happen with even a simple TCP/IP
socket connect) issue a Thread.Abort() from another thread (the one that made
the Thread Object) and you will see that the ThreadAbortExce ption will NOT be
thrown until the Connect call returns.

Another way you can do this is after the connect call is finished and you
start to talk to a server, if you are on a recive data call and the server
stops sending data but never closes the connection, it will block forever.
You will once again not be able to get Thread.Abort() to kill the locked up
thread.

Is there anyone, especially a MSVP who can answer this?

"Ben Voigt [C++ MVP]" wrote:

If you need to terminate a thread while it's running native code, especially
inside a kernel call, you have no way of knowing what state it is modifying
and keeping it coherent. You have to assume the whole process is corrupted.

The only safe way to forcibly end a failed thread like you have is to end
the process containing it.

Is there an unsafe way to kill it? I know it can be done, such tools like
process explorer can let me select a single thread of my app and kill it.

TheSilverHammer wrote:

Because C# has no native SSH class, I am using SharpSSH. Sometimes, for
reasons I do not know, a Connect call will totally lock up the thread and
never return. I am sure it has something to do with weirdness going on with
the server I am talking to. Anyhow, this locked up state happens once in a
while (maybe once per day) and I can't figure out how to deal with the locked
up thread.

If I issue a Thread.Abort() the exception never gets thrown in the thread
because it is locked up. This seems to be the only C# method I know of to
kill a thread. Is there some other way to kill off a thread?

Yes, the unmanaged TerminateThread (). However, this doesn't work, in that it
will kill off the thread, but leave approximately zero chance for your
application to continue running successfully. You are guaranteed to corrupt
internal state with this, especially since the CLR gets no chance to cleanly
release resources associated with that thread. Seriously, don't do this.
Your application will probably just deadlock later on the locks the
terminated thread was holding, if it doesn't just crash on corrupted state.

Also, there's no obvious way to find the thread that's blocking. For one
thing, you can kill off the thread corresponding to the Thread object, but
this is not guaranteed to be the thread doing the actual blocking I/O, it
might just be waiting on another thread. As a result, you've just leaked a
thread that's still busy blocking, and worse, the actual I/O is still in
progress, so the socket is unusable. You don't want to repeat this exercise,
as it's a good way to run out of resources fast.

A way you can simulate this yourself, is create any thread that connects to
a server where the connection takes some time, like 10 to 20 seconds. When
the thread is doing this connect (it will happen with even a simple TCP/IP
socket connect) issue a Thread.Abort() from another thread (the one that made
the Thread Object) and you will see that the ThreadAbortExce ption will NOT be
thrown until the Connect call returns.

Correct. The thread is blocking on I/O, in unmanaged code. You can't end it,
and this is more or less by design. But you shouldn't be too dismayed,
because Thread.Abort() is a bad idea for the same reasons TerminateThread ()
is. If a thread needs to end, it should be designed to have exit points
where the application state is known, and it can check a flag or issue a
wait on a user object at those points. Raising an exception in the middle of
anywhere is a good way of corrupting global state.

Another way you can do this is after the connect call is finished and you
start to talk to a server, if you are on a recive data call and the server
stops sending data but never closes the connection, it will block forever.
You will once again not be able to get Thread.Abort() to kill the locked up
thread.

Same thing.

Is there anyone, especially a MSVP who can answer this?

I'm not an MSVP but I've seen this so many times in our codebase that it's
not funny anymore. The one way to cancel pending I/O on a socket and unwedge
threads blocking on that is to close the socket from another thread and
handle the resulting exceptions. Nothing else will do, at least nothing that
can be called reliable. Of course, this means tearing down the connection,
but that's still a whole lot better than tearing down your process.

The other alternative, which is less straightforward but suits some designs
better, is to make sure that threads never issue I/O which can take
"forever". Almost every I/O call has a timeout parameter, and for those that
don't there's always asynchronous I/O and
ThreadPool.Regi sterWaitForSing leObject(). When the call returns with a
timeout, either poll, decide to wait some more or give up and close the
socket, which you can then do from the same thread that owns the socket,
simplifying error handling.

I understand it's not your code, but trust me: you'll want to rewrite it
anyway, unless you can afford restarting your application every so often.

--
J.

"Jeroen Mostert" wrote:

I'm not an MSVP but I've seen this so many times in our codebase that it's
not funny anymore. The one way to cancel pending I/O on a socket and unwedge
threads blocking on that is to close the socket from another thread and
handle the resulting exceptions. Nothing else will do, at least nothing that
can be called reliable. Of course, this means tearing down the connection,
but that's still a whole lot better than tearing down your process.

The other alternative, which is less straightforward but suits some designs
better, is to make sure that threads never issue I/O which can take
"forever". Almost every I/O call has a timeout parameter, and for those that
don't there's always asynchronous I/O and
ThreadPool.Regi sterWaitForSing leObject(). When the call returns with a
timeout, either poll, decide to wait some more or give up and close the
socket, which you can then do from the same thread that owns the socket,
simplifying error handling.

I understand it's not your code, but trust me: you'll want to rewrite it
anyway, unless you can afford restarting your application every so often.

Grrr.. Damm post thing asked me to login again and ate me post... Anyway...

The SharpSSH code base has a bunch of classes and would take a major effort
to re-write. It is clear that it is unfinished from looking at it. I am
not sure my company wants to fund me re-writing this code set.

However, the solution Peter Bromberg gave on his web site looks good except
for what appears to me to be a big hole or leak. I do not understand how C#
handles this, so maybe it is a non issue. The following is the code segment
from his web site, I hope he doesn't mind me posting it:

public ArrayList DoWorkNeedsTime out(ArrayList alin, int secondsToWait)
{

ArrayList alOut = new ArrayList();

//Create an instance of our delegate, pointing to the helper
method:

DoWorkNeedsTime outDelegate deleg = new
DoWorkNeedsTime outDelegate(DoW orkWithTimeout) ;

// Call BeginInvoke on delegate.
// Note on last two parameters of Delegate BeginInvoke Method:
// 1) callback: not used here, we can pass null
// 2) state: not used, pass an instance of object in the required
parameter location
// Invoke the delegate passing the parameters and get the
IAsyncResult object in "ar":

IAsyncResult ar = deleg.BeginInvo ke(alin, secondsToWait, null,
new object());

// if the WaitOne method times out before we get a result, it
will be false:
if (!ar.AsyncWaitH andle.WaitOne(5 000, false))
{

// handle timeout logging / notification here - Syslog,
Database, Email - whatever you need
alOut.Add("TIME D OUT!");
}

else // we didn't time out:
{
// get the result of the method call here
alOut = deleg.EndInvoke (ar);
}

return alOut;

}

What he is doing is making a delegate to call BeginInvoke with and then
using the IAsyncResult to wait for a time peroid. If the time peroid
expires, then his thread continues on. If it doesn't expire, he calls
EndInvoke(). This looks good except for the issue of dealing with a truely
locked-up thread.

BeginInvoke() uses a thread from the thread-pool right? So what happens if
that thread never returns so you can call End-Invoke? Is it gone from the
thread-pool forever? If you repeat this look 1000s of times and even if 1%
of the time you get a locked up thread, won't you run out of threads?

The only way this can work indefinitly, which it may, is if the Garbage
collector will reclaim the thread once the delegate and other related objects
are out of scope. Is this how it works?

So the basic lesson here is that a locked up thread is unrecoverable. The
only thing you can do about it is abandon the thread and move on. If you
have an application which is supposed to run persistently for days or weeks
at a time, it will have to be restarted to reclaim the resources.

In my case, unless I do major repairs on the SharpSSH class, I will have the
occasional unrecoverable threads.

This kind of stinks. I wonder if there was a way that MS could write a
thread that could be terminated safely. If you can do that with a process,
why can't you do it with a thread? Is there a way to create a process as a
thread that can be killed?

On Thu, 17 Jan 2008 10:33:00 -0800, TheSilverHammer
<Th************ *@discussions.m icrosoft.comwro te:

[...]
In my case, unless I do major repairs on the SharpSSH class, I will have
the
occasional unrecoverable threads.

Yup. One of the risks of using third-party code is that if the code sucks
(whether because it's poorly designed or just a work in progress), there's
not much you can do about it. At least in this case, it sounds like you
_could_ try to fix the library (I don't know anything about the library,
so I'm just taking that from your comments).

This kind of stinks. I wonder if there was a way that MS could write a
thread that could be terminated safely. If you can do that with a
process,
why can't you do it with a thread?

You can't really do it with a process either.

This isn't something that Microsoft can really solve. The lack of safety
has to do with what the code executing in the thread or process is doing,
and in particular the inability for someone outside the code to know for
sure what that is. It is possible to write code that, if interrupted
unexpectedly, leaves things in an indeterminate state.

If you are the one writing the code executing on a thread, there are some
situations in which you could know that aborting the code is safe. But if
you're the one writing the code, there's no need to do so. You can just
design the code correctly, so that it's abortable in a well-defined way
instead.

If you're not the one writing the code, then you don't know whether the
nature of the code is such that it's safe to abort at some arbitrary point
of execution. Thus, it's not safe to do. But there's not really any
practical way for Microsoft to change that. It's not about how the OS
manages the thread, it's about the fact that code executing in a thread
could be doing _anything_.

Pete