473,836 Members | 1,410 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

How to force a thread to stop

Hi all,

Is there a way that the program that created and started a thread also stops
it.
(My usage is a time-out).

E.g.

thread = threading.Threa d(target=Loop.t estLoop)
thread.start() # This thread is expected to finish within a second
thread.join(2) # Or time.sleep(2) ?

if thread.isAlive( ):
# thread has probably encountered a problem and hangs
# What should be here to stop thread ??????

Note that I don't want to change the target (too much), as many possible
targets exist,
together thousands of lines of code.

Thanks,
Hans
Jul 22 '06
51 54901
Paul Rubin wrote:
"Carl J. Van Arsdall" <cv*********@mv ista.comwrites:
>The problem I have is a large distributed system, that's the reality
of it. The short summary, I need to use and control 100+ machines in
a computing farm. They all need to share memory or to actively
communicate with each other via some other mechanism. Without giving
any other details, that's the problem I have to solve.

Have you looked at POSH yet? http://poshmodule.sf.net

There's also an shm module that's older and maybe more reliable.
Or you might be able to just use mmap.
I'm looking at POSH, shm, and stackless right now! :-)

Thanks!

-carl

--

Carl J. Van Arsdall
cv*********@mvi sta.com
Build and Release
MontaVista Software

Jul 26 '06 #21

Carl J. Van Arsdall wrote:
br************* **********@yaho o.com wrote:
Carl J. Van Arsdall wrote:

I don't get what threading and Twisted would to do for
you. The problem you actually have is that you sometimes
need terminate these other process running other programs.
Use spawn, fork/exec* or maybe one of the popens.
I have a strong need for shared memory space in a large distributed
environment.
Distributed shared memory is a tough trick; only a few systems simulate
it.
How does spawn, fork/exec allow me to meet that need?
I have no idea why you think threads or fork/exec will give you
distributed
shared memory.
I'll look into it, but I was under the impression having shared memory
in this situation would be pretty hairy. For example, I could fork of a
50 child processes, but then I would have to setup some kind of
communication mechanism between them where the server builds up a queue
of requests from child processes and then services them in a FIFO
fashion, does that sound about right?
That much is easy. What it has to with what you say you require
remains a mystery.

Threads have little to do with what you say you need.

[...]
I feel like this is something we've established multiple times. Yes, we
want the thread to kill itself. Alright, now that we agree on that,
what is the best way to do that.
Wrong. In your examples, you want to kill other processes. You
can't run external programs such as ssh as Python threads. Ending
a Python thread has essentially nothing to do with it.
There's more going on than ssh here. Since I want to run multiple
processes to multiple devices at one time and still have mass shared
memory I need to use threads.
No, you would need to use something that implements shared
memory across multiple devices. Threads are multiple lines of
execution in the same address space.
There's a mass distributed system that
needs to be controlled, that's the problem I'm trying to solve. You can
think of each ssh as a lengthy IO process that each gets its own
device. I use the threads to allow me to do IO to multiple devices at
once, ssh just happens to be the IO. The combination of threads and ssh
allowed us to have a *primitive* distributed system (and it works too,
so I *can* run external programs in python threads).
No, you showed launching it from a Python thread using os.system().
It's not running in the thread; it's running in a separate process.
I didn't say is
was the best or the correct solution, but it works and its what I was
handed when I was thrown into this project. I'm hoping in fifteen years
or when I get an army of monkeys to fix it, it will change. I'm not
worried about killing processes, that's easy, I could kill all the sshs
or whatever else I want without batting an eye.
After launching it with os.sytem()? Can you show the code?
--
--Bryan

Jul 26 '06 #22
br************* **********@yaho o.com wrote:
Carl J. Van Arsdall wrote:
>br************* **********@yaho o.com wrote:
>>Carl J. Van Arsdall wrote:

I don't get what threading and Twisted would to do for
you. The problem you actually have is that you sometimes
need terminate these other process running other programs.
Use spawn, fork/exec* or maybe one of the popens.

I have a strong need for shared memory space in a large distributed
environment.

Distributed shared memory is a tough trick; only a few systems simulate
it.
Yea, this I understand, maybe I chose some poor words to describe what I
wanted. I think this conversation is getting hairy and confusing so I'm
going to try and paint a better picture of what's going on. Maybe this
will help you understand exactly what's going on or at least what I'm
trying to do, because I feel like we're just running in circles. After
the detailed explanation, if threads are the obvious choice or not, it
will be much easier to pick apart what I need and probably also easier
for me to see your point... so here goes... (sorry its long, but I keep
getting dinged for not being thorough enough).

So, I have a distributed build system. The system is tasked with
building a fairly complex set of packages that form a product. The
system needs to build these packages for 50 architectures using cross
compilation as well as support for 5 different hosts. Say there are
also different versions of this with tweaks for various configurations,
so in the end I might be trying to build 200+ different things at once.
I have a computing farm of 40 machines to do this for me.. That's the
high-level scenario without getting too detailed. There are also
subsystems that help us manage the machines and things, I don't want to
get into that, I'm going to try to focus on a scenario more abstract
than cluster/resource management stuff.

Alright, so manually running builds is going to be crazy and
unmanageable. So what the people who came before me did to manage this
scenario was to fork on thread per build. The threads invoke a series
of calls that look like

os.system(ssh <host<command >)

or for more complex operations they would just spawn a process that ran
another python script)

os.system(ssh <host<script> )

The purpose behind all this was for a couple things:

* The thread constantly needed information about the state of the
system (for example we don't want to end up building the same
architecture twice)
* We wanted a centralized point of control for an entire build
* We needed to be able to use as many machines as possible from a
central location.

Python threads worked very well for this. os.system behaves a lot like
many other IO operations in python and the interpreter gives up the
GIL. Each thread could run remote operations and we didn't really have
any problems. There wasn't much of a need to do fork, all it would have
done is increased the amount of memory used by the system.

Alright, so this scheme that was first put in place kind of worked.
There were some problems, for example when someone did something like

os.system(ssh <host<script> ) we had no good way of knowing what the
hell happened in the script. Now granted, they used shared files to do
some of it over nfs mounts, but I really hate that. It doesn't work
well, its clunky, and difficult to manage. There were other problems
too, but I just wanted to give a sample.

Alright, so things aren't working, I come on board, I have a boss who
wants things done immediately. What we did was created what we called a
"Python Execution Framework". The purpose of the framework was to
mitigate a number of problems we had as well as take the burden of
distribution away from the programmers by providing a few layers of
abstraction (i'm only going to focus on the distributed part of the
framework, the rest is irrelevant to the discussion). The framework
executes and threads modules (or lists of modules). Since we had
limited time, we designed the framework with "distributi on environment"
in mind but realized that if we shoot for the top right away it will
take years to get anything implemented.

Since we knew we eventually wanted a distributed system that could
execute framework modules entirely on remote machines we carefully
design and prepared the system for this. This involves some abstraction
and some simple mechanisms. However right now each ssh call will be
executed from a thread (as they will be done concurrently, just like
before). The threads still need to know about the state of the system,
but we'd also like to be able to issue some type of control that is more
event driven -- this can be sending the thread a terminate message or
sending the thread a message regarding the completion of a dependency
(we use conditions and events to do this synchronization right now). We
hoped that in the case of a catastrophic event or a user 'kill' signal
that the the system could take control of all the threads (or at least,
ask them to go away), this is what started the conversation in the first
place. We don't want to use a polling loop for these threads to check
for messages, we wanted to use something event driven (I mistakenly used
the word interrupt in earlier posts, but I think it still illustrates my
point). Its not only important that the threads die, but that they die
with grace. There's lots of cleanup work that has to be done when
things exit or things end up in an indeterminable state.

So, I feel like I have a couple options,

1) try moving everything to a process oriented configuration - we think
this would be bad, from a resource standpoint as well as it would make
things more difficult to move to a fully distributed system later, when
I get my army of code monkeys.

2) Suck it up and go straight for the distributed system now - managers
don't like this, but maybe its easier than I think its going to be, I dunno

3) See if we can find some other way of getting the threads to terminate.

4) Kill it and clean it up by hand or helper scripts - we don't want to
do this either, its one of the major things we're trying to get away from.

Alright, that's still a fairly high-level description. After all that,
if threads are still stupid then I think I'll much more easily see it
but I hope this starts to clear up confused. I don't really need a
distributed shared memory environment, but right now I do need shared
memory and it needs to be used fairly efficiently. For a fully
distributed environment I was going to see what various technologies
offered to pass data around, I figured that they must have some
mechanism for doing it or at least accessing memory from a central
location (we're setup to do this now we threads, we just need to expand
the concept to allow nodes to do it remotely). Right now, based on what
I have to do I think threads are the right choice until I can look at a
better implementation (i hear twisted is good at what I ultimately want
to do, but I don't know a thing about it).

Alright, if you read all that, thanks, and thanks for your input.
Whether or not I've agreed with anything, me and a few colleagues
definitely discuss each idea as its passed to us. For that, thanks to
the python list!

-carl
--

Carl J. Van Arsdall
cv*********@mvi sta.com
Build and Release
MontaVista Software

Jul 26 '06 #23
"Carl J. Van Arsdall" <cv*********@mv ista.comwrites:
Alright, so manually running builds is going to be crazy and
unmanageable. So what the people who came before me did to manage
this scenario was to fork on thread per build. The threads invoke a
series of calls that look like

os.system(ssh <host<command >)
Instead of using os.system, maybe you want to use one of the popens or
the subprocess module. For each ssh, you'd spawn off a process that
does the ssh and communicates back to the control process through a
set of file descriptors (Unix pipe endpoints or whatever). The
control process could use either threads or polling/select to talk to
the pipes and keep track of what the subprocesses were doing.

I don't think you need anything as complex as shared memory for this.
You're just writing a special purpose chat server.
Jul 26 '06 #24

Paul Rubin wrote:
Have you looked at POSH yet? http://poshmodule.sf.net
Paul, have you used POSH? Does it work well? Any major
gotchas?

I looked at the paper... well, not all 200+ pages, but I checked
how they handle a couple parts that I thought hard and they
seem to have good ideas. I didn't find the SourceForge project
so promising. The status is alpha, the ToDo's are a little scary,
and project looks stalled. Also it's *nix only.
--
--Bryan

Jul 26 '06 #25
On 2006-07-26 17:33:19, Carl J. Van Arsdall wrote:
Alright, if you read all that, thanks, and thanks for your input. Whether
or not I've agreed with anything, me and a few colleagues definitely
discuss each idea as its passed to us. For that, thanks to the python
list!
I think you should spend a few hours and read up on realtime OS features
and multitasking programming techniques. Get a bit away from the bottom
level, forget about the specific features of your OS and your language and
try to come up with a set of requirements and a structure that fits them.

Regarding communicating with a thread (or process, that's about the same,
only the techniques vary), for example -- there are not that many options.
Either the thread/process polls a message queue or it goes to sleep once it
has done whatever it needed to do until something comes in through a queue
or until a semaphore gets set. What is better suited for you depends on
your requirements and overall structure. Both doesn't seem to be too clear.

If you have threads that take too long and need to be killed, then I'd say
fix the code that runs there...

Gerhard

Jul 26 '06 #26
br************* **********@yaho o.com writes:
Have you looked at POSH yet? http://poshmodule.sf.net

Paul, have you used POSH? Does it work well? Any major gotchas?
I haven't used it. I've been wanting to try. I've heard it works ok
in Linux but I've heard of problems with it under Solaris.

Now that I understand what the OP is trying to do, I think POSH is
overkill, and just using pipes or sockets is fine. If he really wants
to use shared memory, hmmm, there used to be an shm module at

http://mambo.peabody.jhu.edu/omr/omi...ource/shm.html

but that site now hangs (and it's not on archive.org), and Python's
built-in mmap module doesn't support any type of locks.

I downloaded the above shm module quite a while ago, so if I can find
it I might upload it to my own site. It was a straightforward
interface to the Sys V shm calls (also *nix-only, I guess). I guess
he also could use mmap with no locks, but with separate memory regions
for reading and writing in each subprocess, using polling loops. I
sort of remember Apache's mod_mmap doing something like that if it has
to.

To really go off the deep end, there are a few different MPI libraries
with Python interfaces.
I looked at the paper... well, not all 200+ pages, but I checked
how they handle a couple parts that I thought hard and they
seem to have good ideas.
200 pages?? The paper I read was fairly short, and I looked at the
code (not too carefully) and it seemed fairly straightforward . Maybe
I missed something, or am not remembering; it's been a while.
I didn't find the SourceForge project
so promising. The status is alpha, the ToDo's are a little scary,
and project looks stalled. Also it's *nix only.
Yeah, using it for anything serious would involve being willing to fix
problems with it as they came up. But I think the delicate parts of
it are parts that aren't that important, so I'd just avoid using
those.
Jul 26 '06 #27
On 2006-07-26 19:08:44, Carl J. Van Arsdall wrote:
Also, threading's condition and event constructs are used a lot
(i talk about it somewhere in that thing I wrote). They are easy to use
and nice and ready for me, with a server wouldn't I have to have things
poll/wait for messages?
How would a thread receive a message, unless it polls some kind of queue or
waits for a message from a queue or at a semaphore? You can't just "push" a
message into a thread; the thread has to "pick it up", one way or another.

Gerhard

Jul 27 '06 #28
Gerhard Fiedler wrote:
On 2006-07-26 19:08:44, Carl J. Van Arsdall wrote:

>Also, threading's condition and event constructs are used a lot
(i talk about it somewhere in that thing I wrote). They are easy to use
and nice and ready for me, with a server wouldn't I have to have things
poll/wait for messages?

How would a thread receive a message, unless it polls some kind of queue or
waits for a message from a queue or at a semaphore? You can't just "push" a
message into a thread; the thread has to "pick it up", one way or another.

Gerhard

Well, I guess I'm thinking of an event driven mechanism, kinda like
setting up signal handlers. I don't necessarily know how it works under
the hood, but I don't poll for a signal. I setup a handler, when the
signal comes, if it comes, the handler gets thrown into action. That's
what I'd be interesting in doing with threads.

-c

--

Carl J. Van Arsdall
cv*********@mvi sta.com
Build and Release
MontaVista Software

Jul 27 '06 #29
On 2006-07-26 21:38:06, Carl J. Van Arsdall wrote:
>>Also, threading's condition and event constructs are used a lot
(i talk about it somewhere in that thing I wrote). They are easy to use
and nice and ready for me, with a server wouldn't I have to have things
poll/wait for messages?

How would a thread receive a message, unless it polls some kind of queue or
waits for a message from a queue or at a semaphore? You can't just "push" a
message into a thread; the thread has to "pick it up", one way or another.

Well, I guess I'm thinking of an event driven mechanism, kinda like
setting up signal handlers. I don't necessarily know how it works under
the hood, but I don't poll for a signal. I setup a handler, when the
signal comes, if it comes, the handler gets thrown into action. That's
what I'd be interesting in doing with threads.
What you call an event handler is a routine that gets called from a message
queue polling routine. You said a few times that you don't want that.

The queue polling routine runs in the context of the thread. If any of the
actions in that thread takes too long, it will prevent the queue polling
routine from running, and therefore the event won't get handled. This is
exactly the scenario that you seem to want to avoid. Event handlers are not
anything multitask or multithread, they are simple polling mechanisms with
an event queue. It just seems that they act preemtively, when you can click
on one button and another button becomes disabled :)

There are of course also semaphores. But they also have to either get
polled like GUI events, or the thread just goes to sleep until the
semaphore wakes it up. You need to understand this basic limitation: a
processor can only execute statements. Either it is doing other things,
then it must, by programming, check the queue -- this is polling. Or it can
suspend itself (the thread or process) and tell the OS (or the thread
handling mechanism) to wake it up when a message arrives in a queue or a
semaphore gets active.

You need to look a bit under the hood, so to speak... That's why I said in
the other message that I think it would do you some good to read up a bit
on multitasking OS programming techniques in general. There are not that
many, in principle, but it helps to understand the basics.

Gerhard

Jul 27 '06 #30

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
2421
by: Olivier Parisy | last post by:
Hi, I like to use thread to simplify the handling of independant, blocking tasks. But controling them from a main thread is not always easy to do in a clean way. So I've written some generic code whose purpose is to start and stop threads in a synchronous (blocking) way from the caller's point of view. Hence, after start() is called you are garanteed that the thread is running, and after stop() you know it has completed
13
3497
by: Andy Fish | last post by:
Hi, I am using COM interop to invoke MS word from my .net app. If word pops up a dialog box, my thread obviously hangs until the dialog is answered. Here's the problem: because the app is running on a server, there is nobody there to answer the dialog, so I have another thread that calls Abort() on the first thread in the event that the operation doesn't complete within a certain amount of time.
16
6742
by: LP | last post by:
Hi, Considering code below. Will it make GC to actually collect. One application creates new instances of a class from 3rd party assembly in a loop (it has to). That class doesn't have .Dispose or any similar method. I want to make sure GC keeps up with the loop. My reasoning if Thread.Sleep(1000) is called; GC will take priority it do its work, right? GC.Collect(); GC.WaitForPendingFinalizers(); System.Threading.Thread.Sleep(1000);
4
2566
by: fred | last post by:
I use a Synclock in a secondary thread and also stop the thread using the abort method. If the abort occurs while the thread is in the Synclock will the SyncLock always be released before the thread stops or do I need to add some extra code to avoid Synclock staying on after the thread has been stopped. Thanks Fred
4
7022
by: kaiteriteri | last post by:
I have a time-consuming VB.net application that i'd like to thread over 2 processors (that's all i've got in my machine!) and, hopefully, get it done in half the time. On running, the application should create a 2nd thread and run it on the other processor (processing a distinct set of data), leaving the current thread to run and process its set of data. But the 2nd thread must run on the free processor, otherwise there's no point... ...
6
5997
by: Joe Jax | last post by:
I have an object that spawns a worker thread to process one of its methods. That method processes methods on a collection of other objects. During this processing, a user may request to cancel the entire operation. I could request abort on the worker thread, but that is a) potentially messy, and b) not guaranteed to take immediate effect anyway. I would rather have some way of allowing the main thread to tell the worker thread that it...
5
2808
by: Alan T | last post by:
I will do several things in my thread: Copy a file to a location Update database record Read the file content Write the content to a log file If I call Thread.Abort(), it may be possible to stop at the middle of the something ?
3
3185
by: yeye.yang | last post by:
hey everybody Does everybody can help me or give me some advise for the cross thread exception catch Here is what I want to do: I have 2 classes "Scenario" and "Step", which have a System.Thread.Timer for each to control their timeout gestion, in my "main program", I start the "Scenario" and "Step", and I do something between "StepStart" and "StepStop", when "Scenario" or "Step" timeout expired, they raise an "exception", then my "main...
1
2541
by: raghudr | last post by:
Hi all, I am displaying a splash screen for which i have created a thread.Since my whole project is launched by windows service and that service will start automatically at the start of the PC and sometimes when i start or restart the PC i have observed that even though the long process is completed splash
0
9820
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
1
10592
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
10254
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
9376
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
7792
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
6979
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5650
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
1
4456
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
3
3116
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.