Hello,
I wrote a program that does essentially the following:
for rule in rules:
for line in line_list:
line = my_apply(rule, line)
line_list contains the lines of some input text.
To "apply a rule" always means to
1. do some regex matches on line
2. substitute something in line
My question is: Given this "architectu re", does it make sense
to use threads? And if so, how?
Klaus 9 1426
Klaus Neuner wrote: Hello,
I wrote a program that does essentially the following:
for rule in rules: for line in line_list: line = my_apply(rule, line)
line_list contains the lines of some input text.
To "apply a rule" always means to
1. do some regex matches on line 2. substitute something in line
My question is: Given this "architectu re", does it make sense to use threads? And if so, how?
Klaus
It depends on how many items are in rules and how long my_apply() takes.
Klaus Neuner wrote: Hello,
I wrote a program that does essentially the following:
for rule in rules: for line in line_list: line = my_apply(rule, line)
line_list contains the lines of some input text.
To "apply a rule" always means to
1. do some regex matches on line 2. substitute something in line
My question is: Given this "architectu re", does it make sense to use threads? And if so, how?
The code is (based on what you give above) "CPU bound",
which means you will not see any advantage in using
threads. Threads don't magically make anything go
faster, and in fact have a certain overhead for the context
switch, so no, it makes no sense to use threads here.
-Peter
Peter Hansen wrote: Klaus Neuner wrote: Hello,
I wrote a program that does essentially the following:
for rule in rules: for line in line_list: line = my_apply(rule, line)
line_list contains the lines of some input text.
To "apply a rule" always means to
1. do some regex matches on line 2. substitute something in line
My question is: Given this "architectu re", does it make sense to use threads? And if so, how?
The code is (based on what you give above) "CPU bound", which means you will not see any advantage in using threads.
Have you ever seen machines with more than one CPU? The code above is
perfectly suited for parallelization .
Mathias
Mathias Waack wrote: Peter Hansen wrote:Klaus Neuner wrote:I wrote a program that does essentially the following:
for rule in rules: for line in line_list: line = my_apply(rule, line)
line_list contains the lines of some input text.
To "apply a rule" always means to
1. do some regex matches on line 2. substitute something in line
My question is: Given this "architectu re", does it make sense to use threads? And if so, how? The code is (based on what you give above) "CPU bound", which means you will not see any advantage in using threads.
Have you ever seen machines with more than one CPU?
Why yes, I have! And have *you* seen an implementation
of Python which will effectively use those multiple CPUs
in code like that above which runs in a single process?
And do you think it likely that the OP is dealing with
a multiple CPU situation, but managed to forget to
mention it? I didn't think it likely, which is why when
I considered the multi-CPU situation I discarded the
idea. Perhaps, however, I was too quick to judge...
The code above is perfectly suited for parallelization .
Yes, it is. Once you take into account implementation
issues (e.g. the GIL), would you still think so?
-Peter
Peter Hansen wrote: Mathias Waack wrote: Have you ever seen machines with more than one CPU?
Why yes, I have! And have *you* seen an implementation of Python which will effectively use those multiple CPUs in code like that above which runs in a single process?
Ok, I've lost: I haven't seen such implementation and don't know much
about the thread-layer of Python. The code above is perfectly suited for parallelization .
Yes, it is. Once you take into account implementation issues (e.g. the GIL), would you still think so?
Depends. Which means: don't know. If I would start thinking about
creating threads to gain a speedup, I would even think about
switching to another programming language.
Mathias
Mathias Waack wrote: I haven't seen such implementation and don't know much about the thread-layer of Python.
Unfortunately, there is something called the Global
Interpreter Lock (GIL), which means that even though
native threads are (generally) used for Python threads,
only one of those threads can be active in the interpreter
at any time, even if there are multiple CPUs present.
Depends. Which means: don't know. If I would start thinking about creating threads to gain a speedup, I would even think about switching to another programming language.
I believe some work has been done in this area to make
Python take advantage of multiple CPU systems, but
I believe your approach (switch languages) is still one of
the best options. Another is to arrange your application
to run as multiple processes, but this isn't quite as
simple as just using multiple threads.
-Peter
"Peter Hansen" <pe***@engcorp. com> wrote in message
news:Kv******** ************@po wergate.ca... The code above is perfectly suited for parallelization .
Yes, it is. Once you take into account implementation issues (e.g. the GIL), would you still think so?
Orrr .... you could cheat and use f.ex. PYRO to disguise a set of *Python
Applications* as Threads; The Applications could then be distributed however
which way you want ;-)
Peter Hansen wrote: Mathias Waack wrote: Depends. Which means: don't know. If I would start thinking about creating threads to gain a speedup, I would even think about switching to another programming language.
I believe some work has been done in this area to make Python take advantage of multiple CPU systems, but I believe your approach (switch languages) is still one of the best options. Another is to arrange your application to run as multiple processes, but this isn't quite as simple as just using multiple threads.
The java people have done a lot to speed up java threads. Without any
real success (just my opinion) - java programs are just slow. There
are classes of problems which can be easily solved using python, and
there are problems not very well suited for pythonic solutions.
Thats a fact and nobody should waste her time to force python into
the wrong direction.
And I think its fair to let other languages live. We should be fair
winners;)
Mathias
Mathias Waack <M.*****@gmx.de > wrote:
... Have you ever seen machines with more than one CPU?
Why yes, I have! And have *you* seen an implementation of Python which will effectively use those multiple CPUs in code like that above which runs in a single process?
Ok, I've lost: I haven't seen such implementation and don't know much about the thread-layer of Python.
There are, as far as I know, three complete implementations of Python
(plus several add-on bits and pieces and unfinished ones): CPython,
Jython, and IronPython. CPython uses its own dedicated virtual machine,
and its threads are subject to a global per-interpreter lock.
However, in lieu of dedicated virtual machines, Jython relies on the
JVM, and IronPython relies on Microsoft CLR's, and I believe both of
those VMs have no global interpreter lock. I have no multi-CPU machine
at hand that can run Microsoft's CLR, but I do have a Powermac with two
CPUs, MacOSX 10.3.5, and a JVM (1.4.2 is the latest one, I believe).
So, if you can suggest a test to show whether Jython there can in fact
effectively use both CPU's, I'll be glad to run it and let everybody
know (I'm a bit rusty on recent Java VMs, so I don't know if I need any
special incantations to tell them to run on many CPUs, or what). I'm
not sure IronPython runs fully on Mono, and neither am I sure the
current release of Mono on MacOSX is able to use multiple CPUs for
threading, but if somebody can find out and suggest a definitive test on
the matter, again I'll be glad to run it and report to the list.
Net of such niggling issues, one might say that _most_ (hey, 2 out of 3,
right?-) current complete implementations of Python can do "free
threading" with no global per-interpreter lock, and thus in theory
should be able to use multiple CPUs productively in multiple CPU-bound
threads of a sinble process -- assuming, say, Java or C# can do so, I
see no reason, in principle, why Python shouldn't be able to, when it
runs on the same underlying VM as Java or C# respectively. The code above is perfectly suited for parallelization .
Yes, it is. Once you take into account implementation issues (e.g. the GIL), would you still think so?
Depends. Which means: don't know. If I would start thinking about creating threads to gain a speedup, I would even think about switching to another programming language.
....or another implementation of Python, if you're currently using
CPython and some limitation in it is a big problem for you...
Alex This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: Przemys³aw Ró¿ycki |
last post by:
Hello,
I have written some code, which creates many threads for each connection
('main connection'). The purpose of this code is to balance the load
between several connections ('pipes'). The number of spawned threads
depends on how many pipes I create (= 2*n+2, where n is the number of
pipes).
For good results I'll presumably share main connection's load between 10
pipes - therefore 22 threads will be spawned. Now if about 50
|
by: Evan David Light |
last post by:
After agonizing over this problem for a few days, I've decided to seek
help. No, not the variety that involes a jacket that zips up the back
but this august body of intrepid individuals.
I've discovered the following:
- My compile line should resemble:
CC -D_PTHREADS -D_POSIX_C_SOURCE=199506L -LANG:std -n32 -mips3 -c
<src> -o <obj>
- My link line should resemble:
CC -LANG:std -n32 -mips3 <objs> -o <executable> -lpthread
|
by: m |
last post by:
Hello,
I have an application that processes thousands of files each day. The
filenames and various related file information is retrieved, related
filenames are associate and placed in a linked list within a single object,
which is then placed on a stack(This cuts down thread creation and deletions
roughly by a factor of 4). I create up to 12 threads, which then process a
single object off of the stack. I use a loop with a boolean...
|
by: Lucas Tam |
last post by:
Is there a document online which details the advantages of using a
Threadpool?
My application uses a user configuration amount of threads which does the
following:
Main Thread Gives Work to Sub Thread
Sub Thread Import/Export Record
Sub Thread Import/Export Record
|
by: Mrinal Kamboj |
last post by:
Hi ,
I had a doubt regarding a piece of code with me , that has to do with
System.Threading.Thread class .
In it user instantiates an array of Thread class and to all of them
assign a method for execution using ThreadStart Delegate say "worker"
method .
Now in the worker method there's a piece of code :
| |
by: mrkbrndck |
last post by:
Please see the code below as I am trying to use multithreading for copying
files to a new location in a way that improves performance of the client
windows application.
The problem occurs when 2 or more threads are created, the ImportOneFile
method attempts to add a previously added file. If I allow 4 maximum threads
and process 4 files, the last file is attempted 4 times and none of the other
files are added to the destination. If I...
|
by: gsimmons |
last post by:
I've been researching multi-threaded WinForms apps and thread
synchronization stuff for a couple days since I'm working on
refactoring a multi-threaded GUI app at work and want to be sure it's
rock solid/thread-safe. I've seen all the posts about using BeginInvoke
to have worker threads interact with the UI. My question is this:
I created a plain old Windows Form application (VS.NET 2005) with a
blank form, built it (release build), ran...
|
by: Ricardo Vazquez |
last post by:
Hello everybody,
I'm programming a TCP server.
During the stress tests I noticed that maybe my socket-receiving thread
became deaf after an hour of heavy stress.
I think that the reason could be this: The processing of the received
messages. Something goes wrong in this processing and the code gets stuck
here. As the processing is within the receiving "while" loop, the loop gets
also stuck, so that we will never reach again the...
|
by: =?ISO-8859-1?Q?Nordl=F6w?= |
last post by:
I am currently designing a synchronized queue used to communicate
between threads. Is the code given below a good solution? Am I
using mutex lock/unlock more than needed?
Are there any resources out there on the Internet on how to design
*thread-safe* *efficient* data-
structures?
/Nordlöw
|
by: Steven Blair |
last post by:
I have been watching an MSDN video on the PFX Task class and have a
question.
Here is my scenario:
TcpListener waits on incoming connections. Once a new connection is
established, a new Task will be created.
So, for my example, 8 connections have been established almost all at
the same time (8 Tasks object).
If my server is a dual core (for arguments sakes, 4 Tasks per CPU) does
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
| |
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it.
Here is my compilation command:
g++-12 -std=c++20 -Wnarrowing bit_field.cpp
Here is the code in...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
|
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
| |
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| |