473,394 Members | 1,802 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,394 software developers and data experts.

GIL, threads and scheduling - performance cost

Hi all,

Wondering if a GIL lock/unlock causes a re-schedule/contect swap when
embedding Python in a multi-threaded C/C++ app on Unix ?

If so, do I have any control or influence on this re-scheduling ?

The app suffers from serious performance degradation (compared to pure
c/C++) and high context switches that I suspect the GIL unlocking may
be aggravating ?

Thanks for any help.

Alan

Aug 29 '05 #1
3 1685
ad*******@eircom.net a écrit :
Hi all,

Wondering if a GIL lock/unlock causes a re-schedule/contect swap when
embedding Python in a multi-threaded C/C++ app on Unix ?

If so, do I have any control or influence on this re-scheduling ?

The app suffers from serious performance degradation (compared to pure
c/C++) and high context switches that I suspect the GIL unlocking may
be aggravating ?
Well, where do you observe this degradation ? When replacing part of the
C++ code by Python's code ? Or on C++ code running parallel to your
Python code ?

Because in the first case, well, this is just something natural ! Python
runtime overhead is much greater than C++ because of its dynamic nature
(it has to resolve all the symbols at runtime ...). And given the policy
for locking/releasing the GIL, I doubt it has serious performance issues
compared to the Python interpreter itself. If current performance is an
issue, consider implementing more in C/C++ ! This will be mainly true if
you currently have some heavy looping in Python. Python is very neat to
put together processor-intensive functions written in other languages,
but not to implement them. (as an exemple looh at that:
http://www.python.org/doc/essays/list2str.html )

Thanks for any help.

Alan


Pierre
Aug 29 '05 #2
Merci Pierre,

Yes I agree and plan to move more to C/C++ and releasing the GIL when
entering C/C++.

I also need to understand my original question re GIL and rescheduling.
I fear that lock/unlock too often is also causing delays due to context
switching.

BTW do you have any hints/comments on SWIG/BOOST etc to glue PY and
C/C++ ?
Alan

Aug 29 '05 #3
ad*******@eircom.net a écrit :
Merci Pierre,

Yes I agree and plan to move more to C/C++ and releasing the GIL when
entering C/C++.

I also need to understand my original question re GIL and rescheduling.
I fear that lock/unlock too often is also causing delays due to context
switching.
Well, concerning GIL and rescheduling, releasing a lock is very likely
to induce rescheduling and context switch, but it won't if the system
considers it worthless. But this is true that lock/unlock has a cost and
so, doing it too much will slow down your app.

BTW do you have any hints/comments on SWIG/BOOST etc to glue PY and
C/C++ ?
Well, for C I would recommand SWIG. For C++, I personnaly use
Boost.Python. However, you must know that SWIG did not support C++ well
when I had to choose between the two, so there was no question for me.
Then, somehow it depends on how you feel with the tools: SWIG depends on
a specific language and has its own tools while Boost is entirely
written in C++ and uses heavily templates. For performances, Boost
generate better code that SWIG, mainly because SWIG relies on Python
code that encapsulate the actual C functions while Boost does everything
in C++. Also, at least with G++, compiling Boost extension is very time-
and memory-consuming due to its heavy use of templates. As I don't use
SWIG, I don't know about the community, but at least for Boost.Python it
is quite active and questions on the mailing list are answered quickly
enough. Well, this is very quick but if you need more detailed
information, I recommend you to visit both websites.


Alan


Pierre
Aug 29 '05 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

11
by: Przemys³aw Ró¿ycki | last post by:
Hello, I have written some code, which creates many threads for each connection ('main connection'). The purpose of this code is to balance the load between several connections ('pipes'). The...
6
by: teedilo | last post by:
We have an application with a SQL Server 2000 back end that is fairly database intensive -- lots of fairly frequent queries, inserts, updates -- the gamut. The application does not make use of...
2
by: yoda | last post by:
Recently I read Charming Python: Implementing Weightless Threads (http://www-128.ibm.com/developerworks/linux/library/l-pythrd.html) by David D. I'm not an authority on threading architectures...
3
by: Muscha | last post by:
Hi, In our application we need to have a high performance scheduling framework. We want to be able to say for item 21 do an update on such and such time. We will have around 1 million items of...
6
by: Quiet Man | last post by:
Hi all, I'm designing a fairly simple service that will run on W2K/SP4 and W2K3 servers. It's job is to be a very specialized database server that listens on a given IP address / TCP port and...
35
by: Carl J. Van Arsdall | last post by:
Alright, based a on discussion on this mailing list, I've started to wonder, why use threads vs processes. So, If I have a system that has a large area of shared memory, which would be better? ...
4
by: Dinsdale | last post by:
I'm looking at adding scheduling features to an application and I wanted to ask the community about any experience with scheduling components, either open source like from code project or from a...
18
by: Jon Slaughter | last post by:
"Instead of just waiting for its time slice to expire, a thread can block each time it initiates a time-consuming activity in another thread until the activity finishes. This is better than...
167
by: darren | last post by:
Hi I have to write a multi-threaded program. I decided to take an OO approach to it. I had the idea to wrap up all of the thread functions in a mix-in class called Threadable. Then when an...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: ryjfgjl | last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
0
by: ryjfgjl | last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
0
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
0
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.