Dear Python dev community,
I'm CTO at a small software company that makes music visualization
software (you can check us out at www.soundspectrum.com). About two
years ago we went with decision to use embedded python in a couple of
our new products, given all the great things about python. We were
close to using lua but for various reasons we decided to go with
python. However, over the last two years, there's been one area of
grief that sometimes makes me think twice about our decision to go
with python...
Some background first... Our software is used for entertainment and
centers around real time, high-performance graphics, so python's
performance, embedded flexibility, and stability are the most
important issues for us. Our software targets a large cross section
of hardware and we currently ship products for Win32, OS X, and the
iPhone and since our customers are end users, our products have to be
robust, have a tidy install footprint, and be foolproof. Basically,
we use embedded python and use it to wrap our high performance C++
class set which wraps OpenGL, DirectX and our own software renderer.
In addition to wrapping our C++ frameworks, we use python to perform
various "worker" tasks on worker thread (e.g. image loading and
processing). However, we require *true* thread/interpreter
independence so python 2 has been frustrating at time, to say the
least. Please don't start with "but really, python supports multiple
interpreters" because I've been there many many times with people.
And, yes, I'm aware of the multiprocessing module added in 2.6, but
that stuff isn't lightweight and isn't suitable at all for many
environments (including ours). The bottom line is that if you want to
perform independent processing (in python) on different threads, using
the machine's multiple cores to the fullest, then you're out of luck
under python 2.
Sadly, the only way we could get truly independent interpreters was to
put python in a dynamic library, have our installer make a *duplicate*
copy of it during the installation process (e.g. python.dll/.bundle ->
python2.dll/.bundle) and load each one explicitly in our app, so we
can get truly independent interpreters. In other words, we load a
fresh dynamic lib for each thread-independent interpreter (you can't
reuse the same dynamic library because the OS will just reference the
already-loaded one).
From what I gather from the python community, the basis for not
offering "real" muti-threaded support is that it'd add to much
internal overhead--and I couldn't agree more. As a high performance C
and C++ guy, I fully agree that thread safety should be at the high
level, not at the low level. BUT, the lack of truly independent
interpreters is what ultimately prevents using python in cool,
powerful ways. This shortcoming alone has caused game developers--
both large and small--to choose other embedded interpreters over
python (e.g. Blizzard chose lua over python). For example, Apple's
QuickTime API is powerful in that high-level instance objects can
leverage performance gains associated with multi-threaded processing.
Meanwhile, the QuickTime API simply lists the responsibilitie s of the
caller regarding thread safety and that's all its needs to do. In
other words, CPython doesn't need to step in an provide a threadsafe
environment; it just needs to establish the rules and make sure that
its own implementation supports those rules.
More than once, I had actually considered expending company resources
to develop a high performance, truly independent interpreter
implementation of the python core language and modules but in the end
estimated that the size of that project would just be too much, given
our company's current resources. Should such an implementation ever
be developed, it would be very attractive for companies to support,
fund, and/or license. The truth is, we just love python as a
language, but it's lack of true interpreter independence (in a
interpreter as well as in a thread sense) remains a *huge* liability.
So, my question becomes: is python 3 ready for true multithreaded
support?? Can we finally abandon our Frankenstein approach of loading
multiple identical dynamic libs to achieve truly independent
interpreters?? I've reviewed all the new python 3 C API module stuff,
and all I have to say is: whew--better late then never!! So, although
that solves modules offering truly independent interpreter support,
the following questions remain:
- In python 3, the C module API now supports true interpreter
independence, but have all the modules in the python codebase been
converted over? Are they all now truly compliant? It will only take
a single static/global state variable in a module to potentially cause
no end of pain in a multiple interpreter environment! Yikes!
- How close is python 3 really to true multithreaded use? The
assumption here is that caller ensures safety (e.g. ensuring that
neither interpreter is in use when serializing data from one to
another).
I believe that true python independent thread/interpreter support is
paramount and should become the top priority because this is the key
consideration used by developers when they're deciding which
interpreter to embed in their app. Until there's a hello world that
demonstrates running independent python interpreters on multiple app
threads, lua will remain the clear choice over python. Python 3 needs
true interpreter independence and multi-threaded support!
Thanks,
Andy O'Meara
Oct 22 '08
114  3925 
On Oct 27, 4:05*am, "Martin v. Löwis" <mar...@v.loewi s.dewrote:
Andy O'Meara wrote:
>
Well, when you're talking about large, intricate data structures
(which include opaque OS object refs that use process-associated
allocators), even a shared memory region between the child process and
the parent can't do the job. *Otherwise, please describe in detail how
I'd get an opaque OS object (e.g. an OS ref that refers to memory-
resident video) from the child process back to the parent process.
WHAT PARENT PROCESS? "In the same address space", to me, means
"a single process only, not multiple processes, and no parent process
anywhere". If you have just multiple threads, the notion of passing
data from a "child process" back to the "parent process" is
meaningless.
I know... I was just responding to you and others here keep beating
the "fork" drum. I just trying make it clear that a shared address
space is the only way to go. Ok, good, so we're in agreement that
threads is the only way to deal with the "intricate and complex" data
set issue in a performance-centric application.
>
Again, the big picture that I'm trying to plant here is that there
really is a serious need for truly independent interpreters/contexts
in a shared address space.
I understand that this is your mission in this thread. However, why
is that your problem? Why can't you just use the existing (limited)
multiple-interpreters machinery, and solve your problems with that?
Because then we're back into the GIL not permitting threads efficient
core use on CPU bound scripts running on other threads (when they
otherwise could). Just so we're on the same page, "when they
otherwise could" is relevant here because that's the important given:
that each interpreter ("context") truly never has any context with
others.
An example would be python scripts that generate video programatically
using an initial set of params and use an in-house C module to
construct frame (which in turn make and modify python C objects that
wrap to intricate codec related data structures). Suppose you wanted
to render 3 of these at the same time, one on each thread (3
threads). With the GIL in place, these threads can't anywhere close
to their potential. Your response thus far is that the C module
should release the GIL before it commences its heavy lifting. Well,
the problem is that if during its heavy lifting it needs to call back
into its interpreter. It's turns out that this isn't an exotic case
at all: there's a *ton* of utility gained by making calls back into
the interpreter. The best example is that since code more easily
maintained in python than in C, a lot of the module "utility" code is
likely to be in python. Unsurprisingly, this is the situation myself
and many others are in: where we want to subsequently use the
interpreter within the C module (so, as I understand it, the proposal
to have the C module release the GIL unfortunately doesn't work as a
general solution).
>
For most
industry-caliber packages, the expectation and convention (unless
documented otherwise) is that the app can make as many contexts as its
wants in whatever threads it wants because the convention is that the
app is must (a) never use one context's objects in another context,
and (b) never use a context at the same time from more than one
thread. *That's all I'm really trying to look at here.
And that's indeed the case for Python, too. The app can make as many
subinterpreters as it wants to, and it must not pass objects from one
subinterpreter to another one, nor should it use a single interpreter
from more than one thread (although that is actually supported by
Python - but it surely won't hurt if you restrict yourself to a single
thread per interpreter).
I'm not following you there... I thought we're all in agreement that
the existing C modules are FAR from being reentrant, regularly making
use of static/global objects. The point I had made before is that
other industry-caliber packages specifically don't have restrictions
in *any* way.
I appreciate your arguments these a PyC concept is a lot of work with
some careful design work, but let's not kill the discussion just
because of that. The fact remains that the video encoding scenario
described above is a pretty reasonable situation, and as more people
are commenting in this thread, there's an increasing need to offer
apps more flexibility when it comes to multi-threaded use.
Andy
Glenn Linderman wrote:
So your 50% number is just a scare tactic, it would seem, based on wild
guesses. Was there really any benefit to the comment?
All I was really trying to say is that it would be a
mistake to assume that the overhead will be negligible,
as that would be just as much a wild guess as 50%.
--
Greg
On Oct 25, 9:46*am, "M.-A. Lemburg" <m...@egenix.co mwrote:
These discussion pop up every year or so and I think that most of them
are not really all that necessary, since the GIL isn't all that bad.
Thing is, if the topic keeps coming up, then that may be an indicator
that change is truly needed. Someone much wiser than me once shared
that a measure of the usefulness and quality of a package (or API) is
how easily it can be added to an application--of any flavors--without
the application needing to change.
So in the rising world of idle cores and worker threads, I do see an
increasing concern over the GIL. Although I recognize that the debate
is lengthy, heated, and has strong arguments on both sides, my reading
on the issue makes me feel like there's a bias for the pro-GIL side
because of the volume of design and coding work associated with
considering various alternatives (such as Glenn's "Py*" concepts).
And I DO respect and appreciate where the pro-GIL people come from:
who the heck wants to do all that work and recoding so that a tiny
percent of developers can benefit? And my best response is that as
unfortunate as it is, python needs to be more multi-threaded app-
friendly if we hope to attract the next generation of app developers
that want to just drop python into their app (and not have to change
their app around python). For example, Lua has that property, as
evidenced by its rapidly growing presence in commercial software
(Blizzard uses it heavily, for example).
>
Furthermore, there are lots of ways to tune the CPython VM to make
it more or less responsive to thread switches via the various sys.set*()
functions in the sys module.
Most computing or I/O intense C extensions, built-in modules and object
implementations already release the GIL for you, so it usually doesn't
get in the way all that often.
The main issue I take there is that it's often highly useful for C
modules to make subsequent calls back into the interpreter. I suppose
the response to that is to call the GIL before reentry, but it just
seems to be more code and responsibility in scenarios where it's no
necessary. Although that code and protocol may come easy to veteran
CPython developers, let's not forget that an important goal is to
attract new developers and companies to the scene, where they get
their thread-independent code up and running using python without any
unexpected reengineering. Again, why are companies choosing Lua over
Python when it comes to an easy and flexible drop-in interpreter? And
please take my points here to be exploratory, and not hostile or
accusatory, in nature.
Andy
On Oct 27, 10:55*pm, Glenn Linderman <v+pyt...@g.nev cal.comwrote:
And I think we still are miscommunicatin g! *Or maybe communicating anyway!
So when you said "object", I actually don't know whether you meant
Python object or something else. *I assumed Python object, which may not
have been correct... but read on, I think the stuff below clears it up.
Then when you mentioned thousands of objects, I imagined thousands of
Python objects, and somehow transforming the blob into same... and back
again. *
My apologies to you and others here on my use of "objects" -- I'm use
the term generically and mean it to *not* refer to python objects (for
the all the reasons discussed here). Python only makes up a small
part of our app, hence my habit of "objects" to refer to other APIs'
allocated and opaque objects (including our own and OS APIs). For all
the reasons we've discussed, in our world, python objects don't travel
around outside of our python C modules -- when python objects need to
be passed to other parts of the app, they're converted into their non-
python (portable) equivalents (ints, floats, buffers, etc--but most of
the time, the objects are PyCObjects, so they can enter and leave a
python context with negligible overhead). I venture to say this is
pretty standard when any industry app uses a package (such as python),
for various reasons:
- Portability/Future (e.g. if we do decode to drop Python and go
with Lua, the changes are limited to only one region of code).
- Sanity (having any API's objects show up in places "far away"
goes against easy-to-follow code).
- MT flexibility (because we always never use static/global
storage, we have all kinds of options when it comes to
multithreading) . For example, recall that by throwing python in
multiple dynamic libs, we were able to achieve the GIL-less
interpreter independence that we want (albeit ghetto and a pain).
Andy
On Oct 28, 9:30*am, "Andy O'Meara" <and...@gmail.c omwrote:
On Oct 25, 9:46*am, "M.-A. Lemburg" <m...@egenix.co mwrote:
These discussion pop up every year or so and I think that most of them
are not really all that necessary, since the GIL isn't all that bad.
Thing is, if the topic keeps coming up, then that may be an indicator
that change is truly needed. *Someone much wiser than me once shared
that a measure of the usefulness and quality of a package (or API) is
how easily it can be added to an application--of any flavors--without
the application needing to change.
So in the rising world of idle cores and worker threads, I do see an
increasing concern over the GIL. *Although I recognize that the debate
is lengthy, heated, and has strong arguments on both sides, my reading
on the issue makes me feel like there's a bias for the pro-GIL side
because of the volume of design and coding work associated with
considering various alternatives (such as Glenn's "Py*" concepts).
And I DO respect and appreciate where the pro-GIL people come from:
who the heck wants to do all that work and recoding so that a tiny
percent of developers can benefit? *And my best response is that as
unfortunate as it is, python needs to be more multi-threaded app-
friendly if we hope to attract the next generation of app developers
that want to just drop python into their app (and not have to change
their app around python). *For example, Lua has that property, as
evidenced by its rapidly growing presence in commercial software
(Blizzard uses it heavily, for example).
Furthermore, there are lots of ways to tune the CPython VM to make
it more or less responsive to thread switches via the various sys.set*()
functions in the sys module.
Most computing or I/O intense C extensions, built-in modules and object
implementations already release the GIL for you, so it usually doesn't
get in the way all that often.
The main issue I take there is that it's often highly useful for C
modules to make subsequent calls back into the interpreter. I suppose
the response to that is to call the GIL before reentry, but it just
seems to be more code and responsibility in scenarios where it's no
necessary. *Although that code and protocol may come easy to veteran
CPython developers, let's not forget that an important goal is to
attract new developers and companies to the scene, where they get
their thread-independent code up and running using python without any
unexpected reengineering. *Again, why are companies choosing Lua over
Python when it comes to an easy and flexible drop-in interpreter? *And
please take my points here to be exploratory, and not hostile or
accusatory, in nature.
Andy
Okay, here's the bottom line:
* This is not about the GIL. This is about *completely* isolated
interpreters; most of the time when we want to remove the GIL we want
a single interpreter with lots of shared data.
* Your use case, although not common, is not extraordinarily rare
either. It'd be nice to support.
* If CPython had supported it all along we would continue to maintain
it.
* However, since it's not supported today, it's not worth the time
invested, API incompatibility , and general breakage it would imply.
* Although it's far more work than just solving your problem, if I
were to remove the GIL I'd go all the way and allow shared objects.
So there's really only two options here:
* get a short-term bodge that works, like hacking the 3rd party
library to use your shared-memory allocator. Should be far less work
than hacking all of CPython.
* invest yourself in solving the *entire* problem (GIL removal with
shared python objects).
Because then we're back into the GIL not permitting threads efficient
core use on CPU bound scripts running on other threads (when they
otherwise could).
Why do you think so? For C code that is carefully written, the GIL
allows *very well* to write CPU bound scripts running on other threads.
(please do get back to Jesse's original remark in case you have lost
the thread :-)
An example would be python scripts that generate video programatically
using an initial set of params and use an in-house C module to
construct frame (which in turn make and modify python C objects that
wrap to intricate codec related data structures). Suppose you wanted
to render 3 of these at the same time, one on each thread (3
threads). With the GIL in place, these threads can't anywhere close
to their potential. Your response thus far is that the C module
should release the GIL before it commences its heavy lifting. Well,
the problem is that if during its heavy lifting it needs to call back
into its interpreter.
So it should reacquire the GIL then. Assuming the other threads
all do their heavy lifting, it should immediately get the GIL,
fetch some data, release the GIL, and continue to do heavy lifting.
If it's truly CPU-bound, I hope it doesn't spend most of its time
in Python API, but in true computation.
It's turns out that this isn't an exotic case
at all: there's a *ton* of utility gained by making calls back into
the interpreter. The best example is that since code more easily
maintained in python than in C, a lot of the module "utility" code is
likely to be in python.
You should really reconsider writing performance-critical code in
Python. Regardless of the issue under discussion, a lot of performance
can be gained by using "flattened" data structures, less pointer,
less reference counting, less objects, and so on - in the inner loops
of the computation. You didn't reveal what *specific* computation you
perform, so it's difficult to give specific advise.
Unsurprisingly, this is the situation myself
and many others are in: where we want to subsequently use the
interpreter within the C module (so, as I understand it, the proposal
to have the C module release the GIL unfortunately doesn't work as a
general solution).
Not if you do the actual computation in Python, no. However, this
subthread started with Jesse's remark that you *can* release the GIL
in C code.
Again, if you do heavy-lifting in Python, you should consider to rewrite
the performance-critical parts in C. You may find that the need for
multiple CPUs goes even away.
I appreciate your arguments these a PyC concept is a lot of work with
some careful design work, but let's not kill the discussion just
because of that.
Any discussion in this newsgroup is futile, except when it either
a) leads to a solution that is already possible, and the OP didn't
envision, or
b) is followed up by code contributions from one of the participants.
If neither is likely to result, killing the discussion is the most
productive thing we can do.
Regards,
Maritn
Close, I work currently for EastWest :)
Well, I actually like almost everything else about CPython,
considering my audio work the only major problem I've had is with the
GIL. I like the purist community, and I like the code, since
integrating it on both platforms has been relatively clean, and
required *zero* support. Frankly, with the exception of some windows
deployment issues relating to static linking of libpython and some
extensions, it's been a dream lib to use.
Further, I really appreciate the discussions that happen in these
lists, and I think that this particular problem is a wonderful example
of a situation that requires tons of miscellaneous opinions and input
from all angles - especially at this stage. I think that this problem
has lots of standing discussion and lots of potential solutions and/or
workarounds, and it would be cool for someone to aggregate and
paraphrase that stuff into a page to assist those thinking about doing
some patching. That's probably something that the coder would do
themselves though.
On Fri, Oct 24, 2008 at 10:25 AM, Andy O'Meara <an****@gmail.c omwrote:
>>
So we are sitting this music platform with unimaginable possibilities
in the music world (of which python does not play a role), but those
little CPU spikes caused by the GIL at low latencies won't let us have
it. AFAIK, there is no music scripting language out there that would
come close, and yet we are sooooo close! This is a big deal.
Perfectly said, Patrick. It pains me to know how widespread python
*could* be in commercial software!
Also, good points about people being longwinded and that "code talks".
Sadly, the time alone I've spend in the last couple days on this
thread is scary, but I'm committed now, I guess. :^( I look at the
length of the posts of some of these guys and I have to wonder what
the heck they do for a living!
As I mentioned, however, I'm close to just blowing the whistle on this
crap and start making CPythonES (as I call it, in the spirit of the
"ES" in "OpenGLES") . Like you, we just want the core features of
python in a clean, tidy, *reliable* fashion--something that we can
ship and not lose sleep (or support hours) over. Basically, I imagine
developing an interpreter designed for dev houses like yours and mine
(you're Ableton or Propellerhead, right?)--a python version of lua, if
you will. The nice thing about it is that is could start fresh and
small, but I have a feeling it would really catch on because every
commercial dev house would choose it over CPython any day of the week
and it would be completely disjoint form CPython.
Andy
Wow, man. Excellent post. You want a job?
The gui could use PyA threads for sure, and the audio thread could use
PyC threads. It would not be a problem to limit the audio thread to
only reentrant libraries.
This kind of thought is what I had in mind about finding a compromise,
especially in the way that PyD would not break old code assuming that
it could eventually be ported.
On Fri, Oct 24, 2008 at 11:02 AM, Glenn Linderman <v+******@g.nev cal.comwrote:
On approximately 10/24/2008 8:42 AM, came the following characters from the
keyboard of Andy O'Meara:
>>
Glenn, great post and points!
Thanks. I need to admit here that while I've got a fair bit of professional
programming experience, I'm quite new to Python -- I've not learned its
internals, nor even the full extent of its rich library. So I have some
questions that are partly about the goals of the applications being
discussed, partly about how Python is constructed, and partly about how the
library is constructed. I'm hoping to get a better understanding of all of
these; perhaps once a better understanding is achieved, limitations will be
understood, and maybe solutions be achievable.
Let me define some speculative Python interpreters; I think the first is
today's Python:
PyA: Has a GIL. PyA threads can run within a process; but are effectively
serialized to the places where the GIL is obtained/released. Needs the GIL
because that solves lots of problems with non-reentrant code (an example of
non-reentrant code, is code that uses global (C global, or C static)
variables – note that I'm not talking about Python vars declared global....
they are only module global). In this model, non-reentrant code could
include pieces of the interpreter, and/or extension modules.
PyB: No GIL. PyB threads acquire/release a lock around each reference to a
global variable (like "with" feature). Requires massive recoding of all code
that contains global variables. Reduces performance significantly by the
increased cost of obtaining and releasing locks.
PyC: No locks. Instead, recoding is done to eliminate global variables
(interpreter requires a state structure to be passed in). Extension modules
that use globals are prohibited... this eliminates large portions of the
library, or requires massive recoding. PyC threads do not share data between
threads except by explicit interfaces.
PyD: (A hybrid of PyA & PyC). The interpreter is recoded to eliminate global
variables, and each interpreter instance is provided a state structure.
There is still a GIL, however, because globals are potentially still usedby
some modules. Code is added to detect use of global variables by a module,
or some contract is written whereby a module can be declared to be reentrant
and global-free. PyA threads will obtain the GIL as they would today. PyC
threads would be available to be created. PyC instances refuse to call
non-reentrant modules, but also need not obtain the GIL... PyC threads would
have limited module support initially, but over time, most modules can be
migrated to be reentrant and global-free, so they can be used by PyC
instances. Most 3rd-party libraries today are starting to care about
reentrancy anyway, because of the popularity of threads.
The assumptions here are that:
Data-1) A Python interpreter doesn't provide any mechanism to share normal
data among threads, they are independent... but message passing works.
Data-2) A Python interpreter could be extended to provide mechanisms to
share special data, and the data would come with an implicit lock.
Data-3) A Python interpreter could be extended to provide unlocked accessto
special data, requiring the application to handle the synchronization
between threads. Data of type 2 could be used to control access to data of
type 3. This type of data could be large, or frequently referenced data, but
only by a single thread at a time, with major handoffs to a different thread
synchronized by the application in whatever way it chooses.
Context-1) A Python interpreter would know about threads it spawns, and
could pass in a block of context (in addition to the state structure) as a
parameter to a new thread. That block of context would belong to the thread
as long as it exists, and return to the spawner when the thread completes..
An embedded interpreter would also be given a block of context (in addition
to the state structure). This would allow application context to be created
and passed around. Pointers to shared memory structures, might be typical
context in the embedded case.
Context-2) Embedded Python interpreters could be spawned either as PyA
threads or PyC threads. PyC threads would be limited to modules that are
reentrant.
I think that PyB and PyC are the visions that people see, which argue
against implementing independent interpreters. PyB isn't truly independent,
because data are shared, recoding is required, and performance suffers. Ick.
PyC requires "recoding the whole library" potentially, if it is the only
solution. PyD allows access to the whole standard library of modules,
exactly like today, but the existing limitations still obtain for PyA
threads using that model – very limited concurrency. But PyC threads would
execute in their own little environments, and not need locking. Pure Python
code would be immediately happy there. Properly coded (reentrant,
global-free) extensions would be happy there. Lots of work could be done
there, to use up multi-core/multi-CPU horsepower (shared-memory
architecture).
Questions for people that know the Python internals: Is PyD possible? How
hard? Is a PyC thread an effective way of implementing a Python sandbox? If
it is, and if it would attract the attention of Brett Cannon, who at least
once wanted to do a thesis on Python sandboxes, he could be a helpful
supporter.
Questions for Andy: is the type of work you want to do in independent
threads mostly pure Python? Or with libraries that you can control to some
extent? Are those libraries reentrant? Could they be made reentrant? How
much of the Python standard library would need to be available in reentrant
mode to provide useful functionality for those threads? I think you want PyC
Questions for Patrick: So if you had a Python GUI using the whole standard
library -- would it likely runs fine in PyA threads, and still be able to
use PyC threads for the audio scripting language? Would it be a problem for
those threads to have limited library support (only reentrant modules)?
>That's the rub... In our case, we're doing image and video
manipulation--stuff not good to be messaging from address space to
address space. The same argument holds for numerical processing with
large data sets. The workers handing back huge data sets via
messaging isn't very attractive.
In the module multiprocessing environment could you not use shared memory,
then, for the large shared data items?
>Our software runs in real time (so performance is paramount),
interacts with other static libraries, depends on worker threads to
perform real-time image manipulation, and leverages Windows and Mac OS
API concepts and features. Python's performance hits have generally
been a huge challenge with our animators because they often have to go
back and massage their python code to improve execution performance.
So, in short, there are many reasons why we use python as a part
rather than a whole.
The other area of pain that I mentioned in one of my other posts is
that what we ship, above all, can't be flaky. The lack of module
cleanup (intended to be addressed by PEP 3121), using a duplicate copy
of the python dynamic lib, and namespace black magic to achieve
independent interpreters are all examples that have made using python
for us much more challenging and time-consuming then we ever
anticipated.
Again, if it turns out nothing can be done about our needs (which
appears to be more and more like the case), I think it's important for
everyone here to consider the points raised here in the last week.
Moreover, realize that the python dev community really stands to gain
from making python usable as a tool (rather than a monolith). This
fact alone has caused lua to *rapidly* rise in popularity with
software companies looking to embed a powerful, lightweight
interpreter in their software.
As a python language fan an enthusiast, don't let lua win! (I say
this endearingly of course--I have the utmost respect for both
communities and I only want to see CPython be an attractive pick when
a company is looking to embed a language that won't intrude upon their
app's design).
Thanks for the further explanations.
--
Glenn -- http://nevcal.com/
=============== ============
A protocol is complete when there is nothing left to remove.
-- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking
--
http://mail.python.org/mailman/listinfo/python-list
On Fri, Oct 24, 2008 at 12:51 PM, Andy O'Meara <an****@gmail.c omwrote:
>
Another great post, Glenn!! Very well laid-out and posed!! Thanks for
taking the time to lay all that out.
>>
Questions for Andy: is the type of work you want to do in independent
threads mostly pure Python? Or with libraries that you can control to
some extent? Are those libraries reentrant? Could they be made
reentrant? How much of the Python standard library would need to be
available in reentrant mode to provide useful functionality for those
threads? I think you want PyC
I think you've defined everything perfectly, and you're you're of
course correct about my love for for the PyC model. :^)
Like any software that's meant to be used without restrictions, our
code and frameworks always use a context object pattern so that
there's never and non-const global/shared data). I would go as far to
say that this is the case with more performance-oriented software than
you may think since it's usually a given for us to have to be parallel
friendly in as many ways as possible. Perhaps Patrick can back me up
there.
And I will.
>
As to what modules are "essential" ... As you point out, once
reentrant module implementations caught on in PyC or hybrid world, I
think we'd start to see real effort to whip them into compliance--
there's just so much to be gained imho. But to answer the question,
there's the obvious ones (operator, math, etc), string/buffer
processing (string, re), C bridge stuff (struct, array), and OS basics
(time, file system, etc). Nice-to-haves would be buffer and image
decompression (zlib, libpng, etc), crypto modules, and xml. As far as
I can imagine, I have to believe all of these modules already contain
little, if any, global data, so I have to believe they'd be super easy
to make "PyC happy". Patrick, what would you see you guys using?
We don't need anything :) Since our goal is just to use python as a
scripting language/engine to our MIDI application, all we really need
is to make calls to the api that we expose using __builtins__.
You know, the standard python library is pretty siiiiiick, but the
syntax, object model, and import mechanics of python itself is an
**equally exportable function** of the code. Funny that I'm lucky
enough to say:
"Screw the extension modules - I just want the LANGUAGE". But, I can't have it.
>
That's the rub... In our case, we're doing image and video
manipulation--stuff not good to be messaging from address space to
address space. The same argument holds for numerical processing with
large data sets. The workers handing back huge data sets via
messaging isn't very attractive.
In the module multiprocessing environment could you not use shared
memory, then, for the large shared data items?
As I understand things, the multiprocessing puts stuff in a child
process (i.e. a separate address space), so the only to get stuff to/
from it is via IPC, which can include a shared/mapped memory region.
Unfortunately, a shared address region doesn't work when you have
large and opaque objects (e.g. a rendered CoreVideo movie in the
QuickTime API or 300 megs of audio data that just went through a
DSP). Then you've got the hit of serialization if you're got
intricate data structures (that would normally would need to be
serialized, such as a hashtable or something). Also, if I may speak
for commercial developers out there who are just looking to get the
job done without new code, it's usually always preferable to just a
single high level sync object (for when the job is complete) than to
start a child processes and use IPC. The former is just WAY less
code, plain and simple.
Andy
--
http://mail.python.org/mailman/listinfo/python-list
On 28 Okt, 21:03, Rhamphoryncus <rha...@gmail.c omwrote:
* get a short-term bodge that works, like hacking the 3rd party
library to use your shared-memory allocator. *Should be far less work
than hacking all of CPython.
Did anyone come up with a reason why shared memory couldn't be used
for the purpose described by the inquirer? With the disadvantages of
serialisation circumvented, that would leave issues of contention, and
on such matters I have to say that I'm skeptical about solutions which
try and make concurrent access to CPython objects totally transparent,
mostly because it appears to be quite a lot of work to get right (as
POSH illustrates, and as your own safethread work shows), and also
because systems where contention is spread over a large "surface" (any
object can potentially be accessed by any process at any time) are
likely to incur a lot of trouble for the dubious benefit of being
vague about which objects are actually being shared.
Paul This thread has been closed and replies have been disabled. Please start a new discussion. | |
