473,217 Members | 2,129 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,217 software developers and data experts.

how to write thread-safe module ? and pytz

Does someone know if the module pytz
(http://sourceforge.net/projects/pytz/) is thread-safe ?
I have not seen it explicitely stated, and just wanted to be sure, as I
want to use it.

That's because in the file pytz/tzinfo.py, I see global variables
_timedelta_cache, _datetime_cache, _ttinfo_cache, which are
dictionnaries and are used as cache.
I always thought that you must protect shared data with locks when
multithreading, but I don't see any lock anywhere in pytz.
However, pytz seems to work well with multiple threads creating various
timezone objects at the same time.
I don't understand where the trick is, that allows multiple threads to
access this module without any locking and that all this seems to work
without any problem...

Does this mean that there is a means to write a module that is
thread-safe, without importing the threading module and creating locks
?

Or have I not understood something ...?

Can someone give me a hint ?

Aug 10 '05 #1
5 2519
nicolas_riesch wrote:
Does someone know if the module pytz
(http://sourceforge.net/projects/pytz/) is thread-safe ?
On that, I don't know.
That's because in the file pytz/tzinfo.py, I see global variables
_timedelta_cache, _datetime_cache, _ttinfo_cache, which are
dictionnaries and are used as cache.
I always thought that you must protect shared data with locks when
multithreading, but I don't see any lock anywhere in pytz.
Definitely stick with what you always thought.
However, pytz seems to work well with multiple threads creating various
timezone objects at the same time.
Buggy threading can seem to work well for along time. It may be
a billion-to-one shot that a thread switch happens at just the
wrong time.
I don't understand where the trick is, that allows multiple threads to
access this module without any locking and that all this seems to work
without any problem...

Does this mean that there is a means to write a module that is
thread-safe, without importing the threading module and creating locks
?

Or have I not understood something ...?

Can someone give me a hint ?


In the current Python implementation, more things are atomic
than the language guarantees to be atomic. Programmers should
not depend on that behavior. Again, I don't know anything about
pytz, but we wouldn't bother with locks and semaphores and such
if we could make the problems go away just by ignoring them.
--
--Bryan
Aug 10 '05 #2
nicolas_riesch wrote:
Does someone know if the module pytz
(http://sourceforge.net/projects/pytz/) is thread-safe ?
I have not seen it explicitely stated, and just wanted to be sure, as I
want to use it.

That's because in the file pytz/tzinfo.py, I see global variables
_timedelta_cache, _datetime_cache, _ttinfo_cache, which are
dictionnaries and are used as cache.
I always thought that you must protect shared data with locks when
multithreading, but I don't see any lock anywhere in pytz.
Dictionaries (and probably most other Python types that are implemented
in C) are inherently thread safe.

This applies only to the individual methods of dictionaries. The
following code would still require a lock:
if mydict.has_key (keyval):
variable = mydict [keyval]
because a second thread could delete the entry between the calls to
has_key and __getvalue__.

mydict [keyval] = mydict.get (keyval, 0) + 1
is also an candidate for problems.
However, pytz seems to work well with multiple threads creating various
timezone objects at the same time.


'Seems to work' is never a good argument with regard to threads.
Especially if you're testing on a single CPU machine.

Daniel
Aug 10 '05 #3
Daniel Dittmar wrote:
Dictionaries (and probably most other Python types that are implemented
in C) are inherently thread safe.


That sounds like a dangerous assumption to me.

Are you relying on the Global Interpreter Lock?
Is is guaranteed?
Does that safety transfer to Jython?
How can I tell if any particular object is thread-safe?

I don't know the answers to these questions, and I have the
feeling that it is probably best to play safe and always use your
own explicit locking.

The Cog
Aug 11 '05 #4
nicolas_riesch wrote:
Does someone know if the module pytz
(http://sourceforge.net/projects/pytz/) is thread-safe ?
I have not seen it explicitely stated, and just wanted to be sure, as I
want to use it.
pytz is thread safe.
That's because in the file pytz/tzinfo.py, I see global variables
_timedelta_cache, _datetime_cache, _ttinfo_cache, which are
dictionnaries and are used as cache. I always thought that you must protect shared data with locks when
multithreading, but I don't see any lock anywhere in pytz.
However, pytz seems to work well with multiple threads creating various
timezone objects at the same time.
I don't understand where the trick is, that allows multiple threads to
access this module without any locking and that all this seems to work
without any problem...


Thanks to the global interpreter lock, with the Python builtin types you
only need to maintain a lock if there is a race condition, or if you care
about the race condition. For example, the following is thread safe code:
from threading import Thread
import time
stack = []
stack2 = []
def doit(i): .... stack.append(i)
.... time.sleep(0.1)
.... stack2.append(stack.pop())
.... threads = [Thread(target=doit, args=(i,)) for i in range(0,100)]
for t in threads: t.start() .... for t in threads: t.join() .... len(stack2) 100 stack2 [99, 95, 98, 94, 93, 97, 92, 91, 96, 88, 87, 86, 85, 84, 83, 90, 79, 78, 77,
76, 74, 73, 72, 71, 70, 75, 82, 81, 80, 89, 69, 67, 66, 65, 64, 68, 60, 59,
58, 57, 56, 55, 63, 62, 61, 49, 54, 53, 52, 51, 46, 45, 44, 50, 48, 47, 29,
28, 35, 34, 33, 43, 42, 41, 40, 39, 38, 32, 37, 31, 30, 36, 27, 26, 25, 24,
23, 22, 21, 20, 19, 18, 17, 12, 16, 15, 14, 13, 11, 10, 9, 8, 7, 6, 4, 3, 2,
1, 0, 5]

Note that the value being appended to 'stack2' might not be the value that
was appended to 'stack' in any particular thread - in this case, we don't
care (but is the sort of thing you might need to watch out for).

In the code you mention in pytz, there *is* a race condition. However, if
this condition occurs the side effects are so trivial as to not worry about
locking. ie. if a number of threads call memorized_timedelta(seconds=60)
simultaneously, there is a slight chance that each thread will get a
different timedelta instance. This is extremely unlikely, and the rest of
the code doesn't care at all. If pytz compared the timedeltas using 'is'
instead of '==' at any point, it would be a bug (but it doesn't, so it isn't).

So you can write thread safe Python code without locks provided you are
using the builtin types, and keep a close eye out for race conditions. This
might sound error prone, but it is quite doable provided the critical areas
that are accessing shared objects are kept isolated, short and simple.

Here is an thread unsafe example. Here the mistake is made that the length
of stack will not change after checking it. Also because we don't use the
atomic stack.pop(), two threads might add the same value to stack2:
from threading import Thread
import time
stack = range(0, 50)
stack2 = []
def doit(): .... if len(stack) > 0:
.... stack2.append(stack[-1])
.... time.sleep(0.1)
.... del stack[-1]
.... threads = [Thread(target=doit) for i in range(0, 100)]
for t in threads: t.start()

....
Exception in thread Thread-249:
Traceback (most recent call last):
File "/usr/lib/python2.4/threading.py", line 442, in __bootstrap
self.run()
File "/usr/lib/python2.4/threading.py", line 422, in run
self.__target(*self.__args, **self.__kwargs)
File "<stdin>", line 5, in doit
IndexError: list assignment index out of range

--
Stuart Bishop <st****@stuartbishop.net>
http://www.stuartbishop.net/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)

iD8DBQFC/t4hAfqZj7rGN0oRAop2AJ4udkfv8TlAtQt7ya0v/dh55l8ntACdG9PH
m2WJx2WTUZnNh7HmAMQcils=
=WyDW
-----END PGP SIGNATURE-----

Aug 14 '05 #5
Thank you very much for all your explanation !
Your pytz module is great !

Aug 15 '05 #6

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

18
by: jas | last post by:
Hi, I would like to start a new process and be able to read/write from/to it. I have tried things like... import subprocess as sp p = sp.Popen("cmd.exe", stdout=sp.PIPE)...
0
by: Dmitri Shvetsov | last post by:
Hi, Does anybody know how to write to richTextBox from thread? All variables using by thread should be static, including the thread method (body). The richTextBox control is used on a main...
0
by: Abubakar | last post by:
Hi, try { int x = ns.Read(readbuffer, 0, readbuffer.Length); } catch (System.IO.IOException ioexception) { UINotifications.ServerMessageDisplay(ioexception.ToString( )); }
3
by: David Thielen | last post by:
Hi; I created a virtual directory in IIS 6.0 and my asp.net app runs fine. But when it tries to write a file I get: Access to the path is denied. - C:\Inetpub\wwwroot\RunReportASP\images ...
6
by: Zytan | last post by:
Make a WebBrowser control, and make a bunch of timers / threads that continually call WebBrowser.Document.Write. Eventually, this happens: DisconnectedContext was detected Message: Context...
2
by: Flyzone | last post by:
I have a list of parameters. I need to run in thread a command, one thread for one parameter. So i made a for loop, creating 5 threads and waiting their stop with: for parameter in parameters ...
7
by: Boki | last post by:
Multi-thread read/write to a single file. I have two processing threads, thread A and thread B; and I called my queue as Q. Thread A will feed data into Q by user input, the timing is random....
3
by: Ryan Liu | last post by:
Will TcpClient.GetStream().Read()/ReadByte() block until at least one byte of data can be read? In a Client/Server application, what does it mean at the end of stream/no more data available? ...
16
by: AAaron123 | last post by:
I have a timer. At each tick, say 0.1 second, I write a file. If it takes more than 0.1 second to write the file the app will not work correctly. How can I tell in the tick event if the...
0
by: Leo Jay | last post by:
I'd like to read and write the same socket in different threads. one thread is only used to read from the socket, and the other is only used to write to the socket. But I always get a 10022...
1
isladogs
by: isladogs | last post by:
The next online meeting of the Access Europe User Group will be on Wednesday 6 Dec 2023 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, Mike...
0
by: VivesProcSPL | last post by:
Obviously, one of the original purposes of SQL is to make data query processing easy. The language uses many English-like terms and syntax in an effort to make it easy to learn, particularly for...
3
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 3 Jan 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). For other local times, please check World Time Buddy In...
0
by: jianzs | last post by:
Introduction Cloud-native applications are conventionally identified as those designed and nurtured on cloud infrastructure. Such applications, rooted in cloud technologies, skillfully benefit from...
0
by: mar23 | last post by:
Here's the situation. I have a form called frmDiceInventory with subform called subfrmDice. The subform's control source is linked to a query called qryDiceInventory. I've been trying to pick up the...
0
by: abbasky | last post by:
### Vandf component communication method one: data sharing ​ Vandf components can achieve data exchange through data sharing, state sharing, events, and other methods. Vandf's data exchange method...
2
by: jimatqsi | last post by:
The boss wants the word "CONFIDENTIAL" overlaying certain reports. He wants it large, slanted across the page, on every page, very light gray, outlined letters, not block letters. I thought Word Art...
0
Git
by: egorbl4 | last post by:
Скачал я git, хотел начать настройку, а там вылезло вот это Что это? Что мне с этим делать? ...
1
by: davi5007 | last post by:
Hi, Basically, I am trying to automate a field named TraceabilityNo into a web page from an access form. I've got the serial held in the variable strSearchString. How can I get this into the...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.