473,698 Members | 2,047 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Use a Thread to reload a Module?

Hi Python Experts,

I hope I can explain this right. I'll try.

Background:
I have a module that I leave running in a server role. It has a
module which has data in it that can change. So every nth time a
function in the server gets called, I want to reload the module so it
has the freshest data. But there's a lot of data so it takes 5-10
seconds to do a reload.

My question is:
Would I be able to launch a seperate thread to reload the module while
the server does it work? Hopefully it would be using the old module
data right up until the thread was finished reloading.

Thanks in advance,

Greg

Here's some psuedo code that might illustrate what I'm talking about:

import lotsa_data

def serve():
reload(lotsa_da ta) #can this launch a thread so "do_stuff" runs right away?
do_stuff(lotsa_ data.data)

while 1:
listen_for_requ ests()
Dec 22 '06 #1
3 1998
Gregory Piñero wrote:
Hi Python Experts,

I hope I can explain this right. I'll try.

Background:
I have a module that I leave running in a server role. It has a
module which has data in it that can change. So every nth time a
function in the server gets called, I want to reload the module so it
has the freshest data. But there's a lot of data so it takes 5-10
seconds to do a reload.

My question is:
Would I be able to launch a seperate thread to reload the module while
the server does it work? Hopefully it would be using the old module
data right up until the thread was finished reloading.

Thanks in advance,

Greg

Here's some psuedo code that might illustrate what I'm talking about:

import lotsa_data

def serve():
reload(lotsa_da ta) #can this launch a thread so "do_stuff" runs rightaway?
do_stuff(lotsa_ data.data)

while 1:
listen_for_requ ests()
Using a thread for this purpose is no problem. Using a module: yep,
that's a problem. (I'd say using a module in this way, to update data,
is very far from best practice, but its convenience justifies simple
uses. You are going beyond simple now, though.)

Not knowing more about your program, I'd say the simplest way is:

1. exec, don't reload, your data file (with the standard warning that
exec should only be used on carefully contructed code, or to
deliberately give the user the power to input python code--of course
the same warning applies when reloading a dynamically generated
module).

2. Store the new data somewhere (such as a Queue) waiting for a good
time to update.

3. At a convenient time, overwrite the old data in the module.

I'm going to assume that your server has heretofore been
single-threaded; therefore you don't need locks or queues or semaphores
in your main code. Here, then, is a very improvable example to
consider. Notice that the lotsa_data module is empty. Instead, you
call load_data() to exec the file where the data really is, and it puts
the loaded data into a queue. Next time you wait for a new request, it
checks to see if there are any data updates in the queue, and updates
the date in lotsa_data module if so.

import Queue
import threading
import lotsa_data ## empty!

data_update_que ue = Queue.Queue()

def serve():
if request_count % n:
threading.Threa d(target=load_d ata).start()
do_stuff(lotsa_ data.data)
request_count += 1

def load_data()
d = {}
exec "/path/to/data/file" in d
data_update_que ue.put(d)

load_data() # run once in main thread to load data initially
while True:
try:
d = data_update_que ue.get_nowait(d )
except Queue.Empty:
pass
else:
for key,value in d:
setattr(losta_d ata,key,value)
listen_for_requ ests()
There is much room for improvement. For example, can requests come in
fast enough to spawn another load_data before the first had ended? You
should consider trying to acquire a threading.Lock in load_data and
waiting and/or returning immediately if you can't. Other things can go
wrong, too. Using threads requires care. But hopefully this'll get
you started.
Carl Banks

Dec 23 '06 #2
On 22 Dec 2006 20:02:31 -0800, Carl Banks <pa************ @gmail.comwrote :
....
There is much room for improvement. For example, can requests come in
fast enough to spawn another load_data before the first had ended? You
should consider trying to acquire a threading.Lock in load_data and
waiting and/or returning immediately if you can't. Other things can go
wrong, too. Using threads requires care. But hopefully this'll get
you started.
Thanks, Carl. That gives me a lot to consider. I won't be able to
work on this project more until Wed. but I'll probably run into some
follow up questions then.

A few more details if it helps ...

That module deals with accessing data from QuickBooks, marshaling it,
and providing methods to access the marshaled data. Having the server
program keep that module and all of its data in memory makes the
server run really fast, but yeah, it does get complicated now that
it's storing 100's of MB of data. I guess most people go to a
database at this point?
It's just so easy to store the data in the Python objects I already
need them in instead of converting to tables in a DB and then
converting back. So maybe this threading will work for me.

Thanks again,

Greg
Dec 23 '06 #3
In article <ma************ *************** ************@py thon.org>,
=?ISO-8859-1?Q?Gregory_Pi= F1ero?= <gr********@gma il.comwrote:
>
That module deals with accessing data from QuickBooks, marshaling it,
and providing methods to access the marshaled data. Having the server
program keep that module and all of its data in memory makes the
server run really fast, but yeah, it does get complicated now that
it's storing 100's of MB of data. I guess most people go to a
database at this point?
Very, very yes. Try using a SQLite in-memory database, maybe?
>It's just so easy to store the data in the Python objects I already
need them in instead of converting to tables in a DB and then
converting back. So maybe this threading will work for me.
You might well run into problems with the import lock. I strongly
advise against your current approach.
--
Aahz (aa**@pythoncra ft.com) <* http://www.pythoncraft.com/

"I support family values -- Addams family values" --www.nancybutton s.com
Dec 24 '06 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

66
3885
by: Ellinghaus, Lance | last post by:
> > Other surprises: Deprecating reload() >Reload doesn't work the way most people think >it does: if you've got any references to the old module, >they stay around. They aren't replaced. >It was a good idea, but the implementation simply >doesn't do what the idea promises. I agree that it does not really work as most people think it does, but how
4
1805
by: David MacQuigg | last post by:
I'm going to be teaching EEs some basic Python using the first few chapters of Learning Python, 2nd ed. by Mark Lutz. The discussion on Reloading Modules starting on page 266 is confusing and I believe incorrect. On page 266 it says that a reload "changes the existing module object in place." That's a little vague, but on page 267 it says "every reference to a module object anywhere in your program is automatically affected by a...
1
2142
by: Emmanuel | last post by:
Hi, I use a 'reload all' feature in my app, that allow to reload every module. I first try this version : import sys def Reload():
0
1319
by: Mustafa Thamer | last post by:
Hi, I'm using import hooks according to PEP 302, in order to load python files from a game PAK file. The game is C++ using embedded and extended Python (v2.33) and Boost. The importing works fine, but after modules are imported I can't reload them. I've tried 'reload(foo)' and 'PyImport_ReloadModule(pModPtr)', but both return 'ImportError: No module named foo'. Is it safe to assume that reload doesn't respect the import hook? That...
4
1798
by: test1dellboy3 | last post by:
Hi, I am a beginner using the python interpreter. To reduce typing effort, I created a module called "aliases.py" containing some aliases for objects I commonly use like - aliases.py : import filecmp, os, commands
4
4654
by: Lonnie Princehouse | last post by:
So, it turns out that reload() fails if the module being reloaded isn't in sys.path. Maybe it could fall back to module.__file__ if the module isn't found in sys.path?? .... or reload could just take an optional path parameter... Or perhaps I'm the only one who thinks this is silly: >>> my_module = imp.load_module(module_name, *imp.find_module(module_name,path))
3
2181
by: John Salerno | last post by:
I understand that after you import something once, you can reload it to pick up new changes. But does reload work with from statements? I tried this: from X import * and then did my testing. I changed X and tried to reload it, but that didn't seem to work. I figure the reason is because the module itself doesn't exist as an object, only its names do. But I couldn't figure out how to pick up my new changes at this point. I think...
3
1551
by: Stef Mientki | last post by:
hello, I've a graphical application (wxPython), where the code in the main GUI loop is given below. 1 JAL_Loaded = False 2 while len(App_Running) 0: 3 if JALsPy_globals.State == SS_Run: 4 try:
0
1903
by: Rafe | last post by:
Hi, This seems to be an old question, and I've read back a bit, but rather than assume the answer is "you can't do that", I'd thought I'd post my version of the question along with a reproducible error to illustrate my confusion. My problem is that I'm using Python inside XSI (a 3D graphics application). If I want to restart Python, I have to restart XSI. This is no small amount of time wasted.
0
8674
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
8603
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
9157
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
7723
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
4366
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
4619
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
3045
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
2328
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
2001
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.