473,597 Members | 2,092 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Import from database

I'm trying to load module code from a database, which stores for each
module its full name, code, load date and a Boolean indicating whether
it's a package or not.

The following simple program:

import dbimp, sys

if __name__ == "__main__":
dbimp.install()
#import bsddb.db
import a.b.c.d
import bsddb

gives a traceback from its last line. The whole output is

$ python -i test.py
Accepted *db*
found a in db
load_module: a
a loaded: <module 'a' from 'db:a'> pkg: 1
found a.b in db
load_module: a.b
a.b loaded: <module 'a.b' from 'db:a.b'> pkg: 1
found a.b.c in db
load_module: a.b.c
a.b.c loaded: <module 'a.b.c' from 'db:a.b.c'> pkg: 1
found a.b.c.d in db
load_module: a.b.c.d
a.b.c.d loaded: <module 'a.b.c.d' from 'db:a.b.c.d'> pkg: 0
found bsddb in db
load_module: bsddb
found weakref in db
load_module: weakref
weakref loaded: <module 'weakref' from 'db:weakref'> pkg: 0
Traceback (most recent call last):
File "test.py", line 7, in ?
import bsddb
File "/c/steve/Projects/Python/dbimp/dbimp.py", line 49, in load_module
exec code in module.__dict__
File "db:bsddb", line 62, in ?
File "/usr/lib/python2.4/os.py", line 133, in ?
from os.path import (curdir, pardir, sep, pathsep, defpath, extsep,
altsep,
ImportError: No module named path


It appears that for some reason execution of the code in
bsddb/__init__.pyc (loaded from the database using my importer) is
causing the os module to execute again although it has already run and
been cached in sys.modules.

Any explanations would be welcome. The importer module is shown below (a
posting with an attachment to avoid code folding didn't appear to make
it out)

Also if someone can suggest how associate the source file with
module/package code from the database that would make tracebacks look
more conventional, though I'd like to retain the indication the module
was loaded by dbimp somehow.

#
# Import modules from a database
#
import sys, db, marshal, imp, new
conn = db.conn()
curs = conn.cursor()

curs.execute("s elect modName from module")
impdict = {}
for n in [x[0] for x in curs.fetchall()]:
impdict[n] = 1
class dbimporter(obje ct):

def __init__(self, item, *args, **kw):
if item != "*db*":
raise ImportError
print "Accepted", item

def find_module(sel f, fullname, path=None):
#print "find_modul e:", fullname, "from", path
if fullname not in impdict:
#print "Bailed on", fullname
return None
else:
print "found", fullname, "in db"
return self

def load_module(sel f, modname):
print "load_modul e:", modname
if modname in sys.modules:
return sys.modules[modname]
curs.execute("s elect modCode, modPackage from module where
modName=%s", (modname, ))
row = curs.fetchone() # should only BE one ...S
if not row:
#print modname, "not found in db"
raise ImportError, "DB module %s not found in modules"
code, package = row
code = marshal.loads(c ode)
module = new.module(modn ame)
sys.modules[modname] = module
module.__name__ = modname
exec code in module.__dict__
module.__file__ = "db:%s" % modname
module.__loader __ = dbimporter
if package:
module.__path__ = sys.path
exec code in module.__dict__
print modname, "loaded:", repr(module), "pkg:", package
return module

def install():
sys.path_hooks. append(dbimport er)
sys.path_import er_cache.clear( ) # probably not necessary
sys.path.insert (0, "*db*") # probably not needed with a metea-path
hook?
regards
Steve
--
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/
Holden Web LLC +1 703 861 4237 +1 800 494 3119
Jul 18 '05 #1
4 3575
Steve,

I believe you have to put ntpath, macpath and posixpath in the module
database for os.path to work.

I tried it with zipimporter builtin and I got the same traceback till I
added ntpath.py to my zip file. (Of course, I renamed the original
ntpath to _ntpath so that the original did not get imported)
Thanks,
--Kartic

Jul 18 '05 #2
Kartic wrote:
Steve,

I believe you have to put ntpath, macpath and posixpath in the module
database for os.path to work.

I tried it with zipimporter builtin and I got the same traceback till I
added ntpath.py to my zip file. (Of course, I renamed the original
ntpath to _ntpath so that the original did not get imported)
Thanks,
--Kartic

I'm not sure I understand, as I am currently relying on the system
implementations of any modules that happen to be loaded before my code
picks up. As to what's in the database

SELECT *
FROM `module`
WHERE modName LIKE '%path%'

shows that I have macpathm macurlpath, ntpath, nturlpath, os2emxpath and
posixpath all there (but, as I say, I've already executed the standard
modules by the time anything of mine gets to run). If I alter my test
program to:

import dbimp, sys

if __name__ == "__main__":
dbimp.install()
k = sys.modules.key s()
k.sort()
for kk in k:
print kk
#import bsddb.db
import a.b.c.d
import bsddb

then I get as output

MySQLdb
MySQLdb.MySQLdb
...
MySQLdb.types
UserDict
__builtin__
__main__
_codecs
_locale
_mysql
_mysql_exceptio ns
_sre
array
cPickle
codecs
copy
copy_reg
db
dbimp
...
mx.Misc
mx.Misc.LazyMod ule
new
nt
ntpath
os
os.path
re
...
zipimport
Accepted *db*
found a in db
load_module: a
a loaded: <module 'a' from 'db:a'> pkg: 1
found a.b in db
load_module: a.b
a.b loaded: <module 'a.b' from 'db:a.b'> pkg: 1
found a.b.c in db
load_module: a.b.c
a.b.c loaded: <module 'a.b.c' from 'db:a.b.c'> pkg: 1
found a.b.c.d in db
load_module: a.b.c.d
a.b.c.d loaded: <module 'a.b.c.d' from 'db:a.b.c.d'> pkg: 0
found bsddb in db
load_module: bsddb
found weakref in db
load_module: weakref
weakref loaded: <module 'weakref' from 'db:weakref'> pkg: 0
Traceback (most recent call last):
File "test.py", line 11, in ?
import bsddb
File "/c/steve/Projects/Python/dbimp/dbimp.py", line 49, in load_module
exec code in module.__dict__
File "db:bsddb", line 62, in ?
File "/usr/lib/python2.4/os.py", line 133, in ?
from os.path import (curdir, pardir, sep, pathsep, defpath, extsep,
altsep,
ImportError: No module named path

In other words, the os module is /already/ in sys.smodules, as is
os.path, yet the interpreter is complaining (I presume) that os.path is
not a module. I don't even know *why* os is being executed a second
time. I can only assume it's being imported as some other name like
"bsddb.os" and some element of the import system is actually doing an
import rather than refusing to guess.

I presume I need to control the import process more closely to make sure
that this import attempt is rejected, but I can't see how to do that.

regards
Steve

Jul 18 '05 #3
Steve,

Hmmm... Yes, I see what you are saying. Could you post your create
table statement so that I can create a db and play around with
dbimport?

Thanks,
--Kartic

Jul 18 '05 #4
Kartic wrote:
Steve,

Hmmm... Yes, I see what you are saying. Could you post your create
table statement so that I can create a db and play around with
dbimport?

Thanks,
--Kartic

Here it is, plus also the loader program I used to suck in the standard
library (in case it's that that's faulty):

CREATE TABLE `module` (
`modName` varchar(25) NOT NULL default '',
`modPackage` tinyint(1) NOT NULL default '0',
`modCompileTime ` timestamp(14) NOT NULL,
`modCode` longtext,
PRIMARY KEY (`modName`)
) TYPE=MyISAM;

#### WARNING: email client may wrap some lines ...
#
# Establish standard library in database
#
import db
import os
import glob
import sys
import marshal

conn = db.conn()
curs = conn.cursor()

if sys.argv[1] == "-d":
curs.execute("d elete from module")
print "All existing modules cleared"
del sys.argv[1]

def importpy(path, modname, package):
print "Importing" , path, modname
c = compile(file(pa th).read(), "db:%s" % modname, "exec")
curs.execute("" "delete from module where modName = %s""", (modname,))
curs.execute("" "insert into module (modName, modCode, modPackage,
modCompileTime)
values (%s, %s, %s, now())""", (modname,
marshal.dumps(c ), package))
print "Added", modname
conn.commit()

def importall(path, modlist):
os.chdir(path)
for f in glob.glob("*"):
if os.path.isdir(f ):
fn = os.path.join(pa th, f, "__init__.p y")
if os.path.exists( fn):
ml = modlist + [f]
importpy(fn, ".".join(ml ), 1)
importall(os.pa th.join(path, f), ml)
elif f.endswith('.py ') and '.' not in f[:-3] and f !=
"__init__.p y":
importpy(os.pat h.join(path, f), ".".join(modlis t+[f[:-3]]), 0)

if __name__ == "__main__":
for path in sys.argv[1:]:
importall(path, [])

regards
Steve

Jul 18 '05 #5

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

5
2466
by: Steve Holden | last post by:
This is even stranger: it makes it if I import the module a second time: import dbimp as dbimp import sys if __name__ == "__main__": dbimp.install() #k = sys.modules.keys() #k.sort() #for kk in k:
1
1451
by: Xif | last post by:
Hello Everyone! Here's a problem with relative imports: Suppose I have a package called some_package (in a separate directory included in the PYTHONPATH, with an __init__.py file etc.) This package has a module inside the directory, called "database", and therefore residing in the file some_package/database.py.
3
6264
by: Doug Baroter | last post by:
Hi, One of my clients has the following situation. They use Access DB for data update etc. some business functions while they also want to view the Access data quickly and more efficiently in SQL Server 2000. Huge Access db with over 100 user tables, over 60 MB data. The DTS package that comes with SQL Server 2000 seems pretty "messy" in the sense that it assumes that one needs to do one time import only or accurately it does not...
15
5059
by: DesignGuy | last post by:
I've inherited a site that has 1000+ product pages that follow a similar layout. I would like to import these pages into a database for ease of future updating. The pages have the usual banners, navigation menus, etc. and other extraneous code that must be removed. Does software exist to import existing HTML pages into a database - even a rough import would work as I can massage the data later. Ideas/suggestions will be appreciated.
2
3069
by: David Berry | last post by:
Hi All. I'm looking for any help or sample code that can show me how to make a file import wizard in ASP.NET (VB preferred) like the one that MS Access uses. I'm working on a web site where the user has the ability to upload a file (.txt or .csv). The data in the file may be comma delaminated, tab delaminated, fixed width etc (we don't know). What I'd like to create is something like MS Access uses to import an Excel file into the...
1
6690
by: Dan | last post by:
Could someone please help me with auto importing a series of data files into an Access table. I tried to follow code given below in a previous messagebut i'm getting error messages. Here's my database stats: Path: C:\Database (contains the database and all the text files to be imported) Text files to import: (SampleData4.txt and SampleData3.txt as testing examples)
4
3015
by: Steve Jorgensen | last post by:
I'm restarting this thread with a different focus. The project I'm working on now id coming along and will be made to work, and it's too late to start over with a new strategy. Still, I'm not coming to a firm conclusion over whether it was the better approach, and wonder if I should do it differently the next time I'm faced with a similar issue. I needed an app to automatically import from spreadsheets with a semi-dynamic structure,...
3
3987
by: Iavor Raytchev | last post by:
Hello, We a situation with a central database that contains the data that needs to be presented at N off-line terminals (N can be 5 000 can be 15 000). Each terminal presents unique data. The central database is used for data preparation. Then the data for each terminal is exported as separate SQL file. The terminals run the same application that is used to prepare the data - only in user mode. So what we do is - we export the full database...
0
9771
by: A3AN | last post by:
Hi. I receive a database backup on a daily basis. I then import this dump on another server which I use for software development. There is two db's being hosted on this server. We test software upgrades against these two db's. The "refresh" needs to be completed at least once a day, sometimes 4-5 times a day. I need the quickest way of doing this task. I have extremely limited knowledge on scripting. What I need is something that I can just...
0
7884
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
8270
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
8380
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
8029
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
8258
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
6683
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
5845
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
1
1493
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
0
1230
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.