473,322 Members | 1,510 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,322 software developers and data experts.

Import from database

I'm trying to load module code from a database, which stores for each
module its full name, code, load date and a Boolean indicating whether
it's a package or not.

The following simple program:

import dbimp, sys

if __name__ == "__main__":
dbimp.install()
#import bsddb.db
import a.b.c.d
import bsddb

gives a traceback from its last line. The whole output is

$ python -i test.py
Accepted *db*
found a in db
load_module: a
a loaded: <module 'a' from 'db:a'> pkg: 1
found a.b in db
load_module: a.b
a.b loaded: <module 'a.b' from 'db:a.b'> pkg: 1
found a.b.c in db
load_module: a.b.c
a.b.c loaded: <module 'a.b.c' from 'db:a.b.c'> pkg: 1
found a.b.c.d in db
load_module: a.b.c.d
a.b.c.d loaded: <module 'a.b.c.d' from 'db:a.b.c.d'> pkg: 0
found bsddb in db
load_module: bsddb
found weakref in db
load_module: weakref
weakref loaded: <module 'weakref' from 'db:weakref'> pkg: 0
Traceback (most recent call last):
File "test.py", line 7, in ?
import bsddb
File "/c/steve/Projects/Python/dbimp/dbimp.py", line 49, in load_module
exec code in module.__dict__
File "db:bsddb", line 62, in ?
File "/usr/lib/python2.4/os.py", line 133, in ?
from os.path import (curdir, pardir, sep, pathsep, defpath, extsep,
altsep,
ImportError: No module named path


It appears that for some reason execution of the code in
bsddb/__init__.pyc (loaded from the database using my importer) is
causing the os module to execute again although it has already run and
been cached in sys.modules.

Any explanations would be welcome. The importer module is shown below (a
posting with an attachment to avoid code folding didn't appear to make
it out)

Also if someone can suggest how associate the source file with
module/package code from the database that would make tracebacks look
more conventional, though I'd like to retain the indication the module
was loaded by dbimp somehow.

#
# Import modules from a database
#
import sys, db, marshal, imp, new
conn = db.conn()
curs = conn.cursor()

curs.execute("select modName from module")
impdict = {}
for n in [x[0] for x in curs.fetchall()]:
impdict[n] = 1
class dbimporter(object):

def __init__(self, item, *args, **kw):
if item != "*db*":
raise ImportError
print "Accepted", item

def find_module(self, fullname, path=None):
#print "find_module:", fullname, "from", path
if fullname not in impdict:
#print "Bailed on", fullname
return None
else:
print "found", fullname, "in db"
return self

def load_module(self, modname):
print "load_module:", modname
if modname in sys.modules:
return sys.modules[modname]
curs.execute("select modCode, modPackage from module where
modName=%s", (modname, ))
row = curs.fetchone() # should only BE one ...S
if not row:
#print modname, "not found in db"
raise ImportError, "DB module %s not found in modules"
code, package = row
code = marshal.loads(code)
module = new.module(modname)
sys.modules[modname] = module
module.__name__ = modname
exec code in module.__dict__
module.__file__ = "db:%s" % modname
module.__loader__ = dbimporter
if package:
module.__path__ = sys.path
exec code in module.__dict__
print modname, "loaded:", repr(module), "pkg:", package
return module

def install():
sys.path_hooks.append(dbimporter)
sys.path_importer_cache.clear() # probably not necessary
sys.path.insert(0, "*db*") # probably not needed with a metea-path
hook?
regards
Steve
--
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/
Holden Web LLC +1 703 861 4237 +1 800 494 3119
Jul 18 '05 #1
4 3557
Steve,

I believe you have to put ntpath, macpath and posixpath in the module
database for os.path to work.

I tried it with zipimporter builtin and I got the same traceback till I
added ntpath.py to my zip file. (Of course, I renamed the original
ntpath to _ntpath so that the original did not get imported)
Thanks,
--Kartic

Jul 18 '05 #2
Kartic wrote:
Steve,

I believe you have to put ntpath, macpath and posixpath in the module
database for os.path to work.

I tried it with zipimporter builtin and I got the same traceback till I
added ntpath.py to my zip file. (Of course, I renamed the original
ntpath to _ntpath so that the original did not get imported)
Thanks,
--Kartic

I'm not sure I understand, as I am currently relying on the system
implementations of any modules that happen to be loaded before my code
picks up. As to what's in the database

SELECT *
FROM `module`
WHERE modName LIKE '%path%'

shows that I have macpathm macurlpath, ntpath, nturlpath, os2emxpath and
posixpath all there (but, as I say, I've already executed the standard
modules by the time anything of mine gets to run). If I alter my test
program to:

import dbimp, sys

if __name__ == "__main__":
dbimp.install()
k = sys.modules.keys()
k.sort()
for kk in k:
print kk
#import bsddb.db
import a.b.c.d
import bsddb

then I get as output

MySQLdb
MySQLdb.MySQLdb
...
MySQLdb.types
UserDict
__builtin__
__main__
_codecs
_locale
_mysql
_mysql_exceptions
_sre
array
cPickle
codecs
copy
copy_reg
db
dbimp
...
mx.Misc
mx.Misc.LazyModule
new
nt
ntpath
os
os.path
re
...
zipimport
Accepted *db*
found a in db
load_module: a
a loaded: <module 'a' from 'db:a'> pkg: 1
found a.b in db
load_module: a.b
a.b loaded: <module 'a.b' from 'db:a.b'> pkg: 1
found a.b.c in db
load_module: a.b.c
a.b.c loaded: <module 'a.b.c' from 'db:a.b.c'> pkg: 1
found a.b.c.d in db
load_module: a.b.c.d
a.b.c.d loaded: <module 'a.b.c.d' from 'db:a.b.c.d'> pkg: 0
found bsddb in db
load_module: bsddb
found weakref in db
load_module: weakref
weakref loaded: <module 'weakref' from 'db:weakref'> pkg: 0
Traceback (most recent call last):
File "test.py", line 11, in ?
import bsddb
File "/c/steve/Projects/Python/dbimp/dbimp.py", line 49, in load_module
exec code in module.__dict__
File "db:bsddb", line 62, in ?
File "/usr/lib/python2.4/os.py", line 133, in ?
from os.path import (curdir, pardir, sep, pathsep, defpath, extsep,
altsep,
ImportError: No module named path

In other words, the os module is /already/ in sys.smodules, as is
os.path, yet the interpreter is complaining (I presume) that os.path is
not a module. I don't even know *why* os is being executed a second
time. I can only assume it's being imported as some other name like
"bsddb.os" and some element of the import system is actually doing an
import rather than refusing to guess.

I presume I need to control the import process more closely to make sure
that this import attempt is rejected, but I can't see how to do that.

regards
Steve

Jul 18 '05 #3
Steve,

Hmmm... Yes, I see what you are saying. Could you post your create
table statement so that I can create a db and play around with
dbimport?

Thanks,
--Kartic

Jul 18 '05 #4
Kartic wrote:
Steve,

Hmmm... Yes, I see what you are saying. Could you post your create
table statement so that I can create a db and play around with
dbimport?

Thanks,
--Kartic

Here it is, plus also the loader program I used to suck in the standard
library (in case it's that that's faulty):

CREATE TABLE `module` (
`modName` varchar(25) NOT NULL default '',
`modPackage` tinyint(1) NOT NULL default '0',
`modCompileTime` timestamp(14) NOT NULL,
`modCode` longtext,
PRIMARY KEY (`modName`)
) TYPE=MyISAM;

#### WARNING: email client may wrap some lines ...
#
# Establish standard library in database
#
import db
import os
import glob
import sys
import marshal

conn = db.conn()
curs = conn.cursor()

if sys.argv[1] == "-d":
curs.execute("delete from module")
print "All existing modules cleared"
del sys.argv[1]

def importpy(path, modname, package):
print "Importing", path, modname
c = compile(file(path).read(), "db:%s" % modname, "exec")
curs.execute("""delete from module where modName = %s""", (modname,))
curs.execute("""insert into module (modName, modCode, modPackage,
modCompileTime)
values (%s, %s, %s, now())""", (modname,
marshal.dumps(c), package))
print "Added", modname
conn.commit()

def importall(path, modlist):
os.chdir(path)
for f in glob.glob("*"):
if os.path.isdir(f):
fn = os.path.join(path, f, "__init__.py")
if os.path.exists(fn):
ml = modlist + [f]
importpy(fn, ".".join(ml), 1)
importall(os.path.join(path, f), ml)
elif f.endswith('.py') and '.' not in f[:-3] and f !=
"__init__.py":
importpy(os.path.join(path, f), ".".join(modlist+[f[:-3]]), 0)

if __name__ == "__main__":
for path in sys.argv[1:]:
importall(path, [])

regards
Steve

Jul 18 '05 #5

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

5
by: Steve Holden | last post by:
This is even stranger: it makes it if I import the module a second time: import dbimp as dbimp import sys if __name__ == "__main__": dbimp.install() #k = sys.modules.keys() #k.sort() #for...
1
by: Xif | last post by:
Hello Everyone! Here's a problem with relative imports: Suppose I have a package called some_package (in a separate directory included in the PYTHONPATH, with an __init__.py file etc.) This...
3
by: Doug Baroter | last post by:
Hi, One of my clients has the following situation. They use Access DB for data update etc. some business functions while they also want to view the Access data quickly and more efficiently in...
15
by: DesignGuy | last post by:
I've inherited a site that has 1000+ product pages that follow a similar layout. I would like to import these pages into a database for ease of future updating. The pages have the usual banners,...
2
by: David Berry | last post by:
Hi All. I'm looking for any help or sample code that can show me how to make a file import wizard in ASP.NET (VB preferred) like the one that MS Access uses. I'm working on a web site where the...
1
by: Dan | last post by:
Could someone please help me with auto importing a series of data files into an Access table. I tried to follow code given below in a previous messagebut i'm getting error messages. Here's my...
4
by: Steve Jorgensen | last post by:
I'm restarting this thread with a different focus. The project I'm working on now id coming along and will be made to work, and it's too late to start over with a new strategy. Still, I'm not...
3
by: Iavor Raytchev | last post by:
Hello, We a situation with a central database that contains the data that needs to be presented at N off-line terminals (N can be 5 000 can be 15 000). Each terminal presents unique data. The...
0
by: A3AN | last post by:
Hi. I receive a database backup on a daily basis. I then import this dump on another server which I use for software development. There is two db's being hosted on this server. We test software...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.