By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
446,266 Members | 1,816 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 446,266 IT Pros & Developers. It's quick & easy.

Hey, get this! [was: import from database]

P: n/a
This is even stranger: it makes it if I import the module a second time:

import dbimp as dbimp
import sys

if __name__ == "__main__":
dbimp.install()
#k = sys.modules.keys()
#k.sort()
#for kk in k:
#print kk
#import bsddb.db
import a.b.c.d
import smtplib
import ftplib
import fileinput
try:
print "first import"
import bsddb
except:
print "second import"
import bsddb
print "Done!"

$ python -i test.py
dbimporter: item: *db* args: () keywords: {}
Accepted *db*
dbimporter: item: /c/steve/Projects/Python/dbimp args: () keywords: {}
dbimporter: item: /c/steve/Projects/Python/dbimp/c args: () keywords: {}
dbimporter: item: /c/steve/Projects/Python/dbimp/\code args: () keywords: {}
dbimporter: item: /usr/lib/python24.zip args: () keywords: {}
dbimporter: item: /usr/lib/python2.4 args: () keywords: {}
dbimporter: item: /usr/lib/python2.4/plat-cygwin args: () keywords: {}
dbimporter: item: /usr/lib/python2.4/lib-tk args: () keywords: {}
dbimporter: item: /usr/lib/python2.4/lib-dynload args: () keywords: {}
dbimporter: item: /usr/lib/python2.4/site-packages args: () keywords: {}
dbimporter: item: /usr/lib/python2.4/site-packages/a args: () keywords: {}
dbimporter: item: /usr/lib/python2.4/site-packages/a/b args: () keywords: {}
dbimporter: item: /usr/lib/python2.4/site-packages/a/b/c args: ()
keywords: {}
found smtplib in db
load_module: smtplib
found socket in db
load_module: socket
socket loaded: <module 'socket' from 'db:socket'> pkg: 0
found rfc822 in db
load_module: rfc822
rfc822 loaded: <module 'rfc822' from 'db:rfc822'> pkg: 0
found base64 in db
load_module: base64
base64 loaded: <module 'base64' from 'db:base64'> pkg: 0
found hmac in db
load_module: hmac
hmac loaded: <module 'hmac' from 'db:hmac'> pkg: 0
dbimporter: item: /usr/lib/python2.4/email args: () keywords: {}
found random in db
load_module: random
random loaded: <module 'random' from 'db:random'> pkg: 0
found quopri in db
load_module: quopri
quopri loaded: <module 'quopri' from 'db:quopri'> pkg: 0
smtplib loaded: <module 'smtplib' from 'db:smtplib'> pkg: 0
found ftplib in db
load_module: ftplib
dbimporter: item: /usr/lib/python2.4/site-packages/PIL args: () keywords: {}
dbimporter: item: /usr/lib/python2.4/site-packages/piddle args: ()
keywords: {}
ftplib loaded: <module 'ftplib' from 'db:ftplib'> pkg: 0
found fileinput in db
load_module: fileinput
fileinput loaded: <module 'fileinput' from 'db:fileinput'> pkg: 0
first import
found bsddb in db
load_module: bsddb
found weakref in db
load_module: weakref
weakref loaded: <module 'weakref' from 'db:weakref'> pkg: 0
second import
Done!


So it's clearly something pretty funky. It now "works" (for some value
of "work" wiht both MySQL and sqlite. I hope I have this sorted out
before PyCon ... I'm currently a bit confused!

regards
Steve
--
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/
Holden Web LLC +1 703 861 4237 +1 800 494 3119

Jul 18 '05 #1
Share this Question
Share on Google+
5 Replies


P: n/a
Steve Holden wrote:
This is even stranger: it makes it if I import the module a second time:


[second import seems to succeed]

Maybe you are experiencing some version confusion? What you describe looks
much like the normal Python 2.3 behaviour (with no import hook involved)
whereas you seem to operate on the 2.4 library.
A partially initialized module object is left behind in sys.modules and seen
by further import attempts.

$ cat arbitrary.py

import not_there

def f():
print "you ain't gonna see that"

$ python
Python 2.3.3 (#1, Apr 6 2004, 01:47:39)
[GCC 3.3.3 (SuSE Linux)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
import arbitrary Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "arbitrary.py", line 2, in ?
import not_there
ImportError: No module named not_there import arbitrary
arbitrary.f() Traceback (most recent call last):
File "<stdin>", line 1, in ?
AttributeError: 'module' object has no attribute 'f'
I have no experience with import hooks, but for normal imports that has been
fixed in Python 2.4:

$ py24
Python 2.4 (#5, Jan 4 2005, 10:14:01)
[GCC 3.3.3 (SuSE Linux)] on linux2
Type "help", "copyright", "credits" or "license" for more information. import arbitrary Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "arbitrary.py", line 2, in ?
import not_there
ImportError: No module named not_there import arbitrary Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "arbitrary.py", line 2, in ?
import not_there
ImportError: No module named not_there


Peter

Jul 18 '05 #2

P: n/a
Peter Otten wrote:
Steve Holden wrote:

This is even stranger: it makes it if I import the module a second time:

[second import seems to succeed]

Maybe you are experiencing some version confusion? What you describe looks
much like the normal Python 2.3 behaviour (with no import hook involved)
whereas you seem to operate on the 2.4 library.
A partially initialized module object is left behind in sys.modules and seen
by further import attempts.

I agree that this is 2.3-like behavior, but Python cannot lie ...

sholden@dellboy ~/Projects/Python/dbimp
$ python
Python 2.4 (#1, Dec 4 2004, 20:10:33)
[GCC 3.3.3 (cygwin special)] on cygwin
Type "help", "copyright", "credits" or "license" for more information.
$ cat arbitrary.py

import not_there

def f():
print "you ain't gonna see that"

$ python
Python 2.3.3 (#1, Apr 6 2004, 01:47:39)
[GCC 3.3.3 (SuSE Linux)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
import arbitrary
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "arbitrary.py", line 2, in ?
import not_there
ImportError: No module named not_there
import arbitrary
arbitrary.f()
Traceback (most recent call last):
File "<stdin>", line 1, in ?
AttributeError: 'module' object has no attribute 'f'
I have no experience with import hooks, but for normal imports that has been
fixed in Python 2.4:

$ py24
Python 2.4 (#5, Jan 4 2005, 10:14:01)
[GCC 3.3.3 (SuSE Linux)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
import arbitrary
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "arbitrary.py", line 2, in ?
import not_there
ImportError: No module named not_there
import arbitrary


Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "arbitrary.py", line 2, in ?
import not_there
ImportError: No module named not_there
Peter

$ cat arbitrary.py
import not_there

def f():
print "you ain't gonna see that"
sholden@dellboy ~/Projects/Python/dbimp
$ python
Python 2.4 (#1, Dec 4 2004, 20:10:33)
[GCC 3.3.3 (cygwin special)] on cygwin
Type "help", "copyright", "credits" or "license" for more information. import arbitrary Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "arbitrary.py", line 1, in ?
import not_there
ImportError: No module named not_there import arbitrary Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "arbitrary.py", line 1, in ?
import not_there
ImportError: No module named not_there


Yup, looks like 2.4 (despite this funny cygwin stuff, could that make a
difference).

Let me try it under Windows [ferkle, ferkle ...]

Does the same thing there.

This problem also seems to depend what's already loaded. I wrote a
program to write a test program that looks like this:

import dbimp
dbimp.install()

print "Trying aifc"
try:
import aifc
except:
print "%Failed: aifc"
print "Trying anydbm"
try:
import anydbm
except:
print "%Failed: anydbm"
print "Trying asynchat"
try:
import asynchat
except:
print "%Failed: asynchat"

...

import dbimp
dbimp.install()

print "Trying aifc"
try:
import aifc
except:
print "%Failed: aifc"
print "Trying anydbm"
try:
import anydbm
except:
print "%Failed: anydbm"
print "Trying asynchat"
try:
import asynchat
except:
print "%Failed: asynchat"

The two platforms give expectedly close results. I'm storing compiled
code, so a version incompatibility would be a problem, I agree, but I
have checked the program that loaded the database, and loaded it again
from the Windows source rather than the CygWin source, just to see
whether there were any unnoticed platform dependencies. The results were
exactly the same using either library, and Windows and Cygwin showed
only minor variations.

exhaustCyg24.txt:%Failed: bsddb.dbtables
exhaustCyg24.txt:%Failed: bsddb.test.test_all
exhaustCyg24.txt:%Failed: bsddb.test.test_associate
exhaustCyg24.txt:%Failed: bsddb.test.test_basics
exhaustCyg24.txt:%Failed: bsddb.test.test_compat
exhaustCyg24.txt:%Failed: bsddb.test.test_dbobj
exhaustCyg24.txt:%Failed: bsddb.test.test_dbshelve
exhaustCyg24.txt:%Failed: bsddb.test.test_dbtables
exhaustCyg24.txt:%Failed: bsddb.test.test_env_close
exhaustCyg24.txt:%Failed: bsddb.test.test_get_none
exhaustCyg24.txt:%Failed: bsddb.test.test_join
exhaustCyg24.txt:%Failed: bsddb.test.test_lock
exhaustCyg24.txt:%Failed: bsddb.test.test_misc
exhaustCyg24.txt:%Failed: bsddb.test.test_queue
exhaustCyg24.txt:%Failed: bsddb.test.test_recno
exhaustCyg24.txt:%Failed: bsddb.test.test_thread
exhaustCyg24.txt:%Failed: tzparse
exhaustWin24.txt:%Failed: bsddb.dbtables
exhaustWin24.txt:%Failed: bsddb.test.test_all
exhaustWin24.txt:%Failed: bsddb.test.test_associate
exhaustWin24.txt:%Failed: bsddb.test.test_basics
exhaustWin24.txt:%Failed: bsddb.test.test_compat
exhaustWin24.txt:%Failed: bsddb.test.test_dbobj
exhaustWin24.txt:%Failed: bsddb.test.test_dbshelve
exhaustWin24.txt:%Failed: bsddb.test.test_dbtables
exhaustWin24.txt:%Failed: bsddb.test.test_env_close
exhaustWin24.txt:%Failed: bsddb.test.test_get_none
exhaustWin24.txt:%Failed: bsddb.test.test_join
exhaustWin24.txt:%Failed: bsddb.test.test_lock
exhaustWin24.txt:%Failed: bsddb.test.test_misc
exhaustWin24.txt:%Failed: bsddb.test.test_queue
exhaustWin24.txt:%Failed: bsddb.test.test_recno
exhaustWin24.txt:%Failed: bsddb.test.test_thread
exhaustWin24.txt:%Failed: pty
exhaustWin24.txt:%Failed: rlcompleter
exhaustWin24.txt:%Failed: tzparse

As a workaround I can for the moment just omit everything that show the
least suspicion of not working from the database, but I'd really rather
know what's failing.

If you have any clues I'd be grateful.

regards
Steve
--
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/
Holden Web LLC +1 703 861 4237 +1 800 494 3119
Jul 18 '05 #3

P: n/a
Steve Holden wrote:
Peter Otten wrote:
Steve Holden wrote:

This is even stranger: it makes it if I import the module a second time:


[second import seems to succeed]

Maybe you are experiencing some version confusion? What you describe
looks
much like the normal Python 2.3 behaviour (with no import hook involved)
whereas you seem to operate on the 2.4 library.
A partially initialized module object is left behind in sys.modules
and seen
by further import attempts.

I agree that this is 2.3-like behavior, but Python cannot lie ...

sholden@dellboy ~/Projects/Python/dbimp
$ python
Python 2.4 (#1, Dec 4 2004, 20:10:33)
[GCC 3.3.3 (cygwin special)] on cygwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

Just to make things simpler, and (;-) to appeal to a wider audience,
here is a program that doesn't use database at all (it loads the entire
standard library into a dict) and still shows the error.

What *I* would like to know is: who is allowing the import of bsddb.os,
thereby somehow causing the code of the os library module to be run a
second time.

#
# Establish standard library in dict
#
import os
import glob
import sys
import marshal
import new

def importpy(dct, path, modname, package):
c = compile(file(path).read(), path, "exec")
dct[modname] = (marshal.dumps(c), package, path)
if package:
print "Package", modname, path
else:
print "Module", modname, path

def importall(dct, path, modlist):
os.chdir(path)
for f in glob.glob("*"):
if os.path.isdir(f):
fn = os.path.join(path, f, "__init__.py")
if os.path.exists(fn):
ml = modlist + [f]
importpy(dct, fn, ".".join(ml), 1)
importall(dct, os.path.join(path, f), ml)
elif f.endswith('.py') and '.' not in f[:-3] and f !=
"__init__.py":
importpy(dct, os.path.join(path, f),
".".join(modlist+[f[:-3]]), 0)

class dbimporter(object):

def __init__(self, item, *args, **kw):
##print "dbimporter: item:", item, "args:", args, "keywords:", kw
if item != "*db*":
raise ImportError
print "Accepted", item

def find_module(self, fullname, path=None):
print "find_module:", fullname, "from", path
if fullname not in impdict:
#print "Bailed on", fullname
return None
else:
print "found", fullname, "in db"
return self

def load_module(self, modname):
print "load_module:", modname
if modname in sys.modules:
return sys.modules[modname]
try:
row = impdict[modname]
except KeyError:
#print modname, "not found in db"
raise ImportError, "DB module %s not found in modules"
code, package, path = row
code = marshal.loads(code)
module = new.module(modname)
sys.modules[modname] = module
module.__name__ = modname
module.__file__ = path # "db:%s" % modname
module.__loader__ = dbimporter
if package:
module.__path__ = sys.path
exec code in module.__dict__
print modname, "loaded:", repr(module), "pkg:", package
return module

def install():
sys.path_hooks.append(dbimporter)
sys.path_importer_cache.clear() # probably not necessary
sys.path.insert(0, "*db*") # probably not needed with a metea-path
hook?

if __name__ == "__main__":
impdict = {}
for path in sys.argv[1:]:
importall(impdict, path, [])
install()
import bsddb

Running this against a copy of the Python 2.4 standard library in C:\Lib
gives me

[...]
Module _strptime C:\Lib\_strptime.py
Module _threading_local C:\Lib\_threading_local.py
Module __future__ C:\Lib\__future__.py
Accepted *db*
find_module: bsddb from None
found bsddb in db
load_module: bsddb
find_module: bsddb._bsddb from None
find_module: bsddb.sys from None
find_module: bsddb.os from None
find_module: bsddb.nt from None
find_module: bsddb.ntpath from None
find_module: bsddb.stat from None
Traceback (most recent call last):
File "C:\Steve\Projects\Python\dbimp\dictload.py", line 79, in ?
import bsddb
File "C:\Steve\Projects\Python\dbimp\dictload.py", line 65, in
load_module
exec code in module.__dict__
File "C:\Lib\bsddb\__init__.py", line 62, in ?
import sys, os
File "C:\Python24\lib\os.py", line 133, in ?
from os.path import (curdir, pardir, sep, pathsep, defpath, extsep,
altsep,
ImportError: No module named path

The 2.3 bsddb library doesn't cause the same problems (and even loads
into 2.4 quite nicely). Lots of modules *will* import, and most packages
don't seem to cause problems. Anyone give me a pointer here?

regards
Steve
--
Meet the Python developers and your c.l.py favorites March 23-25
Come to PyCon DC 2005 http://www.python.org/pycon/2005/
Steve Holden http://www.holdenweb.com/
Jul 18 '05 #4

P: n/a
Steve Holden <st***@holdenweb.com> writes:
What *I* would like to know is: who is allowing the import of bsddb.os,
thereby somehow causing the code of the os library module to be run a
second time.
I would guess (without actually running the code) that this part is
responsible:
if package:
module.__path__ = sys.path


You usually should initialize a package's __path__ to an empty list.
The __init__ module will take care of modifying it if necessary.

Bernhard

--
Intevation GmbH http://intevation.de/
Skencil http://skencil.org/
Thuban http://thuban.intevation.org/
Jul 18 '05 #5

P: n/a
Bernhard Herzog wrote:
Steve Holden <st***@holdenweb.com> writes:

What *I* would like to know is: who is allowing the import of bsddb.os,
thereby somehow causing the code of the os library module to be run a
second time.

I would guess (without actually running the code) that this part is
responsible:

if package:
module.__path__ = sys.path

You usually should initialize a package's __path__ to an empty list.
The __init__ module will take care of modifying it if necessary.

Bernhard

Thanks, Bernhard, that appeared to fix the problem. The error certainly
went away, anyway ... now I can do more experimentation.

regards
Steve
--
Meet the Python developers and your c.l.py favorites March 23-25
Come to PyCon DC 2005 http://www.python.org/pycon/2005/
Steve Holden http://www.holdenweb.com/
Jul 18 '05 #6

This discussion thread is closed

Replies have been disabled for this discussion.