473,320 Members | 1,861 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

Hey, get this! [was: import from database]

This is even stranger: it makes it if I import the module a second time:

import dbimp as dbimp
import sys

if __name__ == "__main__":
dbimp.install()
#k = sys.modules.keys()
#k.sort()
#for kk in k:
#print kk
#import bsddb.db
import a.b.c.d
import smtplib
import ftplib
import fileinput
try:
print "first import"
import bsddb
except:
print "second import"
import bsddb
print "Done!"

$ python -i test.py
dbimporter: item: *db* args: () keywords: {}
Accepted *db*
dbimporter: item: /c/steve/Projects/Python/dbimp args: () keywords: {}
dbimporter: item: /c/steve/Projects/Python/dbimp/c args: () keywords: {}
dbimporter: item: /c/steve/Projects/Python/dbimp/\code args: () keywords: {}
dbimporter: item: /usr/lib/python24.zip args: () keywords: {}
dbimporter: item: /usr/lib/python2.4 args: () keywords: {}
dbimporter: item: /usr/lib/python2.4/plat-cygwin args: () keywords: {}
dbimporter: item: /usr/lib/python2.4/lib-tk args: () keywords: {}
dbimporter: item: /usr/lib/python2.4/lib-dynload args: () keywords: {}
dbimporter: item: /usr/lib/python2.4/site-packages args: () keywords: {}
dbimporter: item: /usr/lib/python2.4/site-packages/a args: () keywords: {}
dbimporter: item: /usr/lib/python2.4/site-packages/a/b args: () keywords: {}
dbimporter: item: /usr/lib/python2.4/site-packages/a/b/c args: ()
keywords: {}
found smtplib in db
load_module: smtplib
found socket in db
load_module: socket
socket loaded: <module 'socket' from 'db:socket'> pkg: 0
found rfc822 in db
load_module: rfc822
rfc822 loaded: <module 'rfc822' from 'db:rfc822'> pkg: 0
found base64 in db
load_module: base64
base64 loaded: <module 'base64' from 'db:base64'> pkg: 0
found hmac in db
load_module: hmac
hmac loaded: <module 'hmac' from 'db:hmac'> pkg: 0
dbimporter: item: /usr/lib/python2.4/email args: () keywords: {}
found random in db
load_module: random
random loaded: <module 'random' from 'db:random'> pkg: 0
found quopri in db
load_module: quopri
quopri loaded: <module 'quopri' from 'db:quopri'> pkg: 0
smtplib loaded: <module 'smtplib' from 'db:smtplib'> pkg: 0
found ftplib in db
load_module: ftplib
dbimporter: item: /usr/lib/python2.4/site-packages/PIL args: () keywords: {}
dbimporter: item: /usr/lib/python2.4/site-packages/piddle args: ()
keywords: {}
ftplib loaded: <module 'ftplib' from 'db:ftplib'> pkg: 0
found fileinput in db
load_module: fileinput
fileinput loaded: <module 'fileinput' from 'db:fileinput'> pkg: 0
first import
found bsddb in db
load_module: bsddb
found weakref in db
load_module: weakref
weakref loaded: <module 'weakref' from 'db:weakref'> pkg: 0
second import
Done!


So it's clearly something pretty funky. It now "works" (for some value
of "work" wiht both MySQL and sqlite. I hope I have this sorted out
before PyCon ... I'm currently a bit confused!

regards
Steve
--
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/
Holden Web LLC +1 703 861 4237 +1 800 494 3119

Jul 18 '05 #1
5 2435
Steve Holden wrote:
This is even stranger: it makes it if I import the module a second time:


[second import seems to succeed]

Maybe you are experiencing some version confusion? What you describe looks
much like the normal Python 2.3 behaviour (with no import hook involved)
whereas you seem to operate on the 2.4 library.
A partially initialized module object is left behind in sys.modules and seen
by further import attempts.

$ cat arbitrary.py

import not_there

def f():
print "you ain't gonna see that"

$ python
Python 2.3.3 (#1, Apr 6 2004, 01:47:39)
[GCC 3.3.3 (SuSE Linux)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
import arbitrary Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "arbitrary.py", line 2, in ?
import not_there
ImportError: No module named not_there import arbitrary
arbitrary.f() Traceback (most recent call last):
File "<stdin>", line 1, in ?
AttributeError: 'module' object has no attribute 'f'
I have no experience with import hooks, but for normal imports that has been
fixed in Python 2.4:

$ py24
Python 2.4 (#5, Jan 4 2005, 10:14:01)
[GCC 3.3.3 (SuSE Linux)] on linux2
Type "help", "copyright", "credits" or "license" for more information. import arbitrary Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "arbitrary.py", line 2, in ?
import not_there
ImportError: No module named not_there import arbitrary Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "arbitrary.py", line 2, in ?
import not_there
ImportError: No module named not_there


Peter

Jul 18 '05 #2
Peter Otten wrote:
Steve Holden wrote:

This is even stranger: it makes it if I import the module a second time:

[second import seems to succeed]

Maybe you are experiencing some version confusion? What you describe looks
much like the normal Python 2.3 behaviour (with no import hook involved)
whereas you seem to operate on the 2.4 library.
A partially initialized module object is left behind in sys.modules and seen
by further import attempts.

I agree that this is 2.3-like behavior, but Python cannot lie ...

sholden@dellboy ~/Projects/Python/dbimp
$ python
Python 2.4 (#1, Dec 4 2004, 20:10:33)
[GCC 3.3.3 (cygwin special)] on cygwin
Type "help", "copyright", "credits" or "license" for more information.
$ cat arbitrary.py

import not_there

def f():
print "you ain't gonna see that"

$ python
Python 2.3.3 (#1, Apr 6 2004, 01:47:39)
[GCC 3.3.3 (SuSE Linux)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
import arbitrary
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "arbitrary.py", line 2, in ?
import not_there
ImportError: No module named not_there
import arbitrary
arbitrary.f()
Traceback (most recent call last):
File "<stdin>", line 1, in ?
AttributeError: 'module' object has no attribute 'f'
I have no experience with import hooks, but for normal imports that has been
fixed in Python 2.4:

$ py24
Python 2.4 (#5, Jan 4 2005, 10:14:01)
[GCC 3.3.3 (SuSE Linux)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
import arbitrary
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "arbitrary.py", line 2, in ?
import not_there
ImportError: No module named not_there
import arbitrary


Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "arbitrary.py", line 2, in ?
import not_there
ImportError: No module named not_there
Peter

$ cat arbitrary.py
import not_there

def f():
print "you ain't gonna see that"
sholden@dellboy ~/Projects/Python/dbimp
$ python
Python 2.4 (#1, Dec 4 2004, 20:10:33)
[GCC 3.3.3 (cygwin special)] on cygwin
Type "help", "copyright", "credits" or "license" for more information. import arbitrary Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "arbitrary.py", line 1, in ?
import not_there
ImportError: No module named not_there import arbitrary Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "arbitrary.py", line 1, in ?
import not_there
ImportError: No module named not_there


Yup, looks like 2.4 (despite this funny cygwin stuff, could that make a
difference).

Let me try it under Windows [ferkle, ferkle ...]

Does the same thing there.

This problem also seems to depend what's already loaded. I wrote a
program to write a test program that looks like this:

import dbimp
dbimp.install()

print "Trying aifc"
try:
import aifc
except:
print "%Failed: aifc"
print "Trying anydbm"
try:
import anydbm
except:
print "%Failed: anydbm"
print "Trying asynchat"
try:
import asynchat
except:
print "%Failed: asynchat"

...

import dbimp
dbimp.install()

print "Trying aifc"
try:
import aifc
except:
print "%Failed: aifc"
print "Trying anydbm"
try:
import anydbm
except:
print "%Failed: anydbm"
print "Trying asynchat"
try:
import asynchat
except:
print "%Failed: asynchat"

The two platforms give expectedly close results. I'm storing compiled
code, so a version incompatibility would be a problem, I agree, but I
have checked the program that loaded the database, and loaded it again
from the Windows source rather than the CygWin source, just to see
whether there were any unnoticed platform dependencies. The results were
exactly the same using either library, and Windows and Cygwin showed
only minor variations.

exhaustCyg24.txt:%Failed: bsddb.dbtables
exhaustCyg24.txt:%Failed: bsddb.test.test_all
exhaustCyg24.txt:%Failed: bsddb.test.test_associate
exhaustCyg24.txt:%Failed: bsddb.test.test_basics
exhaustCyg24.txt:%Failed: bsddb.test.test_compat
exhaustCyg24.txt:%Failed: bsddb.test.test_dbobj
exhaustCyg24.txt:%Failed: bsddb.test.test_dbshelve
exhaustCyg24.txt:%Failed: bsddb.test.test_dbtables
exhaustCyg24.txt:%Failed: bsddb.test.test_env_close
exhaustCyg24.txt:%Failed: bsddb.test.test_get_none
exhaustCyg24.txt:%Failed: bsddb.test.test_join
exhaustCyg24.txt:%Failed: bsddb.test.test_lock
exhaustCyg24.txt:%Failed: bsddb.test.test_misc
exhaustCyg24.txt:%Failed: bsddb.test.test_queue
exhaustCyg24.txt:%Failed: bsddb.test.test_recno
exhaustCyg24.txt:%Failed: bsddb.test.test_thread
exhaustCyg24.txt:%Failed: tzparse
exhaustWin24.txt:%Failed: bsddb.dbtables
exhaustWin24.txt:%Failed: bsddb.test.test_all
exhaustWin24.txt:%Failed: bsddb.test.test_associate
exhaustWin24.txt:%Failed: bsddb.test.test_basics
exhaustWin24.txt:%Failed: bsddb.test.test_compat
exhaustWin24.txt:%Failed: bsddb.test.test_dbobj
exhaustWin24.txt:%Failed: bsddb.test.test_dbshelve
exhaustWin24.txt:%Failed: bsddb.test.test_dbtables
exhaustWin24.txt:%Failed: bsddb.test.test_env_close
exhaustWin24.txt:%Failed: bsddb.test.test_get_none
exhaustWin24.txt:%Failed: bsddb.test.test_join
exhaustWin24.txt:%Failed: bsddb.test.test_lock
exhaustWin24.txt:%Failed: bsddb.test.test_misc
exhaustWin24.txt:%Failed: bsddb.test.test_queue
exhaustWin24.txt:%Failed: bsddb.test.test_recno
exhaustWin24.txt:%Failed: bsddb.test.test_thread
exhaustWin24.txt:%Failed: pty
exhaustWin24.txt:%Failed: rlcompleter
exhaustWin24.txt:%Failed: tzparse

As a workaround I can for the moment just omit everything that show the
least suspicion of not working from the database, but I'd really rather
know what's failing.

If you have any clues I'd be grateful.

regards
Steve
--
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/
Holden Web LLC +1 703 861 4237 +1 800 494 3119
Jul 18 '05 #3
Steve Holden wrote:
Peter Otten wrote:
Steve Holden wrote:

This is even stranger: it makes it if I import the module a second time:


[second import seems to succeed]

Maybe you are experiencing some version confusion? What you describe
looks
much like the normal Python 2.3 behaviour (with no import hook involved)
whereas you seem to operate on the 2.4 library.
A partially initialized module object is left behind in sys.modules
and seen
by further import attempts.

I agree that this is 2.3-like behavior, but Python cannot lie ...

sholden@dellboy ~/Projects/Python/dbimp
$ python
Python 2.4 (#1, Dec 4 2004, 20:10:33)
[GCC 3.3.3 (cygwin special)] on cygwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

Just to make things simpler, and (;-) to appeal to a wider audience,
here is a program that doesn't use database at all (it loads the entire
standard library into a dict) and still shows the error.

What *I* would like to know is: who is allowing the import of bsddb.os,
thereby somehow causing the code of the os library module to be run a
second time.

#
# Establish standard library in dict
#
import os
import glob
import sys
import marshal
import new

def importpy(dct, path, modname, package):
c = compile(file(path).read(), path, "exec")
dct[modname] = (marshal.dumps(c), package, path)
if package:
print "Package", modname, path
else:
print "Module", modname, path

def importall(dct, path, modlist):
os.chdir(path)
for f in glob.glob("*"):
if os.path.isdir(f):
fn = os.path.join(path, f, "__init__.py")
if os.path.exists(fn):
ml = modlist + [f]
importpy(dct, fn, ".".join(ml), 1)
importall(dct, os.path.join(path, f), ml)
elif f.endswith('.py') and '.' not in f[:-3] and f !=
"__init__.py":
importpy(dct, os.path.join(path, f),
".".join(modlist+[f[:-3]]), 0)

class dbimporter(object):

def __init__(self, item, *args, **kw):
##print "dbimporter: item:", item, "args:", args, "keywords:", kw
if item != "*db*":
raise ImportError
print "Accepted", item

def find_module(self, fullname, path=None):
print "find_module:", fullname, "from", path
if fullname not in impdict:
#print "Bailed on", fullname
return None
else:
print "found", fullname, "in db"
return self

def load_module(self, modname):
print "load_module:", modname
if modname in sys.modules:
return sys.modules[modname]
try:
row = impdict[modname]
except KeyError:
#print modname, "not found in db"
raise ImportError, "DB module %s not found in modules"
code, package, path = row
code = marshal.loads(code)
module = new.module(modname)
sys.modules[modname] = module
module.__name__ = modname
module.__file__ = path # "db:%s" % modname
module.__loader__ = dbimporter
if package:
module.__path__ = sys.path
exec code in module.__dict__
print modname, "loaded:", repr(module), "pkg:", package
return module

def install():
sys.path_hooks.append(dbimporter)
sys.path_importer_cache.clear() # probably not necessary
sys.path.insert(0, "*db*") # probably not needed with a metea-path
hook?

if __name__ == "__main__":
impdict = {}
for path in sys.argv[1:]:
importall(impdict, path, [])
install()
import bsddb

Running this against a copy of the Python 2.4 standard library in C:\Lib
gives me

[...]
Module _strptime C:\Lib\_strptime.py
Module _threading_local C:\Lib\_threading_local.py
Module __future__ C:\Lib\__future__.py
Accepted *db*
find_module: bsddb from None
found bsddb in db
load_module: bsddb
find_module: bsddb._bsddb from None
find_module: bsddb.sys from None
find_module: bsddb.os from None
find_module: bsddb.nt from None
find_module: bsddb.ntpath from None
find_module: bsddb.stat from None
Traceback (most recent call last):
File "C:\Steve\Projects\Python\dbimp\dictload.py", line 79, in ?
import bsddb
File "C:\Steve\Projects\Python\dbimp\dictload.py", line 65, in
load_module
exec code in module.__dict__
File "C:\Lib\bsddb\__init__.py", line 62, in ?
import sys, os
File "C:\Python24\lib\os.py", line 133, in ?
from os.path import (curdir, pardir, sep, pathsep, defpath, extsep,
altsep,
ImportError: No module named path

The 2.3 bsddb library doesn't cause the same problems (and even loads
into 2.4 quite nicely). Lots of modules *will* import, and most packages
don't seem to cause problems. Anyone give me a pointer here?

regards
Steve
--
Meet the Python developers and your c.l.py favorites March 23-25
Come to PyCon DC 2005 http://www.python.org/pycon/2005/
Steve Holden http://www.holdenweb.com/
Jul 18 '05 #4
Steve Holden <st***@holdenweb.com> writes:
What *I* would like to know is: who is allowing the import of bsddb.os,
thereby somehow causing the code of the os library module to be run a
second time.
I would guess (without actually running the code) that this part is
responsible:
if package:
module.__path__ = sys.path


You usually should initialize a package's __path__ to an empty list.
The __init__ module will take care of modifying it if necessary.

Bernhard

--
Intevation GmbH http://intevation.de/
Skencil http://skencil.org/
Thuban http://thuban.intevation.org/
Jul 18 '05 #5
Bernhard Herzog wrote:
Steve Holden <st***@holdenweb.com> writes:

What *I* would like to know is: who is allowing the import of bsddb.os,
thereby somehow causing the code of the os library module to be run a
second time.

I would guess (without actually running the code) that this part is
responsible:

if package:
module.__path__ = sys.path

You usually should initialize a package's __path__ to an empty list.
The __init__ module will take care of modifying it if necessary.

Bernhard

Thanks, Bernhard, that appeared to fix the problem. The error certainly
went away, anyway ... now I can do more experimentation.

regards
Steve
--
Meet the Python developers and your c.l.py favorites March 23-25
Come to PyCon DC 2005 http://www.python.org/pycon/2005/
Steve Holden http://www.holdenweb.com/
Jul 18 '05 #6

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
by: James | last post by:
I have a from with 2 fields: Company & Name Depening which is completed, one of the following queries will be run: if($Company){ $query = "Select C* From tblsample Where ID = $Company...
5
by: Scott D | last post by:
I am trying to check and see if a field is posted or not, if not posted then assign $location which is a session variable to $location_other. If it is posted then just assign it to...
2
by: Nick | last post by:
Can someone please tell me how to access elements from a multiple selection list? From what ive read on other posts, this is correct. I keep getting an "Undefined variable" error though... Form...
2
by: Alexander Ross | last post by:
I have a variable ($x) that can have 50 different (string) values. I want to check for 7 of those values and do something based on it ... as I see it I have 2 options: 1) if (($x=="one") ||...
0
by: Dan Foley | last post by:
This script runs fine, but I'd like to know why it's so slow.. Thanks for any help out there on how i can make it faster (it might take up to 5 min to write these 3 export files whith 15 records...
5
by: Lee Redeem | last post by:
Hi there I've created abd uploaded this basic PHP script: <html> <head> <title>PHP Test</title> </head> <body> <H1 align="center">
5
by: christopher vogt | last post by:
Hi, i'm wondering if there is something like $this-> to call a method inside another method of the same class without using the classname in front. I actually use class TEST { function...
6
by: Phil Powell | last post by:
Ok guys, here we go again! SELECT s.nnet_produkt_storrelse_navn FROM nnet_produkt_storrelse s, nnet_produkt_varegruppe v, nnet_storrelse_varegruppe_assoc sv, nnet_produkt p WHERE...
1
by: Michel | last post by:
a site like this http://www.dvdzone2.com/dvd Can you make it in PHP and MySQL within 6 weeks? If so, send me your price 2 a r a (at) p a n d o r a . b e
1
by: Acorn Tutors | last post by:
What Im trying to do is select a single entry from a table called HEADPIC where the current date and time are after a datetime value in a field called POSTED and before a datetime value in a field...
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.