468,483 Members | 2,611 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 468,483 developers. It's quick & easy.

Database module & multithreading

Hi guys!!!
Just one quick question... Which database module should I use when I
want to use multi threading as my application requires lots of data
from internet ???? I also want this database module to be fast, simple
n efficient, in any case multi threading capabilities are # 1
requirement.
Thanx Guys

Anil

Mar 10 '07 #1
19 2541
jupiter wrote:
Hi guys!!!
Just one quick question... Which database module should I use when I
want to use multi threading as my application requires lots of data
from internet ???? I also want this database module to be fast, simple
n efficient, in any case multi threading capabilities are # 1
requirement.
Thanx Guys

Anil
See the threadsafety flag of Python DB-API V2 compliant database modules.
0 = no thread safe
1 = threads may share module
2 = + may share connections
3 = + may share cursors

A+

Laurent.

Mar 10 '07 #2
On Mar 10, 7:29 pm, Laurent Pointal <laurent.poin...@wanadoo.fr>
wrote:
jupiter wrote:
Hi guys!!!
Just one quick question... Which database module should I use when I
want to use multi threading as my application requires lots of data
from internet ???? I also want this database module to be fast, simple
n efficient, in any case multi threading capabilities are # 1
requirement.
Thanx Guys
Anil

See the threadsafety flag of Python DB-API V2 compliant database modules.
0 = no thread safe
1 = threads may share module
2 = + may share connections
3 = + may share cursors

A+

Laurent.
Thanx for this pointer buddy! I have done my homework. Some Database
modules are not actively maintained some modules does not work with
Python 2.5. At this moment I am using Sqlite3 which is pretty fast but
it dosent allow me to use multi threading so which database module is
better in terms of multithreading

@

Mar 10 '07 #3
"jupiter" <an***********@gmail.comwrites:
Hi guys!!!
Just one quick question... Which database module should I use when I
want to use multi threading as my application requires lots of data
from internet ???? I also want this database module to be fast, simple
n efficient, in any case multi threading capabilities are # 1
requirement.
For which database server?

--
Jorge Godoy <jg****@gmail.com>
Mar 10 '07 #4
>
Thanx for this pointer buddy! I have done my homework. Some Database
modules are not actively maintained some modules does not work with
Python 2.5. At this moment I am using Sqlite3 which is pretty fast but
it dosent allow me to use multi threading so which database module is
better in terms of multithreading
Actually, I think sqlite3 just requires that each thread have its own
connection and cursor.
Mar 10 '07 #5
On Mar 10, 8:16 pm, Michael Bentley <rmc...@gmail.comwrote:
Thanx for this pointer buddy! I have done my homework. Some Database
modules are not actively maintained some modules does not work with
Python 2.5. At this moment I am using Sqlite3 which is pretty fast but
it dosent allow me to use multi threading so which database module is
better in terms of multithreading

Actually, I think sqlite3 just requires that each thread have its own
connection and cursor.
I tried this approach with

for each new thread
self.c1=sqlite3.connect('urls')
self.c=self.c1.cursor()
check condition
insert if true else ignore

but no success
Mar 10 '07 #6
On Mar 10, 8:16 pm, Michael Bentley <rmc...@gmail.comwrote:
Thanx for this pointer buddy! I have done my homework. Some Database
modules are not actively maintained some modules does not work with
Python 2.5. At this moment I am using Sqlite3 which is pretty fast but
it dosent allow me to use multi threading so which database module is
better in terms of multithreading

Actually, I think sqlite3 just requires that each thread have its own
connection and cursor.
Exception in thread Thread-2:
Traceback (most recent call last):
File "D:\Python25\lib\threading.py", line 460, in __bootstrap
self.run()
File "url_part1.py", line 161, in run
dbacc(self.o, self.dbase).dataenter()
File "url_part1.py", line 64, in dataenter
self.c.execute('insert into '+self.odbase+' values
(?,?,?,?,?,?,?,?,?,?,?,?,
?)',self.odata)
InterfaceError: Error binding parameter 3 - probably unsupported type.

this is the error I am getting................

Mar 10 '07 #7
jupiter <an***********@gmail.comwrote:
...
Python 2.5. At this moment I am using Sqlite3 which is pretty fast but
it dosent allow me to use multi threading so which database module is
better in terms of multithreading
Perhaps psycopg2 (with PostgreSQL as the engine), according to
<http://cheeseshop.python.org/pypi/psycopg2/2.0.2it is specifically
"designed for heavily multi-threaded applications that create and
destroy lots of cursors and make a conspicuous number of concurrent
INSERTs or UPDATEs".
Alex
Mar 10 '07 #8
On Mar 10, 8:42 pm, a...@mac.com (Alex Martelli) wrote:
jupiter <anil.jupit...@gmail.comwrote:

...
Python 2.5. At this moment I am using Sqlite3 which is pretty fast but
it dosent allow me to use multi threading so which database module is
better in terms of multithreading

Perhaps psycopg2 (with PostgreSQL as the engine), according to
<http://cheeseshop.python.org/pypi/psycopg2/2.0.2it is specifically
"designed for heavily multi-threaded applications that create and
destroy lots of cursors and make a conspicuous number of concurrent
INSERTs or UPDATEs".

Alex
conn = psycopg2.connect("dbname=test user=test")

Traceback (most recent call last):
File "<pyshell#8>", line 1, in <module>
conn = psycopg2.connect("dbname=test user=test")
OperationalError: could not connect to server: Connection refused
(0x0000274D/10061)
Is the server running on host "???" and accepting
TCP/IP connections on port 5432?

this is the error I am getting with psycopg2 I want to access a local
file

Mar 10 '07 #9
"jupiter" <an***********@gmail.comwrites:
conn = psycopg2.connect("dbname=test user=test")

Traceback (most recent call last):
File "<pyshell#8>", line 1, in <module>
conn = psycopg2.connect("dbname=test user=test")
OperationalError: could not connect to server: Connection refused
(0x0000274D/10061)
Is the server running on host "???" and accepting
TCP/IP connections on port 5432?

this is the error I am getting with psycopg2 I want to access a local
file
PostgreSQL is a server, it access the files. You talk to the RDBMS server.
Is it running? Is it accepting connections from your host?

--
Jorge Godoy <jg****@gmail.com>
Mar 10 '07 #10
Michael Bentley wrote:
>>
Thanx for this pointer buddy! I have done my homework. Some Database
modules are not actively maintained some modules does not work with
Python 2.5. At this moment I am using Sqlite3 which is pretty fast but
it dosent allow me to use multi threading so which database module is
better in terms of multithreading


Actually, I think sqlite3 just requires that each thread have its own
connection and cursor.
SQlite is thread-safe but not concurrent; only one thread
at a time can be doing database updates to a database. The whole
database is locked during each transaction that changes the database.
However, multiple reads (SELECTs) can be done simultaneously. So
mostly-read databases can still get some concurrency going.

The question is, how many transactions per second do you
need to process? This sounds like someone writing their first
database program, not someone who needs thousands of transactions
per second and has a farm of dedicated servers to support them.
Any of the usual choices should work: SQLite, MySQL, or Postgres.

As for "some modules not being maintained", it really is sad
that MySQLdb is kind of behind. If you're running Windows and need
MySQL, you're either stuck with Python 2.4, or have to get a 2.5 version
from a World of Warcraft guild. The source version of MySQLdb, though,
does build properly on Red Hat Fedora Core 6 for Python 2.5.

John Nagle
Mar 10 '07 #11
In article <54*****************@newssvr21.news.prodigy.net> , John Nagle wrote:
As for "some modules not being maintained", it really is sad
that MySQLdb is kind of behind. If you're running Windows and need
MySQL, you're either stuck with Python 2.4
Looks like that's changed:

http://sourceforge.net/project/showf...ckage_id=15775
Mar 10 '07 #12
Jon Ribbens wrote:
In article <54*****************@newssvr21.news.prodigy.net> , John Nagle wrote:
> As for "some modules not being maintained", it really is sad
that MySQLdb is kind of behind. If you're running Windows and need
MySQL, you're either stuck with Python 2.4


Looks like that's changed:

http://sourceforge.net/project/showf...ckage_id=15775
Yes, that's a step forward.

Although it's an ".egg" file. If renamed to ".zip" and unpacked, it
seems to work.

Is the "egg" packaging gimmick considered mainstream Python, or just some
wierd idea from the "Peak" people? It seems to complicate installation
without adding much value.

John Nagle
Mar 10 '07 #13
On Mar 10, 6:14 am, "jupiter" <anil.jupit...@gmail.comwrote:
Just one quick question... Which database module
should I use when I want to use multi threading
as my application requires lots of data from
internet ???? I also want this database module
to be fast, simple n efficient, in any case
multi threading capabilities are # 1 requirement.
Any discussion of multithreaded database access should also address
isolation levels and locking scenarios. Have a look at the bottom of
http://projects.amor.org/docs/dejavu...tml#comparison
.... especially the footnotes. Note that Firebird and SQLite
immediately raise errors on locking conflicts, so you need to write
your own retry mechanisms. SQLServer and MySQL, in contrast, have
their own (configurable) timeout settings.
Robert Brewer
System Architect
Amor Ministries
fu******@amor.org

Mar 10 '07 #14
In article <2u******************@newssvr17.news.prodigy.net >, John Nagle wrote:
Is the "egg" packaging gimmick considered mainstream Python, or just some
wierd idea from the "Peak" people? It seems to complicate installation
without adding much value.
I don't know, but stock Python 2.5 seems to stick mysterious '.egg'
files in the site-packages directory when you install things.
Mar 11 '07 #15
En Sat, 10 Mar 2007 22:44:43 -0300, Jon Ribbens
<jo********@unequivocal.co.ukescribió:
I don't know, but stock Python 2.5 seems to stick mysterious '.egg'
files in the site-packages directory when you install things.
Which "stock" Python? Not the one from www.python.org... If you are
talking about some Linux distro, ask the packagers...

--
Gabriel Genellina

Mar 11 '07 #16
In article <ma***************************************@python. org>, Gabriel Genellina wrote:
>I don't know, but stock Python 2.5 seems to stick mysterious '.egg'
files in the site-packages directory when you install things.

Which "stock" Python? Not the one from www.python.org...
Yes, the one from www.python.org.
Mar 11 '07 #17
En Sun, 11 Mar 2007 07:55:15 -0300, Jon Ribbens
<jo********@unequivocal.co.ukescribió:
In article <ma***************************************@python. org>,
Gabriel Genellina wrote:
>>I don't know, but stock Python 2.5 seems to stick mysterious '.egg'
files in the site-packages directory when you install things.

Which "stock" Python? Not the one from www.python.org...

Yes, the one from www.python.org.
Those .egg must come from other installed packages...

--
Gabriel Genellina

Mar 11 '07 #18
In article <ma***************************************@python. org>, Gabriel Genellina wrote:
>>>I don't know, but stock Python 2.5 seems to stick mysterious '.egg'
files in the site-packages directory when you install things.

Which "stock" Python? Not the one from www.python.org...

Yes, the one from www.python.org.

Those .egg must come from other installed packages...
My mistake, they are ".egg-info". And they certainly do come from
stock Python 2.5 from www.python.org and not anything else. They
are installed by distutils/command/install_egg_info, which is
automatically called by 'python setup.py install'.
Mar 11 '07 #19
En Sun, 11 Mar 2007 09:19:51 -0300, Jon Ribbens
<jo********@unequivocal.co.ukescribió:
In article <ma***************************************@python. org>,
Gabriel Genellina wrote:
>>>>I don't know, but stock Python 2.5 seems to stick mysterious '.egg'
files in the site-packages directory when you install things.

Which "stock" Python? Not the one from www.python.org...

Yes, the one from www.python.org.

Those .egg must come from other installed packages...

My mistake, they are ".egg-info". And they certainly do come from
stock Python 2.5 from www.python.org and not anything else. They
are installed by distutils/command/install_egg_info, which is
automatically called by 'python setup.py install'.
Ah! Those files contain metadata about the installed packages. See
http://docs.python.org/whatsnew/whatsnew25.html , but I think the actual
version is http://www.python.org/dev/peps/pep-0345/ . Anyway, nowhere says
that the PKG-INFO files are actually copied as packagename.egg-info

--
Gabriel Genellina

Mar 11 '07 #20

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

8 posts views Thread by Michael | last post: by
47 posts views Thread by mihai | last post: by
6 posts views Thread by a | last post: by
reply views Thread by joop renes | last post: by
reply views Thread by Gabriel Genellina | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.