Obscure bug in pyPgSQL

[Frank Millman]

Hi all

Below is the text of a message I was about to send in connection with
an obscure problem. I have now almost got to the bottom of it, and am
fairly confident that it is a bug in pyPgSQL. I hope this is the
correct place to report such bugs. If not, please advise the correct
forum.

I have left the original message intact as it provides important
information. At the bottom, I have recorded the new information that I
have discovered.

-------------------------------------------------------------------------------

Here is an obscure problem. I am not sure if I can explain the
symptoms properly, but I will do my best.

I have a Linux box (RH9) and an MSW box (W2K Pro) networked to each
other. Both are running Python 2.3.3.

Linux is the server. It is running PostgreSQL, which listens on port
5432. It is also running a socket server program that I have written
using socket.socket() , listening on port 6543. I am using pyPgSQL to
connect to the database.

MSW is the client. From time to time it sends messages to my socket
server. Messages are strings consisting of a single digit identifier,
followed by a cPickle'd tuple containing various data fields. The
client uses cPickle.dumps to prepare the string before sending, and
the server uses cPickle.loads to unpickle it after receiving.

Most times it works perfectly, but in one particular situation the
server terminates with an error. The error is consistent - it always
gives the same error resulting from the same message. However, I do
not think it is caused by the message itself, as exactly the same
message is sent in different circumstances without generating an
error.

The traceback is as follows -

File "./chag_server.py" , line 190, in ?
dat = cPickle.loads(d ata[1:])
File "/usr/local/lib/python2.3/site-packages/pyPgSQL/libpq/__init__.py",
line 43, in _I8
return PgInt8(value)
NameError: ("global name 'PgInt8' is not defined", <function _I8
at 0x403a04c4>, ('17',))

The error occurs at the point of unpickling - I have moved this
around, and it always fails at this point.

A typical example of the tuple being unpickled is (2,'Ar',17,1) - it
is all normal strings and integers.

I have no idea why pyPgSQL is involved at all - I am certainly not
invoking it directly. In fact the error comes from my server program,
and that does not even import pyPgSQL.

Here are the relevant lines from pyPgSQL/libpq/__init__.py -

from libpq import *
from libpq import __version__

HAVE_LONG_LONG_ SUPPORT = dir().count('Pg Int8') == 1

#---------------------------------------------------------------+
# Add support to pickle the following pyPgSQL objects: |
# PgInt2, PgInt8, PgBoolean |
#---------------------------------------------------------------+

def _B(value):
return PgBoolean(value )

def _I2(value):
return PgInt2(value)

def _I8(value):
return PgInt8(value)

Somehow pyPgSQL is responding to my unpickling command, and raising an
error because PgInt8 is not defined.

Here is another fact, which may provide a clue or may add confusion.
If I run exactly the same program from the Linux box, using it as the
client as well as the server, the error does not appear. I connect via
the external IP address, not via 127.0.0.1, so I would have thought it
would behave the same, but it does not.

-------------------------------------------------------------------------------

Ok, here is the reason. I mentioned above that a typical tuple
consists of (2,'Ar',17,1). I have found that under some circumstances
the 3rd element is of type 'long' instead of type 'int', and this is
what generates the error.

Whether it is a long or an int is determined by pyPgSQL itself. It is
a column value, where the column is of type 'serial', which means that
PostgreSQL will generate a next number if the row is being inserted.
If the row exists, the column value is returned along with all the
other columns, and is returned as type int. If the row does not exist,
I insert it, and then to retrieve the id generated, I have the
following piece of code -

cur = db.cursor()
cur.execute("se lect currval('%s_%s_ Seq')" % (tableid,column id) )
rowid = cur.fetchone()[0]

In this case, rowid contains the correct value, but is of type long.

So this raises various questions -

Why is PgInt8 not defined?
Why does pyPgSQL affect my server program, which does not import any
of pyPgSQL?
Why does the error only occur when I connect from a remote machine?

Now that I have identified the problem, I can work around it, so this
has become a matter of correctness rather than a matter of urgency.

For those that have read this far, thanks for your patience.

Frank Millman

[Dennis Lee Bieber]

On 6 Aug 2004 00:31:39 -0700, fr***@chagford. com (Frank Millman)
declaimed the following in comp.lang.pytho n:

Somehow pyPgSQL is responding to my unpickling command, and raising an
error because PgInt8 is not defined.
I'd suspect the pickled data itself has a reference to the
PgInt8(), AND I'd suspect that is an OS-dependent function -- did you
notice that pseudo-constant about HAVE_LONG_LONG. ..?
Here is another fact, which may provide a clue or may add confusion.
If I run exactly the same program from the Linux box, using it as the
client as well as the server, the error does not appear. I connect via
the external IP address, not via 127.0.0.1, so I would have thought it
would behave the same, but it does not.
Which would be reasonable if the changed OS doesn't use/need
PgInt8() when it does the pickling.
Why is PgInt8 not defined?
Because the OS/compiler may have supported the data type
natively?
Why does pyPgSQL affect my server program, which does not import any
of pyPgSQL?
Hypothesis: The pickled object includes a reference to the
data-type?
Why does the error only occur when I connect from a remote machine?

See comment regarding definition...
-- =============== =============== =============== =============== == <
wl*****@ix.netc om.com | Wulfraed Dennis Lee Bieber KD6MOG <
wu******@dm.net | Bestiaria Support Staff <
=============== =============== =============== =============== == <
Home Page: <http://www.dm.net/~wulfraed/> <
Overflow Page: <http://wlfraed.home.ne tcom.com/> <

[Gerhard Häring]

Frank Millman wrote:

Hi all

Below is the text of a message I was about to send in connection with
an obscure problem. I have now almost got to the bottom of it, and am
fairly confident that it is a bug in pyPgSQL. I hope this is the
correct place to report such bugs. If not, please advise the correct
forum. [...]

Please file a bug on the pyPgSQL bug tracker on Sourceforge. This way,
it's less likely that this will get lost:

http://sourceforge.net/projects/pypgsql

-- Gerhard