Solution for architecure dependence in Numeric ?

[Johannes Nix |Johannes.Nix]

Hi,

I have a tricky problem with Numeric. Some time ago, I have generated
a huge and complex data structure, and stored it using the cPickle
module. Now I want to evaluate it quickly again on a workstation
cluster with 64-Bit Opteron CPUs - I have no more than three days to
do this. Compiling Python and running Numeric has been no problem at
all. However, I get an error message when accessing the data pickled
before. (I can load it regularly on 32 bit computers, but it is a
quite complex data object, so I really don't want to store every
element as ASCII data). The problem seems to be with 64 Bit integers
(with 32-bit-floats, no problem was observed).

This looks like that (from the Unix command shell):
jnix@32bithost: ~> python ~/python/test_npickle.py -dump test.pck
jnix@32bithost: ~> python ~/python/test_npickle.py test.pck
[0 1 2 3 4 5 6 7 8 9]
jnix@32bithost: ~> ssh 64bithost python ~/python/test_npickle.py test.pck
Traceback (most recent call last):
File "/home/jnix/python/test_npickle.py ", line 16, in ?
a = cPickle.load(fi le(filename))
File "/home/jnix/lib/python2.4/SuSE-9.0/x86_64/Numeric/Numeric.py", line 520, in array_construct or
x.shape = shape
ValueError: ('total size of new array must be unchanged', <function array_construct or at 0x2a960b0b18>, ((10,), 'l', '\x00\x00\x00\x 00\x01\x00\x00\ x00\x02\x00\x00 \x00\x03\x00\x0 0\x00\x04\x00\x 00\x00\x05\x00\ x00\x00\x06\x00 \x00\x00\x07\x0 0\x00\x00\x08\x 00\x00\x00\t\x0 0\x00\x00', 1))
also I get:
jnix@32bithost: ~> python -c "import Numeric; print Numeric.arange( 0).itemsize()"
4
jnix@64bithost: ~> python -c "import Numeric; print Numeric.arange( 0).itemsize()"
8
The script used to produce the example above is:

-------------------------------------------------------------------------
#/usr/bin/python
# -*- coding: latin1 -*-

import Numeric
import cPickle
import sys

if len(sys.argv) > 1 and sys.argv[1] == '-dump':
filename = sys.argv[2]
binary=1
a = Numeric.arange( 10)
cPickle.dump(a, file(filename,' w',binary))

else:
filename = sys.argv[1]
a = cPickle.load(fi le(filename))
print a
---------------------------------------------------------------------
So what would you suggest ? Can I hack Numeric to assume non-native
32 bit integer numbers ?

Many thanks for any help,

Johannes

[Michael Spencer]

"Johannes Nix|Johannes.Ni x"@uni-oldenburg.de wrote:

Hi,

I have a tricky problem with Numeric. Some time ago, I have generated
a huge and complex data structure, and stored it using the cPickle
module. Now I want to evaluate it quickly again on a workstation
cluster with 64-Bit Opteron CPUs - I have no more than three days to
do this. Compiling Python and running Numeric has been no problem at
all. However, I get an error message when accessing the data pickled
before. (I can load it regularly on 32 bit computers, but it is a
quite complex data object, so I really don't want to store every
element as ASCII data). The problem seems to be with 64 Bit integers
(with 32-bit-floats, no problem was observed).

This looks like that (from the Unix command shell):
jnix@32bithost: ~> python ~/python/test_npickle.py -dump test.pck
jnix@32bithost: ~> python ~/python/test_npickle.py test.pck
[0 1 2 3 4 5 6 7 8 9]
jnix@32bithost: ~> ssh 64bithost python ~/python/test_npickle.py test.pck
Traceback (most recent call last):
File "/home/jnix/python/test_npickle.py ", line 16, in ?
a = cPickle.load(fi le(filename))
File "/home/jnix/lib/python2.4/SuSE-9.0/x86_64/Numeric/Numeric.py", line 520, in array_construct or
x.shape = shape
ValueError: ('total size of new array must be unchanged', <function array_construct or at 0x2a960b0b18>, ((10,), 'l', '\x00\x00\x00\x 00\x01\x00\x00\ x00\x02\x00\x00 \x00\x03\x00\x0 0\x00\x04\x00\x 00\x00\x05\x00\ x00\x00\x06\x00 \x00\x00\x07\x0 0\x00\x00\x08\x 00\x00\x00\t\x0 0\x00\x00', 1))
also I get:
jnix@32bithost: ~> python -c "import Numeric; print Numeric.arange( 0).itemsize()"
4
jnix@64bithost: ~> python -c "import Numeric; print Numeric.arange( 0).itemsize()"
8
The script used to produce the example above is:

-------------------------------------------------------------------------
#/usr/bin/python
# -*- coding: latin1 -*-

import Numeric
import cPickle
import sys

if len(sys.argv) > 1 and sys.argv[1] == '-dump':
filename = sys.argv[2]
binary=1
a = Numeric.arange( 10)
cPickle.dump(a, file(filename,' w',binary))

else:
filename = sys.argv[1]
a = cPickle.load(fi le(filename))
print a
---------------------------------------------------------------------
So what would you suggest ? Can I hack Numeric to assume non-native
32 bit integer numbers ?

Many thanks for any help,

Johannes

It might be worth posting to the Numeric mailing list, mirrored at
http://news.gmane.org/gmane.comp.python.numeric.general

[Paul Rubin]

Johannes Nix |J***********@u ni-oldenburg.de> writes:

So what would you suggest ? Can I hack Numeric to assume non-native
32 bit integer numbers ?

I think it would be better to fix the pickle/cPickle module to
correctly unpack the 32-bit ints on your 64 bit machine.