xml marshal of general (but non Python standard) class

[syd]

Hello all,

In my project, I have container classes holding lists of item classes.
For example, a container class myLibrary might hold a list of item
classes myNation and associated variables like myNation.name='USA' and
myNation.continent='North America'.

Bottom line, I was hoping to use this structure to marshal the classes
to xml.

However, I've got dozens of unique classes (all subclassing the
container and item classes) with unique variables attached, and I do
not want to write rules for each.

I was looking at the source for generic in xml.marshal (from
xml.marshal import generic) which will dump to xml any configuration of
standard Python data types, for example, a tuple of dictionaries
containing lists of strings. This source just writes a rule for each
data type.

Naively, I would hope that there'd be something where the marshaller
could just look at my data class, see what variables were associated,
and just go from there.

I'm moderately experienced with Python, but by no means an expert, and
I'm not an xml pro, either. Would this project (xml marshal of a new
class) be worth my time? If so, what would be best way to proceed?
Any other thoughts?

import xml.marshal
from xml.marshal import generic
generic.dumps(['thank you','comp.lang.python'])

'<?xml version="1.0"?><marshal><list id="i2"><string>thank
you</string><string>comp.lang.python</string></list></marshal>

[Martin v. Löwis]

> In my project, I have container classes holding lists of item classes.

For example, a container class myLibrary might hold a list of item
classes myNation and associated variables like myNation.name='USA' and
myNation.continent='North America'.

Bottom line, I was hoping to use this structure to marshal the classes
to xml.
The center question here is: why? To read it back in later? I would
recommend to use pickle instead.
I'm moderately experienced with Python, but by no means an expert, and
I'm not an xml pro, either. Would this project (xml marshal of a new
class) be worth my time? If so, what would be best way to proceed?
Any other thoughts?

As a starting point, you should ask yourself why you want this, and
then how you want the XML to look like. If "any XML" is fine, you
can relatively easy dump an object through marshal.generic:

class Foo: .... pass
.... f=Foo()
f.name="Hallo"
f.age=10

xml.marshal.generic.dumps(f)

'<?xml version="1.0"?><marshal><object id="i2" module="__main__"
class="Foo"><tuple></tuple><dictionary
id="i3"><string>age</string><int>10</int><string>name</string><string>Hallo</string></dictionary></object></marshal>'

However, the advantage of this format over pickle might be
questionable.

Regards,
Martin

[syd]

Thank you Martin. I had not considered pickle, and I've done my
research. However, I'm still having problems:

Your foo class (for pickle and xml dumps) works fine for me.

f=Foo()
f.thanksTo='Martin'
f.howMany=100
pickle.dumps(f) "(i__main__\nFoo\np0\n(dp1\nS'thanksTo'\np2\nS'Mar tin'\np3\nsS'howMany'\np4\nI100\nsb."

But for my identifiedPeaks class (for instance), it has trouble. This
class contains a list of "peak" classes IdentifiedPeaks.Peak...
from IdentifiedPeaks import IdentifiedPeaks
identifiedPeaks=IdentifiedPeaks()
identifiedPeaks.read(open('input/analysisReport.txt'))
pickle.dumps(identifiedPeaks) Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/usr/lib/python2.3/pickle.py", line 1386, in dumps
Pickler(file, protocol, bin).dump(obj)
File "/usr/lib/python2.3/pickle.py", line 231, in dump
self.save(obj)
File "/usr/lib/python2.3/pickle.py", line 338, in save
self.save_reduce(obj=obj, *rv)
File "/usr/lib/python2.3/pickle.py", line 433, in save_reduce
save(state)
File "/usr/lib/python2.3/pickle.py", line 293, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.3/pickle.py", line 663, in save_dict
self._batch_setitems(obj.iteritems())
File "/usr/lib/python2.3/pickle.py", line 677, in _batch_setitems
save(v)
File "/usr/lib/python2.3/pickle.py", line 293, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.3/pickle.py", line 614, in save_list
self._batch_appends(iter(obj))
File "/usr/lib/python2.3/pickle.py", line 629, in _batch_appends
save(x)
File "/usr/lib/python2.3/pickle.py", line 338, in save
self.save_reduce(obj=obj, *rv)
File "/usr/lib/python2.3/pickle.py", line 415, in save_reduce
save(args)
File "/usr/lib/python2.3/pickle.py", line 293, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.3/pickle.py", line 576, in save_tuple
save(element)
File "/usr/lib/python2.3/pickle.py", line 293, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.3/pickle.py", line 760, in save_global
raise PicklingError(
pickle.PicklingError: Can't pickle <class 'IdentifiedPeaks.Peak'>: it's
not found as IdentifiedPeaks.Peak

xml.marshal has the problem I mentioned before...
xml.marshal.generic.dumps(identifiedPeaks)

Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/usr/lib/python2.3/site-packages/_xmlplus/marshal/generic.py",
line 59, in dumps
L = [self.PROLOGUE + self.DTD] + self.m_root(value, dict)
File "/usr/lib/python2.3/site-packages/_xmlplus/marshal/generic.py",
line 104, in m_root
L = ['<%s>' % name] + self._marshal(value,dict) + ['</%s>' % name]
File "/usr/lib/python2.3/site-packages/_xmlplus/marshal/generic.py",
line 92, in _marshal
return getattr(self, meth)(value, dict)
AttributeError: Marshaller instance has no attribute
'm_IdentifiedPeaks'

Help would be hugely appreciated.

[Martin v. Löwis]

syd wrote:

But for my identifiedPeaks class (for instance), it has trouble. This
class contains a list of "peak" classes IdentifiedPeaks.Peak...
What precisely is the name of the class. You say it is
IdentifiedPeaks.Peak, but...

from IdentifiedPeaks import IdentifiedPeaks

Here you import IdentifiedPeaks.IdentifiedPeaks, not
IdentifiedPeak.Peak.
pickle.PicklingError: Can't pickle <class 'IdentifiedPeaks.Peak'>: it's
not found as IdentifiedPeaks.Peak
Here it claims there is no class IdentifiedPeaks.Peak, and I tend to
believe it. Could it be that this class does not exist under this name?

Python needs to pickle the full class name so that unpickle can find
the class. It uses (klass.__module__).(klass.__name__); if the class
is nested in another class, pickle cannot find out. So I suggest
to move the Peak class toplevel into the module. My guess is
that it is nested inside IdentifiedPeaks. The simplest fix might be
be to put

Peak=IdentifiedPeaks.Peak

into IdentifiedPeaks.py; better would be to move the class.
AttributeError: Marshaller instance has no attribute
'm_IdentifiedPeaks'

That would happen if IdentifiedPeaks is a new-style class (i.e.
inheriting from object). marshal has only generic support for instance
objects; each additional type needs separate support. You can provide
that support by inheriting from Marshaller, adding m_ functions for all
missing types. Each function needs to return a list of XML substrings,
e.g. through

def m_IdentifiedPeaks(self, peaks, dict):
L = [ '<IdentifiedPeaks>' ]
for p in peaks.getPeaks():
L += self._marshal(p)
L += [ '</IdentifiedPeaks>' ]
return L

The dict parameter keeps the object references for cycle and
shared reference detection. Whether or not you need cycle support
depends on your application.

Alternatively, patches to support new-style classes in a more
general way are welcome.

Regards,
Martin