Can XML-RPC performance be improved?

[Sion Arrowsmith]

I've got an established client-server application here where there
is now a need to shovel huge amounts of data (structured as lists of
lists) between the two, and the performance bottleneck has become
the amount of time spent parsing XML (it's taking 100% CPU on one or
other end of the connection and accounting for well over 50% of the
total call time, to the extent that it's having a greater impact on
performance than user interaction). And this is using SGMLOP for
parsing -- I dread to think what it would be like with a slower
parser. Anyone (Fredrik?) got any good ideas for tackling this
problem?

--
\S -- si***@chiark.gr eenend.org.uk -- http://www.chaos.org.uk/~sion/
___ | "Frankly I have no feelings towards penguins one way or the other"
\X/ | -- Arthur C. Clarke
her nu becomeþ se bera eadward ofdun hlæddre heafdes bæce bump bump bump

[Diez B. Roggisch]

Sion Arrowsmith wrote:

I've got an established client-server application here where there
is now a need to shovel huge amounts of data (structured as lists of
lists) between the two, and the performance bottleneck has become
the amount of time spent parsing XML (it's taking 100% CPU on one or
other end of the connection and accounting for well over 50% of the
total call time, to the extent that it's having a greater impact on
performance than user interaction). And this is using SGMLOP for
parsing -- I dread to think what it would be like with a slower
parser. Anyone (Fredrik?) got any good ideas for tackling this
problem?

CORBA. Or TCP/IP directly. I know that isn't really the answer you're
looking for - but there is an overhead parsing XML, and that can't go away.
From what I read at SGMLOP's site I doub't that you can get much faster -
because speed comes with sacrificing standard compliancy, which it already
seems to do.

Regards,

Diez

[Gregory Piñero]

Perhaps there's a hardware solution? Could you run more servers in
parallel? Or have some sort of load distribution system?

(Just some generic server tips, I don't know much about XML-RPC)

-Greg
On 3/21/06, Diez B. Roggisch <de***@nospam.w eb.de> wrote:

Sion Arrowsmith wrote:

I've got an established client-server application here where there
is now a need to shovel huge amounts of data (structured as lists of
lists) between the two, and the performance bottleneck has become
the amount of time spent parsing XML (it's taking 100% CPU on one or
other end of the connection and accounting for well over 50% of the
total call time, to the extent that it's having a greater impact on
performance than user interaction). And this is using SGMLOP for
parsing -- I dread to think what it would be like with a slower
parser. Anyone (Fredrik?) got any good ideas for tackling this
problem?

CORBA. Or TCP/IP directly. I know that isn't really the answer you're
looking for - but there is an overhead parsing XML, and that can't go away.

From what I read at SGMLOP's site I doub't that you can get much faster -

because speed comes with sacrificing standard compliancy, which it already
seems to do.

Regards,

Diez
--
http://mail.python.org/mailman/listinfo/python-list

--
Gregory Piñero
Chief Innovation Officer
Blended Technologies
(www.blendedtechnologies.com)

[Rene Pijlman]

Sion Arrowsmith:

I've got an established client-server application here where there
is now a need to shovel huge amounts of data (structured as lists of
lists) between the two, and the performance bottleneck has become
the amount of time spent parsing XML
http://xmlsucks.org/
Anyone got any good ideas for tackling this problem?

cPickle.
http://docs.python.org/lib/module-cPickle.html

--
René Pijlman

Wat wil jij leren? http://www.leren.nl

[Fredrik Lundh]

Sion Arrowsmith wrote:

I've got an established client-server application here where there
is now a need to shovel huge amounts of data (structured as lists of
lists) between the two, and the performance bottleneck has become
the amount of time spent parsing XML (it's taking 100% CPU on one or
other end of the connection and accounting for well over 50% of the
total call time, to the extent that it's having a greater impact on
performance than user interaction). And this is using SGMLOP for
parsing -- I dread to think what it would be like with a slower
parser. Anyone (Fredrik?) got any good ideas for tackling this
problem?

the cElementTree unmarshaller on this page

http://effbot.org/zone/element-iterp...ental-decoding

is a bit faster than the one in xmlrpclib. (to get more performance,
you probably need a library that allows you to register decoders on
the C level.)

</F>

[gregarican]

Sion Arrowsmith wrote:

shovel huge amounts of data

That right there basically takes XML-RPC off the table as a viable
solution. No matter how much fine tuning you try large amounts of data
don't bode well using XML-RPC. The other posted alternatives are
certainly worth a shot. From CORBA to cPickle to even a basic TCP
socket connection would be more efficient. I have used XML-RPC for very
small recordset transfers but I wouldn't use it for anything large.

[Paul McGuire]

"Diez B. Roggisch" <de***@nospam.w eb.de> wrote in message
news:48******** ****@uni-berlin.de...

Sion Arrowsmith wrote:

I've got an established client-server application here where there
is now a need to shovel huge amounts of data (structured as lists of
lists) between the two, and the performance bottleneck has become
the amount of time spent parsing XML (it's taking 100% CPU on one or
other end of the connection and accounting for well over 50% of the
total call time, to the extent that it's having a greater impact on
performance than user interaction). And this is using SGMLOP for
parsing -- I dread to think what it would be like with a slower
parser. Anyone (Fredrik?) got any good ideas for tackling this
problem?

CORBA. Or TCP/IP directly.

<snip>

Beware, CORBA marshaling/unmarshaling can be similarly deadly, especially if
you are passing Any's and have to marshal complex type codes. But if you
are just sending arrays of octets, or strings of chars, then CORBA
marshal/unmarshal will be quite fast.

Of course, something will have to convert that array to meaningful data
*somewhere*! You can't really eliminate software complexity, you can only
move it around. (not original, I'm quoting Chris Stone, founder of the OMG.
Also from Chris Stone - "I finally came up with a good definition for
middleware. Middleware is the software nobody wants to pay for.")

-- Paul

[skip]

Diez> Sion Arrowsmith wrote:

I've got an established client-server application here where there
is now a need to shovel huge amounts of data (structured as lists of
lists) between the two, and the performance bottleneck has become
the amount of time spent parsing XML ...
Diez> CORBA. Or TCP/IP directly. I know that isn't really the answer
Diez> you're looking for - but there is an overhead parsing XML, and
Diez> that can't go away.

Ah, but if both ends of the pipe happen to be running Python, you can
cheat. ;-) Run your list through marshal.dumps() before passing it to
xmlrpclib, then undo the marshalling on the other end.

biglist = ["abc"] * 50
biglist = [biglist] * 50
import xmlrpclib
import marshal
x = xmlrpclib.dumps ((biglist,))
len(x) 92331 x.count("<") 10310 y = xmlrpclib.dumps ((marshal.dumps (biglist),))
len(y) 20324 y.count("<")

8

If you can swing it, I'd be willing to bet you a beer your XML parsing time
will all but disappear.

If you're not using xmlrpclib with some sort of C helper (like sgmlop), try
that as well. Big difference parsing XML in C and Python.

Skip

[Steve Holden]

gregarican wrote:

Sion Arrowsmith wrote:

shovel huge amounts of data

That right there basically takes XML-RPC off the table as a viable
solution. No matter how much fine tuning you try large amounts of data
don't bode well using XML-RPC. The other posted alternatives are
certainly worth a shot. From CORBA to cPickle to even a basic TCP
socket connection would be more efficient. I have used XML-RPC for very
small recordset transfers but I wouldn't use it for anything large.

<rant>
I am continually surprised that the industry as a whole has become so
enamoured of XML that people persist in using it for tasks it was never
designed nor intended to be used for.
</rant>

I suppose there *was* a good reason for using XML-RPC in the first place?

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC/Ltd www.holdenweb.com
Love me, love my blog holdenweb.blogs pot.com

[skip]

Steve> I suppose there *was* a good reason for using XML-RPC in the
Steve> first place?

I don't know about the OP, but in my case it was a drop-dead simple
cross-language RPC protocol. At one point Mojam's XML-RPC server talked to
clients written in Python, Perl and Objective-C (via a Java bridge, no
less).

Skip