By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
464,822 Members | 1,051 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 464,822 IT Pros & Developers. It's quick & easy.

marshal.dumps quadratic growth and marshal.dump not allowingfile-like objects

P: n/a
I'm stuck on a problem where I want to use marshal for serialization
(yes, yes, I know (c)Pickle is normally recommended here). I favor
marshal for speed for the types of data I use.

However it seems that marshal.dumps() for large objects has a
quadratic performance issue which I'm assuming is that it grows its
memory buffer in constant increments. This causes a nasty slowdown for
marshaling large objects. I thought I would get around this by passing
a cStringIO.StringIO object to marshal.dump() instead but I quickly
learned this is not supported (only true file objects are supported).

Any ideas about how to get around the marshal quadratic issue? Any
hope for a fix for that on the horizon? Thanks for any information.
Jun 27 '08 #1
Share this Question
Share on Google+
10 Replies

P: n/a
On 16:04, domenica 15 giugno 2008 bk*****@gmail.com wrote:
cStringIO.StringIO object to marshal.dump() instead but I quickly
learned this is not supported (only true file objects are supported).

Any ideas about how to get around the marshal quadratic issue? Any
hope for a fix for that on the horizon?
If you zip the cStringIO.StringIO object, would it be possible?

--
Mailsweeper Home : http://it.geocities.com/call_me_not_now/index.html
Jun 27 '08 #2

P: n/a
bk*****@gmail.com wrote:
I'm stuck on a problem where I want to use marshal for serialization
(yes, yes, I know (c)Pickle is normally recommended here). I favor
marshal for speed for the types of data I use.

However it seems that marshal.dumps() for large objects has a
quadratic performance issue which I'm assuming is that it grows its
memory buffer in constant increments. This causes a nasty slowdown for
marshaling large objects. I thought I would get around this by passing
a cStringIO.StringIO object to marshal.dump() instead but I quickly
learned this is not supported (only true file objects are supported).

Any ideas about how to get around the marshal quadratic issue? Any
hope for a fix for that on the horizon? Thanks for any information.
Here's how marshal resizes the string:

newsize = size + size + 1024;
if (newsize 32*1024*1024) {
newsize = size + 1024*1024;
}

Maybe you can split your large objects and marshal multiple objects to keep
the size below the 32MB limit.

Peter
Jun 27 '08 #3

P: n/a
On Jun 15, 1:04*am, bkus...@gmail.com wrote:
However it seems that marshal.dumps() for large objects has a
quadratic performance issue which I'm assuming is that it grows its
memory buffer in constant increments.
Looking at the source in http://svn.python.org/projects/pytho...thon/marshal.c
, it looks like the relevant fragment is in w_more():

. . .
size = PyString_Size(p->str);
newsize = size + size + 1024;
if (newsize 32*1024*1024) {
newsize = size + 1024*1024;
}
if (_PyString_Resize(&p->str, newsize) != 0) {
. . .

When more space is needed, the resize operation over-allocates by
double the previous need plus 1K. This should give amortized O(1)
performance just like list.append().

However, when that strategy requests more than 32Mb, the resizing
becomes less aggressive and grows only in 1MB blocks and giving your
observed nasty quadratic behavior.

Raymond
Jun 27 '08 #4

P: n/a
On Jun 15, 7:47 pm, Peter Otten <__pete...@web.dewrote:
bkus...@gmail.com wrote:
I'm stuck on a problem where I want to use marshal for serialization
(yes, yes, I know (c)Pickle is normally recommended here). I favor
marshal for speed for the types of data I use.
However it seems that marshal.dumps() for large objects has a
quadratic performance issue which I'm assuming is that it grows its
memory buffer in constant increments. This causes a nasty slowdown for
marshaling large objects. I thought I would get around this by passing
a cStringIO.StringIO object to marshal.dump() instead but I quickly
learned this is not supported (only true file objects are supported).
Any ideas about how to get around the marshal quadratic issue? Any
hope for a fix for that on the horizon? Thanks for any information.

Here's how marshal resizes the string:

newsize = size + size + 1024;
if (newsize 32*1024*1024) {
newsize = size + 1024*1024;
}

Maybe you can split your large objects and marshal multiple objects to keep
the size below the 32MB limit.
But that change went into the svn trunk on 11-May-2008; perhaps the OP
is using a production release which would have the previous version,
which is merely "newsize = size + 1024;".

Do people really generate 32MB pyc files, or is stopping doubling at
32MB just a safety valve in case someone/something runs amok?

Cheers,
John
Jun 27 '08 #5

P: n/a
John Machin wrote:
>Here's how marshal resizes the string:

newsize = size + size + 1024;
if (newsize 32*1024*1024) {
newsize = size + 1024*1024;
}

Maybe you can split your large objects and marshal multiple objects to
keep the size below the 32MB limit.

But that change went into the svn trunk on 11-May-2008; perhaps the OP
is using a production release which would have the previous version,
which is merely "newsize = size + 1024;".
That is indeed much worse. Depending on what the OP means by "large objects"
the problem may be fixed in subversion then.
Do people really generate 32MB pyc files, or is stopping doubling at
32MB just a safety valve in case someone/something runs amok?
A 32MB pyc would correspond to a module of roughly the same size. So
someone/something runs amok in either case.

Peter
Jun 27 '08 #6

P: n/a
Raymond Hettinger wrote:
When more space is needed, the resize operation over-allocates by
double the previous need plus 1K. This should give amortized O(1)
performance just like list.append().

However, when that strategy requests more than 32Mb, the resizing
becomes less aggressive and grows only in 1MB blocks and giving your
observed nasty quadratic behavior.
The marshal code has been revamped in Python 2.6. The old code in Python
2.5 uses a linear growth strategy:

size = PyString_Size(p->str);
newsize = size + 1024;
if (_PyString_Resize(&p->str, newsize) != 0) {
p->ptr = p->end = NULL;
}

Anyway marshal should not be used by user code to serialize objects.
It's only meant for Python byte code. Please use the pickle/cPickle
module instead.

Christian

Jun 27 '08 #7

P: n/a
On Jun 15, 3:16*am, John Machin <sjmac...@lexicon.netwrote:
But that change went into the svn trunk on 11-May-2008; perhaps the OP
is using a production release which would have the previous version,
which is merely "newsize = size + 1024;".

Do people really generate 32MB pyc files, or is stopping doubling at
32MB just a safety valve in case someone/something runs amok?
Indeed. I (the OP) am using a production release which has the 1k
linear growth.
I am seeing the problems with ~5MB and ~10MB sizes.
Apparently this will be improved greatly in Python 2.6, at least up to
the 32MB limit.

Thanks all for responding.
Jun 27 '08 #8

P: n/a
On Jun 16, 1:08 am, bkus...@gmail.com wrote:
On Jun 15, 3:16 am, John Machin <sjmac...@lexicon.netwrote:
But that change went into the svn trunk on 11-May-2008; perhaps the OP
is using a production release which would have the previous version,
which is merely "newsize = size + 1024;".
Do people really generate 32MB pyc files, or is stopping doubling at
32MB just a safety valve in case someone/something runs amok?

Indeed. I (the OP) am using a production release which has the 1k
linear growth.
I am seeing the problems with ~5MB and ~10MB sizes.
Apparently this will be improved greatly in Python 2.6, at least up to
the 32MB limit.
Apparently you intend to resist good advice and persist [accidental
pun!] with marshal -- how much slower is cPickle for various sizes of
data? What kinds of objects are you persisting?

Jun 27 '08 #9

P: n/a
On Jun 15, 8:08*am, bkus...@gmail.com wrote:
Indeed. I (the OP) am using a production release which has the 1k
linear growth.
I am seeing the problems with ~5MB and ~10MB sizes.
Apparently this will be improved greatly in Python 2.6, at least up to
the 32MB limit.
I've just fixed this for Py2.5.3 and Py2.6. No more quadratic
behavior.
Raymond
Jun 27 '08 #10

P: n/a
>
Anywaymarshalshould not be used by user code to serialize objects.
It's only meant for Python byte code. Please use the pickle/cPickle
module instead.

Christian
Just for yucks let me point out that marshal has
no real security concerns of interest to the non-paranoid,
whereas pickle is a security disaster waiting to happen
unless you are extremely cautious... yet again.

Sorry, I know a even a monkey learns after 3 times...

-- Aaron Watters

===
http://www.xfeedme.com/nucular/pydis...ETEXT=disaster
Jun 27 '08 #11

This discussion thread is closed

Replies have been disabled for this discussion.