is there a safe marshaler?

[Irmen de Jong]

Pickle and marshal are not safe. They can do harmful
things if fed maliciously constructed data.
That is a pity, because marshal is fast.
I need a fast and safe (secure) marshaler.
Is xdrlib the only option?
I would expect that it is fast and safe because
it (the xdr spec) has been around for so long.

Or are there better options (perhaps 3rd party libraries)?

Thanks

Irmen.

[Aahz]

In article <ma************ *************** ************@py thon.org>,
Skip Montanaro <sk**@pobox.com > wrote:

Carl> but can't effbot's fast cElementree be used for PYROs XML_PICKLE
Carl> and would it be safe and fast enough?

It's not clear to me that if marshal is unsafe how XML could be safe. In
this context they are both just serializations of basic Python data
structures.

The difference is that parsing XML -- even badly malformed -- won't
crash Python.
--
Aahz (aa**@pythoncra ft.com) <*> http://www.pythoncraft.com/

"The joy of coding Python should be in seeing short, concise, readable
classes that express a lot of action in a small amount of clear code --
not in reams of trivial code that bores the reader to death." --GvR

[Fredrik Lundh]

Irmen de Jong wrote:

but can't effbot's fast cElementree be used for PYROs XML_PICKLE and would it be safe and fast
enough?

ElementTree's not a marshaler.
Or has it object (de)serializati on included?

nope. building a serialization layer on top of it is pretty trivial, and the result
is pretty fast, but nowhere close to C speed.

</F>

[Fredrik Lundh]

(repost; gmane seems to have eaten my original post)

Aahz wrote:

It's not clear to me that if marshal is unsafe how XML could be safe. In
this context they are both just serializations of basic Python data
structures.

The difference is that parsing XML -- even badly malformed -- won't
crash Python.

optimist.

import os
os.path.getsize ("BL.xml") 1302 from xml.dom import minidom
x = minidom.parse(" BL.xml")

(have patience. have lots of patience.)

</F>

[Irmen de Jong]

Fredrik Lundh wrote:

import os
os.path.get size("BL.xml")
1302
from xml.dom import minidom
x = minidom.parse(" BL.xml")

(have patience. have lots of patience.)

Hehe, the XML killer file "BillionLaughs" ... correct?

--Irmen

[Irmen de Jong]

Alan Kennedy wrote:

I should learn to keep my mouth zipped :-L
:-D
OK, I really don't have time for a detailed examination of either the
JSON spec or the python impl of same. And I *definitely* don't have time
for a detailed security audit, much though I'd love to.

No problem. The patch you wrote is a very good start, I think!!

Interestingly enough, I just ran across "Flatten":
http://sourceforge.net/project/showf...ckage_id=91311

"...which aids in serializing/unserializing networked data securely,
without having to fear execution of code or the like."

Sounds promising!
--Irmen

[Fredrik Lundh]

Aahz wrote:

It's not clear to me that if marshal is unsafe how XML could be safe. In
this context they are both just serializations of basic Python data
structures.

The difference is that parsing XML -- even badly malformed -- won't
crash Python.

optimist.

import os
os.path.getsize ("BL.xml") 1302 from xml.dom import minidom
x = minidom.parse(" BL.xml")

(have patience. have lots of patience.)

</F>

[Fredrik Lundh]

(repost; gmane seems to have eaten my original post)

Irmen de Jong wrote:

I think marshal could be fixed; the only unsafety I'm aware of is that
it doesn't always act rationally when confronted with incorrect input
like bad type codes or truncated input. It only receives instances of
the built-in types and it never executes user code as a result of
unmarshalling.

So it is not vulnerable in the way that pickle is? That's a start.
The security warning in the marsal doc then makes it sound worse than
it is...

the problem is that the following may or may not reach the "done!" statement,
somewhat depending on python version, memory allocator, and what data you
pass to dumps.

import marshal

data = marshal.dumps(( 1, 2, 3, "hello", 4, 5, 6))

for i in range(len(data) , -1, -1):
try:
print marshal.loads(d ata[:i])
except EOFError:
print "EOFError"
except ValueError:
print "ValueError "

print "done!"

(try different data combinations, to see how far you get on your platform...)

fixing this should be relatively easy, and should result in a safe unmarshaller (your
application will still have to limit the amount of data fed into load/loads, of course).

</F>

[Irmen de Jong]

Fredrik Lundh wrote:

the problem is that the following may or may not reach the "done!" statement,
somewhat depending on python version, memory allocator, and what data you
pass to dumps.

import marshal

data = marshal.dumps(( 1, 2, 3, "hello", 4, 5, 6))

for i in range(len(data) , -1, -1):
try:
print marshal.loads(d ata[:i])
except EOFError:
print "EOFError"
except ValueError:
print "ValueError "

print "done!"

(try different data combinations, to see how far you get on your platform...)
Python 2.4 on my windows box crashes with
Fatal Python error: PyString_Intern InPlace: strings only please!

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.
c:\> _
So indeed it seems that marshal is not safe yet :-|

fixing this should be relatively easy, and should result in a safe unmarshaller (your
application will still have to limit the amount of data fed into load/loads, of course).

Okay.

--Irmen

[Alan Kennedy]

[Irmen de Jong]

Interestingly enough, I just ran across "Flatten":
http://sourceforge.net/project/showf...ckage_id=91311

"...which aids in serializing/unserializing networked data securely,
without having to fear execution of code or the like."

Sounds promising!

Well, I'm always dubious of OSS projects that don't even have any bugs
reported, let alone fixed: no patches submitted, etc, etc.

http://sourceforge.net/tracker/?group_id=82591

Though maybe I'm missing something obvious?

--
alan kennedy
------------------------------------------------------
email alan: http://xhaus.com/contact/alan

[Irmen de Jong]

Alan Kennedy wrote:

[Irmen de Jong]

Interestingly enough, I just ran across "Flatten":
http://sourceforge.net/project/showf...ckage_id=91311

"...which aids in serializing/unserializing networked data securely,
without having to fear execution of code or the like."

Sounds promising!

Well, I'm always dubious of OSS projects that don't even have any bugs
reported, let alone fixed: no patches submitted, etc, etc.

http://sourceforge.net/tracker/?group_id=82591

Though maybe I'm missing something obvious?

Perhaps the SF trackers are simply not used for that project?
Consider my own project:
http://sourceforge.net/tracker/?group_id=18837
I can assure you that I have fixed and applied a huge
amount of bugs and patches during the lifetime of the project.
They are just not entered in the trackers, except for a few.

--Irmen