is there a safe marshaler?

[Irmen de Jong]

Pickle and marshal are not safe. They can do harmful
things if fed maliciously constructed data.
That is a pity, because marshal is fast.
I need a fast and safe (secure) marshaler.
Is xdrlib the only option?
I would expect that it is fast and safe because
it (the xdr spec) has been around for so long.

Or are there better options (perhaps 3rd party libraries)?

Thanks

Irmen.

[Pierre Barbier de Reuille]

Irmen de Jong a écrit :

Pickle and marshal are not safe. They can do harmful
things if fed maliciously constructed data.
That is a pity, because marshal is fast.
I need a fast and safe (secure) marshaler.
Is xdrlib the only option?
I would expect that it is fast and safe because
it (the xdr spec) has been around for so long.

Or are there better options (perhaps 3rd party libraries)?

Thanks

Irmen.

What exactly do you mean by "safe" ? Do you want to ensure your objects
cannot receive corrupted data ? Do you want to ensure no code will be
evaluated during the unmarshalling ?

Please, be more precise,

Pierre

[guido]

Irmen de Jong wrote:

Pickle and marshal are not safe. They can do harmful
things if fed maliciously constructed data.
That is a pity, because marshal is fast.
I think marshal could be fixed; the only unsafety I'm aware of is that
it doesn't always act rationally when confronted with incorrect input
like bad type codes or truncated input. It only receives instances of
the built-in types and it never executes user code as a result of
unmarshalling.

Perhaps someone would be interested in submitting a patch to the
unmarshalling code? Since this is a security fix we'd even accept a fix
for 2.3.
I need a fast and safe (secure) marshaler.
Is xdrlib the only option?
I would expect that it is fast and safe because
it (the xdr spec) has been around for so long.

I don't expect that to be particularly fast, since it mostly operates
at Python speed. I think it could be safe but I would still do a
thorough code review if I were you -- the code is older than my
awareness of the vulnerabilities inherent in this kind of remote data
transfer.

--Guido

[Irmen de Jong]

Pierre Barbier de Reuille wrote:

Irmen de Jong a écrit :

Pickle and marshal are not safe. They can do harmful
things if fed maliciously constructed data.
That is a pity, because marshal is fast.
I need a fast and safe (secure) marshaler.
Is xdrlib the only option?
I would expect that it is fast and safe because
it (the xdr spec) has been around for so long.

Or are there better options (perhaps 3rd party libraries)?

Thanks

Irmen.

What exactly do you mean by "safe" ? Do you want to ensure your objects
cannot receive corrupted data ? Do you want to ensure no code will be
evaluated during the unmarshalling ?

"safe (secure)"
But to be more precise, let's look at the security warning that
is in the marshal documentation:
"The marshal module is not intended to be secure against erroneous or
maliciously constructed data. Never unmarshal data received from an
untrusted or unauthenticated source."

So essentially I want the opposite of that ;-)

I want a marshalar that is okay to use where the data it processes
comes from unknown, external sources (untrusted). It should not crash
on corrupt data and it should not execute arbitrary code when
unmarshaling, so that it is safe against hacking attempts.

Oh, preferrably, it should be fast :)
Some XML-ish thing may be secure but is likely to be not fast at all.

Ideally it should be able to transfer user defined Python types,
but if it is like marshal (can only marshal builtin types) that's
okay too.

--Irmen

[Irmen de Jong]

Hello Guido

gu***@python.or g wrote:

Irmen de Jong wrote:

Pickle and marshal are not safe. They can do harmful
things if fed maliciously constructed data.
That is a pity, because marshal is fast.

I think marshal could be fixed; the only unsafety I'm aware of is that
it doesn't always act rationally when confronted with incorrect input
like bad type codes or truncated input. It only receives instances of
the built-in types and it never executes user code as a result of
unmarshalling.

So it is not vulnerable in the way that pickle is? That's a start.
The security warning in the marsal doc then makes it sound worse than
it is...
Perhaps someone would be interested in submitting a patch to the
unmarshalling code? Since this is a security fix we'd even accept a fix
for 2.3.
That would be nice indeed :)

I need a fast and safe (secure) marshaler.
Is xdrlib the only option?
I would expect that it is fast and safe because
it (the xdr spec) has been around for so long.

I don't expect that to be particularly fast, since it mostly operates
at Python speed.

Ah, I wasn't aware that xdrlib was implemented in Python :)
I thought it used a (standard?) C-implementation.
But I now see that it's a Python module (utilizing struct).
I think it could be safe but I would still do a
thorough code review if I were you -- the code is older than my
awareness of the vulnerabilities inherent in this kind of remote data
transfer.

Thanks for the warning.

--Irmen de Jong

[PA]

On Feb 10, 2005, at 15:01, Irmen de Jong wrote:

Is xdrlib the only option?
I would expect that it is fast and safe because
it (the xdr spec) has been around for so long.

XDR? Like Sun's "XDR: External Data Representation standard"?

http://www.faqs.org/rfcs/rfc1014.html
http://www.faqs.org/rfcs/rfc1832.html

How does XDR copes with Unicode these days?

Alternatively, perhaps there is a ASN.1 DER library in python?

http://asn1.elibel.tm.fr/en/standards/index.htm

Cheers

--
PA, Onnay Equitursay
http://alt.textdrive.com/

[Alan Kennedy]

[Irmen de Jong]

Pickle and marshal are not safe. They can do harmful
things if fed maliciously constructed data.
That is a pity, because marshal is fast.
I need a fast and safe (secure) marshaler.

Hi Irmen,

I'm not necessarily proposing a solution to your problem, but am
interested in your requirement. Is this for pyro?

In the light of pyro, would something JSON be suitable for your need? I
only came across it a week ago (when someone else posted about it here
on c.l.py), and am intrigued by it.

http://json.org

What I find particularly intriguing is the JSON-RPC protocol, which
looks like a nice lightweight alternative to XML-RPC.

http://oss.metaparadigm.com/jsonrpc/

Also interesting is the browser embeddable JSON-RPC client written in
javascript, for which you can see a demo here

http://oss.metaparadigm.com/jsonrpc/demos.html

I thought you might be interested.

regards,

--
alan kennedy
------------------------------------------------------
email alan: http://xhaus.com/contact/alan

[Alan Kennedy]

[Alan Kennedy]

What I find particularly intriguing is the JSON-RPC protocol, which
looks like a nice lightweight alternative to XML-RPC.

http://oss.metaparadigm.com/jsonrpc/

Also interesting is the browser embeddable JSON-RPC client written in
javascript, for which you can see a demo here

http://oss.metaparadigm.com/jsonrpc/demos.html

I should have mentioned as well that there is a python JSON-RPC server
implementation, which incudes a complete JSON<-->python-objects codec.

http://www.json-rpc.org/pyjsonrpc/index.xhtml

regards,

--
alan kennedy
------------------------------------------------------
email alan: http://xhaus.com/contact/alan

[Irmen de Jong]

PA wrote:

On Feb 10, 2005, at 15:01, Irmen de Jong wrote:

Is xdrlib the only option?
I would expect that it is fast and safe because
it (the xdr spec) has been around for so long.

XDR? Like Sun's "XDR: External Data Representation standard"?

http://www.faqs.org/rfcs/rfc1014.html
http://www.faqs.org/rfcs/rfc1832.html

Not "like", but "the".
Or at least, a subset. (the xdrlib module documentation says
"It supports most of the data types described in the RFC").

How does XDR copes with Unicode these days?
Not directly, it seems that you have to encode
your unicode strings yourself first .

Alternatively, perhaps there is a ASN.1 DER library in python?

http://asn1.elibel.tm.fr/en/standards/index.htm

I don't know. Is there?
PS the xdr format is not self-describing in the way that
marshal and pickle streams are. That is a big limitiation
for what I need it for so xdr seems to drop off my radar.
Is an ASN.1 stream self-describing?

--Irmen

[Irmen de Jong]

Alan Kennedy wrote:

[Irmen de Jong]

Pickle and marshal are not safe. They can do harmful
things if fed maliciously constructed data.
That is a pity, because marshal is fast.
I need a fast and safe (secure) marshaler.

Hi Irmen,

I'm not necessarily proposing a solution to your problem, but am
interested in your requirement. Is this for pyro?

Yes and No.
Yes, I'm investigating possible marshaling alternatives
(others than pickle which Pyro uses right now).
No, I'm not changing Pyro yet. It's just that I want to
investigate possible *secure* alternatives to the current
implementation.
(Note that a secure version would also mean that Pyro's
advanced features such as mobile code should go the way
of the dodo, and I don't want to do this yet).
In the light of pyro, would something JSON be suitable for your need? I
only came across it a week ago (when someone else posted about it here
on c.l.py), and am intrigued by it.

http://json.org
Looks very interesting indeed, but in what way would this be
more secure than say, pickle or marshal?
A quick glance at some docs reveal that they are using eval
to process the data... ouch.

I thought you might be interested.

I certainly am but for different reasons.

--Irmen