help on pickle tool

[virg]

Hi,
i have client-server application which is written in python using
XMLRPC protocol. The existing client is a command line. Now client
application we are converting it as Web UI using java. I have seen some
problems in writing a java client. At the server for each request from
client, the server sends a response in hashtable and is serialized
using "pickle". The python function we call at the server is

pickle.dumps(r, 2) -- where r is a hash table which needs to be
serialized

At the client (existing client which in python) we use this call to
get the original data.

r = pickle.loads(result)

Since i am writing this client as Web UI client using java, i am not
able to deserialize this data
using java function "ObjectInputStream" and "ObjectInputStream" which
are most common functions for deserialization and i getting error as
invalid header.

Is it possible to deserialize the data by java which serialized by
Python or is there any compatibility issue. Is there any equivalent
pickle tool on java which supports this operation. so that i can use
across languages.

Because of this problem i am not able to proceed further. Any body has
any pointers for this problem. Any help is highly appreciated

Thanks in advance

regards
Virg

[hanumizzle]

On 5 Oct 2006 21:52:56 -0700, virg <06****@gmail.comwrote:

Hi,
i have client-server application which is written in python using
XMLRPC protocol. The existing client is a command line. Now client
application we are converting it as Web UI using java. I have seen some
problems in writing a java client. At the server for each request from
client, the server sends a response in hashtable and is serialized
using "pickle". The python function we call at the server is...

What kind of data are we talking about? Is it strictly dictionary /
list stuff or is it more complex (functions & other objects)? I can
see 2 possible options: Jython or YAML. (I have been answering a lot
of questions re: serialization today and mentioned YAML in all of them
:))

[Paddy]

virg wrote:

Hi,
i have client-server application which is written in python using
XMLRPC protocol. The existing client is a command line. Now client
application we are converting it as Web UI using java. I have seen some
problems in writing a java client. At the server for each request from
client, the server sends a response in hashtable and is serialized
using "pickle". The python function we call at the server is

pickle.dumps(r, 2) -- where r is a hash table which needs to be
serialized

At the client (existing client which in python) we use this call to
get the original data.

r = pickle.loads(result)

Since i am writing this client as Web UI client using java, i am not
able to deserialize this data
using java function "ObjectInputStream" and "ObjectInputStream" which
are most common functions for deserialization and i getting error as
invalid header.

Is it possible to deserialize the data by java which serialized by
Python or is there any compatibility issue. Is there any equivalent
pickle tool on java which supports this operation. so that i can use
across languages.

Because of this problem i am not able to proceed further. Any body has
any pointers for this problem. Any help is highly appreciated

Thanks in advance

regards
Virg

You might try picking the data with a different pickle formatter that
your Java can use. Maybe an XML pickler
(http://www.gnosis.cx/download/Gnosis...1.2.1.ANNOUNCE
untested by me).
You might also use a JSON/YAML pickler. JSON is now a subset of YAML:
http://cheeseshop.python.org/pypi/python-json/3.4
http://pyyaml.org/wiki/PyYAML
http://jyaml.sourceforge.net/
http://www.yaml.org/
http://www.json.org/
http://www.json.org/java/

- Paddy.

[hanumizzle]

On 5 Oct 2006 22:25:58 -0700, Paddy <pa*******@netscape.netwrote:

You might try picking the data with a different pickle formatter that
your Java can use. Maybe an XML pickler
(http://www.gnosis.cx/download/Gnosis...1.2.1.ANNOUNCE
untested by me).
You might also use a JSON/YAML pickler. JSON is now a subset of YAML:
http://cheeseshop.python.org/pypi/python-json/3.4
http://pyyaml.org/wiki/PyYAML
http://jyaml.sourceforge.net/
http://www.yaml.org/
http://www.json.org/
http://www.json.org/java/

Why a subset?

[MonkeeSage]

hanumizzle wrote:

Why a subset?

I don't think JSON is a subset of YAML.

Regards,
Jordan

[hanumizzle]

On 5 Oct 2006 22:54:46 -0700, MonkeeSage <Mo********@gmail.comwrote:

hanumizzle wrote:

Why a subset?

I don't think JSON is a subset of YAML.

Apparent slip of the fingers by OP. From JSON website:

JSON (JavaScript Object Notation) is a lightweight data-interchange
format. It is easy for humans to read and write. It is easy for
machines to parse and generate. It is based on a subset of the
JavaScript Programming Language, Standard ECMA-262 3rd Edition -
December 1999. JSON is a text format that is completely language
independent but uses conventions that are familiar to programmers of
the C-family of languages, including C, C++, C#, Java, JavaScript,
Perl, Python, and many others. These properties make JSON an ideal
data-interchange language.

I'm happy with my Pythonesque YAML syntax, thank you. :)

[MonkeeSage]

On Oct 6, 1:06 am, hanumizzle <[e-mail]:

I'm happy with my Pythonesque YAML syntax, thank you. :)

YAML is a little more complex, and a little more mature. But JSON
should not be ruled out. I actually like JSON personally.

Regards,
Jordan

[hanumizzle]

On 5 Oct 2006 23:19:18 -0700, MonkeeSage <Mo********@gmail.comwrote:

>

On Oct 6, 1:06 am, hanumizzle <[e-mail]:

I'm happy with my Pythonesque YAML syntax, thank you. :)

YAML is a little more complex, and a little more mature. But JSON
should not be ruled out. I actually like JSON personally.

I guess I'll keep an open mind. But I like editing YAML for the same
reason that I like editing Python. (Although I admit I usually I gen
my data structures in ipython and then dump them.)

-- Theerasak

[Fredrik Lundh]

MonkeeSage wrote:

YAML is a little more complex

a little? when did you last look at the spec?

and a little more mature.

than JavaScript's expression syntax? are you sure you're not confusing
libraries with standards here? (has anyone even managed to write a YAML
library that's small and simple enough to be "obviously correct"?)

</F>

[hanumizzle]

On 10/6/06, Fredrik Lundh <fr*****@pythonware.comwrote:

MonkeeSage wrote:

YAML is a little more complex

a little? when did you last look at the spec?

and a little more mature.

than JavaScript's expression syntax? are you sure you're not confusing
libraries with standards here? (has anyone even managed to write a YAML
library that's small and simple enough to be "obviously correct"?)

Tell. I'm interested in knowing.

-- Theerasak

[Fredrik Lundh]

hanumizzle wrote:

I guess I'll keep an open mind. But I like editing YAML for the same
reason that I like editing Python.

JSON is almost identical to Python's expression syntax, of course, while
YAML isn't even close.

</F>

[hanumizzle]

On 10/6/06, Fredrik Lundh <fr*****@pythonware.comwrote:

hanumizzle wrote:

I guess I'll keep an open mind. But I like editing YAML for the same
reason that I like editing Python.

JSON is almost identical to Python's expression syntax, of course, while
YAML isn't even close.

Getting the source now. S'pose it isn't too late to convert my project over...

-- Theerasak

[MonkeeSage]

On Oct 6, 1:28 am, Fredrik Lundh <[e-mail]:

when did you last look at the spec?

I'm fairly versed in JS objects, having written 10 or so extensions for
firefox; but I've only used YAML for trivial tasks like config files.
So I can't really say how they stack up in "the big picture". But from
what I have seen, JSON is a simpler and easier to use format than YAML.

Regards,
Jordan

[hanumizzle]

On 5 Oct 2006 23:43:50 -0700, MonkeeSage <Mo********@gmail.comwrote:

>

On Oct 6, 1:28 am, Fredrik Lundh <[e-mail]:

when did you last look at the spec?

I'm fairly versed in JS objects, having written 10 or so extensions for
firefox; but I've only used YAML for trivial tasks like config files.
So I can't really say how they stack up in "the big picture". But from
what I have seen, JSON is a simpler and easier to use format than YAML.

That and it sounds like Strongbad pronouncing the name 'Jason'.

-- Theerasak

[virg]

Hi,
The data is simple dictionary with one or more keys. If i use YAML at
the client (webui) do i have to change serialisation method to YAML at
server also. Without changing serialisation method at server, can i use
any of the deserialisation methods at the client. We cannot change the
serialisation methods at the server since it is not under our control.

Thanks & reagards
virg
hanumizzle wrote:

On 5 Oct 2006 21:52:56 -0700, virg <06****@gmail.comwrote:

Hi,
i have client-server application which is written in python using
XMLRPC protocol. The existing client is a command line. Now client
application we are converting it as Web UI using java. I have seen some
problems in writing a java client. At the server for each request from
client, the server sends a response in hashtable and is serialized
using "pickle". The python function we call at the server is...

What kind of data are we talking about? Is it strictly dictionary /
list stuff or is it more complex (functions & other objects)? I can
see 2 possible options: Jython or YAML. (I have been answering a lot
of questions re: serialization today and mentioned YAML in all of them
:))

[hanumizzle]

On 6 Oct 2006 01:41:48 -0700, virg <06****@gmail.comwrote:

Hi,
The data is simple dictionary with one or more keys. If i use YAML at
the client (webui) do i have to change serialisation method to YAML at
server also. Without changing serialisation method at server, can i use
any of the deserialisation methods at the client. We cannot change the
serialisation methods at the server since it is not under our control.

Oh, poopy.

What do you have at the server end?

-- Theerasak

[Steve Holden]

Fredrik Lundh wrote:

MonkeeSage wrote:

>>YAML is a little more complex

a little? when did you last look at the spec?

>>and a little more mature.

than JavaScript's expression syntax? are you sure you're not confusing
libraries with standards here? (has anyone even managed to write a YAML
library that's small and simple enough to be "obviously correct"?)

I have to agree that YAML, having started out with simplicity in mind,
has become a monster that threatens to collapse under its own weight.
The very existence of JSON is a good indicator that YAML has failed to
meet its design goals for a significant proportion of application
developers.

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://holdenweb.blogspot.com
Recent Ramblings http://del.icio.us/steve.holden

[virg]

At the server, based on client request it does some computations , it
sends the result as dictionary (serialized) to the client.

hanumizzle wrote:

On 6 Oct 2006 01:41:48 -0700, virg <06****@gmail.comwrote:

Hi,
The data is simple dictionary with one or more keys. If i use YAML at
the client (webui) do i have to change serialisation method to YAML at
server also. Without changing serialisation method at server, can i use
any of the deserialisation methods at the client. We cannot change the
serialisation methods at the server since it is not under our control.

Oh, poopy.

What do you have at the server end?

-- Theerasak

[hanumizzle]

On 10/6/06, Steve Holden <st***@holdenweb.comwrote:

I have to agree that YAML, having started out with simplicity in mind,
has become a monster that threatens to collapse under its own weight.
The very existence of JSON is a good indicator that YAML has failed to
meet its design goals for a significant proportion of application
developers.

I am looking at JSON, but YAML does work fine for my purposes w/ no
discernable disadvantages. Of course I didn't implement the library,
so I can't really speak with any kind of expertise on this matter. I'm
just saying that, at least at the front end, it's pretty simple to
use. (JMO)

-- Theerasak

[hanumizzle]

On 6 Oct 2006 02:03:07 -0700, virg <06****@gmail.comwrote:

At the server, based on client request it does some computations , it
sends the result as dictionary (serialized) to the client.

If I interpret your message correctly, you are receiving a Python
dictionary object from the server. Yes? In this case, I guess it might
be necessary to use Jython. I'm drawing a blank otherwise.

(This may be naive, but can you write the client in Python as well?)

-- Theerasak

[virg]

Yes your are right. I will send a dictionary object from the server to
the client.
I already have client which is written in python. But we are migrating
the python client which is a command line tool to Web UI client
(java). If it is possible to call python function from java, i need to
read more about jython, i am new to this.
hanumizzle wrote:

On 6 Oct 2006 02:03:07 -0700, virg <06****@gmail.comwrote:

At the server, based on client request it does some computations , it
sends the result as dictionary (serialized) to the client.

If I interpret your message correctly, you are receiving a Python
dictionary object from the server. Yes? In this case, I guess it might
be necessary to use Jython. I'm drawing a blank otherwise.

(This may be naive, but can you write the client in Python as well?)

-- Theerasak

[hanumizzle]

On 6 Oct 2006 02:29:59 -0700, virg <06****@gmail.comwrote:

Yes your are right. I will send a dictionary object from the server to
the client.
I already have client which is written in python. But we are migrating
the python client which is a command line tool to Web UI client
(java).

Please explain 'Web UI'. Can Python perform an equivalent function?

-- Theerasak

[virg]

Hi,

Yes, using python client we are able deserialize data using

r = pickle.loads(result).

where result is a response from the server and r is a dictionary after
deserialization.
For serialisation at the server written in python using
pickle.dumps(result, 2)

Now we are developing web based Client using java. So we are writing
client in java. If server and client are in python we dont see any
problems since we are using same serialisation tool "pickle". Now we
have seen problems because we are writing client in java. we did not
find equivalent function on java for this tool pickle. If i use
standard java desrialisation functions i am getting error as invalid
header becasue of incompatibility between python and java. Please help
me if you have any clue

regards,
- Virg

hanumizzle wrote:

On 6 Oct 2006 02:29:59 -0700, virg <06****@gmail.comwrote:

Yes your are right. I will send a dictionary object from the server to
the client.
I already have client which is written in python. But we are migrating
the python client which is a command line tool to Web UI client
(java).

Please explain 'Web UI'. Can Python perform an equivalent function?

-- Theerasak

[Paddy]

hanumizzle wrote:

On 5 Oct 2006 22:25:58 -0700, Paddy <pa*******@netscape.netwrote:

You might try picking the data with a different pickle formatter that
your Java can use. Maybe an XML pickler
(http://www.gnosis.cx/download/Gnosis...1.2.1.ANNOUNCE
untested by me).
You might also use a JSON/YAML pickler. JSON is now a subset of YAML:

>
Why a subset?

I was referring to comments ike:
http://redhanded.hobix.com/inspect/yamlIsJson.html

http://redhanded.hobix.com/inspect/j...hitespace.html

Happy coding - Paddy.

[Paddy]

Paddy wrote:

hanumizzle wrote:

On 5 Oct 2006 22:25:58 -0700, Paddy <pa*******@netscape.netwrote:

You might try picking the data with a different pickle formatter that
your Java can use. Maybe an XML pickler
(http://www.gnosis.cx/download/Gnosis...1.2.1.ANNOUNCE
untested by me).
You might also use a JSON/YAML pickler. JSON is now a subset of YAML:

Why a subset?

I was referring to comments ike:
http://redhanded.hobix.com/inspect/yamlIsJson.html

http://redhanded.hobix.com/inspect/j...hitespace.html

Happy coding - Paddy.

Oh, and this might also be of use:

http://www-128.ibm.com/developerwork...s46/index.html

- Pad.

[Kirill Simonov]

Hi,

I'm somewhat late to this discussion, but as the author of PyYAML, I'd
like to put my 2c in.
On Thu, Oct 05, 2006 at 09:52:56PM -0700, virg wrote:

Is it possible to deserialize the data by java which serialized by
Python or is there any compatibility issue. Is there any equivalent
pickle tool on java which supports this operation. so that i can use
across languages.

You may serialize/deserialize your data to YAML using PyYAML
(http://pyyaml.org/wiki/PyYAML) on the Python side and jvyaml
(https://jvyaml.dev.java.net/) on the Java side. You may also check
JSON (http://json.org/) as other posters suggested.
On Thu, Oct 05, 2006 at 10:54:46PM -0700, MonkeeSage wrote:

hanumizzle wrote:

Why a subset?

I don't think JSON is a subset of YAML.

It is.
On Fri, Oct 06, 2006 at 08:36:07AM +0200, Fredrik Lundh wrote:

JSON is almost identical to Python's expression syntax, of course,
while YAML isn't even close.

A valid Python list/dict expression is likely to be a valid YAML
expression. For instance

{ "odd": [1, 3, 5, 7, 9], "even": [2, 4, 6, 8] }

is both valid Python and valid YAML.
On Fri, Oct 06, 2006 at 08:28:29AM +0200, Fredrik Lundh wrote:

than JavaScript's expression syntax? are you sure you're not
confusing libraries with standards here? (has anyone even managed to
write a YAML library that's small and simple enough to be "obviously
correct"?)

I've written a complete YAML parser and I must admit it hasn't been
extremely difficult. The YAML syntax is very close to Python and I just
needed to rewrite YAML grammar using Python as a model. The PyYAML
parser is LL(1), which is as simple as it could be.
--
xi