I have a pair of programs which trade python data back and forth by
pickling up lists of objects on one side (using
pickle.HIGHEST_ PROTOCOL), and sending that data over a TCP socket
connection to the receiver, who unpickles the data and uses it.
So far this has been working fine, but I now need a way of separating
multiple chunks of pickled binary data in the stream being sent back and
forth.
Questions:
Is it safe to do what I'm doing? I didn't think there was anything
fundamentally wrong with sending binary pickled data, especially in the
closed, safe environment these programs operate under...but maybe I'm
making a poor assumption?
I was going to separate the chunks of pickled data with some well-formed
string, but couldn't that string potentially randomly appear in the
pickled data? Do I just pick an extremely
unlikely-to-be-randomly-generated string as the separator? Is there some
string that will definitely NEVER show up in pickled binary data?
I thought about base64 encoding the data, and then decoding on the
opposite side (like what xmlrpclib does), but that turns out to be a
very expensive operation, which I want to avoid, speed is of the essence
in this situation.
Is there a reliable way to determine the byte count of some pickled
binary data? Can I rely on len(<pickled data>) == bytes?
Thanks for all responses,
-David
--
Presenting:
mediocre nebula. 4 3740
David Hirschfield <da****@ilm.com writes:
Is there a reliable way to determine the byte count of some pickled
binary data? Can I rely on len(<pickled data>) == bytes?
Huh? Yes, of course len gives you the length.
As for the network representation, DJB proposes this format: http://cr.yp.to/proto/netstrings.txt
Paul Rubin <httpwrote:
As for the network representation, DJB proposes this format: http://cr.yp.to/proto/netstrings.txt
Netstrings are cool and you'll find some python implementations if you
search.
But it is basically "number:string, ", ie "12:hello world!,"
Or you could use escaping which is what I usually do. This has the
advantage that you don't need to know how long the data is in advance.
Eg, these are from a scheme which uses \t to seperate arguments and
\r or \n to seperate transactions. These are then escaped in the
actual data using these functions
def escape(s):
"""This escapes the string passed in, changing CR, LF, TAB and \\ into
\\r, \\n, \\t and \\\\"""
s = s.replace("\\", "\\\\")
s = s.replace("\r", "\\r")
s = s.replace("\n", "\\n")
s = s.replace("\t", "\\t")
return s
def unescape(s, _unescape_mappi ng = string.maketran s('tnr','\t\n\r '), _unescape_re = re.compile(r'\\ ([(rnt\\)])')):
"""This unescapes the string passed in, changing \\r, \\n, \\t and \\any_char into
CR, LF, TAB and any_char"""
def _translate(m):
return m.group(1).tran slate(_unescape _mapping)
return _unescape_re.su b(_translate, s)
(These functions have been through the optimisation mill which is why
they may not look immediately like how you might first think of
writing them!)
--
Nick Craig-Wood <ni**@craig-wood.com-- http://www.craig-wood.com/nick
David Hirschfield wrote:
I have a pair of programs which trade python data back and forth by
pickling up lists of objects on one side (using
pickle.HIGHEST_ PROTOCOL), and sending that data over a TCP socket
connection to the receiver, who unpickles the data and uses it.
So far this has been working fine, but I now need a way of separating
multiple chunks of pickled binary data in the stream being sent back and
forth.
Questions:
Is it safe to do what I'm doing? I didn't think there was anything
fundamentally wrong with sending binary pickled data, especially in the
closed, safe environment these programs operate under...but maybe I'm
making a poor assumption?
I was going to separate the chunks of pickled data with some well-formed
string, but couldn't that string potentially randomly appear in the
pickled data? Do I just pick an extremely
unlikely-to-be-randomly-generated string as the separator? Is there some
string that will definitely NEVER show up in pickled binary data?
I thought about base64 encoding the data, and then decoding on the
opposite side (like what xmlrpclib does), but that turns out to be a
very expensive operation, which I want to avoid, speed is of the essence
in this situation.
Is there a reliable way to determine the byte count of some pickled
binary data? Can I rely on len(<pickled data>) == bytes?
Instead of communicating directly with the TCP socket, you could talk
to it via an object which precedes each chunk with a byte count, and if
you're working with multiple streams of picked data, then each chunk
could also have an identifier which specified which stream it belonged
to.
David Hirschfield wrote:
I have a pair of programs which trade python data back and forth by
pickling up lists of objects on one side (using
pickle.HIGHEST_ PROTOCOL), and sending that data over a TCP socket
connection to the receiver, who unpickles the data and uses it.
So far this has been working fine, but I now need a way of separating
multiple chunks of pickled binary data in the stream being sent back and
forth.
[...]
Save yourself the trouble of implementing some sort of IPC mechanism
over sockets, and give Pyro a swing: http://pyro.sourceforge.net
In Pyro almost all of the nastyness that is usually associated with socket
programming is shielded from you and you'll get much more as well
(a complete pythonic IPC library).
It may be a bit heavy for what you are trying to do but it may
be the right choice to avoid troubles later when your requirements
get more complex and/or you discover problems with your networking code.
Hth,
---Irmen de Jong This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: coder_1024 |
last post by:
I'm trying to send a packet of binary data to a UDP server. If I send
a text string, it works fine. If I attempt to send binary data, it
sends a UDP packet with 0 bytes of data (just the headers). I can see
this because I'm running Ethereal and watching the packets.
I'm defining the packets as shown below:
$text_msg = "Hello, world\r\n";
$binary_msg = chr(0x01).chr(0x02).chr(0x03).chr(0x00).chr(0xA0);
$binary_msg_size = 5;
|
by: Ot?vio |
last post by:
Hi!
I'm havin a problem about sending a binary file with fsockopen. My
problem is solved when i do:
-------------------->8------CODE-----------------------------
<form action="200.120...../file.php" method="POST"
enctype="multipart/form-data">
<form name="file" />
<form name="ip_01" />
|
by: Sami Viitanen |
last post by:
Hello,
How can I check if a file is binary or text?
There was some easy way but I forgot it..
Thanks in adv.
|
by: Andrew McLean |
last post by:
I am writing a CGI script on an account that only has Python 1.5.2. It's
principal purpose is to take some user input from a form, do some
manipulation and return some information to the user. However, I would
also like it to send some information to an e-mail account (behind the
scenes).
The e-mail could be quite simple, but I wouldn't rule out wanting to
send a pickled object or two.
Now if I was using a recent Python I would just...
|
by: Maurizio Berlusconi |
last post by:
Hi all
I have (my mistake) overwritten the beginning of a file where a long
dictionary was pickled with another (pickled and much smaller) dictionary.
Since only 10% or so of the original and important data was overwritten, I
would be happy if I could somehow recover at least the part that I can
still see.
The situation is now the following. If I open the file in vi I can see :
| |
by: Schwarty |
last post by:
I hope I posted this to the correct group. If not, please let me know and I
will get it posted in the correct section.
I have a web application developed in ASP.NET using C# for the code behind.
I need to be able to print shipping labels from a web page within this
application. The shipping label data is pulled from a 3rd party application
and comes in binary format specific for certain brands of label printers. I
have a class library...
|
by: thorley |
last post by:
Greetings, since there was no reponse to my previous post about an
existing FastCGI server in python, I've taken to writing my own. (which
of course I'll share--*if* there's something to share ;)
My problem now, is that I need to send certain binary data over a
socket. That is, I want to make some bytes, and stuff them in a TCP
packet, send them down the pipe, and then listen for a response.
socket.send, as best I can tell, will only...
|
by: Miro |
last post by:
VB 2003
at the end of the code, this works great.
bytCommand = Encoding.ASCII.GetBytes("testing hello send text")
udpClient.Send(bytCommand, bytCommand.Length)
and this recieves it
Dim strReturnData As String = _
System.Text.Encoding.ASCII.GetString(receiveBytes)
|
by: tomPee |
last post by:
Hi,
I've bumbed into a slight problem now, and I just don't seem to know how to fix it. What I want to do is the following:
Send over a socket:
1. Number of files to be send (not as an integer, just as a string)
then for each file to be send:
2. Length of Filename (again as a string)
3. Filename
4. File as binary data.
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look !
Part I. Meaning of...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
| |
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it.
Here is my compilation command:
g++-12 -std=c++20 -Wnarrowing bit_field.cpp
Here is the code in...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |