Hello,
I would like to develop a new network protocol, where the server and the
clients are Python programs.
I think to be effective, I need to use TCP_NODELAY, and manually
buffered transfers.
I would like to create a general messaging object that has methods like
sendinteger
recvinteger
sendstring
recvstring
To be more secure, I think I can use this loads function to transfer
more elaborate python stuctures:
def loads(s):
"""Loads an object from a string.
@param s: The string to load the object from.
@return: The object loaded from the string. This function will not
unpickle globals and instances.
"""
f = cStringIO.Strin gIO(s)
p = cPickle.Unpickl er(f)
p.find_global = None
return p.load()
Am I on the right way to develop a new protocol?
Are there any common mistakes that programmers do?
Is there a howto where I can read more about this?
Thanks
Les 12 5531
> Am I on the right way to develop a new protocol? Are there any common mistakes that programmers do? Is there a howto where I can read more about this?
If you _must_ develop your own protocol, use at least twisted. But I'd
go for an existing solutions out there - namely pyro. No need to invent
wheels again :)
Regards,
Diez
On Mon, 12 Dec 2005, Laszlo Zsolt Nagy wrote: I think to be effective, I need to use TCP_NODELAY, and manually buffered transfers.
Why?
I would like to create a general messaging object that has methods like
sendinteger recvinteger sendstring recvstring
Okay. So you're really developing a marshalling layer, somewhere between
the transport and application layers - fair enough, there are a lot of
protocols that do that.
To be more secure,
Do you really mean secure? I don't think using pickle will give you
security. If you want security, run your protocol over an TLS/SSL
connection.
If, however, you mean robustness, then this is a reasonable thing to do -
it reduces the amount of code you have to write, and so reduces the number
of bugs you'll write! One thing to watch out for, though, is the
compatibility of the pickling at each end - i have no idea what the
backwards- and forwards-compatibility of the pickle protocols is like, but
you might find that if they're on different python versions, the ends
won't understand each other. Defining your own protocol down to the
bits-on-the-socket level would preclude that possibility.
I think I can use this loads function to transfer more elaborate python stuctures:
def loads(s): """Loads an object from a string. @param s: The string to load the object from. @return: The object loaded from the string. This function will not unpickle globals and instances. """ f = cStringIO.Strin gIO(s) p = cPickle.Unpickl er(f) p.find_global = None return p.load()
I don't know the pickle module, so i can't comment on the code.
Am I on the right way to develop a new protocol?
Aside from the versioning issue i mention above, you should bear in mind
that using pickle will make it insanely hard to implement this protocol in
any language other than python (unless someone's implemented a python
pickle library in it - is there such a beast for any other language?).
Personally, i'd steer clear of doing it like this, and try to use an
existing, language-neutral generic marshalling layer. XML and ASN.1 would
be the obvious ones, but i wouldn't advise using either of them, as
they're abominations. JSON would be a good choice: http://www.json.org/
If it's expressive enough for your objects. This is a stunningly simple
format, and there are libraries for working with it for a wide range of
languages.
Are there any common mistakes that programmers do?
The key one, i'd say, is not thinking about the future. Make sure your
protocol is able to grow - use a version number, so peers can figure out
what language they're talking, and perhaps an option negotiation
mechanism, if you're doing anything complex enough to warrant it (hey, you
could always start without it and add it in a later version!). Try to
allow for addition of new commands, message types or whatever, and for
extension of existing ones (within reason).
Is there a howto where I can read more about this?
Not really - protocol design is a bit of a black art. Someone asked about
this on comp.protocols. tcp-ip a while ago: http://groups.google.co.uk/group/com...ca111d67768b83
And didn't get much in the way of answers. Someone did point to this,
though: http://www.internet2.edu/~shalunov/w...ol-design.html
Although i don't agree with much of what that says.
tom
--
limited to concepts that are meta, generic, abstract and philosophical --
IEEE SUO WG
Tom Anderson wrote: I think to be effective, I need to use TCP_NODELAY, and manually buffered transfers.
Why?
Because of the big delays when sending small messages (size < 1500 bytes).
Personally, i'd steer clear of doing it like this, and try to use an existing, language-neutral generic marshalling layer. XML and ASN.1 would be the obvious ones, but i wouldn't advise using either of them, as they're abominations. JSON would be a good choice:
http://www.json.org/
I need to send Python objects too. They are too elaborate to convert
them to XML. (They are using cyclic weak references and other Python
specific stuff.) I can be sure that on both sides, there are Python
programs. Is there any advantage in using XML if I already need to send
Python objects? Those objects cannot be represented in XML, unless
pickled into a CDATA string.
And didn't get much in the way of answers. Someone did point to this, though:
http://www.internet2.edu/~shalunov/w...ol-design.html
Hmm, this was very helpful. Thank you!
Les
Il 2005-12-12, Laszlo Zsolt Nagy <ga*****@design aproduct.biz> ha scritto: Hello,
I would like to develop a new network protocol, where the server and the clients are Python programs.
You should use Twisted for this:
Writing clients http://twistedmatrix.com/projects/co...o/clients.html
Writing servers http://twistedmatrix.com/projects/co...o/servers.html
I think to be effective, I need to use TCP_NODELAY, and manually buffered transfers. I would like to create a general messaging object that has methods like
sendinteger recvinteger sendstring recvstring
You can inherit from twisted.interne t.protocol.Prot ocol or one of its
subclasses, they handle buffering and all sort of these things for
you. Cannot have to reinvent the wheel.
To be more secure, I think I can use this loads function to transfer more elaborate python stuctures:
def loads(s): """Loads an object from a string.
@param s: The string to load the object from. @return: The object loaded from the string. This function will not unpickle globals and instances. """ f = cStringIO.Strin gIO(s) p = cPickle.Unpickl er(f) p.find_global = None return p.load()
Using untrusted pickle loading is *NOT* more secure: http://www.python.org/doc/2.2.3/lib/pickle-sec.html
and http://www.livejournal.com/users/jcalderone/15864.html
--
Lawrence - http://www.oluyede.org/blog
"Anyone can freely use whatever he wants but the light at the end
of the tunnel for most of his problems is Python"
Il 2005-12-13, Laszlo Zsolt Nagy <ga*****@design aproduct.biz> ha scritto: I need to send Python objects too. They are too elaborate to convert them to XML. (They are using cyclic weak references and other Python specific stuff.) I can be sure that on both sides, there are Python programs. Is there any advantage in using XML if I already need to send Python objects? Those objects cannot be represented in XML, unless pickled into a CDATA string.
If you have to send Python objects over the wire the best way is to have
twisted on both ends and use twisted's PerspectiveBrok er, see: http://twistedmatrix.com/projects/co.../pb-intro.html
--
Lawrence - http://www.oluyede.org/blog
"Anyone can freely use whatever he wants but the light at the end
of the tunnel for most of his problems is Python"
Laszlo Zsolt Nagy wrote: I need to send Python objects too. They are too elaborate to convert them to XML. (They are using cyclic weak references and other Python specific stuff.) I can be sure that on both sides, there are Python programs. Is there any advantage in using XML if I already need to send Python objects? Those objects cannot be represented in XML, unless pickled into a CDATA string.
Try Pyro http://pyro.sourceforge.net
before rolling your own Python-specific protocol.
--Irmen Try Pyro http://pyro.sourceforge.net before rolling your own Python-specific protocol.
You are right. I wanted to use pyro before, because it is well tested
and it has nice features.
Unfortunately, it is not good for me. :-(
I already have my own classes. My objects are in object ownership trees,
and they are referencing to each other (weakly and strongly). These
classes have their own streaming methods, and they can be pickled
safely. This is something that pyro cannot handle. It cannot handle
these ownership object trees with weak references. Pyro can distribute
distinct objects only, and it uses an 'object naming scheme'. This is
not what I want. My objects do not have a name, and I do not want to
create 'technical names' just to force the Pyro style access. (Another
problem is that I do not want to rewrite my classes and inherit them
from the Pyro base object class.)
Thanks for the comment. I'm going to check the Pyro documentation again.
I might find something useful.
Les
Laszlo Zsolt Nagy <ga*****@design aproduct.biz> writes: I already have my own classes. My objects are in object ownership trees, and they are referencing to each other (weakly and strongly). These classes have their own streaming methods, and they can be pickled safely.
Standard warning: if you're accepting requests from potentially
hostile sources, don't use pickle.
Paul Rubin wrote: Laszlo Zsolt Nagy <ga*****@design aproduct.biz> writes:
I already have my own classes. My objects are in object ownership trees, and they are referencing to each other (weakly and strongly). These classes have their own streaming methods, and they can be pickled safely.
Standard warning: if you're accepting requests from potentially hostile sources, don't use pickle.
Yes, I know. Not talking about TLS/SSL - there can be hostile persons,
knowing a valid password and using a modified client program.
But how can I transfer pure python objects otherwise? Pyro also uses
Pickle and it also transfers bytecode.
Well, Pyro has an option to use XML messaging, but that is very
restricted, you cannot publish arbitrary python objects with XML. :-(
I read somewhere that Pickle had a security problem before Python 2.2,
but after 2.2 it has been solved.
BTW how CORBA or COM does this? They can do object marshaling safely.
Can we do the same with Python?
Isn't it enough to implement find_global of a cPickler ?
Les This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: Tom Plunket |
last post by:
Hey all-
A friend of mine has built this interesting (to me) bit of
hardware that's intended to communicate with a PC over ethernet.
The hardware has an OS written in C, and has some PC-side testing
software written in Delphi, all of which I have access to. The
hardware communicates with the PC via UDP.
I've been working on porting the PC "test" software to Python so
that 1) I understand the code, 2) don't need to go buy Delphi to
|
by: DaveC |
last post by:
I know this is the wrong group but I need a little pointing in a direction
please.
I need to write an implementation of a yet to be published protocol. It is
transported over the internet via IP. Can any one help me with ideas of how
to write such drivers in Windows? or if that's not doable should I develop
it in Linux? This is just for testing.
Thanks
DaveC
|
by: John Walton |
last post by:
Hello, everyone. I just began school, and they
already assigned us science fair. Since I'm in 8th
grade, I get to do demonstrations for our projects.
I'm probably going to demonstrate Python's networking
capabilities by writing a simple instant messenger
program. I only have a few problems:
1. I know squat about Python network Programming
2. I know nothing about networks
|
by: Roland Hedberg |
last post by:
Hi!
I need a fast protocol to use between a client and a server, both
sides written i Python.
What the protocol has to accomplish is extremely simple; the client
sends a number of lines (actually a RDF) and the server accepts or
rejects the packet. That's all !
Now, presently I'm using Twisted and XMLRPC ( why so is a long
|
by: David W. Fenton |
last post by:
I'm no stranger to this error message, but I have a client who is
experiencing it, but, fortunately, without any actual data
corruption, and it's driving them made. Their inability to grasp
that the cause is fundamentally a networking problem is also making
me pull my hair out.
Some history:
The network is composed of five computers dating back to 1995, each
purchased one at a time, and each having a different OS, etc. Until
| |
by: Ivan Voras |
last post by:
I have a simple network protocol client (it's a part of this:
http://sqlcached.sourceforge.net) implemented in Python, PHP and C.
Everything's fine, except that the Python implementation is the slowest
- up to 30% slower than the PHP version (which implements exactly the
same logic, in a class).
In typical usage (also in the benchmark), an object is created and
..query is called repeatedly. Typical numbers for the benchmark are:
For...
|
by: diffuser78 |
last post by:
I am a newbie in python. I want to learn and implement a small
networking concept. Please help me. Every help is appreciated.
I have one Linux Box and one Windows PC. I want to have a daemon
running on Windows PC which listens on some specicif port number. I
want to send a TCP/IP or UDP/IP packet from Linux box to Windows PC to
start some application. As Windows PC recieves such a packet from Linux
Box it executes a certain .exe file. I...
|
by: DanielJohnson |
last post by:
I was wondering if anyblody can suggest me a network simulator written
in python in which I can add on my own code and extend its
functionality.
I am looking for a simulator which will simualte TCP, UDP, RTP and
most networking protocol. The learning curve for ns2 and other
simulator is too high for me at this time and thats why I am
considering Python for it.
Every help will be appreciated.
|
by: aki |
last post by:
Hi all,
i am writing codes for implementing network protocols.
i have written a method which is receiving a packet from network.
i have assumed that the packet i will receive will be of type char*.
i need to test my method and
for that i want to create a network buffer which will contain a
packet of format as->
example of Configuration req packet
uint8_t code;
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
|
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it.
Here is my compilation command:
g++-12 -std=c++20 -Wnarrowing bit_field.cpp
Here is the code in...
| |
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
|
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: adsilva |
last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
|
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |