473,379 Members | 1,032 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,379 software developers and data experts.

Socket module bug on OpenVMS

Hi,

Recently I was bitten by an apparent bug in the BSD socket layer
on Open VMS. Specifically, it appears that VMS defines MSG_WAITALL
in socket.h but does not implement it (it is not in the documentation).
And I use the socket.MSG_WAITALL flag on my recv() calls... and
then they crash on OpenVMS.

I don't have access to an OpenVMS machine myself so could someone
else that has (or has more knowledge of it) shed some light on it?

This also raises the question to what extent Python itself should
work around platform specific "peculiarities", such as this one.
There's another problem with socket code on Windows and VMS systems,
where you get strange exceptions when using a "too big" recv() buffer.

Things like this force me into writing all sorts of error checking
code or platform specific functions, to work around these bugs.
Just for what was supposed to be a simple socket recv() and a
simple socket send()...

In my opinion Python's socket module itself could implement these
workarounds. That would make user code a lot cleaner and less
error prone, and more portable. What do other people think?
Regards
--Irmen de Jong
Oct 22 '06 #1
9 2129
Hi,
>
Recently I was bitten by an apparent bug in the BSD socket layer
on Open VMS. Specifically, it appears that VMS defines MSG_WAITALL
in socket.h but does not implement it (it is not in the documentation).
And I use the socket.MSG_WAITALL flag on my recv() calls... and
then they crash on OpenVMS.
Which Python version, OpenVMS version, IP stack and stack version?

If you think this is a Python on OpenVMS problem, send me a small
reproduced anf I will take a look.

If you think it's a OpenVMS problem and if you can provide a simple
reproducer and have a support contract I suggest you call HP, but I
suspect that if it's not documented the reply would be not (yet?) supported.
I don't have access to an OpenVMS machine myself so could someone
else that has (or has more knowledge of it) shed some light on it?

It appear that the only place, in the Python source code, where
MSG_WAITALL is used is in socketmodule.c:
#ifdef MSG_WAITALL
PyModule_AddIntConstant(m, "MSG_WAITALL", MSG_WAITALL);
#endif

May be a workaround is to not use MSG_WAITALL (currently) on OpenVMS and
in next version I will not defined MSG_WAITALL in the socket module on
OpenVMS.

If you need an access to an OpenVMS let me know.
>
This also raises the question to what extent Python itself should
work around platform specific "peculiarities", such as this one.
There's another problem with socket code on Windows and VMS systems,
where you get strange exceptions when using a "too big" recv() buffer.
Old Python version on OpenVMS have bugs in the socket module which have
been fixed in newer version. Be sure to check using the latest kit.
Things like this force me into writing all sorts of error checking
code or platform specific functions, to work around these bugs.
Just for what was supposed to be a simple socket recv() and a
simple socket send()...

In my opinion Python's socket module itself could implement these
workarounds. That would make user code a lot cleaner and less
error prone, and more portable. What do other people think?
Regards
--Irmen de Jong
Jean-Francois Pieronne
Oct 22 '06 #2
Jean-François Piéronne wrote:
Which Python version, OpenVMS version, IP stack and stack version?
OpenVMS 7.3-2, Python 2.3.5, no idea about IP stack version.
If you think this is a Python on OpenVMS problem, send me a small
reproduced anf I will take a look.
I don't have any small case lying around (I can't reproduce it myself
because I lack access to an openVMS machine) but I think you should
be able to reproduce it by slightly altering one of the socket examples
that comes with Python. Just add the MSG_WAITALL to the recv() call:

something = sock.recv(somesize, socket.MSG_WAITALL)

and you should see it crash with a socket exception.

Mail me offline if you still need running example code (that I think
would expose the problem).
If you think it's a OpenVMS problem and if you can provide a simple
reproducer and have a support contract I suggest you call HP, but I
suspect that if it's not documented the reply would be not (yet?) supported.
I don't have anything to do with HP... the person that reported the
problem to me has, however. He's already aware of the problem.
May be a workaround is to not use MSG_WAITALL (currently) on OpenVMS and
in next version I will not defined MSG_WAITALL in the socket module on
OpenVMS.
How can I detect that I'm running on OpenVMS?
--Irmen
Oct 22 '06 #3
Irmen de Jong a écrit :
Jean-François Piéronne wrote:
>Which Python version, OpenVMS version, IP stack and stack version?

OpenVMS 7.3-2, Python 2.3.5, no idea about IP stack version.
Thanks, may be upgrade to Python 2.5 will solve the problem.
>If you think this is a Python on OpenVMS problem, send me a small
reproduced anf I will take a look.

I don't have any small case lying around (I can't reproduce it myself
because I lack access to an openVMS machine) but I think you should
be able to reproduce it by slightly altering one of the socket examples
that comes with Python. Just add the MSG_WAITALL to the recv() call:

something = sock.recv(somesize, socket.MSG_WAITALL)

and you should see it crash with a socket exception.
Ok, I will try.
Mail me offline if you still need running example code (that I think
would expose the problem).
Ok
>If you think it's a OpenVMS problem and if you can provide a simple
reproducer and have a support contract I suggest you call HP, but I
suspect that if it's not documented the reply would be not (yet?)
supported.

I don't have anything to do with HP... the person that reported the
problem to me has, however. He's already aware of the problem.
You or the other person can also try to use

http://forums1.itrc.hp.com/service/f...o?familyId=288

>May be a workaround is to not use MSG_WAITALL (currently) on OpenVMS and
in next version I will not defined MSG_WAITALL in the socket module on
OpenVMS.

How can I detect that I'm running on OpenVMS?

for example

import sys
print sys.platform
'OpenVMS'
JF
Oct 22 '06 #4
Irmen de Jong wrote:
This also raises the question to what extent Python itself should
work around platform specific "peculiarities", such as this one.
There's another problem with socket code on Windows and VMS systems,
where you get strange exceptions when using a "too big" recv() buffer.
what's a "strange exception" and a "too big" buffer?
Things like this force me into writing all sorts of error checking
code or platform specific functions, to work around these bugs.
Just for what was supposed to be a simple socket recv() and a
simple socket send()...
if you want buffering, use makefile(). relying on platform-specific
behaviour isn't a good way to write portable code.

</F>

Oct 23 '06 #5
Fredrik Lundh wrote:
Irmen de Jong wrote:
>This also raises the question to what extent Python itself should
work around platform specific "peculiarities", such as this one.
There's another problem with socket code on Windows and VMS systems,
where you get strange exceptions when using a "too big" recv() buffer.

what's a "strange exception" and a "too big" buffer?
The exceptions are MemoryError (I know this one for sure)
and a socket.error I believe (can't remember exactly, and I don't
have a VMS machine to try to reproduce).
Too big buffer means anything above 64 kilobyte or so.

You can find a lot of reports about this happening on Windows at least.
From user reports I've learned that VMS also has similar problems with
recv buffer sizes above a certain size.
>Things like this force me into writing all sorts of error checking
code or platform specific functions, to work around these bugs.
Just for what was supposed to be a simple socket recv() and a
simple socket send()...

if you want buffering, use makefile(). relying on platform-specific
behaviour isn't a good way to write portable code.
I'm not sure if makefile() would shield me from the problems I experienced
(I could try I suppose) but your second remark is exactly my point!
I don't *want* to code around platform-specific behavior. I *want* my
code to be portable. And I expected it to be portable by just using
a regular recv() call... However as it is now, Python's socket module
exposes platform specific (and troublesome) behavior, so I have to write
various workarounds to make my code run without errors on multiple platforms...

Regards,

--Irmen de Jong
Oct 23 '06 #6
Irmen de Jong schrieb:
In my opinion Python's socket module itself could implement these
workarounds. That would make user code a lot cleaner and less
error prone, and more portable. What do other people think?
It depends: when VMS defines MSG_WAITALL, but doesn't implement it
correctly, then Python probably shouldn't expose it.

Implementing recv() "splitting" is *not* something that Python
could do itself. The question is how you do error reporting
for partial results, and neither exception handling nor error
codes are particularly well-suited to such an error model: these
APIs all assume an all-or-none failure model (i.e. if receiving
fails, nothing has happened so far). If you were to split a
large recv call into multiple smaller ones, there wouldn't be
a good way to communicate both the partial result and the error
that you got on the last call.

Perhaps you had some different work-around in mind?

Regards,
Martin
Oct 23 '06 #7
Martin v. Löwis wrote:
Irmen de Jong schrieb:
>In my opinion Python's socket module itself could implement these
workarounds. That would make user code a lot cleaner and less
error prone, and more portable. What do other people think?

It depends: when VMS defines MSG_WAITALL, but doesn't implement it
correctly, then Python probably shouldn't expose it.
That would be one way of solving at least my troubles, because I
already check for the availability of MSG_WAITALL and revert to
a custom recv() loop otherwise.
As others have already reported, at this time Python (well, the build
process) merely checks for the availability of the MSG_WAITALL symbol
in the socket.h C-header file....
Implementing recv() "splitting" is *not* something that Python
could do itself. The question is how you do error reporting
for partial results
[..snip..]

Mm, tricky. Hadn't thought of that.

Although I'm not yet convinced about the all-or-nothing failure
model you were talking about. How can I know if recv() fails
with a MemoryError (or socket error) that it actually didn't
receive anything? Is that even an assumption I can rely on?
If it isn't, splitting up the recv() internally (into smaller
blocks) wouldn't be any different, right?
Perhaps you had some different work-around in mind?
Nope, I was indeed thinking about splitting up the recv() into
smaller blocks internally.
Maybe a more elaborate check in Python's build system (to remove
the MSG_WAITALL on VMS) would qualify as some sort of work-around
as well. Although that would not yet solve the errors you get when
using too big recv() buffer sizes.
Cheers,
--Irmen.
Oct 23 '06 #8
Irmen de Jong schrieb:
Although I'm not yet convinced about the all-or-nothing failure
model you were talking about. How can I know if recv() fails
with a MemoryError (or socket error) that it actually didn't
receive anything? Is that even an assumption I can rely on?
Yes, you can. It can't produce a MemoryError (AFAICT, there is
no code path which possibly could (*)). If there is a socket
error, then the operating system guarantees that no data
were written into the buffer, and that subsequent calls
will yield any data that might have been available.

(*) It can, of course, produce a MemoryError if the result buffer
can't be allocated. If that happens, no recv(2) call has been
made, so the MemoryError indicates that nothing has happened -
you need to provide the result buffer before calling recv; that's
why you need to provide the size of the message you want to
receive.
If it isn't, splitting up the recv() internally (into smaller
blocks) wouldn't be any different, right?
Right. However, directly calling operating system calls and directly
exposing their results allows for such all-or-nothing error model,
as the operating system API itself is designed with the goal of
atomicity for each operation (or, if there are partial results,
has a way of reporting them, e.g. for partial writes).
Nope, I was indeed thinking about splitting up the recv() into
smaller blocks internally.
Look at shutil.copytree for an example of what this leads to.
This routine originally traversed the input tree, copied all
files it could, and skipped over those it couldn't. It ignored
the basic principle "errors should never pass silently".

I then tried fixing it, and the result is really ugly: It
keeps a list of all exceptions that occurred, and then
raises a single exception that contains them all. The
caller is supposed to know this, and perhaps print out
multiple error messages.

shutil.rmtree takes a different approach, which could be
considered just as ugly: it provides an onerror call-back
function which is invoked whenever some remove operation
fails.

I also think there is some API that returns the partial
result inside the exception if there is an exception.
The caller has to know this, and look at any data that
are already in the exception. Of course, changing an
existing API to switch to such a convention should
be considered as an incompatible change.
Maybe a more elaborate check in Python's build system (to remove
the MSG_WAITALL on VMS) would qualify as some sort of work-around
as well.
Before that is done, somebody should confirm that MSG_WAITALL
is really not behaving correctly on VMS. Then, the check doesn't
have to be all that elaborate: a simple #ifdef would do.

Regards,
Martin
Oct 24 '06 #9
Irmen de Jong wrote:
Martin v. Löwis wrote:
[snip]
>Perhaps you had some different work-around in mind?

Nope, I was indeed thinking about splitting up the recv() into
smaller blocks internally.
Maybe a more elaborate check in Python's build system (to remove
the MSG_WAITALL on VMS) would qualify as some sort of work-around
as well. Although that would not yet solve the errors you get when
using too big recv() buffer sizes.
I have contact some guys from HP, unofficial reply is if it's not
documented it's not supported... Why it's defined in socket.h may be
because the TCP/IP stack from HP on OpenVMS is a port of the Tru64 Unix
stack and the flag was present in Tru64.
We will also investigate if the behaviour on OpenVMS when you use
MSG_WAITALL is broken or not.

So I will, in a forthcoming kit don't export this symbol under OpenVMS
(if it's not documented it's not supported).
I have also take a look at the problem of using big recv() buffer and
found a bug when the buffer size is 32767, only the first 32767 bytes
are read, this will be also fixed in the next kit (only Python 2.5).

I expect to have a new kit which include all these fixes be available
before the end of the week.
JF
Oct 24 '06 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: Koen Vossen | last post by:
Hi, I came across some exceptional behaviour (pun intended) when I was programming a small chat server. I've simplified my original code to the following minimal version that still exhibits the...
0
by: Jacob Lee | last post by:
I'm getting a rather bizarre error while using the socket module. If I start out disconnected from the net and then connect while the program or interpreter session is open, I do not always gain...
1
by: Roy Smith | last post by:
Is there any way to access the RFC 3542 defined macros such as ICMP6_FILTER_SETPASSALL() from within the socket module? A dir() on the socket module (Python 2.4.1, built on a solaris box with IPv6...
0
by: Carl Waldbieser | last post by:
I am trying to understand how the SSL object in the socket module is supposed to be used. From looking at the documentation, I can work out that you need to pass a socket into the ssl() function,...
1
by: bobano | last post by:
Hi everyone, I am writing a POP3 Client program in Perl. You connect to a POP3 Server and have a running conversation with the mail server using commands from the RFC 1939 Post Office Protocol....
2
by: Mirko Vogt | last post by:
Hey, it seems that the socket-module behaves differently on unix / windows when a timeout is set. Here an example: # test.py import socket...
0
by: Gabriel Genellina | last post by:
En Wed, 09 Jul 2008 15:02:56 -0300, Mirko Vogt <lists@nanl.deescribi�: Which Python version? Which Windows version? I've tried 2.3.4, 2.4.4, 2.5.1 and 3.0a4, all on WinXP SP2, and in all...
0
by: Mirko Vogt | last post by:
Gabriel Genellina wrote: Hey, this is strange. Linux: $ python --version Python 2.5.2 $ Windows:
2
by: joefazee | last post by:
I`m very new to wxpython, my idea is to create a simple GUI based network application using the python socket module. can someone please demonstrate a 'hello world' server/client using wxpython....
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...
0
by: ryjfgjl | last post by:
In our work, we often need to import Excel data into databases (such as MySQL, SQL Server, Oracle) for data analysis and processing. Usually, we use database tools like Navicat or the Excel import...
0
by: aa123db | last post by:
Variable and constants Use var or let for variables and const fror constants. Var foo ='bar'; Let foo ='bar';const baz ='bar'; Functions function $name$ ($parameters$) { } ...
0
by: ryjfgjl | last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.