473,903 Members | 4,410 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

BaseHTTPServer weirdness

I'm trying to figure out how to use BaseHTTPServer. Here's my little
test app:

=============== =============== ===

#!/usr/bin/python

from BaseHTTPServer import *

import cgi

class myHandler(BaseH TTPRequestHandl er):

def do_GET(r):
s = ''
try:
s = cgi.parse_qs(r. rfile.read(int( r.headers.get
("Content-length"))), 1)
except:
pass

r.send_response (200)
r.send_header(" Content-type", "text/html")
r.end_headers()
r.wfile.write(" ""
<form method=post action=foo>
<input type=text name=text1 value="">
<input type=text name=text2 value="">
<input type=submit>
</form%s
""" % s)

def do_POST(r):
r.do_GET()
d = HTTPServer(('', 1024), myHandler)
d.serve_forever ()

=============== =============== =====

Two questions:

1. The line:

s = cgi.parse_qs(r. rfile.read(int( r.headers.get(" Content-length"))), 1)

feels like a horrible hack. It seems like this would be a better
alternative:

s = cgi.parse(r.rfi le)

but that doesn't actually work. Why? What is the Right Way to parse
form data in a BaseHTTPServer?

2. Despite the fact that I'm passing a 1 for the keep_blank_valu es
argument to cgi.parse_qs, it doesn't actually keep blank values. Is
this a bug, or am I doing something wrong?

Thanks,
rg
Sep 11 '06 #1
13 2675
Ron Garret wrote:
I'm trying to figure out how to use BaseHTTPServer. Here's my little
test app:

=============== =============== ===

#!/usr/bin/python

from BaseHTTPServer import *

import cgi

class myHandler(BaseH TTPRequestHandl er):

def do_GET(r):
s = ''
try:
s = cgi.parse_qs(r. rfile.read(int( r.headers.get
("Content-length"))), 1)
except:
pass

r.send_response (200)
r.send_header(" Content-type", "text/html")
r.end_headers()
r.wfile.write(" ""
<form method=post action=foo>
<input type=text name=text1 value="">
<input type=text name=text2 value="">
<input type=submit>
</form%s
""" % s)

def do_POST(r):
r.do_GET()
d = HTTPServer(('', 1024), myHandler)
d.serve_forever ()

=============== =============== =====

Two questions:

1. The line:

s = cgi.parse_qs(r. rfile.read(int( r.headers.get(" Content-length"))), 1)

feels like a horrible hack. It seems like this would be a better
alternative:

s = cgi.parse(r.rfi le)

but that doesn't actually work. Why? What is the Right Way to parse
form data in a BaseHTTPServer?
The normal way is

s = cgi.parse()

since the CGI script sees the client network socket (after consumption
of HTTP headers) as its standard input. However I'm not sure how much it
currently does in the way on handling strange inputs like gzip
compressed data.
2. Despite the fact that I'm passing a 1 for the keep_blank_valu es
argument to cgi.parse_qs, it doesn't actually keep blank values. Is
this a bug, or am I doing something wrong?
Sounds like a bug, but then since your parsing looks buggy I'm surprised
you get anything at all. Try using a keyword argument
keep_blank_valu es=1 just in case the order has changed or something
daft. But fix your parsing first.

The other thing to note is that since you are putting a dictionary's
string representation out straight into your HTML if there are odd
characters in it this may give you strange output in the browser, so you
should view the page source to ensure that's not the case. Which it
probably isn't ...

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://holdenweb.blogspot.com
Recent Ramblings http://del.icio.us/steve.holden

Sep 11 '06 #2
In article <ma************ *************** **********@pyth on.org>,
Steve Holden <st***@holdenwe b.comwrote:
The normal way is

s = cgi.parse()

since the CGI script sees the client network socket (after consumption
of HTTP headers) as its standard input.
Doesn't work. (I even tried sys.stdin=r.rfi le; s=cgi.parse()) Don't
forget, this is not a CGI script, it's a handler for a BaseHTTPServer.

2. Despite the fact that I'm passing a 1 for the keep_blank_valu es
argument to cgi.parse_qs, it doesn't actually keep blank values. Is
this a bug, or am I doing something wrong?
Sounds like a bug, but then since your parsing looks buggy I'm surprised
you get anything at all. Try using a keyword argument
keep_blank_valu es=1 just in case the order has changed or something
daft. But fix your parsing first.

The other thing to note is that since you are putting a dictionary's
string representation out straight into your HTML if there are odd
characters in it this may give you strange output in the browser, so you
should view the page source to ensure that's not the case. Which it
probably isn't ...
I know that's not a problem because it does work when I use parse_qs.
(I know about escaping HTML and all that, but this is just a little test
program.)

rg
Sep 11 '06 #3
Ron Garret wrote:
In article <ma************ *************** **********@pyth on.org>,
Steve Holden <st***@holdenwe b.comwrote:

>>The normal way is

s = cgi.parse()

since the CGI script sees the client network socket (after consumption
of HTTP headers) as its standard input.


Doesn't work. (I even tried sys.stdin=r.rfi le; s=cgi.parse()) Don't
forget, this is not a CGI script, it's a handler for a BaseHTTPServer.

Right. My bad. However there's clearly something screwy going on,
because otherwise you'd expect to see at least an empty dictionary in
the output.

>
>>>2. Despite the fact that I'm passing a 1 for the keep_blank_valu es
argument to cgi.parse_qs, it doesn't actually keep blank values. Is
this a bug, or am I doing something wrong?

Sounds like a bug, but then since your parsing looks buggy I'm surprised
you get anything at all. Try using a keyword argument
keep_blank_va lues=1 just in case the order has changed or something
daft. But fix your parsing first.
Reading the source of the 2.4.3 library shows that someone added an
environ=os.envi ron argument, which will be the second argument on a
positional call, so that clears that mystery up. The doicumentation
should really show these as keyword arguments rather than implying they
are positionals. It'd be nice if you could report this as a
documentation bug - though I believe by now the 2.5rc2 release will be
frozen.
>>The other thing to note is that since you are putting a dictionary's
string representation out straight into your HTML if there are odd
characters in it this may give you strange output in the browser, so you
should view the page source to ensure that's not the case. Which it
probably isn't ...


I know that's not a problem because it does work when I use parse_qs.
(I know about escaping HTML and all that, but this is just a little test
program.)
I suspect that the remainder of your problems (cgi_parse appears to be
returning a *string*, dammit) are due to the fact that the process you
are running the HTTP server in doesn't have the environment variables
set that a server would set if it really were being called in a CGI
context, and which the CGI library expects to be set. You could try
passing them as an explicit environ argument and see if that worked.

But basically, you aren't providing a CGI environment, and that's why
cgi.parse() isn't working.

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://holdenweb.blogspot.com
Recent Ramblings http://del.icio.us/steve.holden

Sep 11 '06 #4
In article <ma************ *************** **********@pyth on.org>,
Steve Holden <st***@holdenwe b.comwrote:
But basically, you aren't providing a CGI environment, and that's why
cgi.parse() isn't working.
Clearly. So what should I be doing? Surely I'm not the first person to
have this problem?

I have managed to work around this for now by copying and modifying the
code in cgi.parse, but this still feels like a Horrible Hack to me.

rg
Sep 11 '06 #5
>But basically, you aren't providing a CGI environment, and that's why
>cgi.parse() isn't working.

Clearly. So what should I be doing?
Probably you'll need to read the source of cgi.parse_qs (like Steve did) and
see what it needs from os.environ and then provide that (either in
os.environ or in a custom environ dictionary).

BUT why don't you use WSGI?

--
damjan
Sep 11 '06 #6
Ron Garret wrote:
In article <ma************ *************** **********@pyth on.org>,
Steve Holden <st***@holdenwe b.comwrote:

>>But basically, you aren't providing a CGI environment, and that's why
cgi.parse() isn't working.


Clearly. So what should I be doing? Surely I'm not the first person to
have this problem?

I have managed to work around this for now by copying and modifying the
code in cgi.parse, but this still feels like a Horrible Hack to me.
Let me get this right. You are aware that CGIHTTPServer module exists.
But you don't want to use that. Instead you want to use your own code.
So you have ended up duplicating some of the functionality of the cgi
library. And it feels like a hack.

Have I missed anything? :-)

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://holdenweb.blogspot.com
Recent Ramblings http://del.icio.us/steve.holden

Sep 12 '06 #7
Steve Holden wrote:
Ron Garret wrote:
>In article <ma************ *************** **********@pyth on.org>,
Steve Holden <st***@holdenwe b.comwrote:

>>But basically, you aren't providing a CGI environment, and that's why
cgi.parse() isn't working.

Clearly. So what should I be doing? Surely I'm not the first person to
have this problem?

I have managed to work around this for now by copying and modifying the
code in cgi.parse, but this still feels like a Horrible Hack to me.
Let me get this right. You are aware that CGIHTTPServer module exists.
But you don't want to use that. Instead you want to use your own code.
So you have ended up duplicating some of the functionality of the cgi
library. And it feels like a hack.

Have I missed anything? :-)
Hey, be nice. Wanting to write a request handler that actually handles a
POST request doesn't seem so unreasonable.

Except...when there are about a bazillion Python web frameworks to
choose from, why start from BaseHTTPServer? Why not use one of the
simpler frameworks like Karrigell or Snakelets or CherryPy?

Here is the query-handling code from Karrigell's CustomHTTPServe r.py,
good at least for a second opinion:

def do_POST(self):
"""Begin serving a POST request. The request data must be readable
on a file-like object called self.rfile"""
ctype, pdict =
cgi.parse_heade r(self.headers. getheader('cont ent-type'))
self.body = cgi.FieldStorag e(fp=self.rfile ,
headers=self.he aders, environ = {'REQUEST_METHO D':'POST'},
keep_blank_valu es = 1, strict_parsing = 1)
# throw away additional data [see bug #427345]
while select.select([self.rfile._soc k], [], [], 0)[0]:
if not self.rfile._soc k.recv(1):
break
self.handle_dat a()

Here is CherryPy's version from CP 2.1:

# Create a copy of headerMap with lowercase keys because
# FieldStorage doesn't work otherwise
lowerHeaderMap = {}
for key, value in request.headerM ap.items():
lowerHeaderMap[key.lower()] = value

# FieldStorage only recognizes POST, so fake it.
methenv = {'REQUEST_METHO D': "POST"}
try:
forms = _cpcgifs.FieldS torage(fp=reque st.rfile,
headers=lowerHe aderMap,
environ=methenv ,
keep_blank_valu es=1)

where _cpcgifs.FieldS torage is cgi.FieldStorag e with some extra accessors.

HTH,
Kent
Sep 12 '06 #8
In article <ma************ *************** **********@pyth on.org>,
Steve Holden <st***@holdenwe b.comwrote:
Ron Garret wrote:
In article <ma************ *************** **********@pyth on.org>,
Steve Holden <st***@holdenwe b.comwrote:

>But basically, you aren't providing a CGI environment, and that's why
cgi.parse() isn't working.

Clearly. So what should I be doing? Surely I'm not the first person to
have this problem?

I have managed to work around this for now by copying and modifying the
code in cgi.parse, but this still feels like a Horrible Hack to me.
Let me get this right. You are aware that CGIHTTPServer module exists.
But you don't want to use that.
That's right. I don't want to run CGI scripts. I don't want to launch
a new process for every request. I want all requests handled in the
server process.
Instead you want to use your own code.
No, the whole reason I'm asking this question is because I *don't* want
to write my own code. It seems to me that the code to do what I want
ought to be out there (or in there) somewhere and I shouldn't have to
reinvent this wheel. But I can't find it.
So you have ended up duplicating some of the functionality of the cgi
library. And it feels like a hack.
Yep.

rg
Sep 12 '06 #9
In article <3q************ ***@newsreading 01.news.tds.net >,
Kent Johnson <ke**@kentsjohn son.comwrote:
Steve Holden wrote:
Ron Garret wrote:
In article <ma************ *************** **********@pyth on.org>,
Steve Holden <st***@holdenwe b.comwrote:
But basically, you aren't providing a CGI environment, and that's why
cgi.parse() isn't working.

Clearly. So what should I be doing? Surely I'm not the first person to
have this problem?

I have managed to work around this for now by copying and modifying the
code in cgi.parse, but this still feels like a Horrible Hack to me.
Let me get this right. You are aware that CGIHTTPServer module exists.
But you don't want to use that. Instead you want to use your own code.
So you have ended up duplicating some of the functionality of the cgi
library. And it feels like a hack.

Have I missed anything? :-)

Hey, be nice. Wanting to write a request handler that actually handles a
POST request doesn't seem so unreasonable.

Except...when there are about a bazillion Python web frameworks to
choose from, why start from BaseHTTPServer? Why not use one of the
simpler frameworks like Karrigell or Snakelets or CherryPy?
It may come to that. I just thought that what I'm trying to do is so
basic that it ought to be part of the standard library. I mean, what do
people use BaseHTTPServer for if you can't parse form input?
Here is the query-handling code from Karrigell's CustomHTTPServe r.py,
good at least for a second opinion:

def do_POST(self):
"""Begin serving a POST request. The request data must be readable
on a file-like object called self.rfile"""
ctype, pdict =
cgi.parse_heade r(self.headers. getheader('cont ent-type'))
self.body = cgi.FieldStorag e(fp=self.rfile ,
headers=self.he aders, environ = {'REQUEST_METHO D':'POST'},
keep_blank_valu es = 1, strict_parsing = 1)
# throw away additional data [see bug #427345]
while select.select([self.rfile._soc k], [], [], 0)[0]:
if not self.rfile._soc k.recv(1):
break
self.handle_dat a()

Here is CherryPy's version from CP 2.1:

# Create a copy of headerMap with lowercase keys because
# FieldStorage doesn't work otherwise
lowerHeaderMap = {}
for key, value in request.headerM ap.items():
lowerHeaderMap[key.lower()] = value

# FieldStorage only recognizes POST, so fake it.
methenv = {'REQUEST_METHO D': "POST"}
try:
forms = _cpcgifs.FieldS torage(fp=reque st.rfile,
headers=lowerHe aderMap,
environ=methenv ,
keep_blank_valu es=1)

where _cpcgifs.FieldS torage is cgi.FieldStorag e with some extra accessors.
Here's what I actually ended up doing:

def parse(r):
ctype = r.headers.get(' content-type')
if not ctype: return None
ctype, pdict = cgi.parse_heade r(ctype)
if ctype == 'multipart/form-data':
return cgi.parse_multi part(r.rfile, pdict)
elif ctype == 'application/x-www-form-urlencoded':
clength = int(r.headers.g et('Content-length'))
if maxlen and clength maxlen:
raise ValueError, 'Maximum content length exceeded'
return cgi.parse_qs(r. rfile.read(clen gth), 1)
else:
return None

which is copied more or less directly from cgi.py. But it still seems
to me like this (or something like it) ought to be standardized in one
of the *HTTPServer.py modules.

But what do I know?

rg
Sep 12 '06 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
2029
by: Joshua W. Biagio | last post by:
Hello all, I am having difficulty getting the HTTP/1.1 support (i.e. pipelining and persistent connections) for the BaseHTTPServer library to work. When I change the line for the protocol in BaseHTTPServer from "HTTP/1.0" to "HTTP/1.1", the SimpleHTTPServer program can still serve directory listings and even graphics files, but it hangs when I try to browse to an HTML file. I have a directory of web pages that I use to test servers (such as...
1
1901
by: Yin | last post by:
Hello. I am using the basehttpserver to implement the HTTP protocol to serve a fairly large lexicon that I have loaded as a dictionary in python. Rather than writing a whole server, I would like to reuse the BaseHTTPserver classes. I am interested in finding a way to serve the dict without loading the whole dict into memory everytime an HTTP request is made. The dict lives in local memory when it is loaded and takes a long time to load.
2
6387
by: Tortelini | last post by:
I am making custom web server using HTTPServer and want to be able to access it simultaneously from different computers. To achieve multithreading, I have been experimenting with ThreadingMixIn from SocketServer, but it doesn't seem to work, when I freeze code in one instance it seems to be frozen in second one too (I experimented with time.sleep and while 1 loop). I am using python 2.4 on Windows XP. Does anyone have any suggestions? Do I...
4
3571
by: amfr | last post by:
>From the BaseHTTPServer module, how do i gget the POST or GET data sent by the client? Is it stired the the file they requested? e.g. objectname.path
5
1720
by: David Thielen | last post by:
Hi; I am creating png files in my ASP .NET app. When I am running under Windows 2003/IIS 6, the file is not given the security permissions it should have. It does not have any permission for several users that the directory it is in has, including IUSR_JASMINE (my system is named jasmine). Here is the weird part. In my aps .net code I have the following: String jname = Request.PhysicalApplicationPath + "images\\java_" + fileNum +...
0
1005
by: Jeff Gercken | last post by:
I want my http server to return a default form, regardless of the GET request, but I'm a bit of a noob and have no idea how. I'm trying to write a platform-independent captured portal for my wireless AP. This is what I have: #!/usr/bin/env python import CGIHTTPServer import BaseHTTPServer class ServerHandler(CGIHTTPServer.CGIHTTPRequestHandler): cgi_directories=
0
1363
by: Ron Garret | last post by:
I'm write a web server using BaseHTTPServer. It can't be a CGI because it has to do some weird server-push stuff as database updates come in. But I still need to process form inputs as if it were a CGI. But the cgi module only works in a CGI environment. Is there something with the equivalent functionality of cgi.FieldStorage for parsing form input in a non-CGI environment such as a BaseHTTPServer? I looked at the cgi module code but...
3
2408
by: Ron Garret | last post by:
I have a fairly large web app written in Python as a CGI fairly elaborate CGI. All of the requests go through a single CGI script which does authentication and session management and then dispatches to one of a number of handlers that generate the various pages. There is one page that is a performance bottleneck (because it is accessed automatically once a second by an XMLHTTPRequest, but that's another story). I have fixed this by...
0
1096
by: samwyse | last post by:
I've just now submitted two issues to the issue tracker: 1491 BaseHTTPServer incorrectly implements response code 100 RFC 2616 sec 8.2.3 states, "An origin server that sends a 100 (Continue) response MUST ultimately send a final status code, once the request body is received and processed, unless it terminates the transport connection prematurely." The obvious way to do this is to invoke the 'send_response' method twice, once with...
0
10007
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
11297
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
1
10990
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
10504
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
9693
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
7215
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5899
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
6100
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
2
4316
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.