i wanted to created a simple CGI script for uploading files to my
http server, and decided to use python for it. it seems to work
fine except for one issue: there is a lot of CPU overhead.

after profiling, it seems like:
self.read_lines_to_outerboundary()
which is called from
cgi.FieldStorage()
is the reason why it's so slow.

when uploading small files, you won't notice a difference, but if
you upload files larger than 2 megs, you can notice it. this
happens on both win2k and freebsd

is there some other way to process CGI, excluding doing it all
manually? will this code be improved in the future?
-- thx, Nehal

Nehal wrote:
[color=blue]
> i wanted to created a simple CGI script for uploading files to my
> http server, and decided to use python for it. it seems to work
> fine except for one issue: there is a lot of CPU overhead.
>
> after profiling, it seems like:
> self.read_lines_to_outerboundary()
> which is called from
> cgi.FieldStorage()
> is the reason why it's so slow.
>
> when uploading small files, you won't notice a difference, but if
> you upload files larger than 2 megs, you can notice it. this
> happens on both win2k and freebsd
>
> is there some other way to process CGI, excluding doing it all
> manually? will this code be improved in the future?
> -- thx, Nehal[/color]

The cgi module is a confusing mess of code munged, over the years, by
many hands. It would take a brave programmer to plunge in and do what's
necessary.

not-brave-enough-ly y'rs - steve
--
http://www.holdenweb.com
http://pydish.holdenweb.com
Holden Web LLC +1 800 494 3119

On Tue, 16 Nov 2004 07:40:17 -0500
Steve Holden <steve@holdenweb.com> wrote:
[color=blue]
> The cgi module is a confusing mess of code munged, over the
> years, by many hands. It would take a brave programmer to plunge
> in and do what's necessary.
>
> not-brave-enough-ly y'rs - steve[/color]
if i were to do it, could i rewrite the functions in the cgi
module?, or would i need to keep the same functions for backward
compatibility?
-- Nehal

Nehal wrote:
[color=blue]
> On Tue, 16 Nov 2004 07:40:17 -0500
> Steve Holden <steve@holdenweb.com> wrote:
>
>[color=green]
>>The cgi module is a confusing mess of code munged, over the
>>years, by many hands. It would take a brave programmer to plunge
>>in and do what's necessary.
>>
>>not-brave-enough-ly y'rs - steve[/color]
>
> if i were to do it, could i rewrite the functions in the cgi
> module?, or would i need to keep the same functions for backward
> compatibility?
> -- Nehal
>
>[/color]
Well, code breakage is regarded as pretty bad, so you would need to
retain the same interfaces that the module supports now.

Of course you could implement them completely differently, as long as
they worked the same, and you could add new interfaces as well.

regards
Steve
--
http://www.holdenweb.com
http://pydish.holdenweb.com
Holden Web LLC +1 800 494 3119

On 17 Nov 2004 05:47:05 -0800
and-google@doxdesk.com (Andrew Clover) wrote:
[color=blue]
> Nehal <nehalmistry@gmx.net> wrote:
>[color=green]
> > when uploading small files, you won't notice a difference, but
> > if you upload files larger than 2 megs, you can notice it.[/color]
>
> Yep. Large file upload in cgi.py is slow. I don't immediately
> see a way to speed it up without re-architecting some of its
> internals.
>
> In any case I dislike(*) the cgi module's interface too, so I
> rewrote the lot:
>
> http://www.doxdesk.com/software/py/form.html
>
> This isn't drop-in compatible, and is getting a bit crusty (I'm
> expecting to rewrite most of it soon to be more
> objecty/threadable, and support WSGI), but in my experience it's
> considerably faster than cgi for very large files. (We were
> commonly using files in the 10-50MB range.)
>
> (* - more then than now; cgi's interface has got slightly better
> since Python 1.5.2's time.)
>
> --
> Andrew Clover[/color]

I have tested Andrew's 'form.py' module, and also upload cgi
scripts from other languages, i did some benchmarking, i tried
uploading a 6 meg file to localhost and writing to an output file
on apache 2.0.52 win32. here are the results (note: i checked the
error log to make sure all scripts were working and processing the
data as expected):

ruby: 2 sec
Andrew Clover's form.py: 2.5 sec
perl: 2.5 sec
tcl (3rd party module): 4.5 sec
python: 8 sec

of course in practice, most people won't be receiving data at 3
megs/sec, so you won't have to process data at such a speed.
nevertheless, it will put a greater load on the CPU, which may be
an issue for many servers.

it would not be a good idea to put Andrew's form.py in the
official python distribution and have 2 different modules for
processing CGI. either it would have to somehow merged into CGI
module, and keeping backward compatibility, or the existing CGI
module must be optimized; maybe the above benchmark data will
motivate someone to do so ;). until then, i'll stick to form.py
-- thx, Nehal

Nehal Mistry <nehalmistry@gmx.net> wrote:
[color=blue]
> nevertheless, it will put a greater load on the CPU, which may be
> an issue for many servers.[/color]

It certainly was for the ricketty old SPARC box we were trying to
upload umpteen-MB files to! Probably not so critical for most people
these days though.
[color=blue]
> it would not be a good idea to put Andrew's form.py in the
> official python distribution and have 2 different modules for
> processing CGI.[/color]

Fully agree. In any case the current interface is not general enough
for std lib use.
[color=blue]
> either it would have to somehow merged into CGI
> module, and keeping backward compatibility[/color]

If there is interest I could certainly look at providing a cgi-alike
interface to the new version. Personally I was not aiming at the
standard library.

cheers,

--
Andrew Clover
mailto:and@doxdesk.com
http://www.doxdesk.com/