urllib problem (maybe bugs?)

[Timothy Wu]

Hi,

I'm trying to fill the form on page
http://www.cbs.dtu.dk/services/TMHMM/ using urllib.

There are two peculiarities. First of all, I am filling in incorrect
key/value pairs in the parameters on purpose because that's the only
way I can get it to work.. For "version" I am suppose to leave it
unchecked, having value of empty string. And for name "outform" I am
suppose to assign it a value of "-short". Instead, I left out
"outform" all together and fill in "-short" for version. I discovered
the method my accident.

After I've done that it works fine for small SEQ values. Then, when I
try to send large amount of data (1.4MB), it fails miserably with
AttributeError exception.

I highly suspect the two problems I have are the result of some bugs
in the urllib module. Any suggestions?

This is my code:

================================================== ==========
fd = open('secretory0_1.txt', 'r')
txt = fd.read()
fd.close()

params = urllib.urlencode({'SEQ': txt, 'configfile':
'/usr/opt/www/pub/CBS/services/TMHMM-2.0/TMHMM2.cf', 'version':
'-short'})
f = urllib.urlopen("http://www.cbs.dtu.dk/cgi-bin/nph-webface", params)
data = f.read()

start = data.find('follow <a href="h') + 16
end = data.find('">This link</a>')
secondurl = data[start:end]

f = urllib.urlopen(secondurl)
print f.read()
================================================== ==========
The value pairs I am suppose to fill in are:

SEQ => some sequence here
configfile => '/usr/opt/www/pub/CBS/services/TMHMM-2.0/TMHMM2.cf'
version => ''
outform => '-short'
The exception I get when sending secretory0_1.txt is:

C:\Documents and Settings\thw\®à*±>python testhttp.py
Traceback (most recent call last):
File "testhttp.py", line 11, in ?
f = urllib.urlopen("http://www.cbs.dtu.dk/cgi-bin/nph-webface", params)
File "C:\Python24\lib\urllib.py", line 79, in urlopen
return opener.open(url, data)
File "C:\Python24\lib\urllib.py", line 182, in open
return getattr(self, name)(url, data)
File "C:\Python24\lib\urllib.py", line 307, in open_http
return self.http_error(url, fp, errcode, errmsg, headers, data)
File "C:\Python24\lib\urllib.py", line 322, in http_error
return self.http_error_default(url, fp, errcode, errmsg, headers)
File "C:\Python24\lib\urllib.py", line 550, in http_error_default
return addinfourl(fp, headers, "http:" + url)
File "C:\Python24\lib\urllib.py", line 836, in __init__
addbase.__init__(self, fp)
File "C:\Python24\lib\urllib.py", line 786, in __init__
self.read = self.fp.read
AttributeError: 'NoneType' object has no attribute 'read'

Timothy

[Tim Roberts]

Timothy Wu <2h*****@gmail.com> wrote:

I'm trying to fill the form on page
http://www.cbs.dtu.dk/services/TMHMM/ using urllib.

There are two peculiarities. First of all, I am filling in incorrect
key/value pairs in the parameters on purpose because that's the only
way I can get it to work.. For "version" I am suppose to leave it
unchecked, having value of empty string. And for name "outform" I am
suppose to assign it a value of "-short". Instead, I left out
"outform" all together and fill in "-short" for version. I discovered
the method my accident.

After I've done that it works fine for small SEQ values. Then, when I
try to send large amount of data (1.4MB), it fails miserably with
AttributeError exception.

I highly suspect the two problems I have are the result of some bugs
in the urllib module. Any suggestions?

The <form> on that page wants multipart/form-data encoding, where each
parameter is embedded in a separate MIME section. That is common with very
large data, and is _required_ when the form includes a file upload. urllib
doesn't do that. It always sends plain old
application/x-www-form-urlencoded data.

I think you're going to have to roll your own sender.
--
- Tim Roberts, ti**@probo.com
Providenza & Boekelheide, Inc.