urllib and urlleb2 modules in Python

[moconno5]

Hello, I am attempting to write a Python module to access a website and upload batched files I have created, but I'm not getting it to work properly. I want to be able to upload my file, and then capture the reply in a simple text file within Python. Here is the code that I have, which so far just copies the content of the webpage. I am using Python version 2.5.

#!/usr/bin/env python
 
import urllib
import sys
 
url = "http://genome-test.cse.ucsc.edu/cgi-bin/hgBlat?command=start&submit file = mouseguts1"
 
##################### connect to remote page #################
try:
    remote = urllib.urlopen(url)
except:
    print "cannot open URL"
    sys.exit()
 
#################### read the content of the remote page ####
content = remote.read()
print remote.info()
print content
 

Thanks for any and all help,

Mark

[ilikepython]

Hello, I am attempting to write a Python module to access a website and upload batched files I have created, but I'm not getting it to work properly. I want to be able to upload my file, and then capture the reply in a simple text file within Python. Here is the code that I have, which so far just copies the content of the webpage. I am using Python version 2.5.

#!/usr/bin/env python
 
import urllib
import sys
 
url = "http://genome-test.cse.ucsc.edu/cgi-bin/hgBlat?command=start&submit file = mouseguts1"
 
##################### connect to remote page #################
try:
    remote = urllib.urlopen(url)
except:
    print "cannot open URL"
    sys.exit()
 
#################### read the content of the remote page ####
content = remote.read()
print remote.info()
print content
 

Thanks for any and all help,

Mark

There is a python module called cgi, you might want to check that. I wish I could help you more but I don't deal with web pages all that much.

[bartonc]

Hiya, Mark. UC Santa Cruz, huh? Welcome to the Python Forum on TheScripts.com.

I'll look around a bit and see if I can't dig up an example for you.

There is an ongoing discussion that hints at problems with cookies may interfere with what the urllib is expecting. I notice that your site is trying to leave a cookie.

Just a thought.

[bartonc]

There is an ongoing discussion that hints at problems with cookies may interfere with what the urllib is expecting. I notice that your site is trying to leave a cookie.

This may be why that link doesn't automate the process in my browser, either.
I suppose that this works when cookies are not blocked?

[moconno5]

hello again, and thanks for the swift replies!

Actually I am studying at George Mason University in Virginia, but yes the website I am using is located in Santa Cruz. I'm studying for a Masters in Bioinformatics there. I originally had the url as:

url = "http://genome-test.cse.ucsc.edu/cgi-bin/hgBlat?command=start"

But then I tried some modifications with no success and yes, I have been leaving cookies activated. I will give the cgi module a look but I have only one semester of Python and this is my first attempt with client-side interaction. The website has a note saying that there is a 15 second delay for users who are automating their searchs, so I know that it is possible to do this, just don't know how yet.

Mark

[bartonc]

hello again, and thanks for the swift replies!

Actually I am studying at George Mason University in Virginia, but yes the website I am using is located in Santa Cruz. I'm studying for a Masters in Bioinformatics there. I originally had the url as:

url = "http://genome-test.cse.ucsc.edu/cgi-bin/hgBlat?command=start"

But then I tried some modifications with no success and yes, I have been leaving cookies activated. I will give the cgi module a look but I have only one semester of Python and this is my first attempt with client-side interaction. The website has a note saying that there is a 15 second delay for users who are automating their searchs, so I know that it is possible to do this, just don't know how yet.

Mark

Let's see if I can make my assumption clear (I do lots of communications work, but no web work):
1) The server wants to leave a cookie
2) The server needs permission to leave a cookie
3) The sever terminates the command if it doesn't get permission to leave a cookie
SO
4) You need to send the command AND handle the request to leave a cookie (probably why there's a delay)

I don't think that the cgi module is the one that you are after...

The 2.5 docs have documentation of the Cookie Module and the cookielib Module

[moconno5]

Thanks for the help. I have dropped the cgi module, and I was playing with the urllib and have come up with the following code:

#!/usr/bin/env python
# written 7/22/2007
# by Mark O'Connor
 
import urllib
 
def ReadSite():
 
    # First, encode the data.
    infile = open ('mouseguts1', 'r')
    data = infile.read()
    #print "Here is your data:"
    #print data
    # Now get that file-like object again, remembering to mention the data.
    f = urllib.urlopen("http://genome-test.cse.ucsc.edu/cgi-bin/hgBlat?command=start", data)
    # Read the results back.
    s = f.read()
    print s
    # s.close()
 

However passing the data doesn't seem to do anything. I know you suggested using a cookie, but I figured there must be a simple way to transmit a file to a page that has a "Submit File" button on it.

Mark

[moconno5]

I have also modified my code and tried to send as a cookie, but getting error messages. Here is the code:

 
#!/usr/bin/env python
# written 7/22/2007
# by Mark O'Connor
 
import urllib
import urllib2
import Cookie
 
def ReadSite():
 
 
    # First, encode the data.
    infile = open ('mouseguts1', 'r')
    data = infile.read()
    C = Cookie.SimpleCookie()
    C["SUBMIT"] = data
    #print "Here is your data:"
    #print data
    # Now get that file-like object again, remembering to mention the data.
    f = urllib2.urlopen("http://genome-test.cse.ucsc.edu/cgi-bin/hgBlat?command=start", C)
    # Read the results back.
    s = f.read()
    print s
    # s.close()
 
 

and the error message:

Traceback (most recent call last):
File "<pyshell#53>", line 1, in <module>
ReadSite()
File "URLReader.py", line 20, in ReadSite
f = urllib2.urlopen("http://genome-test.cse.ucsc.edu/cgi-bin/hgBlat?command=start", C)
File "C:\Python25\lib\urllib2.py", line 121, in urlopen
return _opener.open(url, data)
File "C:\Python25\lib\urllib2.py", line 374, in open
response = self._open(req, data)
File "C:\Python25\lib\urllib2.py", line 392, in _open
'_open', req)
File "C:\Python25\lib\urllib2.py", line 353, in _call_chain
result = func(*args)
File "C:\Python25\lib\urllib2.py", line 1101, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "C:\Python25\lib\urllib2.py", line 1073, in do_open
h.request(req.get_method(), req.get_selector(), req.data, headers)
File "C:\Python25\lib\httplib.py", line 862, in request
self._send_request(method, url, body, headers)
File "C:\Python25\lib\httplib.py", line 888, in _send_request
self.send(body)
File "C:\Python25\lib\httplib.py", line 707, in send
self.sock.sendall(str)
File "<string>", line 1, in sendall
TypeError: sendall() argument 1 must be string or read-only buffer, not SimpleCookie