471,312 Members | 1,796 Online
Bytes | Software Development & Data Engineering Community
Post +

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 471,312 software developers and data experts.

urllib and urlleb2 modules in Python

Hello, I am attempting to write a Python module to access a website and upload batched files I have created, but I'm not getting it to work properly. I want to be able to upload my file, and then capture the reply in a simple text file within Python. Here is the code that I have, which so far just copies the content of the webpage. I am using Python version 2.5.

Expand|Select|Wrap|Line Numbers
  1. #!/usr/bin/env python
  2.  
  3. import urllib
  4. import sys
  5.  
  6. url = "http://genome-test.cse.ucsc.edu/cgi-bin/hgBlat?command=start&submit file = mouseguts1"
  7.  
  8. ##################### connect to remote page #################
  9. try:
  10.     remote = urllib.urlopen(url)
  11. except:
  12.     print "cannot open URL"
  13.     sys.exit()
  14.  
  15. #################### read the content of the remote page ####
  16. content = remote.read()
  17. print remote.info()
  18. print content
  19.  
Thanks for any and all help,

Mark
Jul 21 '07 #1
7 2222
ilikepython
844 Expert 512MB
Hello, I am attempting to write a Python module to access a website and upload batched files I have created, but I'm not getting it to work properly. I want to be able to upload my file, and then capture the reply in a simple text file within Python. Here is the code that I have, which so far just copies the content of the webpage. I am using Python version 2.5.

Expand|Select|Wrap|Line Numbers
  1. #!/usr/bin/env python
  2.  
  3. import urllib
  4. import sys
  5.  
  6. url = "http://genome-test.cse.ucsc.edu/cgi-bin/hgBlat?command=start&submit file = mouseguts1"
  7.  
  8. ##################### connect to remote page #################
  9. try:
  10.     remote = urllib.urlopen(url)
  11. except:
  12.     print "cannot open URL"
  13.     sys.exit()
  14.  
  15. #################### read the content of the remote page ####
  16. content = remote.read()
  17. print remote.info()
  18. print content
  19.  
Thanks for any and all help,

Mark
There is a python module called cgi, you might want to check that. I wish I could help you more but I don't deal with web pages all that much.
Jul 21 '07 #2
bartonc
6,596 Expert 4TB
Hiya, Mark. UC Santa Cruz, huh? Welcome to the Python Forum on TheScripts.com.

I'll look around a bit and see if I can't dig up an example for you.

There is an ongoing discussion that hints at problems with cookies may interfere with what the urllib is expecting. I notice that your site is trying to leave a cookie.

Just a thought.
Jul 21 '07 #3
bartonc
6,596 Expert 4TB
There is an ongoing discussion that hints at problems with cookies may interfere with what the urllib is expecting. I notice that your site is trying to leave a cookie.
This may be why that link doesn't automate the process in my browser, either.
I suppose that this works when cookies are not blocked?
Jul 21 '07 #4
hello again, and thanks for the swift replies!

Actually I am studying at George Mason University in Virginia, but yes the website I am using is located in Santa Cruz. I'm studying for a Masters in Bioinformatics there. I originally had the url as:

url = "http://genome-test.cse.ucsc.edu/cgi-bin/hgBlat?command=start"

But then I tried some modifications with no success and yes, I have been leaving cookies activated. I will give the cgi module a look but I have only one semester of Python and this is my first attempt with client-side interaction. The website has a note saying that there is a 15 second delay for users who are automating their searchs, so I know that it is possible to do this, just don't know how yet.

Mark
Jul 22 '07 #5
bartonc
6,596 Expert 4TB
hello again, and thanks for the swift replies!

Actually I am studying at George Mason University in Virginia, but yes the website I am using is located in Santa Cruz. I'm studying for a Masters in Bioinformatics there. I originally had the url as:

url = "http://genome-test.cse.ucsc.edu/cgi-bin/hgBlat?command=start"

But then I tried some modifications with no success and yes, I have been leaving cookies activated. I will give the cgi module a look but I have only one semester of Python and this is my first attempt with client-side interaction. The website has a note saying that there is a 15 second delay for users who are automating their searchs, so I know that it is possible to do this, just don't know how yet.

Mark
Let's see if I can make my assumption clear (I do lots of communications work, but no web work):
1) The server wants to leave a cookie
2) The server needs permission to leave a cookie
3) The sever terminates the command if it doesn't get permission to leave a cookie
SO
4) You need to send the command AND handle the request to leave a cookie (probably why there's a delay)

I don't think that the cgi module is the one that you are after...

The 2.5 docs have documentation of the Cookie Module and the cookielib Module
Jul 22 '07 #6
Thanks for the help. I have dropped the cgi module, and I was playing with the urllib and have come up with the following code:

Expand|Select|Wrap|Line Numbers
  1. #!/usr/bin/env python
  2. # written 7/22/2007
  3. # by Mark O'Connor
  4.  
  5. import urllib
  6.  
  7. def ReadSite():
  8.  
  9.     # First, encode the data.
  10.     infile = open ('mouseguts1', 'r')
  11.     data = infile.read()
  12.     #print "Here is your data:"
  13.     #print data
  14.     # Now get that file-like object again, remembering to mention the data.
  15.     f = urllib.urlopen("http://genome-test.cse.ucsc.edu/cgi-bin/hgBlat?command=start", data)
  16.     # Read the results back.
  17.     s = f.read()
  18.     print s
  19.     # s.close()
  20.  
However passing the data doesn't seem to do anything. I know you suggested using a cookie, but I figured there must be a simple way to transmit a file to a page that has a "Submit File" button on it.

Mark
Jul 24 '07 #7
I have also modified my code and tried to send as a cookie, but getting error messages. Here is the code:
Expand|Select|Wrap|Line Numbers
  1.  
  2. #!/usr/bin/env python
  3. # written 7/22/2007
  4. # by Mark O'Connor
  5.  
  6. import urllib
  7. import urllib2
  8. import Cookie
  9.  
  10. def ReadSite():
  11.  
  12.  
  13.     # First, encode the data.
  14.     infile = open ('mouseguts1', 'r')
  15.     data = infile.read()
  16.     C = Cookie.SimpleCookie()
  17.     C["SUBMIT"] = data
  18.     #print "Here is your data:"
  19.     #print data
  20.     # Now get that file-like object again, remembering to mention the data.
  21.     f = urllib2.urlopen("http://genome-test.cse.ucsc.edu/cgi-bin/hgBlat?command=start", C)
  22.     # Read the results back.
  23.     s = f.read()
  24.     print s
  25.     # s.close()
  26.  
  27.  
and the error message:

Traceback (most recent call last):
File "<pyshell#53>", line 1, in <module>
ReadSite()
File "URLReader.py", line 20, in ReadSite
f = urllib2.urlopen("http://genome-test.cse.ucsc.edu/cgi-bin/hgBlat?command=start", C)
File "C:\Python25\lib\urllib2.py", line 121, in urlopen
return _opener.open(url, data)
File "C:\Python25\lib\urllib2.py", line 374, in open
response = self._open(req, data)
File "C:\Python25\lib\urllib2.py", line 392, in _open
'_open', req)
File "C:\Python25\lib\urllib2.py", line 353, in _call_chain
result = func(*args)
File "C:\Python25\lib\urllib2.py", line 1101, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "C:\Python25\lib\urllib2.py", line 1073, in do_open
h.request(req.get_method(), req.get_selector(), req.data, headers)
File "C:\Python25\lib\httplib.py", line 862, in request
self._send_request(method, url, body, headers)
File "C:\Python25\lib\httplib.py", line 888, in _send_request
self.send(body)
File "C:\Python25\lib\httplib.py", line 707, in send
self.sock.sendall(str)
File "<string>", line 1, in sendall
TypeError: sendall() argument 1 must be string or read-only buffer, not SimpleCookie
Jul 25 '07 #8

Post your reply

Sign in to post your reply or Sign up for a free account.

Similar topics

11 posts views Thread by Pater Maximus | last post: by
4 posts views Thread by bob_smith_17280 | last post: by
16 posts views Thread by HMS Surprise | last post: by
5 posts views Thread by chrispoliquin | last post: by
reply views Thread by rosydwin | last post: by

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.