473,221 Members | 1,733 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,221 software developers and data experts.

urllib and urlleb2 modules in Python

Hello, I am attempting to write a Python module to access a website and upload batched files I have created, but I'm not getting it to work properly. I want to be able to upload my file, and then capture the reply in a simple text file within Python. Here is the code that I have, which so far just copies the content of the webpage. I am using Python version 2.5.

Expand|Select|Wrap|Line Numbers
  1. #!/usr/bin/env python
  2.  
  3. import urllib
  4. import sys
  5.  
  6. url = "http://genome-test.cse.ucsc.edu/cgi-bin/hgBlat?command=start&submit file = mouseguts1"
  7.  
  8. ##################### connect to remote page #################
  9. try:
  10.     remote = urllib.urlopen(url)
  11. except:
  12.     print "cannot open URL"
  13.     sys.exit()
  14.  
  15. #################### read the content of the remote page ####
  16. content = remote.read()
  17. print remote.info()
  18. print content
  19.  
Thanks for any and all help,

Mark
Jul 21 '07 #1
7 2478
ilikepython
844 Expert 512MB
Hello, I am attempting to write a Python module to access a website and upload batched files I have created, but I'm not getting it to work properly. I want to be able to upload my file, and then capture the reply in a simple text file within Python. Here is the code that I have, which so far just copies the content of the webpage. I am using Python version 2.5.

Expand|Select|Wrap|Line Numbers
  1. #!/usr/bin/env python
  2.  
  3. import urllib
  4. import sys
  5.  
  6. url = "http://genome-test.cse.ucsc.edu/cgi-bin/hgBlat?command=start&submit file = mouseguts1"
  7.  
  8. ##################### connect to remote page #################
  9. try:
  10.     remote = urllib.urlopen(url)
  11. except:
  12.     print "cannot open URL"
  13.     sys.exit()
  14.  
  15. #################### read the content of the remote page ####
  16. content = remote.read()
  17. print remote.info()
  18. print content
  19.  
Thanks for any and all help,

Mark
There is a python module called cgi, you might want to check that. I wish I could help you more but I don't deal with web pages all that much.
Jul 21 '07 #2
bartonc
6,596 Expert 4TB
Hiya, Mark. UC Santa Cruz, huh? Welcome to the Python Forum on TheScripts.com.

I'll look around a bit and see if I can't dig up an example for you.

There is an ongoing discussion that hints at problems with cookies may interfere with what the urllib is expecting. I notice that your site is trying to leave a cookie.

Just a thought.
Jul 21 '07 #3
bartonc
6,596 Expert 4TB
There is an ongoing discussion that hints at problems with cookies may interfere with what the urllib is expecting. I notice that your site is trying to leave a cookie.
This may be why that link doesn't automate the process in my browser, either.
I suppose that this works when cookies are not blocked?
Jul 21 '07 #4
hello again, and thanks for the swift replies!

Actually I am studying at George Mason University in Virginia, but yes the website I am using is located in Santa Cruz. I'm studying for a Masters in Bioinformatics there. I originally had the url as:

url = "http://genome-test.cse.ucsc.edu/cgi-bin/hgBlat?command=start"

But then I tried some modifications with no success and yes, I have been leaving cookies activated. I will give the cgi module a look but I have only one semester of Python and this is my first attempt with client-side interaction. The website has a note saying that there is a 15 second delay for users who are automating their searchs, so I know that it is possible to do this, just don't know how yet.

Mark
Jul 22 '07 #5
bartonc
6,596 Expert 4TB
hello again, and thanks for the swift replies!

Actually I am studying at George Mason University in Virginia, but yes the website I am using is located in Santa Cruz. I'm studying for a Masters in Bioinformatics there. I originally had the url as:

url = "http://genome-test.cse.ucsc.edu/cgi-bin/hgBlat?command=start"

But then I tried some modifications with no success and yes, I have been leaving cookies activated. I will give the cgi module a look but I have only one semester of Python and this is my first attempt with client-side interaction. The website has a note saying that there is a 15 second delay for users who are automating their searchs, so I know that it is possible to do this, just don't know how yet.

Mark
Let's see if I can make my assumption clear (I do lots of communications work, but no web work):
1) The server wants to leave a cookie
2) The server needs permission to leave a cookie
3) The sever terminates the command if it doesn't get permission to leave a cookie
SO
4) You need to send the command AND handle the request to leave a cookie (probably why there's a delay)

I don't think that the cgi module is the one that you are after...

The 2.5 docs have documentation of the Cookie Module and the cookielib Module
Jul 22 '07 #6
Thanks for the help. I have dropped the cgi module, and I was playing with the urllib and have come up with the following code:

Expand|Select|Wrap|Line Numbers
  1. #!/usr/bin/env python
  2. # written 7/22/2007
  3. # by Mark O'Connor
  4.  
  5. import urllib
  6.  
  7. def ReadSite():
  8.  
  9.     # First, encode the data.
  10.     infile = open ('mouseguts1', 'r')
  11.     data = infile.read()
  12.     #print "Here is your data:"
  13.     #print data
  14.     # Now get that file-like object again, remembering to mention the data.
  15.     f = urllib.urlopen("http://genome-test.cse.ucsc.edu/cgi-bin/hgBlat?command=start", data)
  16.     # Read the results back.
  17.     s = f.read()
  18.     print s
  19.     # s.close()
  20.  
However passing the data doesn't seem to do anything. I know you suggested using a cookie, but I figured there must be a simple way to transmit a file to a page that has a "Submit File" button on it.

Mark
Jul 24 '07 #7
I have also modified my code and tried to send as a cookie, but getting error messages. Here is the code:
Expand|Select|Wrap|Line Numbers
  1.  
  2. #!/usr/bin/env python
  3. # written 7/22/2007
  4. # by Mark O'Connor
  5.  
  6. import urllib
  7. import urllib2
  8. import Cookie
  9.  
  10. def ReadSite():
  11.  
  12.  
  13.     # First, encode the data.
  14.     infile = open ('mouseguts1', 'r')
  15.     data = infile.read()
  16.     C = Cookie.SimpleCookie()
  17.     C["SUBMIT"] = data
  18.     #print "Here is your data:"
  19.     #print data
  20.     # Now get that file-like object again, remembering to mention the data.
  21.     f = urllib2.urlopen("http://genome-test.cse.ucsc.edu/cgi-bin/hgBlat?command=start", C)
  22.     # Read the results back.
  23.     s = f.read()
  24.     print s
  25.     # s.close()
  26.  
  27.  
and the error message:

Traceback (most recent call last):
File "<pyshell#53>", line 1, in <module>
ReadSite()
File "URLReader.py", line 20, in ReadSite
f = urllib2.urlopen("http://genome-test.cse.ucsc.edu/cgi-bin/hgBlat?command=start", C)
File "C:\Python25\lib\urllib2.py", line 121, in urlopen
return _opener.open(url, data)
File "C:\Python25\lib\urllib2.py", line 374, in open
response = self._open(req, data)
File "C:\Python25\lib\urllib2.py", line 392, in _open
'_open', req)
File "C:\Python25\lib\urllib2.py", line 353, in _call_chain
result = func(*args)
File "C:\Python25\lib\urllib2.py", line 1101, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "C:\Python25\lib\urllib2.py", line 1073, in do_open
h.request(req.get_method(), req.get_selector(), req.data, headers)
File "C:\Python25\lib\httplib.py", line 862, in request
self._send_request(method, url, body, headers)
File "C:\Python25\lib\httplib.py", line 888, in _send_request
self.send(body)
File "C:\Python25\lib\httplib.py", line 707, in send
self.sock.sendall(str)
File "<string>", line 1, in sendall
TypeError: sendall() argument 1 must be string or read-only buffer, not SimpleCookie
Jul 25 '07 #8

Sign in to post your reply or Sign up for a free account.

Similar topics

11
by: Pater Maximus | last post by:
I am trying to implement the recipe listed at http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/211886 However, I can not get to first base. When I try to run import urllib...
4
by: bob_smith_17280 | last post by:
Hello, I'm doing a small website survey as a consultant for a company that has a large private lan. Basically, I'm trying to determine how many web sites there are on their network and what...
16
by: HMS Surprise | last post by:
I edited environment varialbes and have added C:\Python25\Lib to PYTHONPATH but get the no module message when the statement import urllib is executed. Even tried copying the urllib file to...
5
by: chrispoliquin | last post by:
Hi, I have a small Python script to fetch some pages from the internet. There are a lot of pages and I am looping through them and then downloading the page using urlretrieve() in the urllib...
0
by: veera ravala | last post by:
ServiceNow is a powerful cloud-based platform that offers a wide range of services to help organizations manage their workflows, operations, and IT services more efficiently. At its core, ServiceNow...
0
by: VivesProcSPL | last post by:
Obviously, one of the original purposes of SQL is to make data query processing easy. The language uses many English-like terms and syntax in an effort to make it easy to learn, particularly for...
3
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 3 Jan 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). For other local times, please check World Time Buddy In...
0
by: abbasky | last post by:
### Vandf component communication method one: data sharing ​ Vandf components can achieve data exchange through data sharing, state sharing, events, and other methods. Vandf's data exchange method...
2
by: jimatqsi | last post by:
The boss wants the word "CONFIDENTIAL" overlaying certain reports. He wants it large, slanted across the page, on every page, very light gray, outlined letters, not block letters. I thought Word Art...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: fareedcanada | last post by:
Hello I am trying to split number on their count. suppose i have 121314151617 (12cnt) then number should be split like 12,13,14,15,16,17 and if 11314151617 (11cnt) then should be split like...
0
Git
by: egorbl4 | last post by:
Скачал я git, хотел начать настройку, а там вылезло вот это Что это? Что мне с этим делать? ...
1
by: davi5007 | last post by:
Hi, Basically, I am trying to automate a field named TraceabilityNo into a web page from an access form. I've got the serial held in the variable strSearchString. How can I get this into the...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.