urlgrabber and ClientForm problem

[hp1980]

Hi,
I'm writing a web automation script using ClientForm and urlgrabber.
I use urlgrabber because I need the "http keepalive" which doesn't exist in urllib2.

I'm facing a problem, the "form.click()" returns a urllib2.Request object which I can't pass to urlgrabber.urlopen().

In the ClientForm web page, I see "see HTMLForm.click.__doc__ if you don't have urllib2", but I have no idea what HTMLForm is.
I have tried to search it in google, nothing interesting.

Thanks for any help.

The error message is as follows.

Traceback (most recent call last):
  File "wretch4.py", line 53, in ?
    response2 = urlopen(request2)
  File "/var/lib/python-support/python2.4/urlgrabber/grabber.py", line 605, in urlopen
    return default_grabber.urlopen(url, **kwargs)
  File "/var/lib/python-support/python2.4/urlgrabber/grabber.py", line 881, in urlopen
    (url,parts) = opts.urlparser.parse(url, opts) 
  File "/var/lib/python-support/python2.4/urlgrabber/grabber.py", line 653, in parse
    parts = urlparse.urlparse(url)
  File "/usr/lib/python2.4/urlparse.py", line 50, in urlparse
    tuple = urlsplit(url, scheme, allow_fragments)
  File "/usr/lib/python2.4/urlparse.py", line 89, in urlsplit
    i = url.find(':')
  File "/usr/lib/python2.4/urllib2.py", line 207, in __getattr__
    raise AttributeError, attr
 

My code is as follows.

#!/usr/bin/python
from ClientForm import ParseResponse
from urlgrabber import urlopen
 
 
url = 'http://www.ggggg.com'
 
headers = (
        ('User-Agent','Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.10) Gecko/20071115 Iceweasel/2.0.0.10 (Debian-2.0.0.10-0etch1)'),
        ('Accept','text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5'),
        ('Accept-Language','en-us,en;q=0.5'),
#       ('Accept-Encoding','gzip,deflate'),
        ('Accept-Charset','ISO-8859-1,utf-8;q=0.7,*;q=0.7'),
        ('Keep-Alive','300'),
        ('Connection','keep-alive'),
        ('Referer',url),
        ('Content-Type','application/x-www-form-urlencoded'))
 
response = urlopen(url,http_headers = headers)
forms = ParseResponse(response, backwards_compat=False)
 
form = forms[0]
print form
 
form["passwd"] = "ggggg"
 
 
request2 = form.click()
response2 = urlopen(request2)
 
print response2
 
 

[mcgrete]

Hi,
I'm writing a web automation script using ClientForm and urlgrabber.
I use urlgrabber because I need the "http keepalive" which doesn't exist in urllib2.

I'm facing a problem, the "form.click()" returns a urllib2.Request object which I can't pass to urlgrabber.urlopen().

In the ClientForm web page, I see "see HTMLForm.click.__doc__ if you don't have urllib2", but I have no idea what HTMLForm is.
I have tried to search it in google, nothing interesting.

Thanks for any help.

The error message is as follows.

Traceback (most recent call last):
  File "wretch4.py", line 53, in ?
    response2 = urlopen(request2)
  File "/var/lib/python-support/python2.4/urlgrabber/grabber.py", line 605, in urlopen
    return default_grabber.urlopen(url, **kwargs)
  File "/var/lib/python-support/python2.4/urlgrabber/grabber.py", line 881, in urlopen
    (url,parts) = opts.urlparser.parse(url, opts) 
  File "/var/lib/python-support/python2.4/urlgrabber/grabber.py", line 653, in parse
    parts = urlparse.urlparse(url)
  File "/usr/lib/python2.4/urlparse.py", line 50, in urlparse
    tuple = urlsplit(url, scheme, allow_fragments)
  File "/usr/lib/python2.4/urlparse.py", line 89, in urlsplit
    i = url.find(':')
  File "/usr/lib/python2.4/urllib2.py", line 207, in __getattr__
    raise AttributeError, attr
 

My code is as follows.

#!/usr/bin/python
from ClientForm import ParseResponse
from urlgrabber import urlopen
 
 
url = 'http://www.ggggg.com'
 
headers = (
        ('User-Agent','Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.10) Gecko/20071115 Iceweasel/2.0.0.10 (Debian-2.0.0.10-0etch1)'),
        ('Accept','text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5'),
        ('Accept-Language','en-us,en;q=0.5'),
#       ('Accept-Encoding','gzip,deflate'),
        ('Accept-Charset','ISO-8859-1,utf-8;q=0.7,*;q=0.7'),
        ('Keep-Alive','300'),
        ('Connection','keep-alive'),
        ('Referer',url),
        ('Content-Type','application/x-www-form-urlencoded'))
 
response = urlopen(url,http_headers = headers)
forms = ParseResponse(response, backwards_compat=False)
 
form = forms[0]
print form
 
form["passwd"] = "ggggg"
 
 
request2 = form.click()
response2 = urlopen(request2)
 
print response2
 
 

Hi,

I don't have an answer for you, sorry. I am not an expert Pyton programmer. But I have a problem that I think you may be able to assist me with, if you don't mind.

I have used cookielib to login to a website. I am unable to connect to additional urls within the webiste once I have logged on. I have the cookie, header, and so on setup properly - I believe. However, I notice from using 'HTTPAnalyze' program to check the header, that it is using a KEEPALIVE connection. This is new to me. I note from your problem statement that URLGRABBER may be my solution to addressing KEEPALIVE.

My problem is that I have tried to utilize URLGRABBER to handle the KEEPALIVE. I can acess the webiste and read, open, and grab: ,but I can't figure out how to login ('username' and 'passwd'), and let alone access additional pages inside the site. I am stumped. No examples available for this online after lots of searching. I was hoping that you might assist me with how to do this by providing a simple example. I am a beginner Python programmer/user.

I am using Pyton 2.5, on Windows XP, IE7 (if manually browsing)

I appreciate it very much.

[hp1980]

Hi Mcgrete,
First you got to figure out how to login to the site, I think it might be GET or POST.

You can search http GET and POST in google and the urlgrabber manual can help you too.
http://linux.duke.edu/projects/urlgrabber/help/urlgrabber.grabber.html


P.S: I have overcome the problem I met, finally I use the mechanize module, my problem is nothing to do with "KEEPALIVE".