urlgrabber cookie handling?

[Devraj]

Hi everyone,

I have been battling to make my code work with a HTTPS proxy, current
my code uses urllib2 to to most things and works well, except that
urllib2 doesn't handle HTTPS proxies.

Urlgrabber (http://linux.duke.edu/projects/urlgrabber/help/
urlgrabber.grabber.html) looks very promising except that I can find a
way to handle cookies in urlgrabber. Is there a way urlgrabber can use
a HTTPCookieProcess or cookielib.CookieJar object to handle cookies?

Thanks for sparing your time. Thanks.

--
Code referenced in this email is available at http://code.google.com/p/gdatacopier

[John J. Lee]

Devraj <de****@gmail.comwrites:

Hi everyone,

I have been battling to make my code work with a HTTPS proxy, current
my code uses urllib2 to to most things and works well, except that
urllib2 doesn't handle HTTPS proxies.

Urlgrabber (http://linux.duke.edu/projects/urlgrabber/help/
urlgrabber.grabber.html) looks very promising except that I can find a
way to handle cookies in urlgrabber. Is there a way urlgrabber can use
a HTTPCookieProcess or cookielib.CookieJar object to handle cookies?

I don't see a nice way. But then I don't see any HTTPS proxy support
in urlgrabber... (I looked at version 3.1.0).

There is a recipe or two on ASPN showing how to support HTTPS proxies
with urllib2, which gives an idea how to do it, though the code is a
bit rough (I'd post some code myself, but I did it for work).
John

[Devraj]

Hi John,

Thanks for getting back to me. I did find the ASPN article. If I
figure this out then I will make sure I post the code somewhere for
public consumption.

On Oct 18, 6:13 am, [e-mail] (John J. Lee) wrote:

Devraj <[e-mail]:

Hi everyone,

I have been battling to make my code work with a HTTPS proxy, current
my code uses urllib2 to to most things and works well, except that
urllib2 doesn't handle HTTPS proxies.

Urlgrabber (http://linux.duke.edu/projects/urlgrabber/help/
urlgrabber.grabber.html) looks very promising except that I can find a
way to handle cookies in urlgrabber. Is there a way urlgrabber can use
a HTTPCookieProcess or cookielib.CookieJar object to handle cookies?

I don't see a nice way. But then I don't see any HTTPS proxy support
in urlgrabber... (I looked at version 3.1.0).

There is a recipe or two on ASPN showing how to support HTTPS proxies
with urllib2, which gives an idea how to do it, though the code is a
bit rough (I'd post some code myself, but I did it for work).

John