471,624 Members | 1,790 Online
Bytes | Software Development & Data Engineering Community
Post +

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 471,624 software developers and data experts.

urlgrabber cookie handling?

Hi everyone,

I have been battling to make my code work with a HTTPS proxy, current
my code uses urllib2 to to most things and works well, except that
urllib2 doesn't handle HTTPS proxies.

Urlgrabber (http://linux.duke.edu/projects/urlgrabber/help/
urlgrabber.grabber.html) looks very promising except that I can find a
way to handle cookies in urlgrabber. Is there a way urlgrabber can use
a HTTPCookieProcess or cookielib.CookieJar object to handle cookies?

Thanks for sparing your time. Thanks.

--
Code referenced in this email is available at http://code.google.com/p/gdatacopier

Oct 17 '07 #1
2 2020
Devraj <de****@gmail.comwrites:
Hi everyone,

I have been battling to make my code work with a HTTPS proxy, current
my code uses urllib2 to to most things and works well, except that
urllib2 doesn't handle HTTPS proxies.

Urlgrabber (http://linux.duke.edu/projects/urlgrabber/help/
urlgrabber.grabber.html) looks very promising except that I can find a
way to handle cookies in urlgrabber. Is there a way urlgrabber can use
a HTTPCookieProcess or cookielib.CookieJar object to handle cookies?
I don't see a nice way. But then I don't see any HTTPS proxy support
in urlgrabber... (I looked at version 3.1.0).

There is a recipe or two on ASPN showing how to support HTTPS proxies
with urllib2, which gives an idea how to do it, though the code is a
bit rough (I'd post some code myself, but I did it for work).
John
Oct 17 '07 #2
Hi John,

Thanks for getting back to me. I did find the ASPN article. If I
figure this out then I will make sure I post the code somewhere for
public consumption.

On Oct 18, 6:13 am, j...@pobox.com (John J. Lee) wrote:
Devraj <dev...@gmail.comwrites:
Hi everyone,
I have been battling to make my code work with a HTTPS proxy, current
my code uses urllib2 to to most things and works well, except that
urllib2 doesn't handle HTTPS proxies.
Urlgrabber (http://linux.duke.edu/projects/urlgrabber/help/
urlgrabber.grabber.html) looks very promising except that I can find a
way to handle cookies in urlgrabber. Is there a way urlgrabber can use
a HTTPCookieProcess or cookielib.CookieJar object to handle cookies?

I don't see a nice way. But then I don't see any HTTPS proxy support
in urlgrabber... (I looked at version 3.1.0).

There is a recipe or two on ASPN showing how to support HTTPS proxies
with urllib2, which gives an idea how to do it, though the code is a
bit rough (I'd post some code myself, but I did it for work).

John

Oct 18 '07 #3

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

7 posts views Thread by N.K | last post: by
5 posts views Thread by Alex Hunsley | last post: by
23 posts views Thread by Phil Powell | last post: by
17 posts views Thread by Bruno | last post: by
6 posts views Thread by kelvlam | last post: by
9 posts views Thread by Josh | last post: by

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.