473,382 Members | 1,703 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,382 software developers and data experts.

urlgrabber cookie handling?

Hi everyone,

I have been battling to make my code work with a HTTPS proxy, current
my code uses urllib2 to to most things and works well, except that
urllib2 doesn't handle HTTPS proxies.

Urlgrabber (http://linux.duke.edu/projects/urlgrabber/help/
urlgrabber.grabber.html) looks very promising except that I can find a
way to handle cookies in urlgrabber. Is there a way urlgrabber can use
a HTTPCookieProcess or cookielib.CookieJar object to handle cookies?

Thanks for sparing your time. Thanks.

--
Code referenced in this email is available at http://code.google.com/p/gdatacopier

Oct 17 '07 #1
2 2102
Devraj <de****@gmail.comwrites:
Hi everyone,

I have been battling to make my code work with a HTTPS proxy, current
my code uses urllib2 to to most things and works well, except that
urllib2 doesn't handle HTTPS proxies.

Urlgrabber (http://linux.duke.edu/projects/urlgrabber/help/
urlgrabber.grabber.html) looks very promising except that I can find a
way to handle cookies in urlgrabber. Is there a way urlgrabber can use
a HTTPCookieProcess or cookielib.CookieJar object to handle cookies?
I don't see a nice way. But then I don't see any HTTPS proxy support
in urlgrabber... (I looked at version 3.1.0).

There is a recipe or two on ASPN showing how to support HTTPS proxies
with urllib2, which gives an idea how to do it, though the code is a
bit rough (I'd post some code myself, but I did it for work).
John
Oct 17 '07 #2
Hi John,

Thanks for getting back to me. I did find the ASPN article. If I
figure this out then I will make sure I post the code somewhere for
public consumption.

On Oct 18, 6:13 am, j...@pobox.com (John J. Lee) wrote:
Devraj <dev...@gmail.comwrites:
Hi everyone,
I have been battling to make my code work with a HTTPS proxy, current
my code uses urllib2 to to most things and works well, except that
urllib2 doesn't handle HTTPS proxies.
Urlgrabber (http://linux.duke.edu/projects/urlgrabber/help/
urlgrabber.grabber.html) looks very promising except that I can find a
way to handle cookies in urlgrabber. Is there a way urlgrabber can use
a HTTPCookieProcess or cookielib.CookieJar object to handle cookies?

I don't see a nice way. But then I don't see any HTTPS proxy support
in urlgrabber... (I looked at version 3.1.0).

There is a recipe or two on ASPN showing how to support HTTPS proxies
with urllib2, which gives an idea how to do it, though the code is a
bit rough (I'd post some code myself, but I did it for work).

John

Oct 18 '07 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

7
by: N.K | last post by:
Hi , Python's existing cookie module doesnt supports new cookie headers SetCookie2 , How to submit a patch for that ? I tried emailing person who owns that module.But no response. Thanks,...
5
by: Alex Hunsley | last post by:
I'm using urllib to post data to a web form by issuing a command similar to this: filename, headers = urllib.urlretrieve("http://www.thewebsitenamehere.com/servlet/com.blah.bloo.XmlFeed",...
5
by: Carl | last post by:
Hi, I want to pass the value of the cookie created in VB to ASP. This doesn't work. Any idea? Thanks Carl .... <script language=vbscript> document.cookie="mycookie" '...
4
by: Shannon Jacobs | last post by:
I'm doing some trivial surveys, and I want to know if the same user answers twice. Can't really know that, but at least I thought I could check for the same browser/computer combination by using a...
23
by: Phil Powell | last post by:
// OBTAINED FROM http://www.javascripter.net/faq/settinga.htm // NOTE THAT IF YOU SET days TO -1 THE COOKIE WILL BE SET TO YESTERDAY AND THUS EXPIRE function setCookie(name, value, days, docObj)...
17
by: Bruno | last post by:
I have a feature that is hosted on a different domain from the primary one in a frame, and need to retain values in a cookie. example: A web page at one.com contains a frame which has a page...
6
by: kelvlam | last post by:
Hello all, I'm still a bit new with JavaScript, and I hope the guru here can shed some light for me. It's regarding handling cookie and the case-sensitive nature of JavaScript itself. My...
2
by: hp1980 | last post by:
Hi, I'm writing a web automation script using ClientForm and urlgrabber. I use urlgrabber because I need the "http keepalive" which doesn't exist in urllib2. I'm facing a problem, the...
9
by: Josh | last post by:
I run a Joomla website and am familiar with php in some but not all aspects. Currently I am trying to find some solutions related to session handling. Am I correct in saying that "login" is kept...
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...
0
by: ryjfgjl | last post by:
In our work, we often need to import Excel data into databases (such as MySQL, SQL Server, Oracle) for data analysis and processing. Usually, we use database tools like Navicat or the Excel import...
0
by: taylorcarr | last post by:
A Canon printer is a smart device known for being advanced, efficient, and reliable. It is designed for home, office, and hybrid workspace use and can also be used for a variety of purposes. However,...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: ryjfgjl | last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.