473,288 Members | 1,705 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,288 software developers and data experts.

urlgrabber cookie handling?

Hi everyone,

I have been battling to make my code work with a HTTPS proxy, current
my code uses urllib2 to to most things and works well, except that
urllib2 doesn't handle HTTPS proxies.

Urlgrabber (http://linux.duke.edu/projects/urlgrabber/help/
urlgrabber.grabber.html) looks very promising except that I can find a
way to handle cookies in urlgrabber. Is there a way urlgrabber can use
a HTTPCookieProcess or cookielib.CookieJar object to handle cookies?

Thanks for sparing your time. Thanks.

--
Code referenced in this email is available at http://code.google.com/p/gdatacopier

Oct 17 '07 #1
2 2099
Devraj <de****@gmail.comwrites:
Hi everyone,

I have been battling to make my code work with a HTTPS proxy, current
my code uses urllib2 to to most things and works well, except that
urllib2 doesn't handle HTTPS proxies.

Urlgrabber (http://linux.duke.edu/projects/urlgrabber/help/
urlgrabber.grabber.html) looks very promising except that I can find a
way to handle cookies in urlgrabber. Is there a way urlgrabber can use
a HTTPCookieProcess or cookielib.CookieJar object to handle cookies?
I don't see a nice way. But then I don't see any HTTPS proxy support
in urlgrabber... (I looked at version 3.1.0).

There is a recipe or two on ASPN showing how to support HTTPS proxies
with urllib2, which gives an idea how to do it, though the code is a
bit rough (I'd post some code myself, but I did it for work).
John
Oct 17 '07 #2
Hi John,

Thanks for getting back to me. I did find the ASPN article. If I
figure this out then I will make sure I post the code somewhere for
public consumption.

On Oct 18, 6:13 am, j...@pobox.com (John J. Lee) wrote:
Devraj <dev...@gmail.comwrites:
Hi everyone,
I have been battling to make my code work with a HTTPS proxy, current
my code uses urllib2 to to most things and works well, except that
urllib2 doesn't handle HTTPS proxies.
Urlgrabber (http://linux.duke.edu/projects/urlgrabber/help/
urlgrabber.grabber.html) looks very promising except that I can find a
way to handle cookies in urlgrabber. Is there a way urlgrabber can use
a HTTPCookieProcess or cookielib.CookieJar object to handle cookies?

I don't see a nice way. But then I don't see any HTTPS proxy support
in urlgrabber... (I looked at version 3.1.0).

There is a recipe or two on ASPN showing how to support HTTPS proxies
with urllib2, which gives an idea how to do it, though the code is a
bit rough (I'd post some code myself, but I did it for work).

John

Oct 18 '07 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

7
by: N.K | last post by:
Hi , Python's existing cookie module doesnt supports new cookie headers SetCookie2 , How to submit a patch for that ? I tried emailing person who owns that module.But no response. Thanks,...
5
by: Alex Hunsley | last post by:
I'm using urllib to post data to a web form by issuing a command similar to this: filename, headers = urllib.urlretrieve("http://www.thewebsitenamehere.com/servlet/com.blah.bloo.XmlFeed",...
5
by: Carl | last post by:
Hi, I want to pass the value of the cookie created in VB to ASP. This doesn't work. Any idea? Thanks Carl .... <script language=vbscript> document.cookie="mycookie" '...
4
by: Shannon Jacobs | last post by:
I'm doing some trivial surveys, and I want to know if the same user answers twice. Can't really know that, but at least I thought I could check for the same browser/computer combination by using a...
23
by: Phil Powell | last post by:
// OBTAINED FROM http://www.javascripter.net/faq/settinga.htm // NOTE THAT IF YOU SET days TO -1 THE COOKIE WILL BE SET TO YESTERDAY AND THUS EXPIRE function setCookie(name, value, days, docObj)...
17
by: Bruno | last post by:
I have a feature that is hosted on a different domain from the primary one in a frame, and need to retain values in a cookie. example: A web page at one.com contains a frame which has a page...
6
by: kelvlam | last post by:
Hello all, I'm still a bit new with JavaScript, and I hope the guru here can shed some light for me. It's regarding handling cookie and the case-sensitive nature of JavaScript itself. My...
2
by: hp1980 | last post by:
Hi, I'm writing a web automation script using ClientForm and urlgrabber. I use urlgrabber because I need the "http keepalive" which doesn't exist in urllib2. I'm facing a problem, the...
9
by: Josh | last post by:
I run a Joomla website and am familiar with php in some but not all aspects. Currently I am trying to find some solutions related to session handling. Am I correct in saying that "login" is kept...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.