473,549 Members | 2,815 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

using httplib for authentication

Hello All,

I want to use python to download files from sites where authentication is
required. The page appears to send a form with the login and pass by post. I
would like to log in and keep this session open within python and
download a number of files automatically. Could anyone kindly point me in the
right direction? I have discovered httplib and read the documentation and
looked at the examples, but they don't seem helpful for this.

James

--
James Stroud
UCLA-DOE Institute for Genomics and Proteomics
Box 951570
Los Angeles, CA 90095

http://www.jamesstroud.com/
Aug 3 '05 #1
2 2588

James Stroud wrote:
Hello All,

I want to use python to download files from sites where authentication is
required. The page appears to send a form with the login and pass by post. I
would like to log in and keep this session open within python and
download a number of files automatically. Could anyone kindly point me in the
right direction? I have discovered httplib and read the documentation and
looked at the examples, but they don't seem helpful for this.

Hello James,

httplib is built on top of the socket module - but is still a library
meant for doing fairly low level http operations. For fetching URLs,
urllib2 is probably the way to go. It handles GET and POST with ease.

If you think this will do the job then you could try reading the
urllib2 tutorial at :

http://www.voidspace.org.uk/python/articles.shtml#http

You might wnat to look at other extension modules too - ClientCookie
for handling cookies, ClientForm for automatically filling in the
forms, BeautifulSoup for parsing HTML pages....

All the Best,

Fuzzy
http://www.voidspace.org.uk/python
James

--
James Stroud
UCLA-DOE Institute for Genomics and Proteomics
Box 951570
Los Angeles, CA 90095

http://www.jamesstroud.com/


Aug 3 '05 #2
Thank you Fuzzy, I will look into these things. Maybe the site is setting a
cookie, as you have suggested. I have never delved into the ways of http
except to configure apache and write some very bare-bones web pages, so I
have to say that some very obvious things do not occur to me.

James

On Wednesday 03 August 2005 08:41 am, Fuzzyman wrote:
Hello James,

httplib is built on top of the socket module - but is still a library
meant for doing fairly low level http operations. For fetching URLs,
urllib2 is probably the way to go. It handles GET and POST with ease.

If you think this will do the job then you could try reading the
urllib2 tutorial at :

http://www.voidspace.org.uk/python/articles.shtml#http

You might wnat to look at other extension modules too - ClientCookie
for handling cookies, ClientForm for automatically filling in the
forms, BeautifulSoup for parsing HTML pages....

All the Best,

Fuzzy
http://www.voidspace.org.uk/python

Aug 3 '05 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

7
2964
by: luigipaioro | last post by:
Good morning to all! I'm trying to access on a web page that needs user and password authentication. I'm enabled to access there (I mean that I have an user name and a password to access via web), but I cannot access using an automatic procedure (that is what I need to make a daemon that downloads weekly an ASCII file from that site). ...
4
3797
by: Roger Binns | last post by:
I have just spent several weeks mashing xmlrpc, httplib and SSL (from M2Crypto) to work together. The current standard library has several problems: - Builtin SSL is pretty much useless if you actually care about security - Poor HTTP authentication support - No server side stuff (SSL, HTTP authentication etc) - Pathological coding to...
1
2871
by: Brian Beck | last post by:
Hi. I'm having some problems with code based directly on the following httplib documentation code: http://www.zvon.org/other/python/doc21/lib/httplib-examples.html I've included the code and traceback at the end of this post. The odd thing is, using DEPRECATED FUNCTIONS to perform the same function works fine!
0
1897
by: Leon | last post by:
I use sniffer look,it's status is 200,but run getresponse() can occur error ResponseNotReady I try wait it by 5 seconds,but still occur error ResponseNotReady source code.... import httplib,threading def waitSeconds(): sResponse = httpClient.getresponse() print sResponse.reason
3
3176
by: John Reese | last post by:
Hello there. I've run into some missing functionality with HTTP Digest authentication in the 2.3 library and I was wondering if I'm just missing something. Missing functionality the first: urllib2 1a. You can add "handlers" to your opener indicating that you want to use HTTP Digest auth. This is nice way to handle it, but I don't see...
0
3570
by: Pieter Edelman | last post by:
Hi all, I'm trying to submit some data using a POST request to a HTTP server with BASIC authentication with python, but I can't get it to work. Since it's driving me completely nuts, so here's my cry for help. The server is an elog logbook server (http://midas.psi.ch/elog/). It is protected with a password and an empty username. I can...
0
2916
by: Robert | last post by:
did you solve this problem? It seems to be still present here with py2.3.5. Robert -- From: Manish Jethani <manish.j@gmx.net> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.6b) Gecko/20031205 Thunderbird/0.4 X-Accept-Language: en-us, en
1
4873
by: Michele Simionato | last post by:
Can somebody provide an example of how to retrieve a https url, given username and password? I don't find it in the standard documentation. TIA, Michele Simionato
0
2021
by: emiliano | last post by:
Hey guys, i was just googling some information about how to use the ClientForm package with a page which requires HTTP basic authentication and i got here :P ... So here is the problem, lets see if anyone here can help me please solving this issue First i open the protected page using the FancyURLopener which support the HTTP basic...
0
7551
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main...
0
7477
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language...
0
7750
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
0
6084
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then...
1
5396
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes...
0
3524
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in...
1
1971
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
1
1084
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
0
790
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.