Hello All,
I want to use python to download files from sites where authentication is
required. The page appears to send a form with the login and pass by post. I
would like to log in and keep this session open within python and
download a number of files automatically. Could anyone kindly point me in the
right direction? I have discovered httplib and read the documentation and
looked at the examples, but they don't seem helpful for this.
James
--
James Stroud
UCLA-DOE Institute for Genomics and Proteomics
Box 951570
Los Angeles, CA 90095 http://www.jamesstroud.com/ 2  2605 
James Stroud wrote: Hello All,
I want to use python to download files from sites where authentication is
required. The page appears to send a form with the login and pass by post. I
would like to log in and keep this session open within python and
download a number of files automatically. Could anyone kindly point me in the
right direction? I have discovered httplib and read the documentation and
looked at the examples, but they don't seem helpful for this.
Hello James,
httplib is built on top of the socket module - but is still a library
meant for doing fairly low level http operations. For fetching URLs,
urllib2 is probably the way to go. It handles GET and POST with ease.
If you think this will do the job then you could try reading the
urllib2 tutorial at :
http://www.voidspace.org.uk/python/articles.shtml#http
You might wnat to look at other extension modules too - ClientCookie
for handling cookies, ClientForm for automatically filling in the
forms, BeautifulSoup for parsing HTML pages....
All the Best,
Fuzzy
http://www.voidspace.org.uk/python
James
--
James Stroud
UCLA-DOE Institute for Genomics and Proteomics
Box 951570
Los Angeles, CA 90095
http://www.jamesstroud.com/
Thank you Fuzzy, I will look into these things. Maybe the site is setting a
cookie, as you have suggested. I have never delved into the ways of http
except to configure apache and write some very bare-bones web pages, so I
have to say that some very obvious things do not occur to me.
James
On Wednesday 03 August 2005 08:41 am, Fuzzyman wrote: Hello James,
httplib is built on top of the socket module - but is still a library
meant for doing fairly low level http operations. For fetching URLs,
urllib2 is probably the way to go. It handles GET and POST with ease.
If you think this will do the job then you could try reading the
urllib2 tutorial at :
http://www.voidspace.org.uk/python/articles.shtml#http
You might wnat to look at other extension modules too - ClientCookie
for handling cookies, ClientForm for automatically filling in the
forms, BeautifulSoup for parsing HTML pages....
All the Best,
Fuzzy
http://www.voidspace.org.uk/python This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics | |
by: luigipaioro |
last post by:
Good morning to all!
I'm trying to access on a web page that needs user and password
authentication. I'm enabled to access there (I mean that I have an
user name and a password to access via web), but I cannot access using
an automatic procedure (that is what I need to make a daemon that
downloads weekly an ASCII file from that site).
I've tried using urllib:
|
by: Roger Binns |
last post by:
I have just spent several weeks mashing xmlrpc, httplib and SSL (from
M2Crypto) to work together. The current standard library has several
problems:
- Builtin SSL is pretty much useless if you actually care
about security
- Poor HTTP authentication support
- No server side stuff (SSL, HTTP authentication etc)
- Pathological coding to ensure that at most one request is
sent on a connection, rather than reusing an already open
|
by: Brian Beck |
last post by:
Hi.
I'm having some problems with code based directly on the following
httplib documentation code:
http://www.zvon.org/other/python/doc21/lib/httplib-examples.html
I've included the code and traceback at the end of this post.
The odd thing is, using DEPRECATED FUNCTIONS to perform the same
function works fine!
|
by: Leon |
last post by:
I use sniffer look,it's status is 200,but run getresponse() can occur error
ResponseNotReady
I try wait it by 5 seconds,but still occur error ResponseNotReady
source code....
import httplib,threading
def waitSeconds():
sResponse = httpClient.getresponse()
print sResponse.reason
|
by: John Reese |
last post by:
Hello there. I've run into some missing functionality with HTTP Digest
authentication in the 2.3 library and I was wondering if I'm just
missing something.
Missing functionality the first: urllib2
1a. You can add "handlers" to your opener indicating that you want to
use HTTP Digest auth. This is nice way to handle it, but I don't
see any way to use a custom verb in your URLOpener -- it always
uses either GET or POST depending on...
| | |
by: Pieter Edelman |
last post by:
Hi all,
I'm trying to submit some data using a POST request to a HTTP server with
BASIC authentication with python, but I can't get it to work. Since it's
driving me completely nuts, so here's my cry for help.
The server is an elog logbook server (http://midas.psi.ch/elog/). It is
protected with a password and an empty username. I can login both using
urllib and urllib2 (suppose the password is "foobar", the logbook is
running on port...
|
by: Robert |
last post by:
did you solve this problem? It seems to be still present here with
py2.3.5.
Robert
--
From: Manish Jethani <manish.j@gmx.net>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.6b)
Gecko/20031205 Thunderbird/0.4
X-Accept-Language: en-us, en
|
by: Michele Simionato |
last post by:
Can somebody provide an example of how to retrieve a https url, given
username and password? I don't find it in the standard documentation.
TIA,
Michele Simionato
|
by: emiliano |
last post by:
Hey guys, i was just googling some information about how to use the ClientForm package with a page which requires HTTP basic authentication and i got here :P ... So here is the problem, lets see if anyone here can help me please solving this issue
First i open the protected page using the FancyURLopener which support the HTTP basic authentication and pass this object to the ParseResponse function so it parses the corresponding forms :P
...
|
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it.
Here is my compilation command:
g++-12 -std=c++20 -Wnarrowing bit_field.cpp
Here is the code in...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
| | |
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
| | |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |
