473,320 Members | 1,691 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

Outbound HTML Authentication

Hi,

I was trying to do a simple web scraping tool, but the network they
use at work does some type of internal authentication before it lets
the request out of the network. As a result I'm getting the '401 -
Authentication Error' from the application.

I know when I use a web browser or other application that it uses the
information from my Windows AD to validate my user before it accesses
a website. I'm constantly getting asked to enter in this info before I
use Firefox, and I assume that IE picks it up automatically.

However I'm not sure how to tell the request that I'm building in my
python script to either use the info in my AD account or enter in my
user/pass automatically.

Anyone know how to do this?

Thanks
Nov 29 '07 #1
2 1130
On Nov 29, 2007 2:22 PM, Mudcat <mn******@gmail.comwrote:
Hi,

I was trying to do a simple web scraping tool, but the network they
use at work does some type of internal authentication before it lets
the request out of the network. As a result I'm getting the '401 -
Authentication Error' from the application.

I know when I use a web browser or other application that it uses the
information from my Windows AD to validate my user before it accesses
a website. I'm constantly getting asked to enter in this info before I
use Firefox, and I assume that IE picks it up automatically.

However I'm not sure how to tell the request that I'm building in my
python script to either use the info in my AD account or enter in my
user/pass automatically.
You can configure a proxy for urllib2, but your proxy probably uses
NTLM authentication which urllib2 doesn't support. Your best bet is to
use a local proxy which understands NTLM.
Nov 29 '07 #2
twill is a simple language for browsing the Web. It's designed for
automated testing of Web sites, but it can be used to interact with
Web sites in a variety of ways. In particular, twill supports form
submission, cookies, redirects, and HTTP authentication.

Mudcat wrote:
Hi,

I was trying to do a simple web scraping tool, but the network they
use at work does some type of internal authentication before it lets
the request out of the network. As a result I'm getting the '401 -
Authentication Error' from the application.

I know when I use a web browser or other application that it uses the
information from my Windows AD to validate my user before it accesses
a website. I'm constantly getting asked to enter in this info before I
use Firefox, and I assume that IE picks it up automatically.

However I'm not sure how to tell the request that I'm building in my
python script to either use the info in my AD account or enter in my
user/pass automatically.

Anyone know how to do this?

Thanks

--
Shane Geiger
IT Director
National Council on Economic Education
sg*****@ncee.net | 402-438-8958 | http://www.ncee.net

Leading the Campaign for Economic and Financial Literacy

Nov 29 '07 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: John | last post by:
I am using forms authentication for a website. I plan to use some static html pages (generated with a tool) on the site as well. I would like the html pages to be secured using the forms...
6
by: William F. Zachmann | last post by:
We've got a project going that involves moving an old web site with a massive dll written in C++ that produces most of the output from a SQL 7.0 data base on NT4 onto IIS on Windows 2003 Server...
1
by: John Rossitter | last post by:
Hi Everybody, I’m looking for a way to capture the outbound XML stream of a web service call. I need to be able to save these communications to SQL Server. Is there any way to do this...
0
by: kv | last post by:
Is it possible to make multiple outbound calls using voicexml/ccxml? This is what i want: - An inbound call is received - We get some information form the caller - Depending on informtaion...
1
by: Arun | last post by:
I have a folder “Secured” under the root folder of the project In the project root web.config authentication is given as <authentication mode="Forms"> <forms loginUrl="Login.aspx" timeout="15"...
5
by: nick | last post by:
I need to create a simple asp.net application that use password protect some html pages. The html page provider doesn't know asp.net. And the host doesn't allow me to create user accounts. ...
2
by: nick | last post by:
I have an Asp.Net 2.0 application using form authentication. I want the html pages be protected by the authentication system too. The accessing of html files need to be authenticated in my local...
25
by: bmearns | last post by:
Is it possible to specify which port to use as the outbound port on a connection? I have the IP address and port number for the computer I'm trying to connect to (not listening for), but it's...
1
by: erikcw | last post by:
Python seems to default to the main system IP for outbound connections (such as urllib), but I want to bind to one of my other IPs for outbound connections. Any ideas? Thanks!
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, youll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shllpp 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.