Is it possible to screen scrape a secure site (HTTPS).....

[Ollie]

I know you can screen scrape a website using the System.Net.HttpWebResponse
& System.Net.HttpWebRequest classes.

But how do you screen scrape a secured website (https) that takes a username
& password, I guess what I am asking where does the username & password
go and where do you store any returned token for further requests...

Cheers

Ollie

[Max]

This might work?
https://username:pa******@www.domain.com


"Ollie" <why do they need this!!!!> wrote in message
news:%2******************@TK2MSFTNGP09.phx.gbl...

I know you can screen scrape a website using the System.Net.HttpWebResponse & System.Net.HttpWebRequest classes.

But how do you screen scrape a secured website (https) that takes a username & password, I guess what I am asking where does the username & password
go and where do you store any returned token for further requests...

Cheers

Ollie

[Eric Lawrence [MSFT]]

There's probably a property on the object to allow you to add your
credentials.

If not, it's still pretty easy. First, figure out what protocol the secure
site is using for authentication-- is it BASIC, DIGEST, Negotiate, or
whatever. If it's basic, it's simple-- just add an Authorization header
with the base-64 encoded username:password string. If it's digest or
Negotiate, it gets trickier.

--
Thanks,

Eric Lawrence
Program Manager
Assistance and Worldwide Services

This posting is provided "AS IS" with no warranties, and confers no rights.
"Max" <ma*****@portvista.com> wrote in message
news:pe********************@twister.tampabay.rr.co m...

This might work?
https://username:pa******@www.domain.com


"Ollie" <why do they need this!!!!> wrote in message
news:%2******************@TK2MSFTNGP09.phx.gbl...

I know you can screen scrape a website using the

System.Net.HttpWebResponse

& System.Net.HttpWebRequest classes.

But how do you screen scrape a secured website (https) that takes a

username

& password, I guess what I am asking where does the username & password
go and where do you store any returned token for further requests...

Cheers

Ollie

[Chad Z. Hower aka Kudzu]

"Eric Lawrence [MSFT]" <e_********@hotmail.com> wrote in news:#5pIH88#DHA.808
@TK2MSFTNGP12.phx.gbl:

If not, it's still pretty easy. First, figure out what protocol the secure
site is using for authentication-- is it BASIC, DIGEST, Negotiate, or
whatever. If it's basic, it's simple-- just add an Authorization header
with the base-64 encoded username:password string. If it's digest or
Negotiate, it gets trickier.

Indy supports all of these authentications and more. Its also free.

http://www.indyproject.org/

The only problem is that the .net version doesnt support SSL yet. BUT you
could bundle it with an SSL tunnel if your options are limited.
--
Chad Z. Hower (a.k.a. Kudzu) - http://www.hower.org/Kudzu/
"Programming is an art form that fights back"