HTML Page Scraping

[james]

Hi Guys,

I want to write an app in C# that signs in to a website and grabs some
information. Do you know how complicated that can get (with security
tokens, etc)?

Thanks,
James

[Nicholas Paldino [.NET/C# MVP]]

James,

It can get pretty difficult, depending on the technologies that are
utilized in the page. For example, authentication can be in the form of
HTTP authentication, or forms-based, or maybe it is done through an AJAX
call. Needless to say, you are almost definitely going to have to
specialized depending on the site and the security it uses.
--
- Nicholas Paldino [.NET/C# MVP]
- mv*@spam.guard.caspershouse.com

"james" <ja********@gmail.comwrote in message
news:11**********************@50g2000hsm.googlegro ups.com...

Hi Guys,

I want to write an app in C# that signs in to a website and grabs some
information. Do you know how complicated that can get (with security
tokens, etc)?

Thanks,
James

[james]

Carl,

Would an embedded browser be able to handle security tokens for me?

Also, do you know if I can access the DOM in C# or would I have to use
javascript or regex?

Thanks,
James

[james]

Carl,

Would an embedded browser be able to handle security tokens for me?

Also, do you know if I can access the DOM in C# or would I have to use
javascript or regex?

Thanks,
James

[Carl Daniel [VC++ MVP]]

james wrote:

Carl,

Would an embedded browser be able to handle security tokens for me?

Maybe. It's a better place to start than from scratch, for sure.

>
Also, do you know if I can access the DOM in C# or would I have to use
javascript or regex?

You should be able to access it from C#. It's a COM/OLE Automation
component, so you should be able to access the whole API from C# via COM
interop. I belive that .NET 2.0+ provides a pre-built wrapper for you.

-cd

[Cor Ligthert[MVP]]

James,

You will for sure need MSHTML, it is a very extended class and therefore the
information is hard to get on MSDN. Be aware that you have to cast almost
everything and mostly 2 times over each other to get the right types as you
use it.

Be aware that it can slow down your IDE referencing it.

Cor

[drorgl]

On Oct 9, 12:27 am, james <[e-mail]:

Hi Guys,

I want to write an app in C# that signs in to a website and grabs some
information. Do you know how complicated that can get (with security
tokens, etc)?

Thanks,
James

Like some people stated here, its site dependant.
but here's a list of starting points:

IE Developer toolbar - to look for the commands/html/http/post
requests between the website and the browser (works on IE, for Firefox
you have FireBug)
HttpWebRequest and HttpWebResponse for using http protocol.
for the above - CookieContainer - for cookie based authentication
CredentialCache for NTLM based authentication (perhaps its used for
other things, but this is what I used it for so far)
HtmlAgilityPack - for parsing HTML, perhaps there's a better
component, so do your own research.

Hope its helpful :-)

[james]

Thanks for the advice.

Regards,
James

[alex_f_il]

On Oct 8, 6:27 pm, james <[e-mail]:

Hi Guys,

I want to write an app in C# that signs in to a website and grabs some
information. Do you know how complicated that can get (with security
tokens, etc)?

Thanks,
James

You can also try SWExplorerAutomation (SWEA) from http://webius.net.
SWEA records, replays automation scripts and generates VB.NET or C#
code.