Scrape text from browser surface (not HTHML)

[_BNC]

I need to do the equivalent of 'select all' and 'capture' on a browser
screen. I just want the text as it appears on-screen. Is there a simple
way to automate this?

The thing that may make it a bit awkward is that the browser windows
appear as popup-style windows, with no navigation buttons. That part
may not be so easy to automate.

[Alvin Bruney [MVP]]

Just fire a webrequest at it. The return is an html string which you can
parse (preferrably with a regex expression) to extract what you want. I have
some code somewhere but i believe i plaigerized it from the net so you
should be able to find a lot of code to do just that. regexlib.com will help
you with the appropriate regex expression

--
Regards,
Alvin Bruney
[ASP.NET MVP http://mvp.support.microsoft.com/default.aspx]
Got tidbits? Get it here... http://tinyurl.com/27cok
"_BNC" <_B**@nospam.ne t> wrote in message
news:2a******** *************** *********@4ax.c om...

I need to do the equivalent of 'select all' and 'capture' on a browser
screen. I just want the text as it appears on-screen. Is there a simple
way to automate this?

The thing that may make it a bit awkward is that the browser windows
appear as popup-style windows, with no navigation buttons. That part
may not be so easy to automate.

[_BNC]

>>I need to do the equivalent of 'select all' and 'capture' on a browser

screen. I just want the text as it appears on-screen. Is there a simple
way to automate this?

On Fri, 19 Nov 2004 22:59:53 -0500, "Alvin Bruney [MVP]" <vapor at
steaming post office> wrote:
Just fire a webrequest at it. The return is an html string which you can
parse (preferrably with a regex expression) to extract what you want. I have
some code somewhere but i believe i plaigerized it from the net so you
should be able to find a lot of code to do just that. regexlib.com will help
you with the appropriate regex expression

That sounds like a good idea, Alvin, but I just checked regexlib.com and
came up empty. (Lots of good stuff there though, so thanks for the
pointer.)

I thought there would be a simpler way of bringing up the window and
inducing a text capture programmaticall y, just as if the user had done it.
Of course if there is an efficient regex expression, that would be great,
but it's tough to come up with efficient search keys for googling that
type of thing.

You'd think someone would have a nice C# function for turning ugly html
email into readable text. Or something equivalent.

[Alvin Bruney [MVP]]

well it's not that difficult to concoct a good regex expression to do this.
if you post a request for help in c# newsgroup, i'm sure you can get someone
like chris r to write you one. I steal stuff off of regexlib and modify it
for my devious purposes. i'm no good at writing that stuff from scratch.

i the webrequest code if you want it by the way

--
Regards,
Alvin Bruney
[ASP.NET MVP http://mvp.support.microsoft.com/default.aspx]
Got tidbits? Get it here... http://tinyurl.com/27cok
"_BNC" <_B**@nospam.ne t> wrote in message
news:4k******** *************** *********@4ax.c om...

I need to do the equivalent of 'select all' and 'capture' on a browser
screen. I just want the text as it appears on-screen. Is there a
simple
way to automate this?

On Fri, 19 Nov 2004 22:59:53 -0500, "Alvin Bruney [MVP]" <vapor at
steaming post office> wrote:

Just fire a webrequest at it. The return is an html string which you can
parse (preferrably with a regex expression) to extract what you want. I
have
some code somewhere but i believe i plaigerized it from the net so you
should be able to find a lot of code to do just that. regexlib.com will
help
you with the appropriate regex expression

That sounds like a good idea, Alvin, but I just checked regexlib.com and
came up empty. (Lots of good stuff there though, so thanks for the
pointer.)

I thought there would be a simpler way of bringing up the window and
inducing a text capture programmaticall y, just as if the user had done it.
Of course if there is an efficient regex expression, that would be great,
but it's tough to come up with efficient search keys for googling that
type of thing.

You'd think someone would have a nice C# function for turning ugly html
email into readable text. Or something equivalent.

[_BNC]

(Re extracting just visible text from HTML)

On Mon, 22 Nov 2004 22:20:26 -0500, "Alvin Bruney [MVP]" <vapor at
steaming post office> wrote:

well it's not that difficult to concoct a good regex expression to do this.
if you post a request for help in c# newsgroup, i'm sure you can get someone
like chris r to write you one. I steal stuff off of regexlib and modify it
for my devious purposes. i'm no good at writing that stuff from scratch.
I confess to xposting this to the C# group in the hope that someone would
notice it. I'm sure I've seen something like this at one time, too. But
it would involve breaking down HTML tables, translating symbols like
'nbsp' and all that. I'm sure it's been done, but it soulds like a lot of
wheel-reinventing if I have to do it myself.
i the webrequest code if you want it by the way

That would be nice! Thanks.

BNC

[Alvin Bruney [MVP]]

this is a webservice i used to scrape telephone directory information from
anywho.com. It's pretty rough code but you should get the general idea

[WebMethod]

public string PhoneLookup(str ing strNumber, ref int counter)

{

string strResult = string.Empty, searchtext = string.Empty;

try

{

// Create a new 'Uri' object with the specified string.

Uri myUri =new
Uri("http://www.anywho.com/qry/wp_rl?npa="+str Number.Substrin g(0,3) +
"&telephone ="+ strNumber.Subst ring(3,7) + "&btnsubmit.x=3 6&btnsubmit.y=9 ");

// Creates an HttpWebRequest with the specified URL.

HttpWebRequest myHttpWebReques t = (HttpWebRequest )WebRequest.Cre ate(myUri);

myHttpWebReques t.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
5.1; Q312461; .NET CLR 1.0.3705)";

HttpWebResponse res = (HttpWebRespons e)myHttpWebRequ est.GetResponse ();

StreamReader sr = new StreamReader(re s.GetResponseSt ream(),
System.Text.Enc oding.UTF8);

string pageContent = sr.ReadToEnd();

res.Close();

sr.Close();
int startpos = pageContent.Ind exOf(@"bin/amap.cgi?") + 10;

if(startpos != -1)

{

int endpost = pageContent.Ind exOf("Maps &amp; Directions");

if(endpost != -1)

searchtext = pageContent.Sub string(startpos , endpost - startpos + 17);

searchtext = searchtext.Repl ace(">Maps &amp; Direct"," ");

searchtext = searchtext.Repl ace("gi?lastnam e="," ");

searchtext = searchtext.Repl ace("firstname= ","\n");

searchtext = searchtext.Repl ace("+"," ");

searchtext = searchtext.Repl ace("="," ");

searchtext = searchtext.Repl ace("&"," ");

searchtext = searchtext.Repl ace("\\"," ");

searchtext = searchtext.Repl ace("\""," ");

searchtext = searchtext.Repl ace("city","\n" );

searchtext = searchtext.Repl ace("state","\n ");

searchtext = searchtext.Repl ace("zip","\n") ;

searchtext = searchtext.Repl ace("country"," \n");

searchtext = searchtext.Repl ace("npatelepho ne","\n");

searchtext = searchtext.Repl ace("streetaddr ess","");

strResult = searchtext;

sr.Close();

pageContent = string.Empty;

}

}

catch

{

// return "No records exist";

// count++;

return null;

}

Agent.SelectedI ndex = -1;

if(strResult.Tr im() == string.Empty)

return null;

counter++;

return strResult + "\n\n";

}
--
Regards,
Alvin Bruney
[ASP.NET MVP http://mvp.support.microsoft.com/default.aspx]
Got tidbits? Get it here... http://tinyurl.com/27cok
"_BNC" <_B**@nospam.ne t> wrote in message
news:89******** *************** *********@4ax.c om...

(Re extracting just visible text from HTML)

On Mon, 22 Nov 2004 22:20:26 -0500, "Alvin Bruney [MVP]" <vapor at
steaming post office> wrote:

well it's not that difficult to concoct a good regex expression to do
this.
if you post a request for help in c# newsgroup, i'm sure you can get
someone
like chris r to write you one. I steal stuff off of regexlib and modify it
for my devious purposes. i'm no good at writing that stuff from scratch.

I confess to xposting this to the C# group in the hope that someone would
notice it. I'm sure I've seen something like this at one time, too. But
it would involve breaking down HTML tables, translating symbols like
'nbsp' and all that. I'm sure it's been done, but it soulds like a lot of
wheel-reinventing if I have to do it myself.

i the webrequest code if you want it by the way

That would be nice! Thanks.

BNC