473,326 Members | 2,023 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,326 software developers and data experts.

How to screen scrape for results?

Hi all --

I would like to create an application(call it Application "A") that I would
like to mimic exactly as a form on a foreign system (Application "F").
Application "F" is on the web (so basically I can not control it). I will
have a form exactly on Application "A" as that of Application "F".
Application "A" will submit to the url of the application "F". I would like
to do a screen scrape of the confirmation obtained after submitting the form
on application "F".

I can easily do the screen scraping of a static page but am not sure how to
screen scrape a form result ?

Any ideas?

TIA
Swanand

Apr 3 '06 #1
7 2142


Here are the hints you need:

You will not have the FormPostCollection and FormPostItem objects.
These are simple objects. the FormPostItem basically has the key and value
you want to post.
FormPostCollection is a implemented CollectionBase, which keeps a collection
of FormPostItems

However, minus that, you can figure out what is going on.

This took me about 3 days to figure out (the GET and querystring was easy,
the form/post was the tough one).
So post a "thank you" for this one.

I got the code for everything outside of the FORM/POST part from another
developer, but the form/post stuff was what I Figured out in the equation.


public void WriteTextFile(string Url, string FilePath, long BufferSize )
{

try
{

//create a web request
HttpWebRequest oHttpWebRequest = null;
oHttpWebRequest = (HttpWebRequest) System.Net.WebRequest.Create(Url);

//set the connection timeout
oHttpWebRequest.Timeout = 100;//m_ConnectTimeout;
this.postDataToHttpWebRequest ( oHttpWebRequest ,
myCollectionOfFormPostValues );

//create a response object that we can read a stream from
HttpWebResponse oHttpResponse = (HttpWebResponse)
oHttpWebRequest.GetResponse();
long workingbuffersize = 1;

//if we don't get back anything from the response, throw and exception
if (oHttpResponse == null)
{
throw new Exception("Url is missing or invalid.");
}

//Define the encoding type
try
{
//see if the page will give us back an encoding type
if (oHttpResponse.ContentEncoding.Length > 0)
m_enc = Encoding.GetEncoding(oHttpResponse.ContentEncoding );
else
m_enc = Encoding.GetEncoding(1252);
}
catch
{
// *** Invalid encoding passed
m_enc = Encoding.GetEncoding(1252);
}
//create a stream reader grabbing text we get over HTTP
StreamReader sr = new
StreamReader(oHttpResponse.GetResponseStream(),m_e nc);

//set the variable that we will use as a buffer to store characters in
while the file is downloading
char[] DownloadedCharChunk = new char[BufferSize];

//go ahead and create our streamwriter to write our file
StreamWriter sw = new StreamWriter(FilePath,false,m_enc);

sw.AutoFlush = false;

//when the working buffer size hits 0 then we know that the file has
finished downloading
while (workingbuffersize > 0)
{
//set the working buffer size based on the length of characters we
receive from the stream
//we will also set DownloadedCharChunk to the set of characters we
recieve from the stream
workingbuffersize = sr.Read(DownloadedCharChunk,0,(int) BufferSize);

if (workingbuffersize > 0)
{
//write DownloadedCharChunk to the file on disk
sw.Write(DownloadedCharChunk,0,(int) workingbuffersize );
}

} // while
sr.Close();
sw.Close();

}
catch(Exception e)
{
throw e;
}
}
private string buildPostString ( FormPostCollection formPostCollec)
{

StringBuilder sb = new StringBuilder();

foreach (FormPostItem fpi in formPostCollec)
{
//string postValue = Encode(Request.Form(postKey));
sb.Append( string.Format("{0}={1}&", fpi.Key , fpi.Value ));
}

return sb.ToString();
}

private void postDataToHttpWebRequest ( HttpWebRequest webRequest ,
Collections.FormPostCollection formPostCollec)
{
if (null != formPostCollec )
{

ASCIIEncoding encoding=new ASCIIEncoding();
byte[] data = encoding.GetBytes(this.buildPostString(formPostCol lec));
webRequest.Method = "POST";
webRequest.ContentType="application/x-www-form-urlencoded";
//oHttpWebRequest.ContentType = "text/xml";//Does Not Work

webRequest.ContentLength = data.Length;
Stream newStream=webRequest.GetRequestStream();
// Send the data.
newStream.Write(data,0,data.Length);
newStream.Close();
}

}



"Swanand Mokashi" <sw***********@swanandmokashi.com> wrote in message
news:Or**************@TK2MSFTNGP10.phx.gbl...
Hi all --

I would like to create an application(call it Application "A") that I would like to mimic exactly as a form on a foreign system (Application "F").
Application "F" is on the web (so basically I can not control it). I will
have a form exactly on Application "A" as that of Application "F".
Application "A" will submit to the url of the application "F". I would like to do a screen scrape of the confirmation obtained after submitting the form on application "F".

I can easily do the screen scraping of a static page but am not sure how to screen scrape a form result ?

Any ideas?

TIA
Swanand

Apr 3 '06 #2
And just why would you want to do this suspicious activity ?

--
( OHM ) - One Handed Man
AKA Terry Burns - http://TrainingOn.net

-----------------------------------------------------------

"Swanand Mokashi" <sw***********@swanandmokashi.com> wrote in message
news:Or**************@TK2MSFTNGP10.phx.gbl...
Hi all --

I would like to create an application(call it Application "A") that I
would like to mimic exactly as a form on a foreign system (Application
"F"). Application "F" is on the web (so basically I can not control it). I
will have a form exactly on Application "A" as that of Application "F".
Application "A" will submit to the url of the application "F". I would
like to do a screen scrape of the confirmation obtained after submitting
the form on application "F".

I can easily do the screen scraping of a static page but am not sure how
to screen scrape a form result ?

Any ideas?

TIA
Swanand

Apr 3 '06 #3
Not trying to do anythig suspicious.
Have a client who wants to sync data with another web site. We have no
control over the other web site (to say write a web service or drop an XML
file and ask them to parse). The data that needs to be synched is the same
as that submitted by the form on the other web site.

Not trying to create an auto-posting bot :)

"OHM ( One Handed Man )" <me@mine.com> wrote in message
news:OA**************@TK2MSFTNGP15.phx.gbl...
And just why would you want to do this suspicious activity ?

--
( OHM ) - One Handed Man
AKA Terry Burns - http://TrainingOn.net

-----------------------------------------------------------

"Swanand Mokashi" <sw***********@swanandmokashi.com> wrote in message
news:Or**************@TK2MSFTNGP10.phx.gbl...
Hi all --

I would like to create an application(call it Application "A") that I
would like to mimic exactly as a form on a foreign system (Application
"F"). Application "F" is on the web (so basically I can not control it).
I will have a form exactly on Application "A" as that of Application "F".
Application "A" will submit to the url of the application "F". I would
like to do a screen scrape of the confirmation obtained after submitting
the form on application "F".

I can easily do the screen scraping of a static page but am not sure how
to screen scrape a form result ?

Any ideas?

TIA
Swanand


Apr 3 '06 #4
OK, did sloans idea work, I didnt really read it through ?

--
( OHM ) - One Handed Man
AKA Terry Burns - http://TrainingOn.net

"Swanand Mokashi" <sw***********@swanandmokashi.com> wrote in message
news:OF**************@TK2MSFTNGP14.phx.gbl...
Not trying to do anythig suspicious.
Have a client who wants to sync data with another web site. We have no
control over the other web site (to say write a web service or drop an XML
file and ask them to parse). The data that needs to be synched is the same
as that submitted by the form on the other web site.

Not trying to create an auto-posting bot :)

"OHM ( One Handed Man )" <me@mine.com> wrote in message
news:OA**************@TK2MSFTNGP15.phx.gbl...
And just why would you want to do this suspicious activity ?

--
( OHM ) - One Handed Man
AKA Terry Burns - http://TrainingOn.net

-----------------------------------------------------------

"Swanand Mokashi" <sw***********@swanandmokashi.com> wrote in message
news:Or**************@TK2MSFTNGP10.phx.gbl...
Hi all --

I would like to create an application(call it Application "A") that I
would like to mimic exactly as a form on a foreign system (Application
"F"). Application "F" is on the web (so basically I can not control it).
I will have a form exactly on Application "A" as that of Application
"F". Application "A" will submit to the url of the application "F". I
would like to do a screen scrape of the confirmation obtained after
submitting the form on application "F".

I can easily do the screen scraping of a static page but am not sure how
to screen scrape a form result ?

Any ideas?

TIA
Swanand



Apr 3 '06 #5
Ok I have stated playing with your code. I tried it with an ASP.NET page
and it did not seem to work -- probably not your problem. I have a check for

if (!Page.IsPostBack)

{

}

and with posting to this page with your code, seems to return false for
Page.IsPostBack. May not be a problem as the form I want to post to
ultimately is not ASP.NET form I will test this with ASP form and let you
know.

Thanks for all your help!

Swanand

"sloan" <sl***@ipass.net> wrote in message
news:uk****************@TK2MSFTNGP12.phx.gbl...


Here are the hints you need:

You will not have the FormPostCollection and FormPostItem objects.
These are simple objects. the FormPostItem basically has the key and
value
you want to post.
FormPostCollection is a implemented CollectionBase, which keeps a
collection
of FormPostItems

However, minus that, you can figure out what is going on.

This took me about 3 days to figure out (the GET and querystring was easy,
the form/post was the tough one).
So post a "thank you" for this one.

I got the code for everything outside of the FORM/POST part from another
developer, but the form/post stuff was what I Figured out in the equation.


public void WriteTextFile(string Url, string FilePath, long BufferSize )
{

try
{

//create a web request
HttpWebRequest oHttpWebRequest = null;
oHttpWebRequest = (HttpWebRequest) System.Net.WebRequest.Create(Url);

//set the connection timeout
oHttpWebRequest.Timeout = 100;//m_ConnectTimeout;
this.postDataToHttpWebRequest ( oHttpWebRequest ,
myCollectionOfFormPostValues );

//create a response object that we can read a stream from
HttpWebResponse oHttpResponse = (HttpWebResponse)
oHttpWebRequest.GetResponse();
long workingbuffersize = 1;

//if we don't get back anything from the response, throw and exception
if (oHttpResponse == null)
{
throw new Exception("Url is missing or invalid.");
}

//Define the encoding type
try
{
//see if the page will give us back an encoding type
if (oHttpResponse.ContentEncoding.Length > 0)
m_enc = Encoding.GetEncoding(oHttpResponse.ContentEncoding );
else
m_enc = Encoding.GetEncoding(1252);
}
catch
{
// *** Invalid encoding passed
m_enc = Encoding.GetEncoding(1252);
}
//create a stream reader grabbing text we get over HTTP
StreamReader sr = new
StreamReader(oHttpResponse.GetResponseStream(),m_e nc);

//set the variable that we will use as a buffer to store characters in
while the file is downloading
char[] DownloadedCharChunk = new char[BufferSize];

//go ahead and create our streamwriter to write our file
StreamWriter sw = new StreamWriter(FilePath,false,m_enc);

sw.AutoFlush = false;

//when the working buffer size hits 0 then we know that the file has
finished downloading
while (workingbuffersize > 0)
{
//set the working buffer size based on the length of characters we
receive from the stream
//we will also set DownloadedCharChunk to the set of characters we
recieve from the stream
workingbuffersize = sr.Read(DownloadedCharChunk,0,(int) BufferSize);

if (workingbuffersize > 0)
{
//write DownloadedCharChunk to the file on disk
sw.Write(DownloadedCharChunk,0,(int) workingbuffersize );
}

} // while
sr.Close();
sw.Close();

}
catch(Exception e)
{
throw e;
}
}
private string buildPostString ( FormPostCollection formPostCollec)
{

StringBuilder sb = new StringBuilder();

foreach (FormPostItem fpi in formPostCollec)
{
//string postValue = Encode(Request.Form(postKey));
sb.Append( string.Format("{0}={1}&", fpi.Key , fpi.Value ));
}

return sb.ToString();
}

private void postDataToHttpWebRequest ( HttpWebRequest webRequest ,
Collections.FormPostCollection formPostCollec)
{
if (null != formPostCollec )
{

ASCIIEncoding encoding=new ASCIIEncoding();
byte[] data = encoding.GetBytes(this.buildPostString(formPostCol lec));
webRequest.Method = "POST";
webRequest.ContentType="application/x-www-form-urlencoded";
//oHttpWebRequest.ContentType = "text/xml";//Does Not Work

webRequest.ContentLength = data.Length;
Stream newStream=webRequest.GetRequestStream();
// Send the data.
newStream.Write(data,0,data.Length);
newStream.Close();
}

}



"Swanand Mokashi" <sw***********@swanandmokashi.com> wrote in message
news:Or**************@TK2MSFTNGP10.phx.gbl...
Hi all --

I would like to create an application(call it Application "A") that I

would
like to mimic exactly as a form on a foreign system (Application "F").
Application "F" is on the web (so basically I can not control it). I will
have a form exactly on Application "A" as that of Application "F".
Application "A" will submit to the url of the application "F". I would

like
to do a screen scrape of the confirmation obtained after submitting the

form
on application "F".

I can easily do the screen scraping of a static page but am not sure how

to
screen scrape a form result ?

Any ideas?

TIA
Swanand


Apr 3 '06 #6
Works with ASP !!
Thanks again
Swanand

"Swanand Mokashi" <sw***********@swanandmokashi.com> wrote in message
news:%2****************@TK2MSFTNGP12.phx.gbl...
Ok I have stated playing with your code. I tried it with an ASP.NET page
and it did not seem to work -- probably not your problem. I have a check
for

if (!Page.IsPostBack)

{

}

and with posting to this page with your code, seems to return false for
Page.IsPostBack. May not be a problem as the form I want to post to
ultimately is not ASP.NET form I will test this with ASP form and let you
know.

Thanks for all your help!

Swanand

"sloan" <sl***@ipass.net> wrote in message
news:uk****************@TK2MSFTNGP12.phx.gbl...


Here are the hints you need:

You will not have the FormPostCollection and FormPostItem objects.
These are simple objects. the FormPostItem basically has the key and
value
you want to post.
FormPostCollection is a implemented CollectionBase, which keeps a
collection
of FormPostItems

However, minus that, you can figure out what is going on.

This took me about 3 days to figure out (the GET and querystring was
easy,
the form/post was the tough one).
So post a "thank you" for this one.

I got the code for everything outside of the FORM/POST part from another
developer, but the form/post stuff was what I Figured out in the
equation.


public void WriteTextFile(string Url, string FilePath, long BufferSize )
{

try
{

//create a web request
HttpWebRequest oHttpWebRequest = null;
oHttpWebRequest = (HttpWebRequest) System.Net.WebRequest.Create(Url);

//set the connection timeout
oHttpWebRequest.Timeout = 100;//m_ConnectTimeout;
this.postDataToHttpWebRequest ( oHttpWebRequest ,
myCollectionOfFormPostValues );

//create a response object that we can read a stream from
HttpWebResponse oHttpResponse = (HttpWebResponse)
oHttpWebRequest.GetResponse();
long workingbuffersize = 1;

//if we don't get back anything from the response, throw and exception
if (oHttpResponse == null)
{
throw new Exception("Url is missing or invalid.");
}

//Define the encoding type
try
{
//see if the page will give us back an encoding type
if (oHttpResponse.ContentEncoding.Length > 0)
m_enc = Encoding.GetEncoding(oHttpResponse.ContentEncoding );
else
m_enc = Encoding.GetEncoding(1252);
}
catch
{
// *** Invalid encoding passed
m_enc = Encoding.GetEncoding(1252);
}
//create a stream reader grabbing text we get over HTTP
StreamReader sr = new
StreamReader(oHttpResponse.GetResponseStream(),m_e nc);

//set the variable that we will use as a buffer to store characters in
while the file is downloading
char[] DownloadedCharChunk = new char[BufferSize];

//go ahead and create our streamwriter to write our file
StreamWriter sw = new StreamWriter(FilePath,false,m_enc);

sw.AutoFlush = false;

//when the working buffer size hits 0 then we know that the file has
finished downloading
while (workingbuffersize > 0)
{
//set the working buffer size based on the length of characters we
receive from the stream
//we will also set DownloadedCharChunk to the set of characters we
recieve from the stream
workingbuffersize = sr.Read(DownloadedCharChunk,0,(int) BufferSize);

if (workingbuffersize > 0)
{
//write DownloadedCharChunk to the file on disk
sw.Write(DownloadedCharChunk,0,(int) workingbuffersize );
}

} // while
sr.Close();
sw.Close();

}
catch(Exception e)
{
throw e;
}
}
private string buildPostString ( FormPostCollection formPostCollec)
{

StringBuilder sb = new StringBuilder();

foreach (FormPostItem fpi in formPostCollec)
{
//string postValue = Encode(Request.Form(postKey));
sb.Append( string.Format("{0}={1}&", fpi.Key , fpi.Value ));
}

return sb.ToString();
}

private void postDataToHttpWebRequest ( HttpWebRequest webRequest ,
Collections.FormPostCollection formPostCollec)
{
if (null != formPostCollec )
{

ASCIIEncoding encoding=new ASCIIEncoding();
byte[] data = encoding.GetBytes(this.buildPostString(formPostCol lec));
webRequest.Method = "POST";
webRequest.ContentType="application/x-www-form-urlencoded";
//oHttpWebRequest.ContentType = "text/xml";//Does Not Work

webRequest.ContentLength = data.Length;
Stream newStream=webRequest.GetRequestStream();
// Send the data.
newStream.Write(data,0,data.Length);
newStream.Close();
}

}



"Swanand Mokashi" <sw***********@swanandmokashi.com> wrote in message
news:Or**************@TK2MSFTNGP10.phx.gbl...
Hi all --

I would like to create an application(call it Application "A") that I

would
like to mimic exactly as a form on a foreign system (Application "F").
Application "F" is on the web (so basically I can not control it). I
will
have a form exactly on Application "A" as that of Application "F".
Application "A" will submit to the url of the application "F". I would

like
to do a screen scrape of the confirmation obtained after submitting the

form
on application "F".

I can easily do the screen scraping of a static page but am not sure how

to
screen scrape a form result ?

Any ideas?

TIA
Swanand



Apr 3 '06 #7
You can also try SWExplorerAutomation (SWEA)
(http://www.webunittesting.com). SWEA supports frames, DHTML (AJAX)
pages, windows and HTML dialogs, popup windows, file downloads. SWEA
solutions can be run from ASP.NET pages or windows service.

Apr 3 '06 #8

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: Adrian Lumsden | last post by:
Hello, I have an app where I have to screen scrape to capture an image from a JMF film player. The user is given a dialog with a list of frames that can be exported as images. If the one they...
0
by: Jason Steeves | last post by:
I have one .aspx form that my users fill out and this then takes that information and populates a second .aspx form via session variables. I need to screen scrape the second .aspx form and e-mail...
3
by: Ollie | last post by:
I know you can screen scrape a website using the System.Net.HttpWebResponse & System.Net.HttpWebRequest classes. But how do you screen scrape a secured website (https) that takes a username &...
2
by: Rob Lauer | last post by:
I have written two completely separate web applications that cannot talk directly to one another (applications "A" and "B"). Application "A" has a form that takes some input (radio buttons,...
5
by: crjunk | last post by:
I have a screen scrape page that allows the user to submit a url. When they hit submit, the page is returned back to them on my screen scrape page. Which computer actuall connects to the url to...
7
by: ljr2600 | last post by:
Hello, I'm very new to python and still familiarizing myself with the language, sorry if the post seems moronic or simple. For a side project I'm working on I need to be able to scrape a...
0
by: itfetish | last post by:
We have a web parts intranet at our office here I have been working on. I have been trying to create a way of getting our OCE TDS 600 plotter queue to be a web part on the page, seeing as their...
1
by: nbomike | last post by:
Hello. I want to scrape pages from a site that generates pages from form inputs using this web app . However, the URL of the results page (the page I want to scrape) is masked and is always the same....
1
by: newdev | last post by:
Hi All, Can somebody maybe please help me? - how do i screen scrape data from a dos application / window to .net application by using c#? - how do i screen scrape data from a dos application /...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.