473,408 Members | 1,729 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,408 software developers and data experts.

Screen scraper again

I recently posted a query about screen scraping, but haven't turned up
any leads yet. Here's what I need to do:

The first screen is retrieved via HttpWebRequest/Response. Easy
enough, as no parameters are required. But then I need to fill in
some fields from that initial screen and POST it back to the website
(to get back info on a specific subject).

It seems easier to do a GET, but unfortunately I need to do this via
POST.

Surely this has been done a lot, right? Any clues on where to look?

Are there other newsgroups that would be more appropriate for this
question?

Nov 18 '05 #1
6 1317
Are you saying you're having a problem doing the POST? The following
function posts to a website and returns a response stream. The body of the
post is passed as a string (sPOST). If you need help figuring out what
should be in the post body, spend some time watching what IE passes up using
a tool like www.fiddlertool.com provides.

public static Stream DoHTTPPost(string sURL, string sLang, string sPOST)
{
HttpWebRequest oRequest = (HttpWebRequest)WebRequest.Create(sURL);
oRequest.UserAgent= "Picker/1.0";
oRequest.Headers.Add("Accept-Language", sLang);
oRequest.Method = "POST";
oRequest.ContentType="text/xml";
StreamWriter myWriter = null;
Stream strmPost = oRequest.GetRequestStream();
try
{
myWriter = new StreamWriter(strmPost, System.Text.Encoding.UTF8);
myWriter.Write(sPOST);
}
catch (Exception e)
{
MessageBox.Show(e.Message);
}
finally
{
myWriter.Close();
}

return oRequest.GetResponse().GetResponseStream();
}
--
Thanks,

Eric Lawrence
Program Manager
Assistance and Worldwide Services

This posting is provided "AS IS" with no warranties, and confers no rights.
"_eee_" <_n****@nomail.com> wrote in message
news:7b********************************@4ax.com...
I recently posted a query about screen scraping, but haven't turned up
any leads yet. Here's what I need to do:

The first screen is retrieved via HttpWebRequest/Response. Easy
enough, as no parameters are required. But then I need to fill in
some fields from that initial screen and POST it back to the website
(to get back info on a specific subject).

It seems easier to do a GET, but unfortunately I need to do this via
POST.

Surely this has been done a lot, right? Any clues on where to look?

Are there other newsgroups that would be more appropriate for this
question?

Nov 18 '05 #2

On Thu, 26 Feb 2004 18:49:07 -0800, "Eric Lawrence [MSFT]"
<e_********@hotmail.com> wrote:
Are you saying you're having a problem doing the POST? The following
function posts to a website and returns a response stream.
....


That was exactly what I was looking for, Eric!
Fiddlertool is also very useful.

Now, where did you find out how to do that?
Any books that cover it?
Nov 18 '05 #3
I recommend O'Reilly's "HTTP: The Definitive Reference", which is the book I
used constantly when I was writing Fiddler in C#.

Thanks,

Eric Lawrence
Program Manager
Assistance and Worldwide Services

This posting is provided "AS IS" with no warranties, and confers no rights.

"_eee_" <_n****@nomail.com> wrote in message
news:kh********************************@4ax.com...

On Thu, 26 Feb 2004 18:49:07 -0800, "Eric Lawrence [MSFT]"
<e_********@hotmail.com> wrote:
Are you saying you're having a problem doing the POST? The following
function posts to a website and returns a response stream.
....


That was exactly what I was looking for, Eric!
Fiddlertool is also very useful.

Now, where did you find out how to do that?
Any books that cover it?

Nov 18 '05 #4
On Fri, 27 Feb 2004 13:36:06 -0800, "Eric Lawrence [MSFT]"
<e_********@hotmail.com> wrote:
I recommend O'Reilly's "HTTP: The Definitive Reference", which is the book I
used constantly when I was writing Fiddler in C#.

Thanks,

Eric Lawrence
Program Manager
Assistance and Worldwide Services


You WROTE Fiddler? I should have put that together.
VERY nice program, Eric.

I'll look for the book.

Nov 18 '05 #5
Thanks!

Please feel free to let me know (via the "Contact" link at fiddlertool.com)
if you have any suggestions for future Fiddler enhancements.

Eric Lawrence
Program Manager
Assistance and Worldwide Services

This posting is provided "AS IS" with no warranties, and confers no rights.

"_eee_" <_n****@nomail.com> wrote in message
news:iv********************************@4ax.com...
On Fri, 27 Feb 2004 13:36:06 -0800, "Eric Lawrence [MSFT]"
<e_********@hotmail.com> wrote:
I recommend O'Reilly's "HTTP: The Definitive Reference", which is the book Iused constantly when I was writing Fiddler in C#.

Thanks,

Eric Lawrence
Program Manager
Assistance and Worldwide Services


You WROTE Fiddler? I should have put that together.
VERY nice program, Eric.

I'll look for the book.

Nov 18 '05 #6
On Sat, 28 Feb 2004 00:02:26 -0800, "Eric Lawrence [MSFT]"
<e_********@hotmail.com> wrote:
Thanks!

Please feel free to let me know (via the "Contact" link at fiddlertool.com)
if you have any suggestions for future Fiddler enhancements.

Eric Lawrence
Program Manager
Assistance and Worldwide Services


I shall, Eric. It looks like the immediate problem is solved,
and Fiddler was very helpful for that. I'll probably be using it more
in future Asp.net projects. (Others on this group should take a look)
Nov 18 '05 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: Brian W | last post by:
I was hoping someone here would know something about screen scrapers, spiders, bots. Perhaps it's a stupid question, but here it goes anyway... Is there any way to make data harder for a screen...
14
by: n8 | last post by:
Hi, Hi have to do the followign and have been racking my brain with various solutions that have had no so great results. I want to use the System.Net.WebClient to submit data to a form (log a...
18
by: DavidS | last post by:
Have resW=screen.width; resH=screen.height in javascript. How can I read these values in ASP.NET source code - Page_Load function of code behind? Any suggestions?
4
by: Ronald S. Cook | last post by:
I've been asked to extract data from web pages. Given that they are rendered in HTML and not any sort of XML I'm wondering how to go about "scraping" such a web page of data. Can anyone give me...
1
by: swestenra | last post by:
I am trying to build a screen scraper. But not just a plain screen scraper, it must also automate the entry of data. Background: We have a new intranet system that goes in to production soon. ...
4
by: onetitfemme | last post by:
Say, people would like to log into their hotmail, yahoo and gmail accounts and "keep an eye" on some text/part of a site .. I think something like that should be out there, since not all sites...
7
by: James Stroud | last post by:
Hello, Does anyone know of an example, however modest, of a screenscraper authored in python? I am using Firefox. Basically, I am answering problems via my browser and being scored for each...
2
by: voroojak | last post by:
Hi Does any one have any idead about screen scraper? Or wher can i find good information? thanks alot
1
by: kronecker | last post by:
A screen scraper is a program that removes text only from a web site. I pinched this one from the web: Public Class Form1 Private Sub Form1_Load(ByVal sender As System.Object, _ ByVal e As...
0
BarryA
by: BarryA | last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
0
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
0
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...
0
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing,...
0
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.