473,686 Members | 2,107 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Seeking examples of screen scraping....

Jim
I want to extract data from several websites that I visit daily. I'd like
to condense the info into a single web page that I can visit (instead of the
multiple websites I have to visit now to get the same info). There are no
open APIs or webservices for these websites that I am aware of.

I am using VS 2005 and VB.Net. If you could point out some sample code (or
controls to accomplush the same thing), I'd really appreciate it. (C# - and
even VS 2003 are OK)

Thanks!
Jan 13 '06 #1
29 1929
KJ
A google search on the terms ".net screen scrape html" brings up a
great many options.

Jan 13 '06 #2
Jim,
If you intend to get serious about this you are probably going to want to
learn to use a library. Take a look at Simon Mourier's HtmlAgilityPack .
Peter

--
Co-founder, Eggheadcafe.com developer portal:
http://www.eggheadcafe.com
UnBlog:
http://petesbloggerama.blogspot.com


"Jim" wrote:
I want to extract data from several websites that I visit daily. I'd like
to condense the info into a single web page that I can visit (instead of the
multiple websites I have to visit now to get the same info). There are no
open APIs or webservices for these websites that I am aware of.

I am using VS 2005 and VB.Net. If you could point out some sample code (or
controls to accomplush the same thing), I'd really appreciate it. (C# - and
even VS 2003 are OK)

Thanks!

Jan 13 '06 #3
Jim

"KJ" <n_**********@m ail.com> wrote in message
news:11******** *************@g 47g2000cwa.goog legroups.com...
A google search on the terms ".net screen scrape html" brings up a
great many options.
Gee!! Thanks! I hadn't thought of that.

(Now, for the rest of you with working frontal lobes, I'd still like to see
what you have. Personal recommendations are always better than random
searches.)

Jan 13 '06 #4
Jim,

See this sample on our website.

http://www.vb-tips.com/default.aspx?...f-56dbb63fdf1c

I hope this helps,

Cor
Jan 13 '06 #5
KJ
You know Jim, I actually thought what I wrote was helpful. And I also
think your sarcasm is out of line.

Jan 13 '06 #6
Jim

"KJ" <n_**********@m ail.com> wrote in message
news:11******** **************@ g47g2000cwa.goo glegroups.com.. .
You know Jim, I actually thought what I wrote was helpful. And I also
think your sarcasm is out of line.


And I think your lazy answer is out of line and sarcastic.

I really get tired of seeing people respond to posts by simply saying
"google it".

If you think the poster is so dense that they don't know how to use search
engines, you should probably skip replying at all as it would do little
good.

Posting a reply like "google it" is a waste of bandwidth and time to those
that view these newsgroups.

Helpful and pertinent posts are welcomed and appreciated. "Google it" is
neither helpful nor pertinent.

How many newsgroup users do you think have not heard of or used Google?

BTW.....your precious Google results only give answers (one of which is
repeated at least 4 times in the first 20 examples - with 2 other repeat
answers accounting for 5 more of the first 20 results) that are very
elementary. The reason for posting the request here is to get more in-depth
answers from the knowledgable people that frequent the newsgroups.

If I have need of a simplistic, irrelevant result I will most assuredly
"Goggle it".

Jim
Jan 13 '06 #7
Steven Smith has a useful article on using HTTPWebRequest to collect the
contents of an HTML site here
http://authors.aspalliance.com/steve.../netscrape.asp
--
--- Nick Malik [Microsoft]
MCSD, CFPS, Certified Scrummaster
http://blogs.msdn.com/nickmalik

Disclaimer: Opinions expressed in this forum are my own, and not
representative of my employer.
I do not answer questions on behalf of my employer. I'm just a
programmer helping programmers.
--
"Jim" <re***@groups.p lease> wrote in message
news:uW******** *************@b ignews6.bellsou th.net...
I want to extract data from several websites that I visit daily. I'd like
to condense the info into a single web page that I can visit (instead of
the multiple websites I have to visit now to get the same info). There are
no open APIs or webservices for these websites that I am aware of.

I am using VS 2005 and VB.Net. If you could point out some sample code
(or controls to accomplush the same thing), I'd really appreciate it.
(C# - and even VS 2003 are OK)

Thanks!

Jan 15 '06 #8
You know, you are just getting help that's worth what you paid for it... If
you disagree with the reply, follow your own advice and skip it, no need to
make frontal lobe comments.
If you think the poster is so dense that they don't know how to use search
engines, you should probably skip replying at all as it would do little
good.

Jan 15 '06 #9
Hello Jim,

Those of us who choose to help others on the newsgroup do it not because we
are paid but out of a desire to help fellow coders and maybe because other
coders help us. It's a chain.

Your attitude leaves a lot to be desired. Your question is un-specific,
about a very broad topic, and you have not presented a particular
programming problem. You want an answer that will give you the complete
overview of the solution without making any effort from your side to write
code or evolve a strategy to solve the problem.

Even a very basic search could tell you that you can retrieve the data of a
webpage using the HttpWebRequest object, and from then on it's a question of
logic.

I don't think you should be so rude on the newsgroup to people who care to
answer, or maybe after a while nobody will care to answer.

Regards
Cyril Gupta
Jan 15 '06 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
4351
by: Jonathan Epstein | last post by:
I would like to perform a more classical type of "screen scraping" than what most people now associate with this term. I only want to find all the text on the current screen, and obtain associated screen coordinates. This probably must be done using OCR. This need only run on Windows. A fairly-pure Python solution would be ideal because most of the software which would use this functionality is also written in Python. The ideal...
4
5732
by: Roland Hall | last post by:
Am I correct in assuming screen scraping is just the response text sent to the browser? If so, would that mean that this could not be screen scraped? function moi() { var tag = '<a href='; var tagType1 = '"mail'+'to:', tagType2 = '">', tagType3 = '<\/a>'; var user1 = 'web', user2 = 'master', user3 = '@'; var dom1 = 'danger', dom2 = 'ous', dom3 = 'ly'; var tld = '.us';...
3
2356
by: Jim Giblin | last post by:
I need to scrape specific information from another website, specifically the prices of precious metals from several different vendors. While I will credit the vendors as the data source, I do not want to use the format of their pages, and want the inforamtion consolidated to a single page of my design. I did something like this for a client a couple of years ago in ASP, but it was complex, and I do not have access to the code. A...
4
3453
by: rachel | last post by:
Hello, I am currently contracted out by a real estate agent. He has a page that he has created himself that has a list of homes.. their images and data in html format. He wants me to take this page and reformat it so that it looks different. Do I use screen scraping to do this? Could someone please point me to a good screen scraping
29
2040
by: Jim | last post by:
I want to extract data from several websites that I visit daily. I'd like to condense the info into a single web page that I can visit (instead of the multiple websites I have to visit now to get the same info). There are no open APIs or webservices for these websites that I am aware of. I am using VS 2005 and VB.Net. If you could point out some sample code (or controls to accomplush the same thing), I'd really appreciate it. (C# - and...
4
3315
by: different.engine | last post by:
Folks: I am screen scraping a large volume of data from Yahoo Finance each evening, and parsing with Beautiful Soup. I was wondering if anyone could give me some pointers on how to make it less obvious to Yahoo that this is what I am doing, as I fear that they probably monitor for this type of activity, and will soon ban my IP.
3
5163
by: WFDGW2 | last post by:
I want to write or obtain C++ code that will scrape text from a dialog box within a poker client, and then record that text somewhere else. What do I do? Thanks.
0
9057
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
1
8770
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
8780
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
7603
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
5800
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
4309
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
4534
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
2947
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
2208
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.