473,839 Members | 1,502 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Possible to "scape" a website?

bbu
Hello,
I need to collect data from a website, everyday. I get an email everyday with 6 to 10 links that is can pick form to get to them. I need to automate as much as possible.

My ideas so far: use an Outlook2000 macro
to automatically open the email when it comes in.
to send the links to a threaded web service
the web service thread
opens the link
opens a notepad of "the source"
scans the source for keywords and
puts the strings into an array
Will this work?
Is this the best way?
Thanks
BB
Nov 21 '05 #1
1 1045
Nak
Hi there,

http://www.members.lycos.co.uk/nickp.../soft-wtr2.htm

The above link has source of mine for an applicaton that scans HTML for
links, it would be easily modified to do as you suggest. As for opening
Outlook, that shouldn't be too difficult, you could even create a kind of
"Send to" plug-in that will enable you to open the file. One thing you
might want to think about is the "Referer" of the web page, this should be
set before attempting to download the content as you *might* encounter
problems. Getting the referer shouldn't be that difficult, the homepage of
the people who send you the email might be enough?

Nick.

"bbu" <bb*@socal.rr.c om> wrote in message
news:HZ******** ***********@twi ster.socal.rr.c om...
Hello,
I need to collect data from a website, everyday. I get an email everyday
with 6 to 10 links that is can pick form to get to them. I need to automate
as much as possible.

My ideas so far: use an Outlook2000 macro
to automatically open the email when it
comes in.
to send the links to a threaded web service
the web service thread

opens the link

opens a notepad of "the source"

scans the source for keywords and

puts the strings into an array
Will this work?
Is this the best way?
Thanks
BB
Nov 21 '05 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

83
4255
by: liketofindoutwhy | last post by:
I am learning more and more Prototype and Script.aculo.us and got the Bungee book... and wonder if I should get some books on jQuery (jQuery in Action, and Learning jQuery) and start learning about it too? Once I saw a website comparing Prototype to Java and jQuery to Ruby... but now that I read more and more about Prototype, it is said that Prototype actually came from Ruby on Rails development and the creator of Prototype created it...
0
9855
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
10907
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
1
10647
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
10293
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
9426
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
7828
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
7017
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5682
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
2
4064
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.