473,765 Members | 2,172 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

The opposite of page scraping - what's it called

I'm familiar with the idea of page-scraping - getting data from
websites where there's no nice feed to use.

Is there a name for the opposite i.e. when your program simulates a
human inputting data to a site? I need to do this and I don't know
what to search for. So far, my searches have been fruitless.

The supplier of our IPT system won't support direct updates to their
directory or config database. I'm trying to implement an Identity
Management solution and the only way I can think that we can provision
users into the IPT system is to have a program access the IPT system's
web-based user management system. It's going to be nasty but
otherwise the IM stuff will be half-a-job.

Regards

SSG

Mar 31 '07 #1
4 3271

"ssg3141592 6" <ne**********@g mail.comwrote in message
news:11******** **************@ y66g2000hsf.goo glegroups.com.. .
I'm familiar with the idea of page-scraping - getting data from
websites where there's no nice feed to use.

Is there a name for the opposite i.e. when your program simulates a
human inputting data to a site? I need to do this and I don't know
web crawling?
spidering?
automated web testing?
programmatic access to a website?
what to search for. So far, my searches have been fruitless.
http://msdn2.microsoft.com/en-us/lib...eststream.aspx
http://twill.idyll.org/examples.html
>
The supplier of our IPT system won't support direct updates to their
directory or config database. I'm trying to implement an Identity
Management solution and the only way I can think that we can provision
users into the IPT system is to have a program access the IPT system's
web-based user management system. It's going to be nasty but
otherwise the IM stuff will be half-a-job.

Regards

SSG

Mar 31 '07 #2
On Mar 31, 12:16 pm, "ssg3141592 6" <newsjunkm...@g mail.comwrote:
I'm familiar with the idea of page-scraping- getting data from
websites where there's no nice feed to use.

Is there a name for the opposite i.e. when your program simulates a
human inputting data to a site? I need to do this and I don't know
what to search for. So far, my searches have been fruitless.

The supplier of our IPT system won't support direct updates to their
directory or config database. I'm trying to implement an Identity
Management solution and the only way I can think that we can provision
users into the IPT system is to have a program access the IPT system'sweb-based user management system. It's going to be nasty but
otherwise the IM stuff will be half-a-job.

Regards

SSG
You can try SWExplorerAutom ation (SWEA) from http://webiussoft.com.
SWEA records Web automation scripts and gnerates c# or VB.NET code for
them.

Apr 4 '07 #3
On Mar 31, 6:16 pm, "ssg3141592 6" <newsjunkm...@g mail.comwrote:
I'm familiar with the idea of page-scraping - getting data from
websites where there's no nice feed to use. Is there a name for the opposite i.e. when your program simulates a
human inputting data to a site?

You need automated web form filling. Try the free iMacros for Firefox
tool at http://www.iopus.com/imacros/firefox/

If you need to control this application from within your program (VBS,
VBA. C#,...), you can use the Scripting Edition from the same website.

Francis
Apr 5 '07 #4
On 5 Apr, 09:29, FrancisHean...@ googlemail.com wrote:
On Mar 31, 6:16 pm, "ssg3141592 6" <newsjunkm...@g mail.comwrote:I 'm familiar with the idea of page-scraping - getting data from
websites where there's no nice feed to use. Is there a name for the opposite i.e. when your program simulates a

human inputting data to a site?

You need automated web form filling. Try the free iMacros for Firefox
tool athttp://www.iopus.com/imacros/firefox/

If you need to control this application from within your program (VBS,
VBA. C#,...), you can use the Scripting Edition from the same website.

Francis
Thank you all for your help.

Apr 11 '07 #5

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
4381
by: David Jones | last post by:
Hi, I'm interested in learning about web scraping/site scraping using Python. Does anybody know of some online resources or have any modules that are available to help out. O'Reilly published an interesting book "Spidering Hacks" which covered some great scraping hacks but it is all written in Perl. I don't know Perl and don't want to. I'm new to programing and have been advised to start with Python. So far so good ... but need some...
2
1561
by: cosmic foo | last post by:
how do people prevent (or slow down) page scraping? i was thinking of using a session variable on the server to track the frequency of page requests, and if it seems faster than any human could request, shut them down
5
1635
by: Lorenzo | last post by:
I've a web site with a classic asp login page (https), another where in a textbox i write a sql query and a third that shows the resulset of the query.... Now i want to create an asp.net application that have only one page with 2 textbox for login, a textbox where i'll write the sql query and a textbox where shows the html of the previuos third page with the resultset of the query... i try using the HttpWebRequest Class but when i...
6
1889
by: Skeptical | last post by:
Hello, I am trying to embed html output into my webform but could not figure out how to so far. The form will execute a Perl script with some parameters, and script will output some html code. I need to capture and render this html into my webform. Any ideas?
4
9458
by: Ronald S. Cook | last post by:
I've been asked to extract data from web pages. Given that they are rendered in HTML and not any sort of XML I'm wondering how to go about "scraping" such a web page of data. Can anyone give me any starting place? Thanks, Ron
7
3562
by: ljr2600 | last post by:
Hello, I'm very new to python and still familiarizing myself with the language, sorry if the post seems moronic or simple. For a side project I'm working on I need to be able to scrape a modern computer desktop. Is there any basic material already available to do this? I'd rather not need to write my own to interact with hardware. Thanks!
4
3459
by: onetitfemme | last post by:
Say, people would like to log into their hotmail, yahoo and gmail accounts and "keep an eye" on some text/part of a site .. I think something like that should be out there, since not all sites provide RSS feeds nor are they really interested in providing consistent and informative content (what we (almost) all are looking for). .. I have been mostly programming java lately. THis is how I see such an API could -very basically indeed- be...
4
449
by: lucavilla | last post by:
If you go to http://europe.nokia.com/A4305060, fill the "Enter your product code:" field with the value "0523183" and press "Go" (the ending page URL varies because there's a variable session-ID in the URL-link associated to "Go") you will obtain this string: "Version: RM43_V1.10.030" Is it possible to have a string.php page that just display this string? how can I do it?
11
2920
by: tdmailbox | last post by:
I have a website that I subscribe to that allows me to track tasks. I have an extra computer with a big monitor that i want to always display the task list. the problem is after so many hours I get logged out. I need to need to reload that sign in url every few hours If I load this url it logs me and resets my session variable and redirects me...
0
9568
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
9398
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
10156
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
10007
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
9832
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
1
7375
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
5275
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
2
3531
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
2805
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.