Automated web browing

[mr_marcin]

Hi

Does anybody have some idea how to input some text into inputbox on
one page, than press some button on that page, that will load another
page, and finally read the responde? Suppose I want to write a price
comparision engine, where I would like to parse shops website for
price each time user wants.

I have found similar feature in Symfony framework, called sfBrowser
(or sfTestBrowser). These are made for automated functional testing,
but should provide the functinality I am requesting.

The question is: will this be efficient enough? Maybe there are other
ways to achieve this? Of course I can always try to make it more
manually - look for some pattern in url (search is usually done via
GET), and parse output html.

Thanks for help
Marcin

[Manuel Lemos]

on 01/17/2008 07:52 PM mr_marcin said the following:

Hi

Does anybody have some idea how to input some text into inputbox on
one page, than press some button on that page, that will load another
page, and finally read the responde? Suppose I want to write a price
comparision engine, where I would like to parse shops website for
price each time user wants.

I have found similar feature in Symfony framework, called sfBrowser
(or sfTestBrowser). These are made for automated functional testing,
but should provide the functinality I am requesting.

The question is: will this be efficient enough? Maybe there are other
ways to achieve this? Of course I can always try to make it more
manually - look for some pattern in url (search is usually done via
GET), and parse output html.

You may want to try this HTTP client class. Basically it acts like a
browser accessing pages, submitting forms, collecting cookies, handling
redirection, etc. which seems what you need to retrieve the pages with
the prices you want to grab.

http://www.phpclasses.org/httpclient
--

Regards,
Manuel Lemos

PHP professionals looking for PHP jobs
http://www.phpclasses.org/professionals/

PHP Classes - Free ready to use OOP components written in PHP
http://www.phpclasses.org/

[Jerry Stuckle]

mr_marcin wrote:

Hi

Does anybody have some idea how to input some text into inputbox on
one page, than press some button on that page, that will load another
page, and finally read the responde? Suppose I want to write a price
comparision engine, where I would like to parse shops website for
price each time user wants.

I have found similar feature in Symfony framework, called sfBrowser
(or sfTestBrowser). These are made for automated functional testing,
but should provide the functinality I am requesting.

The question is: will this be efficient enough? Maybe there are other
ways to achieve this? Of course I can always try to make it more
manually - look for some pattern in url (search is usually done via
GET), and parse output html.

Thanks for help
Marcin

cURL will allow you to get or post to pages, and will return the data.
I much prefer it over the HTTPClient class. It's more flexible.

--
=============== ===
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
js*******@attgl obal.net
=============== ===

[Marlin Forbes]

mr_marcin wrote:

Hi

Does anybody have some idea how to input some text into inputbox on
one page, than press some button on that page, that will load another
page, and finally read the responde? Suppose I want to write a price
comparision engine, where I would like to parse shops website for
price each time user wants.

Hi there,

SimpleTest has a class included called SimpleBrowser, which does what
you want, with a very intuitive API. It's not too fast tho...

SimpleTest: http://www.lastcraft.com/simple_test.php

Or, you can interactively setup browsing sessions with the Selenium IDE
and then use the PHP client for the Selenium Remote Control to run them...

Selenium IDE: http://www.openqa.org/selenium-ide/
Selenium RC: http://www.openqa.org/selenium-rc/
PHP Client for Selenium: http://pear.php.net/package/Testing_Selenium

Misc:
http://blog.thinkphp.de/archives/133...-Selenium.html
Regards,
Marlin Forbes
Freelance Developer
Data Shaman
datashaman.com
[phone]

[mr_marcin]

Or, you can interactively setup browsing sessions with the Selenium IDE

and then use the PHP client for the Selenium Remote Control to run them...

Selenium IDE:http://www.openqa.org/selenium-ide/
Selenium RC:http://www.openqa.org/selenium-rc/
PHP Client for Selenium:http://pear.php.net/package/Testing_Selenium

This sounds like a quite easy to use package, but will this be
efficient enough? I will check all options next week.

[mr_marcin]

cURL will allow you to get or post to pages, and will return the data.

I much prefer it over the HTTPClient class. It's more flexible.

I guess this approach requires some manual job, but you are right -
thats the most flexible and probably most effective way.

[R. Rajesh Jeba Anbiah]

On Jan 18, 2:52 am, mr_marcin <[e-mail] rote:

Hi

Does anybody have some idea how to input some text into inputbox on
one page, than press some button on that page, that will load another
page, and finally read the responde? Suppose I want to write a price
comparision engine, where I would like to parse shops website for
price each time user wants.

I have found similar feature in Symfony framework, called sfBrowser
(or sfTestBrowser). These are made for automated functional testing,
but should provide the functinality I am requesting.

The question is: will this be efficient enough? Maybe there are other
ways to achieve this? Of course I can always try to make it more
manually - look for some pattern in url (search is usually done via
GET), and parse output html.

1. If you're looking for client tools http://www.iopus.com/imacros/firefox/
2. Web scraping with cURL or HTTPClient class
3. Look for the Web services (SOAP, XML, etc)

--
<?php echo 'Just another PHP saint'; ?>
Email: rrjanbiah-at-Y!com Blog: http://rajeshanbiah.blogspot.com/

[Manuel Lemos]

Hello,

on 01/17/2008 10:15 PM Jerry Stuckle said the following:

>Does anybody have some idea how to input some text into inputbox on
one page, than press some button on that page, that will load another
page, and finally read the responde? Suppose I want to write a price
comparision engine, where I would like to parse shops website for
price each time user wants.

I have found similar feature in Symfony framework, called sfBrowser
(or sfTestBrowser). These are made for automated functional testing,
but should provide the functinality I am requesting.

The question is: will this be efficient enough? Maybe there are other
ways to achieve this? Of course I can always try to make it more
manually - look for some pattern in url (search is usually done via
GET), and parse output html.

Thanks for help
Marcin

cURL will allow you to get or post to pages, and will return the data. I
much prefer it over the HTTPClient class. It's more flexible.

I wonder which HTTP client you are talking about. The HTTP client I
mentioned wraps around Curl or socket functions depending on which is
more convinient to use in each PHP setup. This is the HTTP client class
I meant:

http://www.phpclasses.org/httpclient

As for Curl being flexible, I wonder what you are talking about.

Personally I find it very odd that you cannot read retrieved pages with
Curl in small chunks at a time without having to use callbacks. This is
bad because it makes very difficult to retrieve and process large pages
without using external files nor exceeding the PHP memory limits.

--

Regards,
Manuel Lemos

PHP professionals looking for PHP jobs
http://www.phpclasses.org/professionals/

PHP Classes - Free ready to use OOP components written in PHP
http://www.phpclasses.org/

[Jerry Stuckle]

Manuel Lemos wrote:

Hello,

on 01/17/2008 10:15 PM Jerry Stuckle said the following:

>>Does anybody have some idea how to input some text into inputbox on
one page, than press some button on that page, that will load another
page, and finally read the responde? Suppose I want to write a price
comparision engine, where I would like to parse shops website for
price each time user wants.

I have found similar feature in Symfony framework, called sfBrowser
(or sfTestBrowser). These are made for automated functional testing,
but should provide the functinality I am requesting.

The question is: will this be efficient enough? Maybe there are other
ways to achieve this? Of course I can always try to make it more
manually - look for some pattern in url (search is usually done via
GET), and parse output html.

Thanks for help
Marcin

cURL will allow you to get or post to pages, and will return the data. I
much prefer it over the HTTPClient class. It's more flexible.

I wonder which HTTP client you are talking about. The HTTP client I
mentioned wraps around Curl or socket functions depending on which is
more convinient to use in each PHP setup. This is the HTTP client class
I meant:

http://www.phpclasses.org/httpclient

The same one.

As for Curl being flexible, I wonder what you are talking about.

I can do virtually anything with it that I can do with a browser, with
the exception of client side scripting. Also much less overhead than
the httpclient class.

Personally I find it very odd that you cannot read retrieved pages with
Curl in small chunks at a time without having to use callbacks. This is
bad because it makes very difficult to retrieve and process large pages
without using external files nor exceeding the PHP memory limits.

So? I never needed to. First of all, I have no need to retrieve huge
pages. The larges I've ever downloaded (a table with lots of info) was
a little over 3MB and Curl and PHP handled it just fine.

But if the text were split, you need to do additional processing to
handle splits at inconvenient locations. Much easier to add everything
to a temporary file and read it back in the way I need to so it.

But that's one of the advantages of cURL - it gives me the option of
doing the callbacks or not.

--
=============== ===
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
js*******@attgl obal.net
=============== ===

[Jerry Stuckle]

Manuel Lemos wrote:

<snip junk>

Manuel,

I'm not going to argue with you about whether the HTTPClass is easier to
use or whatever.

My single point was that cURL is more flexible. You can do anything
with cURL that you can with the HTTPClient class and more. That is
pretty obvious - because the HTTPClient class is built on cURL - so if
cURL can't do it, neither can the HTTPClient class.

But being built on cURL, the HTTPClient class restricts what you can do.
So it is less flexible.

You can sit there and argue all you want as to the other merits of your
class. I won't bite. Because that was not my point.

--
=============== ===
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
js*******@attgl obal.net
=============== ===