HTTP-POST simultaneous requests

[mark]

Hello,

I want to create a php scraper that will get some information from
e.g. 5 sites simultaneously. I tried the following script:
http://www.phpied.com/simultaneuos-h...php-with-curl/
Everything works fine, but what I want is simultaneuos (something to
multithread, when these 5 websites will be loaded not one after
another, but by using different sockets) scraper.

In addition I would like to display the results as soon as it will be
scraped. So when first http-post get answer, it will show the result
and wait for the rest of the pages (not display everything when all
scraping is done).
Any ideas how can I achieve it? Thanks!

regards, Mark

[Jerry Stuckle]

mark wrote:

Hello,

I want to create a php scraper that will get some information from
e.g. 5 sites simultaneously. I tried the following script:
http://www.phpied.com/simultaneuos-h...php-with-curl/
Everything works fine, but what I want is simultaneuos (something to
multithread, when these 5 websites will be loaded not one after
another, but by using different sockets) scraper.

In addition I would like to display the results as soon as it will be
scraped. So when first http-post get answer, it will show the result
and wait for the rest of the pages (not display everything when all
scraping is done).
Any ideas how can I achieve it? Thanks!

regards, Mark

Sorry, PHP doesn't do multithreading very well. Probably the best you
can do is start multiple background processes to do the work then
communicate via a database, shared memory, etc.

As for displaying the contents immediately - again, not guaranteed
possible. You can flush() the buffers in PHP - but that doesn't
guarantee the data will be sent by the webserver to the client
immediately, nor does it guarantee the client will display the data
before it's received.

Sounds like java might be a better fit.

--
=============== ===
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
js*******@attgl obal.net
=============== ===

[=?UTF-8?B?SXbDoW4gU8OhbmNoZXogT3J0ZWdh?=]

mark wrote:

In addition I would like to display the results as soon as it will be
scraped. So when first http-post get answer, it will show the result
and wait for the rest of the pages (not display everything when all
scraping is done).
Any ideas how can I achieve it? Thanks!

Either:
- Run in console and use fork().
- Use raw HTTP and some socket_select() magic.
- curl_multi_exec ().
- Rely on javascript, ajax techniques, and make a web browser launch 5
queries yo your web server, each of one scraping a site.
- Use ignore_user_abo rt() and a mix of raw HTTP with sockets to blindly
launch PHP threads. This one's quite tricky to pull out.

There may be more ways to do this, but unless you know what a critical
section is, please stay away from concurrent (AKA multithread) programming.

Besides, you want IPC to get the results as they appear - to make your life
easier, you should stick with either curl_multi queries or rely on
javascript to individually fetch results as they are ready.
--
----------------------------------
Iván Sánchez Ortega -ivan-algarroba-sanchezortega-punto-es-

Now listening to: Deep Forest - Music.Detected_ (2002) - [4] Computer
Machine (5:12) (99.061996%)

[Manuel Lemos]

Hello,

on 10/04/2008 05:09 PM mark said the following:

Hello,

I want to create a php scraper that will get some information from
e.g. 5 sites simultaneously. I tried the following script:
http://www.phpied.com/simultaneuos-h...php-with-curl/
Everything works fine, but what I want is simultaneuos (something to
multithread, when these 5 websites will be loaded not one after
another, but by using different sockets) scraper.

In addition I would like to display the results as soon as it will be
scraped. So when first http-post get answer, it will show the result
and wait for the rest of the pages (not display everything when all
scraping is done).
Any ideas how can I achieve it? Thanks!

This class can do exactly what you describe:

http://www.phpclasses.org/thread

This other class also uses separate HTTP requests to run multiple
parallel tasks but these are started from the browser side using AJAX
requests:

http://www.phpclasses.org/phpthreader
--

Regards,
Manuel Lemos

Find and post PHP jobs
http://www.phpclasses.org/jobs/

PHP Classes - Free ready to use OOP components written in PHP
http://www.phpclasses.org/

[Jerry Stuckle]

Manuel Lemos wrote:

Hello,

on 10/04/2008 05:09 PM mark said the following:

>Hello,

I want to create a php scraper that will get some information from
e.g. 5 sites simultaneously. I tried the following script:
http://www.phpied.com/simultaneuos-h...php-with-curl/
Everything works fine, but what I want is simultaneuos (something to
multithread, when these 5 websites will be loaded not one after
another, but by using different sockets) scraper.

In addition I would like to display the results as soon as it will be
scraped. So when first http-post get answer, it will show the result
and wait for the rest of the pages (not display everything when all
scraping is done).
Any ideas how can I achieve it? Thanks!

This class can do exactly what you describe:

http://www.phpclasses.org/thread

This other class also uses separate HTTP requests to run multiple
parallel tasks but these are started from the browser side using AJAX
requests:

http://www.phpclasses.org/phpthreader

Why don't you tell him that's your own site you're spamming again, Manuel?

And those are your own classes (which, BTW, aren't worth a damn) you're
spamming?

--
=============== ===
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
js*******@attgl obal.net
=============== ===

[C. (http://symcbean.blogspot.com/)]

On 4 Oct, 21:09, mark <mkazmier...@gm ail.comwrote:

Hello,

I want to create a php scraper that will get some information from
e.g. 5 sites simultaneously. I tried the following script:http://www.phpied.com/simultaneuos-h...php-with-curl/
Everything works fine, but what I want is simultaneuos (something to
multithread, when these 5 websites will be loaded not one after
another, but by using different sockets) scraper.

That's exactly what curl_multi_* does.

In addition I would like to display the results as soon as it will be
scraped. So when first http-post get answer, it will show the result
and wait for the rest of the pages (not display everything when all
scraping is done).
Any ideas how can I achieve it? Thanks!

This is not a trivial bit of coding. It's not impossible but since you
seem to be relying on cut-and-paste coding, do you think you're
overstretching your abilities?

C.

[R. Rajesh Jeba Anbiah]

On Oct 5, 7:44 am, Jerry Stuckle <jstuck...@attg lobal.netwrote:

Manuel Lemos wrote:

<snip>

>

This class can do exactly what you describe:

http://www.phpclasses.org/thread

This other class also uses separate HTTP requests to run multiple
parallel tasks but these are started from the browser side using AJAX
requests:

http://www.phpclasses.org/phpthreader

Why don't you tell him that's your own site you're spamming again, Manuel?

And those are your own classes (which, BTW, aren't worth a damn) you're
spamming?

What's your solution? Do you have better approach?

--
<?php echo 'Just another PHP saint'; ?>
Email: rrjanbiah-at-Y!com Blog: http://rajeshanbiah.blogspot.com/

[Jerry Stuckle]

R. Rajesh Jeba Anbiah wrote:

On Oct 5, 7:44 am, Jerry Stuckle <jstuck...@attg lobal.netwrote:

>Manuel Lemos wrote:

<snip>

>>This class can do exactly what you describe:
http://www.phpclasses.org/thread
This other class also uses separate HTTP requests to run multiple
parallel tasks but these are started from the browser side using AJAX
requests:
http://www.phpclasses.org/phpthreader

Why don't you tell him that's your own site you're spamming again, Manuel?

And those are your own classes (which, BTW, aren't worth a damn) you're
spamming?

What's your solution? Do you have better approach?

--
<?php echo 'Just another PHP saint'; ?>
Email: rrjanbiah-at-Y!com Blog: http://rajeshanbiah.blogspot.com/

Yes, curl_multi_exec (), as Iván indicated.

Manuel is just a spammer - virtually every answer he posts refers to
something on his site. And he doesn't even indicate it's his own site
when he spams it.

Now I wouldn't mind if he were giving good technical advice. But I've
looked at some of his scripts. I've seen relatively new PHP programmers
do better.

--
=============== ===
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
js*******@attgl obal.net
=============== ===

[Michael Fesser]

..oO(Jerry Stuckle)

>R. Rajesh Jeba Anbiah wrote:

>On Oct 5, 7:44 am, Jerry Stuckle <jstuck...@attg lobal.netwrote:

>>>
And those are your own classes (which, BTW, aren't worth a damn) you're
spamming?

What's your solution? Do you have better approach?

Yes, curl_multi_exec (), as Iván indicated.

Manuel is just a spammer

Wrong.

>virtually every answer he posts refers to
something on his site.

Nothing wrong with that. I would also point to my own classes to solve a
given problem if they would be freely available.

>And he doesn't even indicate it's his own site
when he spams it.

Not necessary.

It would be spam if it would be totally OT, but he posts ready-to-use
solutions to PHP problems. It doesn't matter if these solutions are his
own or not. Even if they would be commercial, it wouldn't be spam in the
given context.

>Now I wouldn't mind if he were giving good technical advice. But I've
looked at some of his scripts.

Some. But surely not all. They might not fit your coding standards, but
this doesn't give you the right to discredit them on every chance you
get. If you have a problem with them, come to the point and post exactly
what you don't like. And _prove_ it by posting code samples.

>I've seen relatively new PHP programmers
do better.

If you don't like his solutions, post better ones or simply ignore him.
It's always good to have a choice between various ways to solve a
problem. He's contributing to the community by posting alternatives.

You OTOH are just trolling by attacking him personally on each and every
post. This sucks.

Enough is enough! >:-(

Micha

[salmobytes]

Jerry Stuckle wrote:

Manuel Lemos wrote:

Jerry Stuckle has a personality problem.
He seems to live on comp.lang.php like rat addicted to the cocaine
lever in a laboratory cage. He seems to do nothing else. Does his
employer know how much time he spends insulting people, complaining,
posturing? He seems to be a competent hacker. But also a lonely,
friendless, nasty dispositioned jerk.

Manuel Lemos is a mature, cosiderate and helpful guy by comparison.