Multiple threads/connections in cURL

Hi, I am using cURL with php to submit a form to a website and then retrieve the page that comes back. That part is easy. The problem is that the site I am posting the request to displays a 'please_wait' page which then redirects you to the page that you actually want to see. I can deal with this fine if I am only sending one request, but the trouble is that I am sending lots of requests (i.e. I am posting the form several times with different form data in each request). If I do this one after the other then I have to post the first request, then wait a couple of seconds and then request the page with the info on it. Then post the next request, wait a few seconds, get the page. Send the next.......ZZZzzzzz

So I need to be able to do them all at once...which cURL can do. But the trouble is that the 'please wait' page redirects you using javascript, so cURL can't automatically pick this up i.e. the CURLOPT_FOLLOWLOCATION doesn't work. The only way to do it is to post the request, and get the 'please wait' page, and then send a separate request for the page with the info on it.

And here's the snag: Posting the request and getting the info page all has to be done on the same connection, otherwise the website that I'm getting the page from doesn't know which page to send me!!

So I need to run several simultaneous cURL connections, then wait a couple of seconds and then USING THE SAME CONNECTIONS send 'GET' requests for the resulting pages.

How??

May 10 '07 #1

Subscribe Post Reply

4176

adamalton

I think it might be possible to write a script that posts the request and then requests the resulting page. And then get cURL to access that script on my server lots of times all at once. But I can't work out how to do this.

May 10 '07 #2

adamalton

The answer: There's no need to keep the same connection.

HTTP is 'stateless' meaning that a server has no way of tracking which user is which just by keeping the connection. The only way a server can determine which user is which (or which request is which) is by putting a cookie onto the user's computer to track who they are. (Or a user could be tracked using hidden form fields.) I had made a mistake in the way that I was dealing with the cookies which falsely led me to believe that there was something else affecting how the server was determining which request was which.

This explains it:
http://en.wikipedia.org/wiki/HTTP_se...vs_Server_side

For anyone looking for how to do multiple cURL transfers look at the example at the bottom of this page of the php manual:
http://uk2.php.net/manual/en/functio...multi-exec.php

May 11 '07 #3

Atli

5,058

Expert 4TB

I'm glad you solved your problem and thank you for sharing the solution with us.

May 11 '07 #4

by: Przemys³aw Ró¿ycki | last post by:

Hello, I have written some code, which creates many threads for each connection ('main connection'). The purpose of this code is to balance the load between several connections ('pipes'). The...

Python

Multiple listeners on a TCP/IP port?

by: Paul A. Steckler | last post by:

I need to write a TCP/IP server in C# that can handle multiple connections. My first try was to use TCPListener instances in multiple .NET threads. Of course, I got an exception from...

.NET Framework

Most efficient way to process thousands of files using multiple threads (dealing with thread handling)

by: m | last post by:

Hello, I have an application that processes thousands of files each day. The filenames and various related file information is retrieved, related filenames are associate and placed in a linked...

C# / C Sharp

How to pass messages between multiple threads?

by: Adam | last post by:

My telnet server app. creates a new thread for each incoming Tcp client as shown in the code below. How can the client threads send text messages back and forth to each other? I don't understand...

C# / C Sharp

Asynchronous I/O or multiple threads?

by: Quiet Man | last post by:

Hi all, I'm designing a fairly simple service that will run on W2K/SP4 and W2K3 servers. It's job is to be a very specialized database server that listens on a given IP address / TCP port and...

C# / C Sharp

Multiple worker processes for a single application in a applicationpool

by: Abhishek Srivastava | last post by:

Hello All, In IIS 6.0 We have a concept of worker processes and application pools. As I understand it, we can have multiple worker process per appliction pool. Each worker process is dedicated...

ASP.NET

Unspecified error on Database Connection in Multiple Threads

by: Jerry Spence1 | last post by:

I am starting up a number of threads. Each one specifies a connection to the same database. Sub ReaderThread() Dim cn As New System.Data.OleDb.OleDbConnection cn.ConnectionString =...

Visual Basic .NET

downloading a single file using multiple threads

by: keerthyragavendran | last post by:

hi i'm downloading a single file using multiple threads... how can i specify a particular range of bytes alone from a single large file... for example say if i need only bytes ranging from...

C# / C Sharp

Batch import of multiple excel files into the database

by: ryjfgjl | last post by:

If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...

Data Management

Merging data from multiple Excel files

by: ryjfgjl | last post by:

In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...

Data Management

Migrating Website to Cloud - Emmanuel Katto

by: emmanuelkatto | last post by:

Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel

General

Is that possible of reading the .csv file in column wise and the column have different lengths ?

by: Sonnysonu | last post by:

This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...

C / C++

How to build RAID in BIOS?

by: Hystou | last post by:

There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...

Computer Hardware

Changing the language in Windows 10

by: Hystou | last post by:

Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...

Windows Server

Maximizing Business Potential: The Nexus of Website Design and Digital Marketing

by: jinu1996 | last post by:

In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...

Online Marketing

The easy way to turn off automatic updates for Windows 10/11

by: Hystou | last post by:

Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...

Windows Server

Discussion: How does Zigbee compare with other wireless protocols in smart home applications?

by: tracyyun | last post by:

Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...

General

Multiple threads/connections in cURL

Similar topics