By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
443,983 Members | 1,627 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 443,983 IT Pros & Developers. It's quick & easy.

Having Trouble With CURL Function, Please Help

P: n/a
I am using the below CURL Function and can not figure out why it is not
retruning the results from the post. Can anyone take a look and tell
me what I may be doing wrong? I am just not seeing it.

As you can see at the bottom of the script if (strstr($Line, 'HOUSTON
YAMAHA MOTORSPORTS')) then it is reading the correct page, otherwise if
(strstr($Line, 'Yamaha Dealer Locator')) then it is reading the
incorrect page of the initial URL called
http://www.yamaha-motor.com/sport/de....aspx?ls=sport.
So when I run it what appears to be happening is the it is finding the
http://www.yamaha-motor.com/sport/de....aspx?ls=sport
but not he returned page from the post.

It is a fairly simple script. Your help is much appreciated on how I
can resolve this issue.

Thank you.

<?php

$data="MasterTemplate:_PageTemplate:cphCenterConte nt:CenterContent:fcDropDownList=ATVs&MasterTemplat e:_PageTemplate:cphCenterContent:CenterContent:zip TextBox=77090";

curlgrabber("http://www.yamaha-motor.com/sport/dealers/dealerhome/home.aspx?ls=sport",
$data); // Calls The CURL Function

unset($data);
function curlgrabber ($url, $data)
{
// create a new curl resource
$ch = curl_init();

// set URL and other appropriate options
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 600);
curl_setopt($ch, CURLOPT_TIMEOUT, 600);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);

// grab URL, and return output

$Contents = curl_exec($ch);

// close curl resource, and free up system resources
if (curl_errno($ch)) {
print curl_error($ch);
} else {
curl_close($ch);
unset($ch);
}
$Lines = preg_split("/\n/",$Contents);

$counter=0;
$strTag="False";

foreach($Lines as $Line)
{

if (strstr($Line, 'HOUSTON YAMAHA MOTORSPORTS')) // EXAMPLE ONLY!
{
echo "Post Worked And Is Reading Correct Page";
}
if (strstr($Line, 'Yamaha Dealer Locator')) // EXAMPLE ONLY!
{
echo "Post did not work. This is reading the initial screen at
http://www.yamaha-motor.com/sport/dealers/dealerhome/home.aspx?ls=sport";
}
}// End For Each
}// End Function

?>

Dec 13 '06 #1
Share this Question
Share on Google+
9 Replies


P: n/a
On 13 Dec 2006 10:44:06 -0800, "devranger" <kn**********@hotmail.comwrote:
>$data="MasterTemplate:_PageTemplate:cphCenterCont ent:CenterContent:fcDropDownList=ATVs&MasterTempla te:_PageTemplate:cphCenterContent:CenterContent:zi pTextBox=77090";
[snip]
>curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
What's with the colons in the data? That doesn't look like the sort of data
that you'd normally send in a POST.

And if you capture the POST from visiting that page with a browser, there's a
colossal amount of junk posted along with the form, nearly 2k of the stuff.
Which probably has some meaning to the page. Try it out with LiveHTTPHeaders or
similar.

When pasting all this back in exactly as the $data field it works, but it's so
cryptic I have no idea if it'd continue to work. (And depending on what you're
doing with the data you're getting out of the page the owners of the page may
not be happy about it either - have you just asked them for the information
directly?)

--
Andy Hassall :: an**@andyh.co.uk :: http://www.andyh.co.uk
http://www.andyhsoftware.co.uk/space :: disk and FTP usage analysis tool
Dec 14 '06 #2

P: n/a
Rik
Andy Hassall wrote:
On 13 Dec 2006 10:44:06 -0800, "devranger" <kn**********@hotmail.com>
wrote:
>>
$data="MasterTemplate:_PageTemplate:cphCenterConte nt:CenterContent:fcDropDo
wnList=ATVs&MasterTemplate:_PageTemplate:cphCenter Content:CenterContent:zip
TextBox=77090";
>[snip] curl_setopt($ch, CURLOPT_POSTFIELDS, $data);

What's with the colons in the data? That doesn't look like the sort
of data
that you'd normally send in a POST.
Yup, AFAIK it stil should e ampersands.
Let cURL handle it though:
$topost = array(
'key1' ='value',
'key2' ='value2'
//etc...
);
curl_setopt($ch, CURLOPT_POSTFIELDS, $topost );

--
Rik Wasmus
Dec 14 '06 #3

P: n/a
Rik wrote:
Andy Hassall wrote:
On 13 Dec 2006 10:44:06 -0800, "devranger" <kn**********@hotmail.com>
wrote:
>
$data="MasterTemplate:_PageTemplate:cphCenterConte nt:CenterContent:fcDropDo
wnList=ATVs&MasterTemplate:_PageTemplate:cphCenter Content:CenterContent:zip
TextBox=77090";
[snip] curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
What's with the colons in the data? That doesn't look like the sort
of data
that you'd normally send in a POST.

Yup, AFAIK it stil should e ampersands.
Let cURL handle it though:
$topost = array(
'key1' ='value',
'key2' ='value2'
//etc...
);
curl_setopt($ch, CURLOPT_POSTFIELDS, $topost );

--
Rik Wasmus
Rik -

So you are suggesting I use an array but what do you mean by e
ampersads? Does that mean the ":" should be and &?

The reason I placed the colons is that they are just part of the field
input name I thought was necessary for the post. For instance, one
field on the form is called
MasterTemplate:_PageTemplate:cphCenterContent:Cent erContent:zipTextBox.
Looks real weird to me as well but I do not know any other way to post
to the field than to call it as specified in the input name of the form
i.e. <input
name="MasterTemplate:_PageTemplate:cphCenterConten t:CenterContent:zipTextBox"
type="text" maxlength="5"
id="MasterTemplate__PageTemplate_cphCenterContent_ CenterContent_zipTextBox"
tabindex="1" class="dealertext" style="width:100px;" size="20" />

Is this incorrect?

Thank you.

Dec 14 '06 #4

P: n/a
devranger wrote:
Rik wrote:
>>Andy Hassall wrote:
>>>On 13 Dec 2006 10:44:06 -0800, "devranger" <kn**********@hotmail.com>
wrote:

$data="MasterTemplate:_PageTemplate:cphCenterCon tent:CenterContent:fcDropDo
wnList=ATVs&MasterTemplate:_PageTemplate:cphCent erContent:CenterContent:zip
TextBox=77090";
>>>>[snip] curl_setopt($ch, CURLOPT_POSTFIELDS, $data);

What's with the colons in the data? That doesn't look like the sort
of data
that you'd normally send in a POST.

Yup, AFAIK it stil should e ampersands.
Let cURL handle it though:
$topost = array(
'key1' ='value',
'key2' ='value2'
//etc...
);
curl_setopt($ch, CURLOPT_POSTFIELDS, $topost );

--
Rik Wasmus


Rik -

So you are suggesting I use an array but what do you mean by e
ampersads? Does that mean the ":" should be and &?

The reason I placed the colons is that they are just part of the field
input name I thought was necessary for the post. For instance, one
field on the form is called
MasterTemplate:_PageTemplate:cphCenterContent:Cent erContent:zipTextBox.
Looks real weird to me as well but I do not know any other way to post
to the field than to call it as specified in the input name of the form
i.e. <input
name="MasterTemplate:_PageTemplate:cphCenterConten t:CenterContent:zipTextBox"
type="text" maxlength="5"
id="MasterTemplate__PageTemplate_cphCenterContent_ CenterContent_zipTextBox"
tabindex="1" class="dealertext" style="width:100px;" size="20" />

Is this incorrect?

Thank you.
It looks like your data is correct, but there could be all kinds of
other reasons. For instance, they might be checking the HTTP_REFERER to
ensure it comes from their own page. Or they could have started a
session on the first page and check for it on the second. Or any of
several other things.

And I agree with Andy. It looks like maybe you're trying to scrape
something from their page. They may not like it, but perhaps if you ask
nicely they'll give it to you.

--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
js*******@attglobal.net
==================
Dec 14 '06 #5

P: n/a

Jerry Stuckle wrote:
devranger wrote:
Rik wrote:
>Andy Hassall wrote:

On 13 Dec 2006 10:44:06 -0800, "devranger" <kn**********@hotmail.com>
wrote:
$data="MasterTemplate:_PageTemplate:cphCenterCont ent:CenterContent:fcDropDo
wnList=ATVs&MasterTemplate:_PageTemplate:cphCente rContent:CenterContent:zip
TextBox=77090";

[snip] curl_setopt($ch, CURLOPT_POSTFIELDS, $data);

What's with the colons in the data? That doesn't look like the sort
of data
that you'd normally send in a POST.

Yup, AFAIK it stil should e ampersands.
Let cURL handle it though:
$topost = array(
'key1' ='value',
'key2' ='value2'
//etc...
);
curl_setopt($ch, CURLOPT_POSTFIELDS, $topost );

--
Rik Wasmus

Rik -

So you are suggesting I use an array but what do you mean by e
ampersads? Does that mean the ":" should be and &?

The reason I placed the colons is that they are just part of the field
input name I thought was necessary for the post. For instance, one
field on the form is called
MasterTemplate:_PageTemplate:cphCenterContent:Cent erContent:zipTextBox.
Looks real weird to me as well but I do not know any other way to post
to the field than to call it as specified in the input name of the form
i.e. <input
name="MasterTemplate:_PageTemplate:cphCenterConten t:CenterContent:zipTextBox"
type="text" maxlength="5"
id="MasterTemplate__PageTemplate_cphCenterContent_ CenterContent_zipTextBox"
tabindex="1" class="dealertext" style="width:100px;" size="20" />

Is this incorrect?

Thank you.

It looks like your data is correct, but there could be all kinds of
other reasons. For instance, they might be checking the HTTP_REFERER to
ensure it comes from their own page. Or they could have started a
session on the first page and check for it on the second. Or any of
several other things.

And I agree with Andy. It looks like maybe you're trying to scrape
something from their page. They may not like it, but perhaps if you ask
nicely they'll give it to you.

--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
js*******@attglobal.net
==================
Jerry / Andy

I am just doing this CURL as a test example. See I am new to
programming and I am learning php programming through a book called php
web development. I am catching on and in the book I was reading about
all the cool things you can do with CURL, so I started to do some
testing becasue this is how I learn. I have been sucessful with
posting to a few pages on different sites, but this one gave me trouble
and is kind of driving me crazy on why. I really have no use for the
data just getting it to work or figuring out why it may not work.

Now I see and have learned a couple things. You mentioned HTTP_REFERER
or they could have started a session. See I have learned through you
response becasue I had no idea these were limitation to CURL. That is
the book never mention it. Thank you.

Just for my reference is there any other limitation of CURL I should be
aware of?

Thank you.

Dec 15 '06 #6

P: n/a
Rik
devranger wrote:
I am just doing this CURL as a test example. See I am new to
programming and I am learning php programming through a book called
php
web development. I am catching on and in the book I was reading about
all the cool things you can do with CURL, so I started to do some
testing becasue this is how I learn. I have been sucessful with
posting to a few pages on different sites, but this one gave me
trouble
and is kind of driving me crazy on why. I really have no use for the
data just getting it to work or figuring out why it may not work.

Now I see and have learned a couple things. You mentioned
HTTP_REFERER
or they could have started a session. See I have learned through you
response becasue I had no idea these were limitation to CURL. That is
the book never mention it. Thank you.

Just for my reference is there any other limitation of CURL I should
be
aware of?
It's not a limitation of cURL, it's a check they might have on their own
website. You're better of reading up on the normal HTTP protocol, and how
some major sites check their visitors. cURL can set a HTTP_REFERER for you
without a problem, or open the 'previous page' so get a session started/a
cookie. It's just a pain to figure out what they're checking.

Try to go to the page 'manually', and check all headers sent to & from the
site. If you use Mozilla, check uit LiveHTTPHeaders, for MSIE there is
Fiddler. THey will show you what is sent, try to emulate that with cURL.
--
Rik Wasmus
Dec 15 '06 #7

P: n/a
I have been reading up on what you suggested Rik and testing different
options but I am still not successful in getting the results returned.
I have set the curl_setopt($ch, CURLOPT_REFERER, $url); and also set up
to curl sessions, one to capture the cookie and the other to post with
the cookie (see below): I really feel it may have something to do
with the long input names and : in the input name of my $data variable,
but Jerry said the data vaiable was correct.

Does anyone have any other suggestion on how I could get this working?
Your help is much appreciated, I feel I have learned so much so far.

Thank you!!!!

$data="MasterTemplate:_PageTemplate:cphCenterConte nt:CenterContent:fcDropDownList=ATVs&MasterTemplat e:_PageTemplate:cphCenterContent:CenterContent:zip TextBox=77090";

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURL_ERROR, 1);
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookie.txt");
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
$result = curl_exec ($ch);
curl_close ($ch);

// create a new curl resource

$ch = curl_init();

// set URL and other appropriate options
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURL_ERROR, 1);
curl_setopt($ch, CURLOPT_REFERER, $url);
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookie.txt");
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 600);
curl_setopt($ch, CURLOPT_TIMEOUT, 600);

// grab URL, and return output

$Contents = curl_exec($ch);

Dec 25 '06 #8

P: n/a
devranger wrote:
I have been reading up on what you suggested Rik and testing different
options but I am still not successful in getting the results returned.
I have set the curl_setopt($ch, CURLOPT_REFERER, $url); and also set up
to curl sessions, one to capture the cookie and the other to post with
the cookie (see below): I really feel it may have something to do
with the long input names and : in the input name of my $data variable,
but Jerry said the data vaiable was correct.

Does anyone have any other suggestion on how I could get this working?
Your help is much appreciated, I feel I have learned so much so far.

Thank you!!!!

$data="MasterTemplate:_PageTemplate:cphCenterConte nt:CenterContent:fcDropDownList=ATVs&MasterTemplat e:_PageTemplate:cphCenterContent:CenterContent:zip TextBox=77090";

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURL_ERROR, 1);
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookie.txt");
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
$result = curl_exec ($ch);
curl_close ($ch);

// create a new curl resource

$ch = curl_init();

// set URL and other appropriate options
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURL_ERROR, 1);
curl_setopt($ch, CURLOPT_REFERER, $url);
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookie.txt");
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 600);
curl_setopt($ch, CURLOPT_TIMEOUT, 600);

// grab URL, and return output

$Contents = curl_exec($ch);
You're going to have to ask the owner of the data you wish to mine.
There are all kinds of ways of protecting data from mining, and only he
knows which one(s) he's using.

--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
js*******@attglobal.net
==================
Dec 26 '06 #9

P: n/a
Rik
devranger wrote:
I have been reading up on what you suggested Rik and testing different
options but I am still not successful in getting the results returned.
I have set the curl_setopt($ch, CURLOPT_REFERER, $url); and also set
up
to curl sessions, one to capture the cookie and the other to post with
the cookie (see below): I really feel it may have something to do
with the long input names and : in the input name of my $data
variable,
but Jerry said the data vaiable was correct.

Does anyone have any other suggestion on how I could get this working?
Your help is much appreciated, I feel I have learned so much so far.

Thank you!!!!

$data="MasterTemplate:_PageTemplate:cphCenterConte nt:CenterContent:fcDropDo
wnList=ATVs&MasterTemplate:_PageTemplate:cphCenter Content:CenterContent:zip
TextBox=77090";

Probably:
$data=array(
'MasterTemplate:_P.......ist' ='ATV',
'MasterTemplate:_P...ox' ='77090');

Grtz,
--
Rik Wasmus
Dec 28 '06 #10

This discussion thread is closed

Replies have been disabled for this discussion.