By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
440,874 Members | 1,027 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 440,874 IT Pros & Developers. It's quick & easy.

General Web Scraping Question

P: n/a
I've been working on a web scraping program, and have the basics down.

But I don't understand the parameters.
Normally, you go to a URL (say a reverse yellow pages directory), and enter
some parameters (like area code, phone number, etc.) and POST this back to
the web. Then you parse the response, looking for the data you need.

Ofen I see examples where the data you post contains something like
"AreaCode=503&Number=5551212&x=1&y=2"

Where do the "x=1 and y=2" come from? I have some sites where my post
doesn't work. In one case, you are supposed to enter a contractor's license
number, and then click a button, and the result contains information about
the license. After I post what I think should work, the result coming back
is the same web page, with the contractor's number filled in.

Do the X and Y parameters involve invoking a button? How do you determine
what to use for the parameters?

Thanks in advance for any advice or pointers!
---Selden McCabe
Nov 18 '05 #1
Share this Question
Share on Google+
2 Replies


P: n/a
I suspect X and Y are passed by the browser when the user clicks on an image
map. Have you tried passing &x=1&y=1 in your post?

--
Thanks,

Eric Lawrence
Program Manager
Assistance and Worldwide Services

This posting is provided "AS IS" with no warranties, and confers no rights.
"Selden McCabe" <se*****@msn.com> wrote in message
news:#l**************@TK2MSFTNGP09.phx.gbl...
I've been working on a web scraping program, and have the basics down.

But I don't understand the parameters.
Normally, you go to a URL (say a reverse yellow pages directory), and enter some parameters (like area code, phone number, etc.) and POST this back to
the web. Then you parse the response, looking for the data you need.

Ofen I see examples where the data you post contains something like
"AreaCode=503&Number=5551212&x=1&y=2"

Where do the "x=1 and y=2" come from? I have some sites where my post
doesn't work. In one case, you are supposed to enter a contractor's license number, and then click a button, and the result contains information about
the license. After I post what I think should work, the result coming back is the same web page, with the contractor's number filled in.

Do the X and Y parameters involve invoking a button? How do you determine
what to use for the parameters?

Thanks in advance for any advice or pointers!
---Selden McCabe

Nov 18 '05 #2

P: n/a
Selden McCabe wrote:
I've been working on a web scraping program, and have the basics down.

But I don't understand the parameters.
Normally, you go to a URL (say a reverse yellow pages directory), and
enter some parameters (like area code, phone number, etc.) and POST
this back to the web. Then you parse the response, looking for the
data you need.

Ofen I see examples where the data you post contains something like
"AreaCode=503&Number=5551212&x=1&y=2"

Where do the "x=1 and y=2" come from? I have some sites where my post
doesn't work. In one case, you are supposed to enter a contractor's
license number, and then click a button, and the result contains
information about the license. After I post what I think should
work, the result coming back is the same web page, with the
contractor's number filled in.

Do the X and Y parameters involve invoking a button? How do you
determine what to use for the parameters?


These could be hidden fields used by web application to store session state
on the client. Actually, it's not easy to implement web scraping for
"foreign" web applications where you don't have access to the code or at
least some inside knowledge.

Cheers,

--
Joerg Jooss
jo*********@gmx.net

Nov 18 '05 #3

This discussion thread is closed

Replies have been disabled for this discussion.