468,257 Members | 1,431 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 468,257 developers. It's quick & easy.

Automate access to password protected sites

Hello,

we're fully paid up and signed up to a password protected site, so this
isn't hacking. In this site we can download a price list in CSV
format.

What I'd like to do is have a PHP script log into the site and download
the CSV file so we can then display it on our website.

Is this posible?

Darren

Jan 24 '06 #1
5 2616
>we're fully paid up and signed up to a password protected site, so this
isn't hacking. In this site we can download a price list in CSV
format.

What I'd like to do is have a PHP script log into the site and download
the CSV file so we can then display it on our website.
Are you sure this is a permitted use of the price list?
Being allowed to access the list and publishing it are quite
different from the point of view of copyright law.
Is this posible?


With cURL you can submit the login page with your info filled in,
get back a session cookie (or whatever types the site uses), then
use that cookie to go to other places on the site. It may require
some research as to what variables need to be sent, (including
HTTP_REFERER, maybe) and this can break if the site changes them.

Gordon L. Burditt
Jan 24 '06 #2
d
"Darren" <gr****@darrenludlam.plus.com> wrote in message
news:11**********************@g47g2000cwa.googlegr oups.com...
Hello,

we're fully paid up and signed up to a password protected site, so this
isn't hacking. In this site we can download a price list in CSV
format.

What I'd like to do is have a PHP script log into the site and download
the CSV file so we can then display it on our website.

Is this posible?
It's very possible. As other posts have said - make sure it's not against
the site's rules to do this. Just because you can manually view the list
doesn't mean to say you can do it automatically.

The best way to do it is to use cURL. I say that because unless you feel
particularly comfortable with writing and interpreting HTTP requests
(they're not hard, but you do need to do some reading), cURL will do most of
the work for you. I would do it like this:

Step 1. Read the page where you would log in, using cURL. Get the HTML and
find the form. Get the variable names from the form (for username and
password) - this is done in case they change the variable names, your script
will still work.

Step 2. Create your post variables containing the variable names from step
1 and your own username and password. Submit these variables to the action
of the form you read in step 1.

Step 3. Now, cURL should be remembering your session ID/cookie (you might
have to set the cookie jar/file settings for cURL - this is covered
adequately on the PHP site, http://www.php.net/curl).

Step 4. Using cURL, access the CSV's url - cURL should tell the server what
it needs to hear to accept that you're logged on, and the file should be
available.

I hope that helps.

dave
Darren

Jan 25 '06 #3
*** Darren escribió/wrote (24 Jan 2006 09:11:48 -0800):
we're fully paid up and signed up to a password protected site, so this
isn't hacking. In this site we can download a price list in CSV
format.

What I'd like to do is have a PHP script log into the site and download
the CSV file so we can then display it on our website.


In this case I'd normally do a system() call and use wget (if available).
It takes care itself of all the dirty details.

wget --help

[...]

HTTP options:
--http-user=USER set http user to USER.
--http-passwd=PASS set http password to PASS.

If the site is not using HTTP authentication, though, you'd need to send
the login form data and probably create a cookie. The right tool is curl.

--
-+ Álvaro G. Vicario - Burgos, Spain
++ http://bits.demogracia.com es mi sitio para programadores web
+- http://www.demogracia.com es mi web de humor libre de cloro
--
Jan 25 '06 #4
Darren wrote:
Hello,

we're fully paid up and signed up to a password protected site, so this
isn't hacking. In this site we can download a price list in CSV
format.

What I'd like to do is have a PHP script log into the site and download
the CSV file so we can then display it on our website.

Is this posible?

Darren

How is the site protected? WWW-Auth or regular form? Cookies?

In any case, you can use fsockopen
[http://us3.php.net/manual/en/function.fsockopen.php] to simulate a
connection to the server. You just need to pass the right headers.

I prefer this way because it stays within PHP, without branching to *NIX
command line. This should work just fine on Windows or *NIX.

Jan 26 '06 #5
d
"Carl Vondrick" <us****@carlsoft.net> wrote in message
news:Uj******************@newssvr25.news.prodigy.n et...
Darren wrote:
Hello,

we're fully paid up and signed up to a password protected site, so this
isn't hacking. In this site we can download a price list in CSV
format.

What I'd like to do is have a PHP script log into the site and download
the CSV file so we can then display it on our website.

Is this posible?

Darren

How is the site protected? WWW-Auth or regular form? Cookies?

In any case, you can use fsockopen
[http://us3.php.net/manual/en/function.fsockopen.php] to simulate a
connection to the server. You just need to pass the right headers.

I prefer this way because it stays within PHP, without branching to *NIX
command line. This should work just fine on Windows or *NIX.


It also doesn't need any extensions, like cURL. It does, though, take a bit
more knowledge about HTTP, but I totally agree with you - I use fsockopen
for all my HTTP needs ;)

dave
Jan 26 '06 #6

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

1 post views Thread by Tom Jones | last post: by
3 posts views Thread by Brad Burke | last post: by
4 posts views Thread by Andrew Chanter | last post: by
3 posts views Thread by Lyle Fairfield | last post: by
reply views Thread by kermitthefrogpy | last post: by
reply views Thread by zattat | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.