Using external web sites search form to query database

[ManWithNoName]

Hey guys! Did you miss me? I sure missed you ;). Finished my course today! The next one will begin the coming week so I will have some time to try some stuff out. Because this is my first post in a very long time I would say that it should technically be viewed as a first post. So without further ado…

This is my first post, please be gentle.

I want to use PHP to

1) do a search

2) choose some of the result (open them)

3) save the content.

The third step isn’t a problem.

Regarding the first step: I know that I could use the URI string to query a database with the search terms, however it would be much easier if I could just use the actual form and type my key strings. Why so? Because some pages are stupid and I can’t understand how they are querying the database (I think it's because they use POST).

Any ideas how I could approach this problem?

Regarding the second step: for the moment I can’t seem to find any problems; the search result renders links and all I need to do is to identify the suckers – sounds like a job for REGEX!

To recap: how can I search a web sites database using the web sites own search function [i.e. the search form]?

[Atli]

To recap: how can I search a web sites database using the web sites own search function [i.e. the search form]?

Hi. Welcome back! :)

I'm not sure I fully understand what you are asking.
What is "the web sites own search function"?
Is this a part of some web you created?

It is easy enough to create a <form> element and use it's <input> to search a database, but I am not sure that's what you are asking for.

[ManWithNoName]

Hi. Welcome back! :)

I'm not sure I fully understand what you are asking.
What is "the web sites own search function"?
Is this a part of some web you created?

It is easy enough to create a <form> element and use it's <input> to search a database, but I am not sure that's what you are asking for.

Hey! Thanks for your reply!

Ok, let's see if I can do this right:

Basically what I want to do is to download content from another web site. The content, however, is not the "actual web site" but the database entries it holds (e.g. a Wikipedia article, or maybe a Google search result).

Let's say that the web site is Google, and I want to get the top ten search results for "the Scripts" ;), then I could use something like
http://www.google.com/search?source=...=Google+Search

That would create a html page with the results. I could then filter out what I want (e.g. the links to each result).*

*I haven't actually tried this, but I see no reason why it shouldn't work.

Now my problem: it would seem is that not all web sites use GET (to request the database), so I can't use the URI string to define my search terms; all I'm stuck with is the actual search form (the one you see when you are browsing, i.e. the rendered HTML-form).

In other words: I want to search a external website database, and the only access to that database is -- as far as I can see it -- through that websites [rendered] search form.

The only thing that comes through my mind is that I
1) open the external website in PHP and
2) jump the cursor to the form / or activate the form
3) have a list of search terms PHP uses to write in the input box and then
4) "presses the search button" so that the search is executed.

This however sounds more like a job for macros... No?

I'm out of [good] fresh ideas...

[pbmods]

Heya, ManWithNoName.

You can send POST data using cURL or (sometimes) HTTP stream contexts.

I posted a short example of how to use cURL around here somewhere....

Ah. Here it is.

[ManWithNoName]

I knew that you would eventually come to my rescue ;) It seems to be what I was looking for! Thanks! You’re the man pbmods! :D

Now I have some more information/functions/terms to work with! =D And now I have some more stuff to think about when sitting through my course until the winter holidays ;)