Downloading full page source of a web page

[helpmefinda]

Hello everyone,

I am currently writing a web spider, and I have it working for the
most part using winINet functions. However, I have found that winINet
functions do not get close to retrieving the full page source, and all
I get is the basic html.

Main Question:::::
What I really want is the entire page source as you would see it if
you do a view->page source in firefox. Can this be done using wininet?
or would I need to use another connection method.

I looked through all the flags that you could set while setting all
the functions necessary to retrieve a file, but I did not find any
flags that would do what I wanted...
found here : http://msdn2.microsoft.com/en-us/library/aa385473.aspx

this is my current code for connecting (minus error checking):
hINet = InternetOpen("I netHTTP/1.0", INTERNET_OPEN_T YPE_PRECONFIG,
NULL, NULL, 0);

hConnection = InternetConnect ( hINet,tempsite. c_str(),
INTERNET_DEFAUL T_HTTP_PORT, NULL, NULL, INTERNET_SERVIC E_HTTP, 0, 0);

hData = HttpOpenRequest (hConnection, "GET",
csite.site.subs tr(endposition, csite.site.leng th()).c_str(), NULL,
NULL, NULL, INTERNET_FLAG_K EEP_CONNECTION, 0);

httpSendRequest Succeeded = HttpSendRequest (hData, NULL, 0, 0, 0);

internetReadFil eSucceeded = InternetReadFil e(hData, (LPVOID)buffer,
(ULONG)(BUFFSIZ E-1), &dwBytesRead );
Thanks a lot,
Rob

[Victor Bazarov]

he*********@gma il.com wrote:

I am currently writing a web spider, and I have it working for the
most part using winINet functions. However, I have found that winINet
functions do not get close to retrieving the full page source, and all
I get is the basic html.

Main Question:::::
What I really want is the entire page source as you would see it if
you do a view->page source in firefox. Can this be done using wininet?
[..]

Wrong newsgroup. Whatever "wininet" is, it's not part of C++ language
or the Standard library. You should consider asking in the newsgroup
for your platform or your compiler (if compiler contains that it its
package).

V
--
Please remove capital 'A's when replying by e-mail
I do not respond to top-posted replies, please don't ask

[=?ISO-8859-1?Q?Erik_Wikstr=F6m?=]

On 2007-08-29 05:08, he*********@gma il.com wrote:

Hello everyone,

I am currently writing a web spider, and I have it working for the
most part using winINet functions. However, I have found that winINet
functions do not get close to retrieving the full page source, and all
I get is the basic html.

Main Question:::::
What I really want is the entire page source as you would see it if
you do a view->page source in firefox. Can this be done using wininet?
or would I need to use another connection method.

It seems to me that your understanding of the HTTP protocol and of how
the internet works is a bit lacking. Try to read up on those things and
you'll find that your question will be answered.

--
Erik Wikström