ASP: Building a "page grabber" to saveout text AND images?

[Ken Fine]

My well-heeled clients would like a tool that I could build into a web
application that would suck down the entire contents a remote webpage and
store it on a local filesystem.

I know how to scrape textual content from places on the web using MSXML and
assorted scracping widget. That doesn't help me for photos, nor does it
answer how to encapsulate the photos in a form that I can see them.

This would probably only find use in intranet-type situations, so a IE-only
..mht-type encapsulation of the page might be OK.

I have OK skills working with the FSO, so if there's a widget or method by
which the retrieval can be done, I can probably figure things out.

Has anyone done anything like this? Can someone suggest widgets and/or a
design approach?

Thanks.

-KF

[Ray at]

This isn't an anser to your question, but why? If they want to store Web
sites locally, shouldn't they be looking at proxy server software instead of
an ASP application? I don't really know what it is that your clients want,
but perhaps you can talk them into buy ISA Server or something a bit cheaper
with good proxy services.

Ray at home

"Ken Fine" <ke*****@u.washington.edu> wrote in message
news:c4**********@nntp6.u.washington.edu...

My well-heeled clients would like a tool that I could build into a web
application that would suck down the entire contents a remote webpage and
store it on a local filesystem.

I know how to scrape textual content from places on the web using MSXML and assorted scracping widget. That doesn't help me for photos, nor does it
answer how to encapsulate the photos in a form that I can see them.

This would probably only find use in intranet-type situations, so a IE-only .mht-type encapsulation of the page might be OK.

I have OK skills working with the FSO, so if there's a widget or method by
which the retrieval can be done, I can probably figure things out.

Has anyone done anything like this? Can someone suggest widgets and/or a
design approach?

Thanks.

-KF

[Ken Fine]

I'm not sure if this is a complete answer to your self-described non-answer
:), but...

1)...the pages that the "client" (my employer :) wants don't live forever
on the web. Eventually they're restricted in a way, and we'd to capture and
catalogue them in a local cache before that happens.

2) We want to interactively pick out a very limited selection of a given
site, not the entire site's content. We're only interested in certain pages,
and it requires human judgement to ascertain which pages are best.

Does this help?

-KF



"Ray at <%=sLocation%> [MVP]" <myfirstname at lane34 dot com> wrote in
message news:%2****************@TK2MSFTNGP12.phx.gbl...

This isn't an anser to your question, but why? If they want to store Web
sites locally, shouldn't they be looking at proxy server software instead of an ASP application? I don't really know what it is that your clients want, but perhaps you can talk them into buy ISA Server or something a bit cheaper with good proxy services.

Ray at home

"Ken Fine" <ke*****@u.washington.edu> wrote in message
news:c4**********@nntp6.u.washington.edu...

My well-heeled clients would like a tool that I could build into a web
application that would suck down the entire contents a remote webpage and store it on a local filesystem.

I know how to scrape textual content from places on the web using MSXML

and

assorted scracping widget. That doesn't help me for photos, nor does it
answer how to encapsulate the photos in a form that I can see them.

This would probably only find use in intranet-type situations, so a

IE-only

.mht-type encapsulation of the page might be OK.

I have OK skills working with the FSO, so if there's a widget or method by which the retrieval can be done, I can probably figure things out.

Has anyone done anything like this? Can someone suggest widgets and/or a
design approach?

Thanks.

-KF

[Manohar Kamath [MVP]]

Look for "offline browsers" using Google or www.tucows.com. There are plenty
of apps that do that for you right now.

--
Manohar Kamath
Editor, .netWire
www.dotnetwire.com
"Ken Fine" <ke*****@u.washington.edu> wrote in message
news:c4**********@nntp6.u.washington.edu...

My well-heeled clients would like a tool that I could build into a web
application that would suck down the entire contents a remote webpage and
store it on a local filesystem.

I know how to scrape textual content from places on the web using MSXML and assorted scracping widget. That doesn't help me for photos, nor does it
answer how to encapsulate the photos in a form that I can see them.

This would probably only find use in intranet-type situations, so a IE-only .mht-type encapsulation of the page might be OK.

I have OK skills working with the FSO, so if there's a widget or method by
which the retrieval can be done, I can probably figure things out.

Has anyone done anything like this? Can someone suggest widgets and/or a
design approach?

Thanks.

-KF

[Ken Fine]

Thanks, Manohar. But that's not what I need. I need a method or widget that
I can access programmatically, and that I can build into the works of other
web applications that I'm creating. I don't want a fixed tool, no matter how
good the implementation.

Does anyone have other ideas?

-KF

"Manohar Kamath [MVP]" <mk*****@TAKETHISOUTkamath.com> wrote in message
news:uc**************@tk2msftngp13.phx.gbl...

Look for "offline browsers" using Google or www.tucows.com. There are plenty of apps that do that for you right now.

--
Manohar Kamath
Editor, .netWire
www.dotnetwire.com
"Ken Fine" <ke*****@u.washington.edu> wrote in message
news:c4**********@nntp6.u.washington.edu...

My well-heeled clients would like a tool that I could build into a web
application that would suck down the entire contents a remote webpage and store it on a local filesystem.

I know how to scrape textual content from places on the web using MSXML

and

assorted scracping widget. That doesn't help me for photos, nor does it
answer how to encapsulate the photos in a form that I can see them.

This would probably only find use in intranet-type situations, so a

IE-only

.mht-type encapsulation of the page might be OK.

I have OK skills working with the FSO, so if there's a widget or method by which the retrieval can be done, I can probably figure things out.

Has anyone done anything like this? Can someone suggest widgets and/or a
design approach?

Thanks.

-KF

[Manohar Kamath [MVP]]

One way to do this, is programmatically call these tools. Doing it on your
own might be "reinventing the wheel" and prolonged.

If that's what you want, here are some articles that might help:

http://www.yart.com.au/articles/Yider.asp
http://www.asp101.com/articles/chris/spider/default.asp
--
Manohar Kamath
Editor, .netWire
www.dotnetwire.com
"Ken Fine" <ke*****@u.washington.edu> wrote in message
news:c4**********@nntp6.u.washington.edu...

Thanks, Manohar. But that's not what I need. I need a method or widget that I can access programmatically, and that I can build into the works of other web applications that I'm creating. I don't want a fixed tool, no matter how good the implementation.

Does anyone have other ideas?

-KF

"Manohar Kamath [MVP]" <mk*****@TAKETHISOUTkamath.com> wrote in message
news:uc**************@tk2msftngp13.phx.gbl...

Look for "offline browsers" using Google or www.tucows.com. There are plenty

of apps that do that for you right now.

--
Manohar Kamath
Editor, .netWire
www.dotnetwire.com
"Ken Fine" <ke*****@u.washington.edu> wrote in message
news:c4**********@nntp6.u.washington.edu...

My well-heeled clients would like a tool that I could build into a web
application that would suck down the entire contents a remote webpage and store it on a local filesystem.

I know how to scrape textual content from places on the web using MSXML

and

assorted scracping widget. That doesn't help me for photos, nor does

it answer how to encapsulate the photos in a form that I can see them.

This would probably only find use in intranet-type situations, so a

IE-only

.mht-type encapsulation of the page might be OK.

I have OK skills working with the FSO, so if there's a widget or method by which the retrieval can be done, I can probably figure things out.

Has anyone done anything like this? Can someone suggest widgets and/or

a design approach?

Thanks.

-KF

[Ken Fine]

Thanks again. Programmatically calling existing tools is a great idea, and
I'll look carefully at what's are out there. One limit of the two tools you
linked to is that neither of them deal with retrieving page images, which is
at the core of what I need to do.

I was hoping someone might already know of such a tool.

I don't really need spidering capability. What I need is a method to suck
down the text and images of a single remote page, and ideally a way to
rework the links so that everything doesn't look broken. The IE "Save page
as single file"/.mht is a good model for what I'd ideally like to do.

-KF

"Manohar Kamath [MVP]" <mk*****@TAKETHISOUTkamath.com> wrote in message
news:ez**************@TK2MSFTNGP11.phx.gbl...

One way to do this, is programmatically call these tools. Doing it on your
own might be "reinventing the wheel" and prolonged.

If that's what you want, here are some articles that might help:

http://www.yart.com.au/articles/Yider.asp
http://www.asp101.com/articles/chris/spider/default.asp
--
Manohar Kamath
Editor, .netWire
www.dotnetwire.com
"Ken Fine" <ke*****@u.washington.edu> wrote in message
news:c4**********@nntp6.u.washington.edu...

Thanks, Manohar. But that's not what I need. I need a method or widget that

I can access programmatically, and that I can build into the works of

other

web applications that I'm creating. I don't want a fixed tool, no matter

how

good the implementation.

Does anyone have other ideas?

-KF

"Manohar Kamath [MVP]" <mk*****@TAKETHISOUTkamath.com> wrote in message
news:uc**************@tk2msftngp13.phx.gbl...

Look for "offline browsers" using Google or www.tucows.com. There are

plenty

of apps that do that for you right now.

--
Manohar Kamath
Editor, .netWire
www.dotnetwire.com
"Ken Fine" <ke*****@u.washington.edu> wrote in message
news:c4**********@nntp6.u.washington.edu...
> My well-heeled clients would like a tool that I could build into a web > application that would suck down the entire contents a remote

webpage

and

> store it on a local filesystem.
>
> I know how to scrape textual content from places on the web using

MSXML and
> assorted scracping widget. That doesn't help me for photos, nor does it > answer how to encapsulate the photos in a form that I can see them.
>
> This would probably only find use in intranet-type situations, so a
IE-only
> .mht-type encapsulation of the page might be OK.
>
> I have OK skills working with the FSO, so if there's a widget or method

by

> which the retrieval can be done, I can probably figure things out.
>
> Has anyone done anything like this? Can someone suggest widgets

and/or a > design approach?
>
> Thanks.
>
> -KF
>
>
>

[Ken Fine]

I have learned about a UNIX utility called wget that does precisely what
I've been describing.

Unless there's something comparable in the realm of ASP that doesn't require
launching an external process (as wget does), that's probably my solution.

Thanks for the pointers...

-KF

"Ken Fine" <ke*****@u.washington.edu> wrote in message
news:c4**********@nntp6.u.washington.edu...

Thanks again. Programmatically calling existing tools is a great idea, and
I'll look carefully at what's are out there. One limit of the two tools you linked to is that neither of them deal with retrieving page images, which is at the core of what I need to do.

I was hoping someone might already know of such a tool.

I don't really need spidering capability. What I need is a method to suck
down the text and images of a single remote page, and ideally a way to
rework the links so that everything doesn't look broken. The IE "Save page
as single file"/.mht is a good model for what I'd ideally like to do.

-KF

"Manohar Kamath [MVP]" <mk*****@TAKETHISOUTkamath.com> wrote in message
news:ez**************@TK2MSFTNGP11.phx.gbl...

One way to do this, is programmatically call these tools. Doing it on your

own might be "reinventing the wheel" and prolonged.

If that's what you want, here are some articles that might help:

http://www.yart.com.au/articles/Yider.asp
http://www.asp101.com/articles/chris/spider/default.asp
--
Manohar Kamath
Editor, .netWire
www.dotnetwire.com
"Ken Fine" <ke*****@u.washington.edu> wrote in message
news:c4**********@nntp6.u.washington.edu...

Thanks, Manohar. But that's not what I need. I need a method or widget

that

I can access programmatically, and that I can build into the works of

other

web applications that I'm creating. I don't want a fixed tool, no matter

how

good the implementation.

Does anyone have other ideas?

-KF

"Manohar Kamath [MVP]" <mk*****@TAKETHISOUTkamath.com> wrote in

message news:uc**************@tk2msftngp13.phx.gbl...
> Look for "offline browsers" using Google or www.tucows.com. There are plenty
> of apps that do that for you right now.
>
> --
> Manohar Kamath
> Editor, .netWire
> www.dotnetwire.com
>
>
> "Ken Fine" <ke*****@u.washington.edu> wrote in message
> news:c4**********@nntp6.u.washington.edu...
> > My well-heeled clients would like a tool that I could build into a

web > > application that would suck down the entire contents a remote webpage and
> > store it on a local filesystem.
> >
> > I know how to scrape textual content from places on the web using

MSXML

> and
> > assorted scracping widget. That doesn't help me for photos, nor does it

> > answer how to encapsulate the photos in a form that I can see

them. > >
> > This would probably only find use in intranet-type situations, so a > IE-only
> > .mht-type encapsulation of the page might be OK.
> >
> > I have OK skills working with the FSO, so if there's a widget or

method

by
> > which the retrieval can be done, I can probably figure things out.
> >
> > Has anyone done anything like this? Can someone suggest widgets

and/or

a

> > design approach?
> >
> > Thanks.
> >
> > -KF
> >
> >
> >
>
>