By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
459,932 Members | 1,529 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 459,932 IT Pros & Developers. It's quick & easy.

looking for wget-like module for getching software

P: n/a

just getting started with python, and i'm designing a program for
fetching software from the net, given the package name, version
number and/or date stamp, download method (tarball, CVS, etc.) and so
on. i've already got a shell script doing this, but python would
certainly clean up the code a lot.

first question -- does such a utility/module (or something like it)
already exist? that is, with the ability to say, "go get me gcc,
version 3.4.3, as a tarball"? or "go get me glibc, CVS date stamp
xxxx, from that CVS repo"?

as for the second question, the driving force behind a fair bit of
the current program is "wget", which has the ability to search
recursively and bring back a file of a given name (which "curl"
doesn't appear to have).

that is, i can just say, "go get file gcc-3.4.2.tar.bz2", and start
searching at "ftp://pub.gnu.org/pub/gcc". i may not know how far down
in the directory structure that file is, but wget will happily search
recursively until it finds it.

so, any pointers to python utils/modules that do any or all of the
above? thanks.

rday
Jul 18 '05 #1
Share this Question
Share on Google+
2 Replies


P: n/a

Robert P. J. Day schrieb:
that is, i can just say, "go get file gcc-3.4.2.tar.bz2", and start
searching at "ftp://pub.gnu.org/pub/gcc". i may not know how far down
in the directory structure that file is, but wget will happily search
recursively until it finds it.


That sounds pretty inefficient and should produce some load on the server
side. Imagine everyone doing that. Many FTP-servers provide files like
"ls-R.gz" somewhere, meaning, a list of files and directories. And then,
there's Google. Why do you have to use FTP? Ask Google for
"yourfile.tar.bz2 site:the.domain.name". It will very likely return a
suitable URL for you that you can use for download. Just tried it with
"site:ftp.gnu.org gcc-3.4.2.tar.bz2" - works. You can even use that link
and try to replace "http://" with "ftp://" for download. Should work for
many source repositories.

You can use urllib2 for both querying and download.

Stefan
Jul 18 '05 #2

P: n/a
On Thu, 16 Dec 2004, Stefan Behnel wrote:

Robert P. J. Day schrieb:
that is, i can just say, "go get file gcc-3.4.2.tar.bz2", and start
searching at "ftp://pub.gnu.org/pub/gcc". i may not know how far down
in the directory structure that file is, but wget will happily search
recursively until it finds it.
That sounds pretty inefficient and should produce some load on the
server side. Imagine everyone doing that. Many FTP-servers provide
files like "ls-R.gz" somewhere, meaning, a list of files and
directories.


i realize that and, as much as possible, i try to start the process
with what i *know* is the actual directory location. but
occasionally, if i can't guarantee an exact directory, i just want the
option to start a bit higher up the directory structure. i wasn't
going to make a habit of it, it's more an absolute last resort, if it
comes down to it.
And then, there's Google. Why do you have to use FTP? Ask Google for
"yourfile.tar.bz2 site:the.domain.name". It will very likely return
a suitable URL for you that you can use for download. Just tried it
with "site:ftp.gnu.org gcc-3.4.2.tar.bz2" - works. You can even use
that link and try to replace "http://" with "ftp://" for download.
Should work for many source repositories.

You can use urllib2 for both querying and download.


and that pretty much answers the question i had. much thanks. and
now that i have my answer, i might have to unsubscribe. you guys sure
are a chatty bunch. :)

rday

p.s. just kidding.
Jul 18 '05 #3

This discussion thread is closed

Replies have been disabled for this discussion.