Client side network programming

[Roysun_rohit]

I am interested in making a search engine which takes web sites
iteratively, and downloads the web page or has to perform some search
across the web pages.
I am unsucessful to do so. My machine works through a proxy server and the
internet connection is through 24 hour lease line. when ever i try the
code it gives :

Traceback (innermost last):
File "http-getfile-urllib2.py", line 19, in ?
urllib.urlretrieve(remoteaddr, localname)
File "/usr/lib/python1.5/urllib.py", line 66, in urlretrieve
return _urlopener.retrieve(url, filename, reporthook)
File "/usr/lib/python1.5/urllib.py", line 186, in retrieve
fp = self.open(url)
File "/usr/lib/python1.5/urllib.py", line 159, in open
return getattr(self, name)(url)
File "/usr/lib/python1.5/urllib.py", line 260, in open_http
h = httplib.HTTP(host)
File "/usr/lib/python1.5/httplib.py", line 53, in __init__
if host: self.connect(host, port)
File "/usr/lib/python1.5/httplib.py", line 81, in connect
self.sock.connect(host, port)
IOError: [Errno socket error] (101, 'Network is unreachable')
================================================== ==========
The code is like this:-
#!/usr/bin/env python
import os, sys, urllib, urlparse, socket
showlines = 6
try:
servername, filename = sys.argv[1:3]
except:
servername, filename = 'www.igib.res.in', '/sarsanalysis.html'

remoteaddr = 'http://%s%s' % (servername, filename)
if len(sys.argv) == 4:
localname = sys.argv[3]
else:
(scheme, server, path, parms, query, frag) =
urlparse.urlparse(remoteaddr)
localname = os.path.split(path)[1]

print remoteaddr, localname
urllib.urlretrieve(remoteaddr, localname)
remotedata = open(localname).readlines()
for line in remotedata[:showlines]: print line,
================================================== ==========
I am new to the internet programming as well as python. please guide me,
how to solve this one.

[fishboy]

On Wed, 02 Jun 2004 06:23:20 -0400, "Roysun_rohit"
<ro**********@rediffmail.com> wrote:

I am interested in making a search engine which takes web sites
iteratively, and downloads the web page or has to perform some search
across the web pages.
I am unsucessful to do so. My machine works through a proxy server and the
internet connection is through 24 hour lease line. when ever i try the
code it gives :

Traceback (innermost last):
File "http-getfile-urllib2.py", line 19, in ?
urllib.urlretrieve(remoteaddr, localname)
File "/usr/lib/python1.5/urllib.py", line 66, in urlretrieve
return _urlopener.retrieve(url, filename, reporthook)
File "/usr/lib/python1.5/urllib.py", line 186, in retrieve
fp = self.open(url)
File "/usr/lib/python1.5/urllib.py", line 159, in open
return getattr(self, name)(url)
File "/usr/lib/python1.5/urllib.py", line 260, in open_http
h = httplib.HTTP(host)
File "/usr/lib/python1.5/httplib.py", line 53, in __init__
if host: self.connect(host, port)
File "/usr/lib/python1.5/httplib.py", line 81, in connect
self.sock.connect(host, port)
IOError: [Errno socket error] (101, 'Network is unreachable')
================================================= ===========
The code is like this:-
#!/usr/bin/env python
import os, sys, urllib, urlparse, socket
showlines = 6
try:
servername, filename = sys.argv[1:3]
except:
servername, filename = 'www.igib.res.in', '/sarsanalysis.html'

remoteaddr = 'http://%s%s' % (servername, filename)
if len(sys.argv) == 4:
localname = sys.argv[3]
else:
(scheme, server, path, parms, query, frag) =
urlparse.urlparse(remoteaddr)
localname = os.path.split(path)[1]

print remoteaddr, localname
urllib.urlretrieve(remoteaddr, localname)
remotedata = open(localname).readlines()
for line in remotedata[:showlines]: print line,
================================================= ===========
I am new to the internet programming as well as python. please guide me,
how to solve this one.
urllib will work with proxies. Just set your environment to point at
it before you start python. Like this (copied from the urllib doc):

% http_proxy="http://www.someproxy.com:3128"
% export http_proxy
% python

hth,<{{{*>

[Anand Pillai]

Try downloading HarvestMan from http://harvestman.freezope.org .

-Anand
fishboy <fi*****@spamspamspam.com> wrote in message news:<qs********************************@4ax.com>. ..

On Wed, 02 Jun 2004 06:23:20 -0400, "Roysun_rohit"
<ro**********@rediffmail.com> wrote:

I am interested in making a search engine which takes web sites
iteratively, and downloads the web page or has to perform some search
across the web pages.
I am unsucessful to do so. My machine works through a proxy server and the
internet connection is through 24 hour lease line. when ever i try the
code it gives :

Traceback (innermost last):
File "http-getfile-urllib2.py", line 19, in ?
urllib.urlretrieve(remoteaddr, localname)
File "/usr/lib/python1.5/urllib.py", line 66, in urlretrieve
return _urlopener.retrieve(url, filename, reporthook)
File "/usr/lib/python1.5/urllib.py", line 186, in retrieve
fp = self.open(url)
File "/usr/lib/python1.5/urllib.py", line 159, in open
return getattr(self, name)(url)
File "/usr/lib/python1.5/urllib.py", line 260, in open_http
h = httplib.HTTP(host)
File "/usr/lib/python1.5/httplib.py", line 53, in __init__
if host: self.connect(host, port)
File "/usr/lib/python1.5/httplib.py", line 81, in connect
self.sock.connect(host, port)
IOError: [Errno socket error] (101, 'Network is unreachable')
================================================= ===========
The code is like this:-
#!/usr/bin/env python
import os, sys, urllib, urlparse, socket
showlines = 6
try:
servername, filename = sys.argv[1:3]
except:
servername, filename = 'www.igib.res.in', '/sarsanalysis.html'

remoteaddr = 'http://%s%s' % (servername, filename)
if len(sys.argv) == 4:
localname = sys.argv[3]
else:
(scheme, server, path, parms, query, frag) =
urlparse.urlparse(remoteaddr)
localname = os.path.split(path)[1]

print remoteaddr, localname
urllib.urlretrieve(remoteaddr, localname)
remotedata = open(localname).readlines()
for line in remotedata[:showlines]: print line,
================================================= ===========
I am new to the internet programming as well as python. please guide me,
how to solve this one.

urllib will work with proxies. Just set your environment to point at
it before you start python. Like this (copied from the urllib doc):

% http_proxy="http://www.someproxy.com:3128"
% export http_proxy
% python

hth,

<{{{*>