Getting the size of sourcecode

[mh121]

Hello,

I am trying to input a spreadsheet of possible domain names and output the length of the sourcecode of the webpage (if it exists). In doing this, I have three small questions (I am a newbie and apologize if the questions are simple):

1. How do I convert the length of the page to a string? I have looked around the web for Python 'tostring' and found several individually created functions, but I tried a few and had problems.

2. What is the best way to handle errors when a domain phrase doesn't lead to a good website? This will happen (I think) with the line z=br.open('http://www.'+domainTerm)
for which the domainTerm might not lead to an active website.

3. Instead of getting the total number of characters on the sourcepage (which I get by looking at len(page) ), is there any way to get the number of lines?

Thank you,
Mitch

from mechanize import Browser
import re, time, urllib2

def MakeBrowser():
b = Browser()
headerString = 'mozilla/5.0 (x11; u; linux i686; en-us; rv:1.7.12) ' + \
'gecko/20050922 firefox/1.0.7 (debian package 1.0.7-1)'
h = [('User-agent', headerString)]
b.addheaders = h
b.set_handle_robots(False)
return(b)

f = open('bizornot1.csv','r')
lines = f.readlines()
f.close()
f2 = open('bizornot1_new.csv','w')
f2.write(lines[0].rstrip()+',PageSize'+"\n")
print(lines[0].rstrip()+",PageSize")

for i in range(len(lines)-1):
domainTerm = domainTerms[i]
br = MakeBrowser()
z=br.open('http://www.'+domainTerm)
page=z.read()
f2.write(lines[i+1].rstrip()+','+len(page)+"\n")
print(lines[i+1].rstrip()+','+len(page))

f2.close()

[bvdet]

Hello,

I am trying to input a spreadsheet of possible domain names and output the length of the sourcecode of the webpage (if it exists). In doing this, I have three small questions (I am a newbie and apologize if the questions are simple):

1. How do I convert the length of the page to a string? I have looked around the web for Python 'tostring' and found several individually created functions, but I tried a few and had problems.

2. What is the best way to handle errors when a domain phrase doesn't lead to a good website? This will happen (I think) with the line z=br.open('http://www.'+domainTerm)
for which the domainTerm might not lead to an active website.

3. Instead of getting the total number of characters on the sourcepage (which I get by looking at len(page) ), is there any way to get the number of lines?

Thank you,
Mitch

from mechanize import Browser
import re, time, urllib2

def MakeBrowser():
b = Browser()
headerString = 'mozilla/5.0 (x11; u; linux i686; en-us; rv:1.7.12) ' + \
'gecko/20050922 firefox/1.0.7 (debian package 1.0.7-1)'
h = [('User-agent', headerString)]
b.addheaders = h
b.set_handle_robots(False)
return(b)

f = open('bizornot1.csv','r')
lines = f.readlines()
f.close()
f2 = open('bizornot1_new.csv','w')
f2.write(lines[0].rstrip()+',PageSize'+"\n")
print(lines[0].rstrip()+",PageSize")

for i in range(len(lines)-1):
domainTerm = domainTerms[i]
br = MakeBrowser()
z=br.open('http://www.'+domainTerm)
page=z.read()
f2.write(lines[i+1].rstrip()+','+len(page)+"\n")
print(lines[i+1].rstrip()+','+len(page))

f2.close()

I am not familiar with the 'mechanize' module. You can do the following with the 'urllib' module:

from urllib import urlopen
 
h = urlopen('http://www.somewebsite.com/')
 
# source = h.read() # read page into a string
lineList = h.readlines() # read page into a list of strings
info = h.info()
trueURL = h.geturl()
 
print ('The number of lines is %d' % len(lineList))
 
print 'The number of words is %d' % sum([len(line.strip().split()) for line in lineList])
 
h.close()
 
try:
    h = urlopen('http://www.invalidURL.com/')
except IOError, e:
    print e

>>> The number of lines is 110
The number of words is 807
[Errno socket error] (7, 'getaddrinfo failed')
 
>>> info
<httplib.HTTPMessage instance at 0x00E7A3C8>
>>> print info
Date: Thu, 09 Aug 2007 02:25:43 GMT
Server: Apache
Last-Modified: Tue, 07 Aug 2007 02:41:23 GMT
ETag: "744141-2b29-46b7dbd3"
Accept-Ranges: bytes
Content-Length: 11049
Connection: close
Content-Type: text/html
 
>>> trueURL
'http://www.bvdetailing.com/'
>>> 
'''

[robin746]

1. How do I convert the length of the page to a string? I have looked around the web for Python 'tostring' and found several individually created functions, but I tried a few and had problems.

Please clarify. If you have read the page into a variable, it is (likely) already a string. The str() function converts to a string but I do not think that is what you want.

2. What is the best way to handle errors when a domain phrase doesn't lead to a good website?

What error does your module spit back? Wrap the code you have in a try/except block specifying this error, and then do what you want when it happens. For example, if the error is BadSillyError, do this:

try:
    # code to run
except BadSillyError:
    # what to do on failure
else:
    # continue with rest of cdoe if error does not happen
 

3. Instead of getting the total number of characters on the sourcepage (which I get by looking at len(page) ), is there any way to get the number of lines?

If you know lines are separated by carriage returns you can do something like:

lines = page.split('\n')
number_of_lines = len(lines)
 

But then you might want to eliminate blank lines. Or comments.

[robin746]

I have just published a full Line Of Code Counter that you can adapt to your purpose.

[mh121]

Thank you very much for your comments. What I meant for the destring function was actually just the str() function you provided. I tried out many of your suggestions today and, after trying out more tomorrow, if I continue to have questions, I will repost.