473,322 Members | 1,409 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,322 software developers and data experts.

Advice for a python newbie on parsing whois records?

Hi. I'm stretching my boundaries in programming with a little python
shell-script which is going to loop through a list of domain names,
grab the whois record, parse it, and put the results into a csv.

I've got the results coming back fine, but since I have *no*
experience with python I'm wondering what would be the preferred
"pythonic" way of parsing the whois string into a csv record.

Tips/thoughts/examples more than welcome!
Jun 27 '08 #1
3 4195
Lie
On Jun 10, 9:47*pm, Phillip B Oldham <phillip.old...@gmail.comwrote:
Hi. I'm stretching my boundaries in programming with a little python
shell-script which is going to loop through a list of domain names,
grab the whois record, parse it, and put the results into a csv.

I've got the results coming back fine, but since I have *no*
experience with python I'm wondering what would be the preferred
"pythonic" way of parsing the whois string into a csv record.

Tips/thoughts/examples more than welcome!
Generally, when doing a simple parsing you'd use re module (regular
expression).
More complex parsers would use pyparsing (3rd party module). I've
never used pyparsing myself though. You might download pyparsing
here:http://pyparsing.wikispaces.com/Down...d+Installation
Jun 27 '08 #2
Hello,
Hi. I'm stretching my boundaries in programming with a little python
shell-script which is going to loop through a list of domain names,
grab the whois record, parse it, and put the results into a csv.

I've got the results coming back fine, but since I have *no*
experience with python I'm wondering what would be the preferred
"pythonic" way of parsing the whois string into a csv record.

Tips/thoughts/examples more than welcome!
from os import popen
import re

find_record = re.compile("\s+([^:]+): (.*)\s*").match
for line in popen("whois google.com"):
match = find_record(line)
if not match:
continue
print "%s --%s" % (match.groups()[0], match.groups()[1])

HTH,
--
Miki <mi*********@gmail.com>
http://pythonwise.blogspot.com
Jun 27 '08 #3
On Jun 10, 8:21 pm, Miki <miki.teb...@gmail.comwrote:
Hello,
Hi. I'm stretching my boundaries in programming with a little python
shell-script which is going to loop through a list of domain names,
grab the whois record, parse it, and put the results into a csv.
I've got the results coming back fine, but since I have *no*
experience with python I'm wondering what would be the preferred
"pythonic" way of parsing the whois string into a csv record.
Tips/thoughts/examples more than welcome!

from os import popen
import re

find_record = re.compile("\s+([^:]+): (.*)\s*").match
for line in popen("whois google.com"):
match = find_record(line)
if not match:
continue
print "%s --%s" % (match.groups()[0], match.groups()[1])

HTH,
--
Miki <miki.teb...@gmail.com>http://pythonwise.blogspot.com
OK, here's what I've got so far. I'm treating this as a learning
exercise, so the resulting file isn't so important as understanding
and thinking in python (although I believe the results are adequate
for my needs). I'd appreciate the community's comments as this is my
*first* attempt at python and has taken me a couple of hours
(including googling).

#!/usr/bin/env python
import subprocess
import re

src = open('./domains.txt')

dest = open('./whois.csv', 'w');

def trim( txt ):
x = []
for line in txt.split("\n"):
if line.strip() == "":
continue
if line.strip().startswith('WHOIS'):
continue
if line.strip().startswith('>>>'):
continue
if line.strip().startswith('%'):
continue
if line.startswith("--"):
return ''.join(x)
x.append(" "+line)
return "\n".join(x)

def clean( txt ):
x = []
isok = re.compile("^\s?([^:]+): ").match
for line in txt.split("\n"):
match = isok(line)
if not match:
continue
x.append(line)
return "\n".join(x);

def clean_co_uk( rec ):
rec = rec.replace('Company number:', 'Company number -')
rec = rec.replace("\n\n", "\n")
rec = rec.replace("\n", "")
rec = rec.replace(": ", ":\n")
rec = re.sub("([^(][a-zA-Z']+\s?[a-zA-Z]*:\n)", "\n\g<0>", rec)
rec = rec.replace(":\n", ": ")
rec = re.sub("^[ ]+\n", "", rec)
return rec

def clean_net( rec ):
rec = rec.replace("\n\n", "\n")
rec = rec.replace("\n", "")
rec = rec.replace(": ", ":\n")
rec = re.sub("([a-zA-Z']+\s?[a-zA-Z]*:\n)", "\n\g<0>", rec)
rec = rec.replace(":\n", ": ")
return rec

def clean_info( rec ):
x = []
for line in rec.split("\n"):
x.append(re.sub("^([^:]+):", "\g<0", line))
return "\n".join(x)

def record(domain, record):

## Records are as follows: [ domain, registrant, registrant's address
registrar, type, registered, renewal, updated name servers ]
details = ['','','','','','','','','']
for k, v in record.items():
try:
details[0] = domain.lower()
result = {
"registrant": lambda: 1,
"registrant name": lambda: 1,
"registrant type": lambda: 4,
"registrant's address": lambda: 2,
"registrant address1": lambda: 2,
"registrar": lambda: 3,
"sponsoring registrar": lambda: 3,
"registered on": lambda: 5,
"registered": lambda: 5,
"domain registeration date": lambda: 5,
"renewal date": lambda: 6,
"last updated": lambda: 7,
"domain last updated date": lambda: 7,
"name servers": lambda: 8,
"name server": lambda: 8,
"nameservers": lambda: 8,
"updated date": lambda: 7,
"creation date": lambda: 5,
"expiration date": lambda: 6,
"domain expiration date": lambda: 6,
"administrative contact": lambda: 2
}[k.lower()]()
if v != '':
details[result] = v
except:
continue

dest.write('|'.join(details)+"\n")

## Loop through domains
for domain in src:

domain = domain.strip()

if domain == '':
continue

rec = subprocess.Popen(["whois",domain],
stdout=subprocess.PIPE).communicate()[0]

if rec.startswith("No whois server") == True:
continue

if rec.startswith("This TLD has no whois server") == True:
continue

rec = trim(rec)

if domain.endswith(".net"):
rec = clean_net(rec)

if domain.endswith(".com"):
rec = clean_net(rec)

if domain.endswith(".tv"):
rec = clean_net(rec)

if domain.endswith(".co.uk"):
rec = clean_co_uk(rec)

if domain.endswith(".info"):
rec = clean_info(rec)

rec = clean(rec)

details = {}

try:
for line in rec.split("\n"):
bits = line.split(': ')
a = bits.pop(0)
b = bits.pop(0)
details[a.strip()] = b.strip().replace("\t", ", ")
except:
continue

record(domain, details)

src.close()
dest.close()
Jun 27 '08 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: Dave Brueck | last post by:
Below is some information I collected from a *small* project in which I wrote a Python version of a Java application. I share this info only as a data point (rather than trying to say this data...
3
by: WIWA | last post by:
Hi all, Anybody that could tell me how to execute some Linux commands using Python. The application is that I want to be able from a website to launch a Python script and that python script...
0
by: richard.hubbell | last post by:
I am sure this is old news, the syntax of python is crazy to me. There I said it, I'm sure I'll get over it or around it. I was trying to make a whois script work and was unable. May be...
3
by: Vjay77 | last post by:
As a result from who is I am getting this: > > NOTICE AND TERMS OF USE: You are not authorized to access or query our WHOIS database through the use of high-volume, automated, electronic...
0
by: skip | last post by:
I'm looking to get whois information for a given domain from Python. I'm mostly interested in record update and creation dates at the moment, but eventually might want more. I stumbled upon a...
10
by: Roel | last post by:
Hi all, I want to query WHOIS info from my ASP.NET app. I'd like to query a specific domain (e.g. google.com) and for availability (e.g. google.eu (available/taken), google.nl...
5
by: amjadcsu | last post by:
I am a newbie in python I am trying to parse a xml file and write its content in a txt file. The txt contains null elements. Any reason what iam doing wrong here Here is the code that i wrote...
0
by: eliss | last post by:
Hi everyone, I'm trying to write a python script to whois a few domains twice a day so I get notified when they become available. I did it 2 ways, but neither way is very reliable, so I'm asking...
3
by: Ethan Furman | last post by:
len wrote: I've never had the (mis?)fortune to work with COBOL -- what are the files like? Fixed format, or something like a dBase III style? I
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.