473,562 Members | 2,956 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Advice for a python newbie on parsing whois records?

Hi. I'm stretching my boundaries in programming with a little python
shell-script which is going to loop through a list of domain names,
grab the whois record, parse it, and put the results into a csv.

I've got the results coming back fine, but since I have *no*
experience with python I'm wondering what would be the preferred
"pythonic" way of parsing the whois string into a csv record.

Tips/thoughts/examples more than welcome!
Jun 27 '08 #1
3 4229
Lie
On Jun 10, 9:47*pm, Phillip B Oldham <phillip.old... @gmail.comwrote :
Hi. I'm stretching my boundaries in programming with a little python
shell-script which is going to loop through a list of domain names,
grab the whois record, parse it, and put the results into a csv.

I've got the results coming back fine, but since I have *no*
experience with python I'm wondering what would be the preferred
"pythonic" way of parsing the whois string into a csv record.

Tips/thoughts/examples more than welcome!
Generally, when doing a simple parsing you'd use re module (regular
expression).
More complex parsers would use pyparsing (3rd party module). I've
never used pyparsing myself though. You might download pyparsing
here:http://pyparsing.wikispaces.com/Down...d+Installation
Jun 27 '08 #2
Hello,
Hi. I'm stretching my boundaries in programming with a little python
shell-script which is going to loop through a list of domain names,
grab the whois record, parse it, and put the results into a csv.

I've got the results coming back fine, but since I have *no*
experience with python I'm wondering what would be the preferred
"pythonic" way of parsing the whois string into a csv record.

Tips/thoughts/examples more than welcome!
from os import popen
import re

find_record = re.compile("\s+ ([^:]+): (.*)\s*").match
for line in popen("whois google.com"):
match = find_record(lin e)
if not match:
continue
print "%s --%s" % (match.groups()[0], match.groups()[1])

HTH,
--
Miki <mi*********@gm ail.com>
http://pythonwise.blogspot.com
Jun 27 '08 #3
On Jun 10, 8:21 pm, Miki <miki.teb...@gm ail.comwrote:
Hello,
Hi. I'm stretching my boundaries in programming with a little python
shell-script which is going to loop through a list of domain names,
grab the whois record, parse it, and put the results into a csv.
I've got the results coming back fine, but since I have *no*
experience with python I'm wondering what would be the preferred
"pythonic" way of parsing the whois string into a csv record.
Tips/thoughts/examples more than welcome!

from os import popen
import re

find_record = re.compile("\s+ ([^:]+): (.*)\s*").match
for line in popen("whois google.com"):
match = find_record(lin e)
if not match:
continue
print "%s --%s" % (match.groups()[0], match.groups()[1])

HTH,
--
Miki <miki.teb...@gm ail.com>http://pythonwise.blogspot.com
OK, here's what I've got so far. I'm treating this as a learning
exercise, so the resulting file isn't so important as understanding
and thinking in python (although I believe the results are adequate
for my needs). I'd appreciate the community's comments as this is my
*first* attempt at python and has taken me a couple of hours
(including googling).

#!/usr/bin/env python
import subprocess
import re

src = open('./domains.txt')

dest = open('./whois.csv', 'w');

def trim( txt ):
x = []
for line in txt.split("\n") :
if line.strip() == "":
continue
if line.strip().st artswith('WHOIS '):
continue
if line.strip().st artswith('>>>') :
continue
if line.strip().st artswith('%'):
continue
if line.startswith ("--"):
return ''.join(x)
x.append(" "+line)
return "\n".join(x )

def clean( txt ):
x = []
isok = re.compile("^\s ?([^:]+): ").match
for line in txt.split("\n") :
match = isok(line)
if not match:
continue
x.append(line)
return "\n".join(x );

def clean_co_uk( rec ):
rec = rec.replace('Co mpany number:', 'Company number -')
rec = rec.replace("\n \n", "\n")
rec = rec.replace("\n ", "")
rec = rec.replace(": ", ":\n")
rec = re.sub("([^(][a-zA-Z']+\s?[a-zA-Z]*:\n)", "\n\g<0>", rec)
rec = rec.replace(":\ n", ": ")
rec = re.sub("^[ ]+\n", "", rec)
return rec

def clean_net( rec ):
rec = rec.replace("\n \n", "\n")
rec = rec.replace("\n ", "")
rec = rec.replace(": ", ":\n")
rec = re.sub("([a-zA-Z']+\s?[a-zA-Z]*:\n)", "\n\g<0>", rec)
rec = rec.replace(":\ n", ": ")
return rec

def clean_info( rec ):
x = []
for line in rec.split("\n") :
x.append(re.sub ("^([^:]+):", "\g<0", line))
return "\n".join(x )

def record(domain, record):

## Records are as follows: [ domain, registrant, registrant's address
registrar, type, registered, renewal, updated name servers ]
details = ['','','','','', '','','','']
for k, v in record.items():
try:
details[0] = domain.lower()
result = {
"registrant ": lambda: 1,
"registrant name": lambda: 1,
"registrant type": lambda: 4,
"registrant 's address": lambda: 2,
"registrant address1": lambda: 2,
"registrar" : lambda: 3,
"sponsoring registrar": lambda: 3,
"registered on": lambda: 5,
"registered ": lambda: 5,
"domain registeration date": lambda: 5,
"renewal date": lambda: 6,
"last updated": lambda: 7,
"domain last updated date": lambda: 7,
"name servers": lambda: 8,
"name server": lambda: 8,
"nameserver s": lambda: 8,
"updated date": lambda: 7,
"creation date": lambda: 5,
"expiration date": lambda: 6,
"domain expiration date": lambda: 6,
"administra tive contact": lambda: 2
}[k.lower()]()
if v != '':
details[result] = v
except:
continue

dest.write('|'. join(details)+" \n")

## Loop through domains
for domain in src:

domain = domain.strip()

if domain == '':
continue

rec = subprocess.Pope n(["whois",dom ain],
stdout=subproce ss.PIPE).commun icate()[0]

if rec.startswith( "No whois server") == True:
continue

if rec.startswith( "This TLD has no whois server") == True:
continue

rec = trim(rec)

if domain.endswith (".net"):
rec = clean_net(rec)

if domain.endswith (".com"):
rec = clean_net(rec)

if domain.endswith (".tv"):
rec = clean_net(rec)

if domain.endswith (".co.uk"):
rec = clean_co_uk(rec )

if domain.endswith (".info"):
rec = clean_info(rec)

rec = clean(rec)

details = {}

try:
for line in rec.split("\n") :
bits = line.split(': ')
a = bits.pop(0)
b = bits.pop(0)
details[a.strip()] = b.strip().repla ce("\t", ", ")
except:
continue

record(domain, details)

src.close()
dest.close()
Jun 27 '08 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
3453
by: Dave Brueck | last post by:
Below is some information I collected from a *small* project in which I wrote a Python version of a Java application. I share this info only as a data point (rather than trying to say this data "proves" something) to consider the next time the "Python makes developers more productive" thread starts up again. Background ========== An...
3
9564
by: WIWA | last post by:
Hi all, Anybody that could tell me how to execute some Linux commands using Python. The application is that I want to be able from a website to launch a Python script and that python script should be able to ping to the provided IP address, or should do a whois of that IP address. I want to use the existing Linux commands if possible. ...
0
1429
by: richard.hubbell | last post by:
I am sure this is old news, the syntax of python is crazy to me. There I said it, I'm sure I'll get over it or around it. I was trying to make a whois script work and was unable. May be someone with lint-like eyes can tell what's wrong. Using xemacs I had hoped that python mode would do more for syntax problems, maybe I'm just not using...
3
1529
by: Vjay77 | last post by:
As a result from who is I am getting this: > > NOTICE AND TERMS OF USE: You are not authorized to access or query our WHOIS database through the use of high-volume, automated, electronic processes. The Data in Network Solutions' WHOIS database is provided by Network Solutions for information purposes only, and to assist persons in...
0
1190
by: skip | last post by:
I'm looking to get whois information for a given domain from Python. I'm mostly interested in record update and creation dates at the moment, but eventually might want more. I stumbled upon a couple modules/programs: rwhois.py and whois.py but neither seems to work. I can clearly issue a whois command directly, but I'm hoping there's some...
10
1989
by: Roel | last post by:
Hi all, I want to query WHOIS info from my ASP.NET app. I'd like to query a specific domain (e.g. google.com) and for availability (e.g. google.eu (available/taken), google.nl (available/taken)). Anyone knows a (commercial) component that gets this job done? Kind Regards,
5
4920
by: amjadcsu | last post by:
I am a newbie in python I am trying to parse a xml file and write its content in a txt file. The txt contains null elements. Any reason what iam doing wrong here Here is the code that i wrote import sys,os import xml.sax import xml.sax.handler
0
2715
by: eliss | last post by:
Hi everyone, I'm trying to write a python script to whois a few domains twice a day so I get notified when they become available. I did it 2 ways, but neither way is very reliable, so I'm asking here. 1) I tried just calling "whois %s" using popen, but I found that once every few weeks, I get errors like: connect: No route to host OR...
3
2317
by: Ethan Furman | last post by:
len wrote: I've never had the (mis?)fortune to work with COBOL -- what are the files like? Fixed format, or something like a dBase III style? I
0
7655
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main...
0
7577
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language...
1
7627
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For...
0
7935
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the...
1
5477
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes...
0
5193
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert...
0
3608
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
2073
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
0
903
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.