Help beautify ugly heuristic code

[Stuart D. Gathman]

I have a function that recognizes PTR records for dynamic IPs. There is
no hard and fast rule for this - every ISP does it differently, and may
change their policy at any time, and use different conventions in
different places. Nevertheless, it is useful to apply stricter
authentication standards to incoming email when the PTR for the IP
indicates a dynamic IP (namely, the PTR record is ignored since it doesn't
mean anything except to the ISP). This is because Windoze Zombies are the
favorite platform of spammers.

Here is the very ugly code so far. It offends me to look at it, but
haven't had any better ideas. I have lots of test data from mail logs.

# examples we don't yet recognize:
#
# 1Cust65.tnt4.at l4.da.uu.net at ('67.192.40.65' , 4588)
# 1Cust200.tnt8.b ne1.da.uu.net at ('203.61.67.200 ', 4144)
# 1Cust141.tnt30. rtm1.nld.da.uu. net at ('213.116.154.1 41', 2036)
# user64.net2045. mo.sprint-hsd.net at ('67.77.185.64' , 3901)
# wiley-268-8196.roadrunner .nf.net at ('205.251.174.4 6', 4810)
# 221.fib163.satn et.net at ('200.69.163.22 1', 3301)
# cpc2-ches1-4-0-cust8.lutn.cabl e.ntl.com at ('80.4.105.8', 61099)
# user239.res.ope nband.net at ('65.246.82.239 ', 1392)
# xdsl-2449.zgora.dial og.net.pl at ('81.168.237.14 5', 1238)
# spr1-runc1-4-0-cust25.bagu.bro adband.ntl.com at ('80.5.10.25', 1684)
# user-0c6s7hv.cable.m indspring.com at ('24.110.30.63' , 3720)
# user-0c8hvet.cable.m indspring.com at ('24.136.253.22 1', 4529)
# user-0cdf5j8.cable.m indspring.com at ('24.215.150.10 4', 3783)
# mmds-dhcp-11-143.plateautel. net at ('63.99.131.143 ', 4858)
# ca-santaanahub-cuda3-c6b-134.anhmca.adel phia.net at ('68.67.152.134 ', 62047)
# cbl-sd-02-79.aster.com.do at ('200.88.62.79' , 4153)
# h105n6c2o912.br edband.skanova. com at ('213.67.33.105 ', 3259)

import re

ip3 = re.compile('([0-9]{1,3})[.x-]([0-9]{1,3})[.x-]([0-9]{1,3})')
rehmac = re.compile(
'h[0-9a-f]{12}[.]|pcp[0-9]{6,10}pcs[.]|no-reverse|S[0-9a-f]{16}[.][a-z]{2}[.]'
)

def is_dynip(host,a ddr):
"""Return True if hostname is for a dynamic ip.
Examples:

is_dynip('post3 .fabulousdealz. com','69.60.99. 112') False is_dynip('adsl-69-208-201-177.dsl.emhril. ameritech.net', '69.208.201.177 ') True is_dynip('[1.2.3.4]','1.2.3.4')

True
"""
if host.startswith ('[') and host.endswith(']'):
return True
if addr:
if host.find(addr) >= 0: return True
a = addr.split('.')
ia = map(int,a)
m = ip3.search(host )
if m:
g = map(int,m.group s())
if g == ia[1:] or g == ia[:3]: return True
if g[0] == ia[3] and g[1:] == ia[:2]: return True
g.reverse()
if g == ia[1:] or g == ia[:3]: return True
if rehmac.search(h ost): return True
if host.find("%s." % '-'.join(a[2:])) >= 0: return True
if host.find("w%s. " % '-'.join(a[:2])) >= 0: return True
if host.find("dsl% s-" % '-'.join(a[:2])) >= 0: return True
if host.find(''.jo in(a[:3])) >= 0: return True
if host.find(''.jo in(a[1:])) >= 0: return True
x = "%02x%02x%02x%0 2x" % tuple(ia)
if host.lower().fi nd(x) >= 0: return True
z = [n.zfill(3) for n in a]
if host.find('-'.join(z)) >= 0: return True
if host.find("-%s." % '-'.join(z[2:])) >= 0: return True
if host.find("%s." % ''.join(z[2:])) >= 0: return True
if host.find(''.jo in(z)) >= 0: return True
a.reverse()
if host.find("%s." % '-'.join(a[:2])) >= 0: return True
if host.find("%s." % '.'.join(a[:2])) >= 0: return True
if host.find("%s." % a[0]) >= 0 and \
host.find('.ads l.') > 0 or host.find('.dia l-up.') > 0: return True
return False

if __name__ == '__main__':
import fileinput
for ln in fileinput.input ():
a = ln.split()
if len(a) == 2:
ip,host = a
if host.startswith ('[') and host.endswith(']'):
continue # no PTR
if is_dynip(host,i p):
print ip,host

[Jeremy Sanders]

On Wed, 08 Dec 2004 18:38:14 -0500, Stuart D. Gathman wrote:

Here are the last 20 (which my subjective judgement says are correct):
65.112.76.15 usfshlxmx01.myr eg.net 201.128.108.41

[snip] 80.143.79.97 p508F4F61.dip0. t-ipconnect.de DYN

Looks like you could do something like look for a part of the dns name
which is over a certain length with a high fraction of numbers and
punctuation characters.

Of course, you could build up a probability tree of the chance of a
character being followed by another character in a dynamic name and a
static name.

This might work well, as many are sequences of numbers (high chance of
number followed by number), and mixed characters and numbers, which are
unusual in normal dns names.

Jeremy

[Stuart D. Gathman]

On Thu, 09 Dec 2004 00:01:36 -0800, Lonnie Princehouse wrote:

I believe you can still do this with only compiling a regex once and
then performing a few substitutions on the hostname.

Cool idea. Convert ip matches to fixed patterns before matching a fixed
regex. The leftovers like shaw cable (which has the MAC address of the
cable modem instead of the IP) can still be handled with regex patterns.

I had an idea last night to compile 254 regexes, one for each possible
last IP byte - but I think your idea is better.

Mitja suggested a socring system reminiscent of SpamAssassin.

That gives me a few things to try.

--
Stuart D. Gathman <st****@bmsi.co m>
Business Management Systems Inc. Phone: [phone] Fax: [phone]
"Confutatis maledictis, flamis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

[JanC]

Stuart D. Gathman schreef:

I have a function that recognizes PTR records for dynamic IPs. There
is no hard and fast rule for this - every ISP does it differently, and
may change their policy at any time, and use different conventions in
different places. Nevertheless, it is useful to apply stricter
authentication standards to incoming email when the PTR for the IP
indicates a dynamic IP (namely, the PTR record is ignored since it
doesn't mean anything except to the ISP). This is because Windoze
Zombies are the favorite platform of spammers.

Did you also think about ISPs that use such a PTR record for both dynamic
and fixed IPs?

--
JanC

"Be strict when sending and tolerant when receiving."
RFC 1958 - Architectural Principles of the Internet - section 3.9

[Stuart D. Gathman]

On Fri, 10 Dec 2004 22:03:20 +0000, JanC wrote:

Stuart D. Gathman schreef:

I have a function that recognizes PTR records for dynamic IPs. There

Did you also think about ISPs that use such a PTR record for both dynamic
and fixed IPs?

There seems to be a lot of misunderstandin g about this. I am not blocking
anyones mail because they have a dynamic looking PTR. I simply don't
accept such a PTR as MTA authentication. You see, MTAs *SHOULD* provide a
fully qualified domain as their HELO name which resolves to the IP of the
MTA. Sadly, however, many internet facing MTAs don't do this, but I
accept a meaningful PTR as a substitute. I also waive the requirement for
MTA authentication if the MAIL FROM has an SPF record
(http://spf.pobox.com).

So, if your MTA complies with RFC HELO recommendations , you'll have no
trouble sending me mail. You can even use a dynamic IP with a dynamic DNS
service.

I 'do* block PTR names of "." or "localhost" . I would like to block all
single word HELO names - but there are too many clueless mail admins out
there. People seem to be unsure of what to send for HELO.

--
Stuart D. Gathman <st****@bmsi.co m>
Business Management Systems Inc. Phone: [phone] Fax: [phone]
"Confutatis maledictis, flamis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

[Stuart D. Gathman]

On Thu, 09 Dec 2004 00:01:36 -0800, Lonnie Princehouse wrote:

I believe you can still do this with only compiling a regex once and
then performing a few substitutions on the hostname.

That is a interesting idea. Convert ip matches to fixed patterns, and
*then* match the regex. I think I would convert hex matches to the same
pattern as decimal (and roman numeral). How would you handle zero fill?

1.2.3.4 001002003004foo .isp.com

An idea I had last night is to precompile 254 regexes - one for each of
the possible last ip bytes. However, your idea is cleaner - except, how
would it handle ip bytes that are the same: 1.2.2.2

Mitja has proposed a scoring system reminiscent of SpamAssassin.

This gives me a few things to try.

--
Stuart D. Gathman <st****@bmsi.co m>
Business Management Systems Inc. Phone: [phone] Fax: [phone]
"Confutatis maledictis, flamis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.