473,231 Members | 1,838 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,231 software developers and data experts.

URL parsing for the hard cases

Is there something available that will parse the "netloc" field as
returned by URLparse, including all the hard cases? The "netloc" field
can potentially contain a port number and a numeric IP address. The
IP address may take many forms, including an IPv6 address.

I'm parsing URLs used by hostile sites, and the wierd cases come up
all too frequently.

John Nagle
Jul 22 '07 #1
7 3757
On 7/22/07, John Nagle wrote:
Is there something available that will parse the "netloc" field as
returned by URLparse, including all the hard cases? The "netloc" field
can potentially contain a port number and a numeric IP address. The
IP address may take many forms, including an IPv6 address.
What do you mean by "parse" the field? What do you want to get back
from the parser function?
Jul 22 '07 #2
On 22 Jul, 18:56, John Nagle <na...@animats.comwrote:
Is there something available that will parse the "netloc" field as
returned by URLparse, including all the hard cases? The "netloc" field
can potentially contain a port number and a numeric IP address. The
IP address may take many forms, including an IPv6 address.

I'm parsing URLs used by hostile sites, and the wierd cases come up
all too frequently.
I assume that when you say "netloc" you are referring to the second
field returned by the urlparse module. If this netloc contains an IPv6
address then it will also contain square brackets. The colons inside
the [] belong to the IPv6 address and the single possible colon
outside the brackets belongs to the port number. Of course, you might
want to try to help people who do not follow the RFCs and failed to
wrap the IPv6 address in square brackets. In that case, try...expect
comes in handy. You can try to parse an IPv6 address and if it fails
because of too many segments, then fallback to some other behaviour.

The worst case is a URL like http://2001::123:4567:abcd:8080/something.
Does the 8080 refer to a port number or part of the IPv6 address. If I
had to support non-bracketed IPv6 addresses, then I would interpret
this as http://[2001::123:4567:abcd]:8080/something.

RFC3986 is the reference for correct URL formats.

Once you eliminate IPv6 addresses, parsing is simple. Is there a
colon? Then there is a port number. Does the left over have any
characters not in [0123456789.]? Then it is a name, not an IPv4
address.

--Michael Dillon

Jul 22 '07 #3
me******@yahoo.com wrote:
Once you eliminate IPv6 addresses, parsing is simple. Is there a
colon? Then there is a port number. Does the left over have any
characters not in [0123456789.]? Then it is a name, not an IPv4
address.

--Michael Dillon
You wish. Hex input of IP addresses is allowed:

http://0x525eedda

and

http://0x52.0x5e.0xed.0xda

are both "Python.org". Or just put

0x52.0x5e.0xed.0xda

into the address bar of a browser. All these work in Firefox on Windows and
are recognized as valid IP addresses.

On the other hand,

0x52.com

is a valid domain name, in use by PairNIC.

But

http://test.0xda

is handled by Firefox on Windows as a domain name. It doesn't resolve, but it's
sent to DNS.

So I think the question is whether every term between dots can be parsed as
a decimal or hex number. If all terms can be parsed as a number, and there are
no more than four of them, it's an IP address. Otherwise it's a domain name.

There are phishing sites that pull stuff like this, and I'm parsing a long list
of such sites. So I really do need to get the hard cases right.

Is there any library function that correctly tests for an IP address vs. a
domain name based on syntax, i.e. without looking it up in DNS?

John Nagle
Jul 23 '07 #4
Here's another hard case. This one might be a bug in urlparse:

import urlparse

s = 'ftp://administrator:pa******@64.105.135.30/originals/6 june
07/ebay/login/ebayisapi.html'

urlparse.urlparse(s)

yields:

(u'ftp', u'administrator:pa******@64.105.135.30', u'/originals/6 june
07/ebay/login/ebayisapi.html', '', '', '')

That second field is supposed to be the "hostport" (per the RFC usage
of the term; Python uses the term "netloc"), and the username/password
should have been parsed and moved to the "username" and "password" fields
of the object. So it looks like urlparse doesn't really understand FTP URLs.

That's a real URL, from a search for phishing sites. There are lots
of hostile URLs out there. Some of which can fool some parsers.

John Nagle

John Nagle wrote:
me******@yahoo.com wrote:
>Once you eliminate IPv6 addresses, parsing is simple. Is there a
colon? Then there is a port number. Does the left over have any
characters not in [0123456789.]? Then it is a name, not an IPv4
address.

--Michael Dillon

You wish. Hex input of IP addresses is allowed:

http://0x525eedda

and

http://0x52.0x5e.0xed.0xda

are both "Python.org". Or just put

0x52.0x5e.0xed.0xda

into the address bar of a browser. All these work in Firefox on Windows
and
are recognized as valid IP addresses.

On the other hand,

0x52.com

is a valid domain name, in use by PairNIC.

But

http://test.0xda

is handled by Firefox on Windows as a domain name. It doesn't resolve,
but it's
sent to DNS.

So I think the question is whether every term between dots can be parsed as
a decimal or hex number. If all terms can be parsed as a number, and
there are
no more than four of them, it's an IP address. Otherwise it's a domain
name.

There are phishing sites that pull stuff like this, and I'm parsing a
long list
of such sites. So I really do need to get the hard cases right.

Is there any library function that correctly tests for an IP address vs. a
domain name based on syntax, i.e. without looking it up in DNS?

John Nagle
Jul 23 '07 #5
On 7/23/07, John Nagle wrote:
Here's another hard case. This one might be a bug in urlparse:

import urlparse

s = 'ftp://administrator:pa******@64.105.135.30/originals/6 june
07/ebay/login/ebayisapi.html'

urlparse.urlparse(s)

yields:

(u'ftp', u'administrator:pa******@64.105.135.30', u'/originals/6 june
07/ebay/login/ebayisapi.html', '', '', '')

That second field is supposed to be the "hostport" (per the RFC usage
of the term; Python uses the term "netloc"), and the username/password
should have been parsed and moved to the "username" and "password" fields
of the object. So it looks like urlparse doesn't really understand FTP URLs.
Those values aren't "moved" to the fields; they're extracted on the
fly from the netloc. Use the .hostname property of the result tuple
to get just the hostname.

-Miles
Jul 23 '07 #6
On 7/22/07, John Nagle wrote:
Is there any library function that correctly tests for an IP address vs. a
domain name based on syntax, i.e. without looking it up in DNS?
import re, string

NETLOC_RE = re.compile(r'''^ # start of string
(?:([^@])+@)? # 1:
(?:\[([0-9a-fA-F:]+)\]| # 2: IPv6 addr
([^\[\]:]+)) # 3: IPv4 addr or reg-name
(?::(\d+))? # 4: optional port
$''', re.VERBOSE) # end of string

def normalize_IPv4(netloc):
try: # Assume it's an IP; if it's not, catch the error and return None
host = NETLOC_RE.match(netloc).group(3)
octets = [string.atoi(o, 0) for o in host.split('.')]
assert len(octets) <= 4
for i in range(len(octets), 4):
octets[i-1:] = divmod(octets[i-1], 256**(4-i))
for o in octets: assert o < 256
host = '.'.join(str(o) for o in octets)
except (AssertionError, ValueError, AttributeError): return None
return host
def is_ip(netloc):
if normalize_IPv4(netloc) is None:
match = NETLOC_RE.match(netloc)
# IPv6 validation could be stricter
if match and match.group(2): return True
else: return False
return True

The first function, I'd imagine, is the more interesting of the two.

-Miles
Jul 23 '07 #7
On 7/23/07, Miles wrote:
On 7/22/07, John Nagle wrote:
Is there any library function that correctly tests for an IP address vs. a
domain name based on syntax, i.e. without looking it up in DNS?

import re, string

NETLOC_RE = re.compile(r'''^ # start of string
(?:([^@])+@)? # 1:
(?:\[([0-9a-fA-F:]+)\]| # 2: IPv6 addr
([^\[\]:]+)) # 3: IPv4 addr or reg-name
(?::(\d+))? # 4: optional port
$''', re.VERBOSE) # end of string

def normalize_IPv4(netloc):
try: # Assume it's an IP; if it's not, catch the error and return None
host = NETLOC_RE.match(netloc).group(3)
octets = [string.atoi(o, 0) for o in host.split('.')]
assert len(octets) <= 4
for i in range(len(octets), 4):
octets[i-1:] = divmod(octets[i-1], 256**(4-i))
for o in octets: assert o < 256
host = '.'.join(str(o) for o in octets)
except (AssertionError, ValueError, AttributeError): return None
return host
Apparently this will generally work as well:

import re, socket

NETLOC_RE = ...

def normalize_IPv4(netloc):
try:
host = NETLOC_RE.match(netloc).group(3)
return socket.inet_ntoa(socket.inet_aton(host))
except (AttributeError, socket.error):
return None

Thanks to http://mail.python.org/pipermail/pyt...ly/450317.html

-Miles
Jul 23 '07 #8

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

8
by: Gerrit Holl | last post by:
Posted with permission from the author. I have some comments on this PEP, see the (coming) followup to this message. PEP: 321 Title: Date/Time Parsing and Formatting Version: $Revision: 1.3 $...
2
by: Daniel Kramer | last post by:
Hello, I'm fairly new to python but I've written a script that takes in a special text file (a renderman .rib to be specific).. and filters some of the commands. The .rib file is a simple text...
3
by: Willem Ligtenberg | last post by:
I decided to use SAX to parse my xml file. But the parser crashes on: File "/usr/lib/python2.3/site-packages/_xmlplus/sax/handler.py", line 38, in fatalError raise exception...
9
by: g.kanaka.raju | last post by:
Hi, I wanted to write a function that gets the file path as an argument. The file path make take one of the two forms : (i) Absolute file path e.g., /usr/local/file.log or (ii)it may...
3
by: Natalia DeBow | last post by:
Hi there, I have another question for .NET RegEx experts. I am reading in a C Sharp file line by line and I am trying to detect comments that start with either // of ///. What I am...
1
by: sp | last post by:
i have an xml file (an rss file) <?xml version="1.0" ?> <rss version="2.0"> <channel> <title>CodeGuru.com</title> <link>http://www.codeguru.com/</link> <description>The number one developer...
2
by: aditya.raghunath | last post by:
hi, I'm trying to read text files and then parse them. Some of these files are of several 100 Mbytes or in some cases GBytes. Reading them using the getline function slows down my program a lot,...
15
by: linq936 | last post by:
Hi, I am reading book <<Expert C Programming>>, it has the following quiz, a //* //*/ b In C and C++ compiler what does the above code trun out? I think it is simple for C compiler, it is...
13
by: Chris Carlen | last post by:
Hi: Having completed enough serial driver code for a TMS320F2812 microcontroller to talk to a terminal, I am now trying different approaches to command interpretation. I have a very simple...
0
by: VivesProcSPL | last post by:
Obviously, one of the original purposes of SQL is to make data query processing easy. The language uses many English-like terms and syntax in an effort to make it easy to learn, particularly for...
3
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 3 Jan 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). For other local times, please check World Time Buddy In...
0
by: jianzs | last post by:
Introduction Cloud-native applications are conventionally identified as those designed and nurtured on cloud infrastructure. Such applications, rooted in cloud technologies, skillfully benefit from...
0
by: stefan129 | last post by:
Hey forum members, I'm exploring options for SSL certificates for multiple domains. Has anyone had experience with multi-domain SSL certificates? Any recommendations on reliable providers or specific...
1
by: davi5007 | last post by:
Hi, Basically, I am trying to automate a field named TraceabilityNo into a web page from an access form. I've got the serial held in the variable strSearchString. How can I get this into the...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.