Find urls in plain text files

[Salve =?iso-8859-1?Q?H=E5kedal?=]

What is the best regular expression for finding urls in plain text
files?
(By urls I mean http://www.something.com, but also www.something.com,
or sa***@somewhere.com)

Salve

[Chris Gorospe]

Salve Håkedal wrote:

What is the best regular expression for finding urls in plain text
files?
(By urls I mean http://www.something.com, but also www.something.com,
or sa***@somewhere.com)

Salve

The simplest way is to use the one thing they all have in common. ".com".

strstr($text,'.com');

[Christoph Burschka]

Salve Håkedal wrote:

What is the best regular expression for finding urls in plain text
files?
(By urls I mean http://www.something.com, but also www.something.com,
or sa***@somewhere.com)

Salve

I've used this before, but you're probably better off making your own
expression. Note that it's really loose and will get a lot of false positives -
especially file names - and it will cause havoc if you use it on HTML source. I
deliberately did not enter any Top-Level-Domain filtering, because there are so
many of them. You can replace the [a-z]{2,5} with something like (com|net|org)
if you don't need to worry about country codes.

The following expression should find strings that satisfy these conditions:

- optionally a http protocol identifier
- optionally a username(:password)@ string, which allows pretty much any
characters except for spaces and colons. This isn't RFC-standard, by the way.
- a hostname consisting of at least two and at most 34 labels, the last of which
has 2 to 5 alphabet letters (for weird new ones like aero and museum; you can
shorten it to 3 and still get the most common ones).
- optionally a path containing any characters apart from spaces, and /ending in
a non-punctuation character/. This last bit is vital because it avoids messing
up URLs at the end of a sentence.

(http:\/\/)?([^ :]+(:[^
]+)?@)?[a-z0-9]([a-z0-9i\-]{0,61}[a-z0-9])?(\.[a-z0-9]([a-z0-9\-]{0,61}[a-z0-9])?){0,32}\.[a-z]{2,5}(\/[^
]*[^" \.,;\)])?

(linebreaks are added by email client)

This is a case insensitive pattern, you'll need the i modifier.

--
Christoph Burschka

[Salve =?iso-8859-1?Q?H=E5kedal?=]

On 2007-11-02, Christoph Burschka <ch****************@rwth-aachen.dewrote:

Salve Håkedal wrote:

>What is the best regular expression for finding urls in plain text
files?
(By urls I mean http://www.something.com, but also www.something.com,
or sa***@somewhere.com)

Salve

I've used this before, but you're probably better off making your own
expression. Note that it's really loose and will get a lot of false positives -
especially file names - and it will cause havoc if you use it on HTML source. I
deliberately did not enter any Top-Level-Domain filtering, because there are so
many of them. You can replace the [a-z]{2,5} with something like (com|net|org)
if you don't need to worry about country codes.

The following expression should find strings that satisfy these conditions:

- optionally a http protocol identifier
- optionally a username(:password)@ string, which allows pretty much any
characters except for spaces and colons. This isn't RFC-standard, by the way.
- a hostname consisting of at least two and at most 34 labels, the last of which
has 2 to 5 alphabet letters (for weird new ones like aero and museum; you can
shorten it to 3 and still get the most common ones).
- optionally a path containing any characters apart from spaces, and /ending in
a non-punctuation character/. This last bit is vital because it avoids messing
up URLs at the end of a sentence.

(http:\/\/)?([^ :]+(:[^
]+)?@)?[a-z0-9]([a-z0-9i\-]{0,61}[a-z0-9])?(\.[a-z0-9]([a-z0-9\-]{0,61}[a-z0-9])?){0,32}\.[a-z]{2,5}(\/[^
]*[^" \.,;\)])?

(linebreaks are added by email client)

This is a case insensitive pattern, you'll need the i modifier.

--
Christoph Burschka

Thanks alot! I'll study this closely

--
Salve

[Michael Fesser]

..oO(Chris Gorospe)

>Salve Håkedal wrote:

>What is the best regular expression for finding urls in plain text
files?
(By urls I mean http://www.something.com, but also www.something.com,
or sa***@somewhere.com)

The simplest way is to use the one thing they all have in common. ".com".

strstr($text,'.com');

What about the other TLDs? There are _some_ more ...

Micha

[Sanders Kaufman]

"Salve Håkedal" <ik*************@slogedalen.nowrote in message
news:2-*********************@telenor.com...

On 2007-11-02, Christoph Burschka <ch****************@rwth-aachen.de>
wrote:

>The following expression should find strings that satisfy these
conditions:

- optionally a http protocol identifier
- optionally a username(:password)@ string, which allows pretty much any
characters except for spaces and colons. This isn't RFC-standard, by the
way.
- a hostname consisting of at least two and at most 34 labels, the last
of which
has 2 to 5 alphabet letters (for weird new ones like aero and museum; you
can
shorten it to 3 and still get the most common ones).
- optionally a path containing any characters apart from spaces, and
/ending in
a non-punctuation character/. This last bit is vital because it avoids
messing
up URLs at the end of a sentence.

(http:\/\/)?([^ :]+(:[^
]+)?@)?[a-z0-9]([a-z0-9i\-]{0,61}[a-z0-9])?(\.[a-z0-9]([a-z0-9\-]{0,61}[a-z0-9])?){0,32}\.[a-z]{2,5}(\/[^
]*[^" \.,;\)])?

(linebreaks are added by email client)

This is a case insensitive pattern, you'll need the i modifier.

--
Christoph Burschka

Thanks alot! I'll study this closely

Does it make your eyes and ears bleed,the way it does mine?
Been doin this stuff since the 70's - but regex still makes me cry.

On that note, I am at the moment, writing a function that could sure benefit
from some regex.
I just want to see if a string starts with "http(s)://", "news:", "mailto:",
"ftp:".
That's a pretty simple regex, right?
Hoooowwww?

[Carl]

"Sanders Kaufman" <bu***@kaufman.netwrites:

Does it make your eyes and ears bleed,the way it does mine?
Been doin this stuff since the 70's - but regex still makes me cry.

On that note, I am at the moment, writing a function that could sure benefit
from some regex.
I just want to see if a string starts with "http(s)://", "news:", "mailto:",
"ftp:".
That's a pretty simple regex, right?
Hoooowwww?

This should get you started:

$pattern = '/^(http(s)?:\/\/|news:|mailto:|ftp:)/';

$tests = array('http://www.google.com', 'https://www.google.com',
'news:comp.lang', 'mailto:te**@nowhere.com',
' http://www.google.com',
'bad_http://www.google.com',
'mailtobad:fdsa', 'ftp://ftp.host.net',
'ftpbad:', 'badftp://');

foreach($tests as $v) {
print "'".$v."'" . ' ~ ' .
(preg_match($pattern, $v)
? 'matches'
: 'doesn\'t match')."\n";
}

[Salve =?iso-8859-1?Q?H=E5kedal?=]

On 2007-11-02, Carl <c.******@gmail.comwrote:

"Sanders Kaufman" <bu***@kaufman.netwrites:

>Does it make your eyes and ears bleed,the way it does mine?
Been doin this stuff since the 70's - but regex still makes me cry.

On that note, I am at the moment, writing a function that could sure benefit
from some regex.
I just want to see if a string starts with "http(s)://", "news:", "mailto:",
"ftp:".
That's a pretty simple regex, right?
Hoooowwww?

This should get you started:

$pattern = '/^(http(s)?:\/\/|news:|mailto:|ftp:)/';

$tests = array('http://www.google.com', 'https://www.google.com',
'news:comp.lang', 'mailto:te**@nowhere.com',
' http://www.google.com',
'bad_http://www.google.com',
'mailtobad:fdsa', 'ftp://ftp.host.net',
'ftpbad:', 'badftp://');

foreach($tests as $v) {
print "'".$v."'" . ' ~ ' .
(preg_match($pattern, $v)
? 'matches'
: 'doesn\'t match')."\n";
}

Thank you, Carl, for the script.

But the regexp there is as simple at I could have written myself. What
I need is something that can find urls in a text file, and convert them
to links. And by urls I mean, as I wrote in OT: http://something.org as
well as for example www.someother.anytopdm and the url in the original
text may be in parantheses or for example at the end of a sentence, so
it'll have a . in the end. So on..

Christoph Burschka's is still the most promising, but I haven't had time
to understand and to try it out yet.

Salve

[Carl]

Salve Håkedal <ik*************@slogedalen.nowrites:

On 2007-11-02, Carl <c.******@gmail.comwrote:

>"Sanders Kaufman" <bu***@kaufman.netwrites:

>>Does it make your eyes and ears bleed,the way it does mine?
Been doin this stuff since the 70's - but regex still makes me cry.

On that note, I am at the moment, writing a function that could sure benefit
from some regex.
I just want to see if a string starts with "http(s)://", "news:", "mailto:",
"ftp:".
That's a pretty simple regex, right?
Hoooowwww?

This should get you started:

$pattern = '/^(http(s)?:\/\/|news:|mailto:|ftp:)/';

$tests = array('http://www.google.com', 'https://www.google.com',
'news:comp.lang', 'mailto:te**@nowhere.com',
' http://www.google.com',
'bad_http://www.google.com',
'mailtobad:fdsa', 'ftp://ftp.host.net',
'ftpbad:', 'badftp://');

foreach($tests as $v) {
print "'".$v."'" . ' ~ ' .
(preg_match($pattern, $v)
? 'matches'
: 'doesn\'t match')."\n";
}

Thank you, Carl, for the script.

But the regexp there is as simple at I could have written myself. What
I need ...

--8<-- message cut -->8--
Salve,

My response was a followup to Sanders, not your O.P., I assumed that
the previous posts answered you question already.

The question you posted is quite common, and google'n should turn
up enough examples that you should'nt have to do much to get it working
decently enough.

For starters, this looks promising, though I haven't tested it (from
the 1st results page of a google search)

http://immike.net/blog/2007/04/06/5-...r-should-know/

--
Hope that helps,
Carl.

[John Dunlop]

Salve Håkedal:

What is the best regular expression for finding urls in plain text
files?

Matching URLs of every scheme with a single regular expression would
be
incredibly complex and complicated. For example, matching mailto URLs
would entail the notorious regular expression for matching e-mail
addresses. Even matching URLs of individual schemes would require
careful study of both RFC3986 and the specification that governs that
particular URL scheme. Moreover, if you want to turn partial URLs
such as www.example.com into complete URLs, you would need to define
your own heuristics for doing so.

Upshot is, you're in a world of hurt.

--
Jock