Extract email addresses from big file.

[superc0red]

Hey.

I have a big text file with data,
and i want to extract mail addresses.

How i can do it?

[arne]

Hey.

I have a big text file with data,
and i want to extract mail addresses.

How i can do it?

I guess there are plenty of ways to do it. Any constraints on the tool/language?

[superc0red]

perl / shellscript using awk-sed-cut ??

[arne]

perl / shellscript using awk-sed-cut ??

Perl is certainly a reasonable choice, yes. If I had to do it, I would use it.

[Motoma]

Regular expressions would be a great way to do this. Try looking at the sed tool.

[ghostdog74]

awk '
{
  for (i=1;i<=NF;i++) {
       if ( $i ~ /[[:alpha:]]@[[:alpha:]]/ )  { 
      print $i      
       }
  }
}' "file"
 

[superc0red]

Thanx for the code dude :)

[prn]

It's been quite a while since I did anything with awk, so I wasn't sure how well ghostdog's code would work. It looked like it should handle only alphabetics with no more than one component on each side of the "@". So I made up a test file (test.txt):

this is a test file [e-mail] we are looking for [e-mail] email
addresses inside, [e-mail], a text file with no
particular [e-mail] other [e-mail] restrictions
on the format or locations of the [e-mail] addresses inside the file.
Let's try one at the end [e-mail].
 

I ran ghostdog's awk script on this and got the output:

[e-mail]
[e-mail]
[e-mail],
[e-mail]
[e-mail]
 

Note that this output has FIVE email addresses, but the file has SEVEN so there is something wrong. The two that are omitted have digits just beside the "@" so it looks like I was close but not quite right on how much awk would match with this RE. It catches everything between spaces into $i whenever it matches /[[:alpha:]]@[[:alpha:]]/

But note that it also caught the comma following the third address "[e-mail]," which it should not include in the email address.

Here's a Perl one-liner:

perl -wne'while(/[\w\.]+@[\w\.]+/g){print "$&\n"}' test.txt

This gives the output

[e-mail]
[e-mail]
[e-mail]
[e-mail]
[e-mail]
[e-mail]
[e-mail].

which is almost correct (and does not include the comma following number 3, although it does include the period at the end).

Here's a corrected version:

perl -wne'while(/[\w\.]+@[\w\.]+\w+/g){print "$&\n"}' test.txt

This yields

[e-mail]
[e-mail]
[e-mail]
[e-mail]
[e-mail]
[e-mail]
[e-mail]

I'm sure ghostdog74's awk script could also easily be fixed, but as I said, it's been a long time and I'm not sure how much I want to play with it. ;)

HTH,
Paul

[peripatetic]

Hi.
Thanks for this. I was using it for a while and thought it was wonderful. However it misses the legitimate hyphen character within emails. Here's an updated version.

perl -wne'while(/[\w\.\-]+@[\w\.\-]+\w+/g){print "$&\n"}' emails.txt | sort -u > output.txt

I also piped it through sort to get a sorted, unique list of emails.

[Motoma]

Hi.
Thanks for this. I was using it for a while and thought it was wonderful. However it misses the legitimate hyphen character within emails. Here's an updated version.

perl -wne'while(/[\w\.\-]+@[\w\.\-]+\w+/g){print "$&\n"}' emails.txt | sort -u > output.txt

I also piped it through sort to get a sorted, unique list of emails.

Great catch peripatetic! Thanks for the addition, and welcome to The Scripts!

[HostQ8i]

guys can this perl script be used on websites ? and i replace the file with a web adress ? or how can i do this to get the emails included in a website ?


and let's say i have www.domain.com/aa.php=1 have some emails saved inside
and www.domain.com/aa.php=2 have also some mails .. how can i make a loop to get all the aa.php=variable and get the mails in all the files ?
thanks in advance and sorry for my english

[David Akpan]

I have a big file with many email addresses, how do i extract only the email address, if posible please include the software i can use

[Freakin]

How would I use a script like this on a group of files that are in a directory to retrieve email addresses from all of them?

[gpraghuram]

How would I use a script like this on a group of files that are in a directory to retrieve email addresses from all of them?

Try to combine the find command with xargs and the perl script given here like this.

find . -name "*.txt" | xargs perl <script given here>


Raghu

[RADEP]

I tried the above example but it didn't work for me.

I got the following error:

C:\Documents and Settings\user\Desktop\abc\trunk\docs>perl -wne'while(/[\w\.\-]+@[\w\.\-]+\w+/g){print "$&\n"}'db_em
ails.txt | sort -u > output.txt
Can't find string terminator "'" anywhere before EOF at -e line 1.
-uThe system cannot find the file specified.

[prn]

Hi RADEP,

You are apparently trying to do this in a MS Windows environment rather than a *nix environment. The error you are seeing comes from the Windows command-line parser. I am going to assume (which may get me into trouble) that you entered the offending command on a single line. If so, then the only thing that leaps out at me is the lack of space between the final ' and the apparent input file "db_em". Did you copy and paste your example directly from your command window? If so, the first thing I'd try would be to make sure that you do have a space there.

If that does not help, then we have to look a little further. The prior discussion here has been under a Linux/Unix assumption and the *nix shells do parse command lines differently from Windows. The Perl itself should be OK, especially with peripatetic's modification. However, you may have to run it differently. If the windows command-line parser can't handle this as a one-liner, you can always just put the Perl into a file (e.g., "ExtractEmail.pl") and then you should be able to run it that way as:

C:\...>perl ExtractEmail.pl <db_em

or the like.

Let us know if this meets your needs.

Paul

[smcgimpsey]

Hey Paul,

I'm in the same boat as RADEP. I'm very much able to use the script on my linux machine, but unable to once I try it on my windows vm.

I tried copying the code verbatim into a .pl file and running it from command line per your suggestion with a similar output to RADEP's experience.

As for the "'" terminator, I have no clue, but I am going to guess that windows will not support the 'sort -u' command near the end. What are your thoughts?

And by the way, thanks for everyone's help in this. It's forums like these that help me get through the work day. :)

Cheers,
Scott

[prn]

Hi Scott,

I'm afraid I was too lazy the other day.

If you're going to create a file to do the same job, you have to do the read from STDIN explicitly. So the file ExtractEmail.pl could look like this:

while (<STDIN>) {
    while (/[\w\.\-]+@[\w\.\-]+\w+/g)
        {print "$&\n"}
}

Then you can invoke it like this:

C:\...>perl ExtractEmail.pl <test.txt >out.txt

Of course, the sorting as in a *nix environment is not available in the native windows environment. There are several ways to get the capability. You could install cygwin, which is a port of the bash shell with utilities including sort. (Then you should be able to use the original one-liner.) If you search for "windows unix sort" you should find some advice (which I have not tested) on other ports of the sort utility.

HTH,
Paul

[mbstrlbstr]

I keep getting this error:

syntax error at email.pl line 1, near "){"
Can't find string terminator "'" anywhere before EOF at email.pl line 1

and I'm using this code:

perl -wne'while(/[\w\.\-]+@[\w\.\-]+\w+/g){print "$&\n"}' emails.txt | sort -u > output.txt

[Exenfris]

How about this ...

if for some reason the file lost some spaces or got extra letters and there's this case ....

[e-mail]@ <--- or what about this ... [e-mail]

in those cases i want to create ...
[e-mail] and also the one it catch [e-mail] <-- notice doesn't have the @ at the end
and fix [e-mail] ( which is certainly a public email provider ) but just in case we want to keep that one we found
[e-mail]
and add
[e-mail]

I had many years ago a code that just to do that I'm going to try to find it but if you have a regular expression or short code that can fix that it will be wonderful !

Regards
Angelo

[Exenfris]

Also what about emails like ...

[e-mail] ?

this is very interesting i love it !

[Halle]

Hello,

This one-liner is great! I would love to exclude all email addresses ending in one specific domain from the output (e.g. don't print any address ending with thisdomain.com), is there an easy way to do that? Thanks!

[Obi Wan]

Just pipe the output to e/f/grep with the "invert" option to exclude specific domains e.g. grep -v @DOMAIN

[carl win]

Hi,

Great code. Unfortunately,

perl -wne'while(/[\w\.\-]+@[\w\.\-]+\w+/g){print "$&\n"}' emails.txt | sort -u > output.txt

catches:

[e-mail]...User

Can someone correct this???

Many thanks.

[MaGiK]

Hi,
I know its been a while since last post, but I couldn't just leave it with out an answer.
I did same small upgrades making the code better match emails.
I believe this is what you all were looking for:

perl -ne'if(/[\w\.\-\_]+@([\w\-\_]+\.)+[A-Za-z]{2,4}/g){print "$&\n"}' *.txt
 

Have a nice mail extraction!

[prashantva]

$ egrep -o [A-Za-z0-9_.]*'[-]'*[A-Za-z0-9_.]*@[A-Za-z0-9_.]*[.][A-Za-z]* < filename

This should work in any Unix version. This should take care of all possibilities.

[prn]

I haven't really revisited this topic for a while now, but here are a few comments.

TIMTOWTDI, prashantva. :) That is, "There Is More Than One Way To Do It." Your way appears to be almost equivalent to what we had before. The only difference I note will show up in the case of more than one hyphen in a name. Check out the result from the "j-q-public" entry in the output of the perl program below versus the egrep re. Some experimentation with the re should fix that.

I'm not quite sure what Exenfris wants, but it looks like a database of valid domains. I don't think that's practical in a short program, certianly not in a one-liner. I suppose you could add a piece that checks whois for each domain. I'll happily leave that as an exercise for someone else. :) I can imagine adding a routine to check for valid domains to the perl script, but personally, I'm not prepared to try to correct those that turn out to be invalid.

Halle asks for a way to exclude specific domains. Obi Wan suggests just piping the output through grep using "-v". That would work, but how convenient it would be depends a lot on how many domains you want to exclude and how often you want to run the search. Personally, when I start needing more features, I like to document them by including them into a single file, so I've added this feature into the perl script below. (Obviously, a shell script could work equally well.) For convenience the exclude pattern is prominently defined using the regex quote operator.

Carl win points out a problem when an address is followed/terminated by multiple dots. I've included a fix for this below too. I haven't seen a way to solve this one in a single re, but someone clever may be able to do that.

File: ExtractEmail.pl

#! /usr/bin/perl
use strict;
 
#exclude the following domains:
my $exclude = qr/thisdomain\.com|otherdomain\.com/o;
 
# We need a variable here because we cannot assign to $1
my $address;
 
while (<STDIN>) {
  while ( /([\w\.\-]+@[\w+\.\-]+\.[A-Za-z]{2,4})\W/g ) {
    $address = $1;              # use a variable so we can modify if needed
    $address =~ s/\.\..*// ;    # terminate the address at multiple "."s
    print "$address\n" unless $address =~ $exclude;  
  }
}
exit;
 

And here's a test file that should cover all the cases that have been discussed above.
File: emailtest.txt

this is a test file [e-mail] we are looking for [e-mail] email
addresses inside, [e-mail], a text file with no
particular [e-mail] other [e-mail] restrictions
on the format or locations of the [e-mail] addresses inside the
file. And, let's not forget addresses with hyphens like [e-mail] or
underscores like [e-mail]
Also what about emails like [e-mail]?
Let's also say we want to exclude addresses like [e-mail] from a
specific domain.
Unfortunately, the simpler regex catches: [e-mail]...User so what do we do?
But it should not have a problem with: [e-mail] ... User. right?
Trailing stuff with hyphens or something else after the end of the actual address 
should be ruled out automatically as in [e-mail]--bad, right?
Let's also exclude stuff from otherdomain as in [e-mail], OK?
This version also rules out TLDs that are too long as in [e-mail].
Let's try one at the end [e-mail].
And one at the very end without a final newline [e-mail]

To test this script, you can use

ExtractEmail.pl <emailtest.txt

I have tested the script with this data both in a command prompt window on MSWindows and in Linux.

To test prashantva's egrep version for comparison, you can use

egrep -o [A-Za-z0-9_.]*'[-]'*[A-Za-z0-9_.]*@[A-Za-z0-9_.]*[.][A-Za-z]*  <emailtest.txt

Paul

[sun123]

for i in `cat filename`
do
echo $i >> filename1
done
grep "@" filename

The above code will extract all the email Ids

[prn]

Let's adjust sun123's script to:

for i in `cat emailtest.txt`
  do
    echo $i >> filename1
  done
grep "@" filename1
 

This now uses the name of that we have been using for the original input and the file to be grepped in the last line has been corrected from "filename" to "filename1", i.e., the output from the cat instead of the original input.

The "for" loop simply writes "words" from the original file to separate lines in an intermediate file.

sun123's corrected script can alternatively be simplified by eliminating the unnecessary temporary file:

for i in `cat emailtest.txt`
 do
   echo $i | grep "@"
 done
 

The output from either version is:

[e-mail]
[e-mail]
[e-mail],
[e-mail]
[e-mail]
[e-mail]
[e-mail]
[e-mail]
[e-mail]?
[e-mail]
[e-mail]...User
[e-mail]
[e-mail]--bad,
[e-mail],
[e-mail].
[e-mail].
[e-mail]
 

Note that the result incorrectly includes trailing punctuation, and not just dots, and fails to exclude [e-mail] (where "topleveldomain" is not a valid top level domain, being too long). Obviously further filters could be added, but this one is not there yet.

Best Regards,
Paul

[mdnuts]

Excellent script
I found one error if an email address has two consecutive dots in it. such as:

[e-mail] - will display as Paul.A

I'm using this with sorting and uniq like this.

perl extractemail.pl < myfile | sort | uniq > emaillist.txt

performing a quick bash SED like this

sed -i -e 's/\.\./\./g' myfile

followed by the

perl extractemail.pl < myfile | sort | uniq > emaillist.txt

works great