Monitor a folder. no activity for 1 hour, send an email

[jonathan184]

Hi I am trying to do a script to monitor a dir where files pass through approx every 5 mins but sometimes files may not come in for hours

So the purpose of the script is to monitor by if no files pass through in 1 hour send an email.

So was thinking to use the shell command to grab the time when a file comes in and write to a txtfile.
Now when the files keep coming it will overwrite the old time.
Now if no new files are coming in an hour i got the old time but i having trouble calculating the hour so to trigger out to send an email

this is what i came up with so far

#!/usr/bin/perl
#Purpose to monitor a folder
 
my $IN = "TIMEFILE1";
my $IN2 = "TEST";
 
my $srcdir = "/home/tibco/javastuff/test";
my $tfile = "/home/tibco/scripts/time.txt";
my $tfile2 = "/home/tibco/scripts/time2.txt";
 
open (OUT ,">$tfile") || "Could not open file $!";
open (FILE, ls -lt | awk -F" " '{print $8}');
while (<FILE>) {
    #print $_;
    print OUT "$_";
    #print $_;
}
close OUT;
close FILE;
###### get system time
open (SYSTEM ,">$tfile2") || "Could not open file $!";
open (FILE2, date | awk -F" " '{print $4}');
while (<FILE2>){
    #print $_;
    print OUT "$_";
    #print $_;
}
close SYSTEM;
close FILE2;
 
#END

[miller]

Try something like the following. If the script is going to be running continuously then there is no reason to write information to a file. Just keep it in a variable. Also, it's not even really necessary to find the datetime value of a file, you just must mark the time close to when the file appears. Anyway, take a look at this psuedocode and see if it will work for your purposes.

#!/usr/bin/perl
 
use strict;
 
use constant WAIT => 10; # In Seconds
use constant TIMEOUT => 1 * 3600; # In Seconds
 
my $lastSeen = time;
my $warningMode = 0;
 
for (;;) {
    if (seeFileInDirectory()) {
        $lastSeen = time;
        $warningMode = 0;
 
    } elsif (! $warningMode && time > $lastSeen + TIMEOUT) {
        sendEmailToAdmin();
        $warningMode = 1;
    }
 
    sleep WAIT;
}
 

- Miller

[jonathan184]

Hi Miller

The script that does the transfer does not actually run continuously. even thouight its close. BAsically the script that does the transfer is a shel script that has a cron job that runs every sec. I wanted to make a call to the perl script script to do the monitoring part, would i still be able to do it with your solution?

[prn]

I'm not miller, but I think I can safely answer that miller's script is almost certainly a better solution than a cron job that fires off every second. If you're going to run it that frequently, you might as well just run it once and let it stay put. For one thing, if you restart the job once per second, you really can't rely on scheduling at that level of granularity. When I look at logs for various of my standing cron jobs I often see 1-5 seconds worth of variation. Beyond that, remember that every time you start the job, it will have to go out to disk and reload the script, as well as the perl interpreter/compiler from disk. If you load it once and let it loop and sleep, then you eliminate almost all of that excess disk activity. You also eliminate 100% of the disk writes from your script by keeping the times in a local variable rather than a file. Beyond that, miller's script is just so much simpler and more straightforward than yours.

Just start up miller's script (or a close equivalent) and let it run all the time. Drop it from your crontab and put it in your init scripts (or inittab). It's really a LOT better that way.

Best Regards,
Paul

[jonathan184]

Thanks for the advice prn, i guess what you said did make sense.

Anyways I am trying to use Miller's script but i having some problems in getting it to work.

This is my code for the subroutines , the synax seems ok but i am not sure where i am going wrong here?
I am trying to get the print statemtns to show up but I have no luck in doing this.
If you guys could tell me where i am going wrong or what to try, i would appreciat eit.

Thanks for helping so far.

#!/usr/bin/perl
 
use File::Copy;
 
use constant WAIT => 10; # In Seconds
use constant TIMEOUT => 1 * 3600; # In Seconds
#### Dir to monitor ####
my $srcdir = "d:\\monitor";
 
### Email header/message
my $emailheader = "No DWH files sent int he last hour";
my $emailmsg = "No files were sent int he last hour.";
 
 
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime();
$time = sprintf("%02d%02d", $hour, $min);
 
#the time
print $time;
 
my $lastSeen = $time;
my $warningMode = 0;
 
for (;;) {
    if (seeFileInDirectory()) {
        my $lastSeen = sprintf("%02d%02d", $hour, $min);
        my $warningMode = 0;
    }
    elsif (! $warningMode && $time > $lastSeen + TIMEOUT) {
        sendEmailToAdmin();
        $warningMode = 1;
    }
 
    sleep WAIT;
}
 
sub seeFileInDirectory {
    opendir(DIR, "srcdir") || "Cannot open the dir $!";
    my @files = grep{!/^\.+$/} readdir(DIR);
    close(DIR);
    print "@files";
    close(DIR);
 
    foreach $files(@files) {
        if ($files =~ /^DWH/) {
            print "DWH file found";
            return 0;
        }
        else {
            return 1;
            print "No files found";
        }
    }
}
 
sub sendEmailToAdmin {
    system "mail -s $emailheader test\@domain.com <<EOF $emailmsg EOF";
}
 

[miller]

Hello Jonathan,

Well, first of all don't change provided code for no reason. You don't need or want to format the time. And you've also localized (used my) on the variables within the if statement causing a bug. I'll remove that stuff for you now.

Concerning your own code. You don't need to use \\ in windows file systems, perl will also use / correctly. The seeFileInDirectory() sub should return TRUE for when there are files in the directory. You currently have this logic inverted. Printing after a return statement in a sub won't print. And finally your logic for no files was incorrect when it was included in the for loop.

#!/usr/bin/perl
 
use File::Copy;
 
use strict;
 
use constant WAIT => 10; # In Seconds
use constant TIMEOUT => 1 * 3600; # In Seconds
 
#### Dir to monitor ####
my $srcdir = "d:/monitor";
 
### Email header/message
my $emailheader = "No DWH files sent in the last hour";
my $emailmsg = "No files were sent in the last hour.";
 
my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime();
$time = sprintf("%02d%02d", $hour, $min);
 
#the time
print $time;
 
my $lastSeen = time;
my $warningMode = 0;
 
for (;;) {
    if (seeFileInDirectory()) {
        $lastSeen = time;
        $warningMode = 0;
    }
    elsif (! $warningMode && time > $lastSeen + TIMEOUT) {
        sendEmailToAdmin();
        $warningMode = 1;
    }
 
    sleep WAIT;
}
 
sub seeFileInDirectory {
    opendir(DIR, $srcdir) or "Cant open $sirdir: $!";
    my @files = grep{!/^\.+$/} readdir(DIR);
    close(DIR);
    print "@files";
    close(DIR);
 
    foreach my $file (@files) {
        if ($file =~ /^DWH/) {
            print "DWH file found";
            return 1;
        }
    }
 
    print "No files found";
    return 0;
}
 
sub sendEmailToAdmin {
    system "mail -s $emailheader test\@domain.com <<EOF $emailmsg EOF";
}
 

- Miller

[jonathan184]

Hi Miller

thanks for explaining it to me , i got what you showed me there. I just got one last question that is puzzling me , the value you are giving as time;

It is not a variable but how does it get the time there, that is why i changed it i was not sure how that would work, If you could tell me how that works or refer me to a doc about that i would be very greatful and thanks again for all your help. I have learned alot from the forums.

keep up the good work.

Thanks
Jonathan

[miller]

perldoc time

- Miller

[KevinADC]

this line:

opendir(DIR, "srcdir") || "Cannot open the dir $!";

should be:

opendir(DIR, "$srcdir") || "Cannot open the dir $!";

[miller]

Thanks Kev,

I've edited the code to take that into account.

- M

[jonathan184]

thank you for your help guys.

[KevinADC]

Personally I don't see how the code will know if a file was new in the dir within the last hour. All it seems to do is check if there is a file with DWH in the filename.