Perl FTP issue

[psutoad]

I am somewhat a Perl Noob, but I've done some programming with my animal books at my side. This issue, however doesn't make sense to me. I am grabbing data via FTP corresponding to data I've already downloaded (this is a pre-processing front end to some other fortran code). However, in the part indicated below (the second FTP execution), I get a login failure, even both ftp sites allow anonymous FTP.

Any thoughts or suggestions on how I might run this down would be much appreciated.

Sincerely,
Todd

Code pasted below
----------------------------

#!/usr/bin/perl

use Net::FTP;

$ftp_username = 'anonymous';
$ftp_passwd = 'BLOCKED - this would normally be my email'

my ($tsec, $tmin, $thour, $mday, $mmon, $yyear, $wday, $yday, $isdst) = localtime time;
$yyear += 1900;

#initialize arrays with strings

@month_str = ('01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12');
@day_str = ('01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19', '20', '21', '22', '23', '24', '25', '26', '27', '28', '29', '30', '31');

# set local AMSU data directory

$AMSUdir = 'data/2005';

chdir $AMSUdir or die "cannot change dir to $AMSU_dir: $!";

@AMSUfiles = glob "*.eos";

# loop through each file to grab relevant ancillary data

foreach $AMSU (@AMSUfiles) {

# do not grab more data if this swath is in same day as last swath

if ($AMSU ne $hold_name) {

# capture the year and the doy

if ($AMSU =~ /\w+(\d{4})\.(\d+)\w\d+\w\d+\w\d+\w+\.eos/) {

$year = $1*1;
$doy = $2;

# convert dayofyear to date

if ($year % 4 eq 0) {
@days_by_month = (31, 60, 91, 121, 152, 182, 213, 244, 274, 305, 335, 366);
} else {
@days_by_month = (31, 59, 90, 120, 151, 181, 212, 243, 273, 304, 334, 365);
}

$month = 1;
$dd = $2*1;

foreach $dm (@days_by_month) {
if ($2 <= $dm) {
$day = $dd;
} else {
$dd = $2 - $dm;
$month++;
}

}

$mo_string = $month_str[$month-1];
$day_string = $day_str[$day-1];

# get the SSMI data

$ssmi_dir = "/ssmi/f15/bmaps_v06/y$year/m$mo_string/";
$ssmi_filename = "f15_$year$mo_string$day_stringv6.gz";

$ssmi_site = 'ftp.ssmi.com';
$ftp_obj = Net::FTP -> new ($ssmi_site, Timeout => 20) or
die "Cannot access $ssmi_site via FTP\n";

$ftp_obj -> login($ftp_username, $ftp_passwd) or
die "Invalid user name and/or password SSMI\n";

$ftp_obj -> binary;

$ftp_obj -> cwd ("$ssmi_dir") or
die "Cannot access directory $ssmi_dir\n";

$ftp_obj -> get("$ssmi_filename");

$ftp_obj -> quit;

# get the SST data

$sst_dir = "/sst/daily/tmi_amsre/";
$sst_filename = "tmi_amsre.fusion.$year.$doy.v02.gz";

$sst_site = 'ftp.discover-earth.org';
$ftp_obj -> new ($sst_site, Timeout=>20) or
die "Cannot access $sst_site via FTP\n";

# this next line throws out an invalid login error

$ftp_obj -> login($ftp_username, $ftp_passwd) or
die "Invalid user name and/or password SST\n";

$ftp_obj -> binary;

$ftp_obj -> cwd ("$sst_dir") or
die "Cannot access directory $sst_dir\n";

$ftp_obj -> get("$sst_filename");

$ftp_obj -> quit;

}
}

# set this string for next time through

$hold_name = $AMSU;

# write out the relevant data

print "$year $doy $AMSU $ssmi_filename $sst_filename\n";

}

# make a new data for all data

$data_dir = "data_$thour$tmin_$yday_$yyear";
mkdir $data_dir, 0777;

# move all the data

foreach my $file (glob "*.eos") {

my $newfile = "$data_dir/$file";
rename $file, $newfile;

}

foreach my $file (glob "*.gz") {

my $newfile = "$data_dir/$file";
rename $file, $newfile;

}

[GunnarH]

# this next line throws out an invalid login error

$ftp_obj -> login($ftp_username, $ftp_passwd) or
die "Invalid user name and/or password SST\n";

Then ask Perl why it failed. See in the Net::FTP docs (the SYNOPSIS section) how it can be done.

[psutoad]

Then ask Perl why it failed. See in the Net::FTP docs (the SYNOPSIS section) how it can be done.

That's not helpful - nothing in the synopsis tells me anything more than I already know - that I am unable to log in during the second ftp event. Adding the $ftp_obj -> message to this step returns "connection closed in line 83".

Like I said, I'm relatively new to perl, but I've been programming for 20 years...


So is this something about the FTP package? Something with the use of the variables? Something else? I've been trying stuff for 4 hours.

-T

[miller]

psutoad. I would like to help you, but I must first pick on one thing with regard to your perl programming practices. It is very important that you include the line "use strict;" at the beginning of any perl script that you write. This will force you to always declare your variables with "my", and in this way encourage you to think about the scope of your variables, instead of treating everything as global. Additionally, it as the benefit of helping you find spelling errors immediately, of which you currently have 4 in your script.

Take the below code that I spent 10 minutes reformatting for you. There are a lot more ways that I can clean up the code, but I woudl like you to discover the spelling errors on your own so that you can fix them. After you've done that, I will continue trying to help you debug your problem.

#!/usr/bin/perl
 
use Net::FTP;
 
use strict;
 
my $ftp_username = 'anonymous';
my $ftp_passwd   = 'BLOCKED - this would normally be my email';
 
my ( $tsec, $tmin, $thour, $mday, $mmon, $yyear, $wday, $yday, $isdst ) =
  localtime time;
$yyear += 1900;
 
# set local AMSU data directory
my $AMSUdir = 'data/2005';
 
chdir $AMSUdir or die "cannot change dir to $AMSU_dir: $!";
 
my @AMSUfiles = glob "*.eos";
 
# loop through each file to grab relevant ancillary data
my $hold_name = '';
 
foreach my $AMSU (@AMSUfiles) {
 
    my $year = '';
    my $doy = '';
    my $ssmi_filename = '';
    my $sst_filename = '';
 
    # do not grab more data if this swath is in same day as last swath
 
    if ( $AMSU ne $hold_name ) {
 
        # capture the year and the doy
 
        if ( $AMSU =~ /\w+(\d{4})\.(\d+)\w\d+\w\d+\w\d+\w+\.eos/ ) {
 
            my $year = $1 * 1;
            my $doy  = $2;
 
            # convert dayofyear to date
 
            my @days_by_month = ( $year % 4 eq 0 )
                ? ( 31, 60, 91, 121, 152, 182, 213, 244, 274, 305, 335, 366 )
                : ( 31, 59, 90, 120, 151, 181, 212, 243, 273, 304, 334, 365 );
 
            my $month = 1;
            my $dd    = $2 * 1;
 
            my $day;
            foreach my $dm (@days_by_month) {
                if ( $2 <= $dm ) {
                    $day = $dd;
                }
                else {
                    $dd = $2 - $dm;
                    $month++;
                }
 
            }
 
            my $mo_string  = sprintf "%02s", $month;
            my $day_string = sprintf "%02s", $day;
 
            # get the SSMI data
 
            my $ssmi_dir      = "/ssmi/f15/bmaps_v06/y$year/m$mo_string/";
            my $ssmi_filename = "f15_$year$mo_string$day_stringv6.gz";
 
            my $ssmi_site = 'ftp.ssmi.com';
            my $ftp_obj = Net::FTP->new( $ssmi_site, Timeout => 20 )
              or die "Cannot access $ssmi_site via FTP\n";
 
            $ftp_obj->login( $ftp_username, $ftp_passwd )
              or die "Invalid user name and/or password SSMI\n";
 
            $ftp_obj->binary;
 
            $ftp_obj->cwd("$ssmi_dir")
              or die "Cannot access directory $ssmi_dir\n";
 
            $ftp_obj->get("$ssmi_filename");
 
            $ftp_obj->quit;
 
            # get the SST data
 
            my $sst_dir      = "/sst/daily/tmi_amsre/";
            my $sst_filename = "tmi_amsre.fusion.$year.$doy.v02.gz";
 
            my $sst_site = 'ftp.discover-earth.org';
            $ftp_obj->new( $sst_site, Timeout => 20 )
              or die "Cannot access $sst_site via FTP\n";
 
            # this next line throws out an invalid login error
 
            $ftp_obj->login( $ftp_username, $ftp_passwd )
              or die "Invalid user name and/or password SST\n";
 
            $ftp_obj->binary;
 
            $ftp_obj->cwd("$sst_dir")
              or die "Cannot access directory $sst_dir\n";
 
            $ftp_obj->get("$sst_filename");
 
            $ftp_obj->quit;
 
        }
    }
 
    # set this string for next time through
 
    $hold_name = $AMSU;
 
    # write out the relevant data
 
    print "$year $doy $AMSU $ssmi_filename $sst_filename\n";
 
}
 
# make a new data for all data
 
my $data_dir = "data_$thour$tmin_$yday_$yyear";
mkdir $data_dir, 0777;
 
# move all the data
 
foreach my $file ( glob "*.eos" ) {
    my $newfile = "$data_dir/$file";
    rename $file, $newfile;
}
 
foreach my $file ( glob "*.gz" ) {
    my $newfile = "$data_dir/$file";
    rename $file, $newfile;
}
 

Good night for now.

Note. If you want to explicitly specify a variable in a interpolated string, then you should include it in brackets. "my ${var}is crowded". That reads the variable $var, where as "my $varis crowded" reads the varaible $varis.

Just repost the fixed code, and I will continue helping you as soon as I wake up.

[psutoad]

I would like to help you, but I must first pick on one thing with regard to your perl programming practices. It is very important that you include the line "use strict;" at the beginning of any perl script that you write. This will force you to always declare your variables with "my", and in this way encourage you to think about the scope of your variables, instead of treating everything as global. Additionally, it as the benefit of helping you find spelling errors immediately, of which you currently have 4 in your script.

Take the below code that I spent 10 minutes reformatting for you. There are a lot more ways that I can clean up the code, but I woudl like you to discover the spelling errors on your own so that you can fix them. After you've done that, I will continue trying to help you debug your problem.

Wow, I really appreciate not only the effort, but this commenton 'use strict'. In my Perl books they talk about this in later chapters, but it didn't seem necessary to me and I'd never used the strict command in any scripts I've made over the past 10 years. I will use it from now on, though I really don't understand what purpose it serves other than to find spelling errors.. I will read more on this...

So this is my reworked code - I incorported most of your code reformatting, but not all. There's also a slightly different reordering of commands and a slightly different FTP block. But, the problem remains. Whenever I try to open a second ftp connection within the same loop, it fails every time. I've tried several different combinations... I've changed the $ftp_obj variable to a different one for each FTP block, I've tried troubleshooting every variable by printing them between every command and they all look ok.

Again many thanks for the effort so far, it was more than I expected. I just can't get my head around what I'm doing wrong with these FTP calls.. any continued help is greatly appreciated.

#!/usr/bin/perl
 
use Net::FTP;
use strict;
 
#### USER INPUT PARAMETERS
 
my $ftp_passwd = '[e-mail]'  # your email for anonymous FTP
 
my $AMSUdir = '/home/ellis/work/classwork/AT652/DREAM/test/'; # the directory where your AMSU data is located
 
########### DO NOT CHANGE ANYTHING BELOW THIS LINE ###################
 
my $ftp_username = 'anonymous'; # for anonymous ftp
 
# get current time for naming files and directories
 
my ($tsec, $tmin, $thour, $mday, $mmon, $yyear, $wday, $yday, $isdst) = localtime time;
$yyear += 1900;
 
# initialize output file name
 
my $out_file_name = "DREAMfiles_${thour}${tmin}_${yday}_${yyear}.dat";
open OUTFILE, ">$out_file_name";
$| = 1; # clear buffer after each output
 
# change to AMSU data directory and work here
 
chdir $AMSUdir or die "cannot change dir to $AMSUdir: $!";
 
# load in names of all existing AMSU data
 
my @AMSUfiles = glob "*.eos";
 
# output number of files (for IDL program)
 
print OUTFILE "$#AMSUfiles\n";
 
# make a new data directory for all data files
 
my $data_dir = "data_${thour}${tmin}_${yday}_${yyear}";
mkdir $data_dir, 0777;
 
# output data directory name for IDL program
 
print OUTFILE "$data_dir\n";
 
# initialize holders
 
my $hold_doy = 0;
my $hold_year = 0;
 
# loop through each AMSU file
 
foreach my $AMSU (@AMSUfiles) {
 
 my $ssmi_filename = '';
 my $sst_filename = '';
 my $ncep_uwind_name = '';
 my $ncep_vwind_name = '';
 my $year = '';
 my $doy = '';
 
 # match file name to extract year and day of year
 
 if ($AMSU =~ /\w+(\d{4})\.(\d+)\w\d+\w\d+\w\d+\w+\.eos/) {
 
  my $year = $1 * 1; # converts year to number
  my $doy = $2;
 
  # compare day of year to last time through - only download data if new day
 
  if ($doy !=  $hold_doy) {
 
   # convert dayofyear to date
 
   my @days_by_month = ($year % 4 eq 0)
    ? (31, 60, 91, 121, 152, 182, 213, 244, 274, 305, 335, 366)
    : (31, 59, 90, 120, 151, 181, 212, 243, 273, 304, 334, 365);
 
   my $month = 1;
   my $dd = $2*1;
 
   my $day;
 
   foreach my $dm (@days_by_month) {
    if ($2 <= $dm) {
     $day = $dd;
    } else {
     $dd = $2 - $dm;
     $month++;
    }
 
   }
 
   # grab strings corresponding to date of file
 
   my $mo_string = sprintf "%02s", $month;
   my $day_string = sprintf "%02s", $day;
 
   # get the SSMI data
 
   my $ssmi_dir = "/ssmi/f15/bmaps_v06/y${year}/m${mo_string}/";
   my $ssmi_filename = "f15_${year}${mo_string}${day_string}v6.gz";
 
   my $ssmi_site = 'ftp.ssmi.com';
 
   my $ftp_obj = Net::FTP -> new ($ssmi_site, Timeout => 20) or
    die "Cannot access $ssmi_site via FTP\n";
 
   $ftp_obj -> login($ftp_username, $ftp_passwd) or
    die "Invalid user name and/or password SSMI\n";
 
   $ftp_obj -> binary;
 
   $ftp_obj -> cwd ("$ssmi_dir") or
    die "Cannot access directory $ssmi_dir\n";
 
   $ftp_obj -> get("$ssmi_filename") or die "Get failed $ssmi_filename";
 
   # get the SST data
 
   my $sst_dir = "/sst/daily/tmi_amsre/";
   my $sst_filename = "tmi_amsre.fusion.${year}.${doy}.v02.gz";
 
   $ftp_obj -> binary;
 
   $ftp_obj -> cwd ("$sst_dir") or
    die "Cannot access directory $sst_dir\n";
 
   $ftp_obj -> get("$sst_filename") or die "Cannot get SST file\n";
 
   $ftp_obj -> quit;
 
  }
 
  # if new year, grab NCEP data
 
  if ($year != $hold_year) {
 
   my $ncep_dir = '/Datasets/ncep.reanalysis.dailyavgs/surface/';
   my $ncep_uwind_name = "uwnd.sig995.${year}.nc";
   my $ncep_vwind_name = "vwnd.sig995.${year}.nc";
 
   my $ncep_site = "ftp.cdc.noaa.gov";
 
   my $ncep_obj = Net::FTP -> new ($ncep_site, Timeout => 20) or
    die "Cannot access $ncep_site via FTP\n";
 
   print "$ncep_obj\n";
 
   $ncep_obj -> binary;
 
   print "$ncep_obj\n";
   print "$ncep_dir\n";
 
# error is currently at this step - yet I can manually use FTP to access the directory...
   $ncep_obj -> cwd ("$ncep_dir") or die "Cannot access directory $ncep_dir\n";
 
   $ncep_obj -> get("$ncep_uwind_name") or die "Get failed u-wind\n";
   $ncep_obj -> get("$ncep_vwind_name") or die "Get failed v-wind\n";
 
   $ncep_obj -> quit;
 
  }
 } 
 
 # hold the day of year and year for comparison to next iteration of loop
 
 $hold_doy = $doy;
 $hold_year = $year;
 
 # write out the relevant data
 
 print OUTFILE "$year $doy $AMSU $ssmi_filename $sst_filename $ncep_uwind_name $ncep_vwind_name\n"; 
 
}
 
# move all data to new data directory
 
foreach my $file (glob "*.eos") {
 
 my $newfile = "$data_dir/$file";
 rename $file, $newfile;
 
}
 
foreach my $file (glob "*.gz") {
 
 my $newfile = "$data_dir/$file";
 rename $file, $newfile;
 
}
 
foreach my $file (glob "*.nc") {
 
 my $newfile = "$data_dir/$file";
 rename $file, $newfile;
 
}
 
# end of file
 

Many thanks
-T

[miller]

The main thing that is missing from your script now are the error messages as stated in http://perldoc.perl.org/Net/FTP.html I went ahead and added them, and immediate was returned the following error:

"Cwd failed Please login with USER and PASS"

This is definitely beats the old error message as far as clarity. I therefore added the login statement and everything appears to work just fine. You'll also notice that I changed the formatting of the code some and discovered a couple other minor bugs. There are a lot of other enhancements that I could make to your script or suggestions of ones that you can make. But I think it's just important that you get it running for now.

The main lesson learned is that take advantage of all the debugging information that perl gives you, whether that is from the use of directives, or error messages provided by modules. Both were very useful in getting your script to a working state.

Happy Thanksgiving.

#!/usr/bin/perl
 
use Net::FTP;
use strict;
 
#### USER INPUT PARAMETERS
 
my $ftp_passwd = '[e-mail]';  # your email for anonymous FTP
# Was BUG: missing semicolon
 
my $AMSUdir = '/home/ellis/work/classwork/AT652/DREAM/test/'; # the directory where your AMSU data is located
 
########### DO NOT CHANGE ANYTHING BELOW THIS LINE ###################
 
my $ftp_username = 'anonymous'; # for anonymous ftp
 
# get current time for naming files and directories
 
my ($tsec, $tmin, $thour, $mday, $mmon, $yyear, $wday, $yday, $isdst) = localtime time;
$yyear += 1900;
# REDUNDANT CODE - from http://perldoc.perl.org/functions/localtime.html
# "If EXPR is omitted, localtime() uses the current time (localtime(time))."
 
# initialize output file name
 
my $out_file_name = "DREAMfiles_${thour}${tmin}_${yday}_${yyear}.dat";
open OUTFILE, ">$out_file_name";
my $old_fh = select(OUTFILE); $| = 1; select($old_fh); # clear buffer after each output
# Was BUG: Cleared only STDOUT (not OUTFILE) - http://perldoc.perl.org/functions/select.html
 
# change to AMSU data directory and work here
 
chdir $AMSUdir or die "cannot change dir to $AMSUdir: $!";
 
# load in names of all existing AMSU data
 
my @AMSUfiles = glob "*.eos";
 
# output number of files (for IDL program)
 
print OUTFILE "$#AMSUfiles\n";
 
# make a new data directory for all data files
 
my $data_dir = "data_${thour}${tmin}_${yday}_${yyear}";
mkdir $data_dir, 0777;
 
# output data directory name for IDL program
 
print OUTFILE "$data_dir\n";
 
# initialize holders
 
my $hold_doy = 0;
my $hold_year = 0;
 
# loop through each AMSU file
 
foreach my $AMSU (@AMSUfiles) {
 
    my $ssmi_filename = '';
    my $sst_filename = '';
    my $ncep_uwind_name = '';
    my $ncep_vwind_name = '';
    my $year = '';
    my $doy = '';
 
    # match file name to extract year and day of year
 
    if ($AMSU =~ /\w+(\d{4})\.(\d+)\w\d+\w\d+\w\d+\w+\.eos/) {
 
        my $year = $1 * 1; # converts year to number
        my $doy = $2;
 
        # compare day of year to last time through - only download data if new day
 
        if ($doy !=  $hold_doy) {
 
            # convert dayofyear to date
 
            my @days_by_month = ($year % 4 eq 0)
                ? (31, 60, 91, 121, 152, 182, 213, 244, 274, 305, 335, 366)
                : (31, 59, 90, 120, 151, 181, 212, 243, 273, 304, 334, 365);
 
            my $month = 1;
            my $dd = $2*1;
 
            my $day;
 
            foreach my $dm (@days_by_month) {
                if ($2 <= $dm) {
                    $day = $dd;
                } else {
                    $dd = $2 - $dm;
                    $month++;
                }
            }
 
            # grab strings corresponding to date of file
            my $mo_string = sprintf "%02s", $month;
            my $day_string = sprintf "%02s", $day;
 
            my $ssmi_site = 'ftp.ssmi.com';
 
            my $ftp = Net::FTP->new($ssmi_site, Timeout => 20)
                or die "Cannot connect to $ssmi_site: $@";
            $ftp ->login($ftp_username, $ftp_passwd)
                or die "Login failed " , $ftp->message;
            $ftp->binary;
 
            # get the SSMI data
            $ftp->cwd("/ssmi/f15/bmaps_v06/y${year}/m${mo_string}/")
                or die "Cwd failed ", $ftp->message;
            $ftp->get("f15_${year}${mo_string}${day_string}v6.gz")
                or die "Get failed ", $ftp->message;
 
            # get the SST data
            $ftp->cwd("/sst/daily/tmi_amsre/")
                or die "Cwd failed", $ftp->message;
            $ftp->get("tmi_amsre.fusion.${year}.${doy}.v02.gz")
                or die "Get failed ", $ftp->message;
 
            $ftp->quit;
        }
 
        # if new year, grab NCEP data
 
        if ($year != $hold_year) {
 
            my $ncep_site = "ftp.cdc.noaa.gov";
 
            my $ftp = Net::FTP->new($ncep_site, Timeout => 20)
                or die "Cannot connect to $ncep_site: $@";
            $ftp->login($ftp_username, $ftp_passwd)
                or die "Login failed " , $ftp->message;
            $ftp->binary;
            $ftp->cwd('/Datasets/ncep.reanalysis.dailyavgs/surface/')
                or die "Cwd failed ", $ftp->message;
            $ftp->get("uwnd.sig995.${year}.nc")
                or die "Get failed ", $ftp->message;
            $ftp->get("vwnd.sig995.${year}.nc")
                or die "Get failed ", $ftp->message;
            $ftp->quit;
 
        }
    }
 
    # hold the day of year and year for comparison to next iteration of loop
 
    $hold_doy = $doy;
    $hold_year = $year;
 
    # write out the relevant data
 
    print OUTFILE "$year $doy $AMSU $ssmi_filename $sst_filename $ncep_uwind_name $ncep_vwind_name\n";
 
}
 
# move all data to new data directory
 
foreach my $file (glob "*.eos") {
 
    my $newfile = "$data_dir/$file";
    rename $file, $newfile;
 
}
 
foreach my $file (glob "*.gz") {
 
    my $newfile = "$data_dir/$file";
    rename $file, $newfile;
 
}
 
foreach my $file (glob "*.nc") {
 
    my $newfile = "$data_dir/$file";
    rename $file, $newfile;
 
}
 
# end of file
 

[psutoad]

I cannot thank you enough, both for your assistance is finding the bugs (yes, that error message was much more helpful), but more importantly for helping me take better advantage of all perl has to offer. I feel like I learned a great deal from you, which is significantly more valuable than just having code that works.

I apologize for my late reply, as I was on travel for the holiday. I hope your holidays are meaningful and enjoyable.

Thanks again,
-T