Split a large file and then go back through the smaller chunks

[JayDog]

I have a large data file that I split into smaller more manageable chunks (went from a 12.86 GB file to 500 MB - 1.6 GB chunks).

I now want to add to the PERL script and go back through those more manageable chunks and pull out any invoices within those smaller data files that are larger than 250 MB each and print them to their own files as well.

How do I go about doing that?

Here is what I am currently working with...

#!usr/bin/perl -w
 
my $chunksize = 500000000; # 500MB
my $filenumber = 0;
my $infile = "infile.dat";
my $outsize = 0;
my $eof = 0;
 
open INFILE, $infile;
open OUTFILE, ">outfile_".$filenumber.".dat";
 
while (<INFILE>) {
    chomp;
 
    if( $outsize>$chunksize and /^.{67}11/ ) {
        close OUTFILE;
        $outsize = 0;
        $filenumber++;
        open (OUTFILE, ">outfile_".$filenumber.".dat") or die "Can't open outfile_".$filenumber.".dat";
    }
 
    print OUTFILE "$_\n";
    $outsize += length;
}
close INFILE;
 

[miller]

As Kevin has already noted on TT, you're spamming this thread on multiple forums: Here, PG, TT, and who knows where else. Please note that cross posting will likely net you less help because the most prolific experts are often on multiple forum sites. Cross posters notoriously do not come back for the help that is offered, so people stop offering.

Brigmar was nice enough to provide you with your current script on TT. However, noone is going to do an entire project for you. Even one that is this easy.

The only hard part about this project is the fact that during testing it can take a long time to parse through files more than a meg in size let along a gig. Therefore it helps to break them up into a sample size file for testing purposes before rolling out the production level code.

I would have advised you to break the project up into steps.
1) Count the number of lines in the file
2) Count the number of invoices in the file
3) Count the size of the file and the size of the invoices.
4) Implement the code to output groups of invoices to secondary files.

Neither of these steps is difficult, and they are all you ultimately need to accomplish your goal. The code that you were provided has the fundumentals for this, so it's up to you to put them to together.

If you're having trouble with any particular part, feel free to ask here. However, it's up to you to do the programming yourself.

- Miller