Break up Large file into smaller ones

[coldy]

Hi,

I have a large txt file (1GB) which I need to break into smaller files based on the contents of a column in the file.

The values in the column of intrest starts low, then increases, then starts low again eg
1
2
3
0
3
4

what I need is to break this up into smaller files when the column does not increase
so the eg becomes
file1
1
2
3

file 2
0
3
4

I have code that reads the large file into an array of arrays (takes ages!) then checks consecutive rows of the field of interest to see if they decrease, if not print row to file else print row to new file.

I have some code, any suggestions to increase efficiency much appreciated!!

my(@ibdfile,@tmp) = ();
 open( INFILE, "< $infile" ) or die "Can't open $infile : $!";
 while( <INFILE> ) {
             next unless $. > 0;
             @tmp = split;
             push  @ibdfile, [@tmp];
             }
 
  my $chr=1;
  my $n = @ibdfile;
  for (my $j=0; $j < $n ; $j++){
               if ( $ibdfile[$j][3] >= $ibdfile[$j+1][3]){
                  my $outfile = "$chr chr.ibd";
                  open OUTFILE, ">>$outfile" or die "can't open '$outfile': $!";
                  print OUTFILE "@{$ibdfile[$j]}\n";
                  close OUTFILE;
                 }
            else {
            $chr++;
          }
   }

[numberwhun]

Hi,

I have a large txt file (1GB) which I need to break into smaller files based on the contents of a column in the file.

The values in the column of intrest starts low, then increases, then starts low again eg
1
2
3
0
3
4

what I need is to break this up into smaller files when the column does not increase
so the eg becomes
file1
1
2
3

file 2
0
3
4

I have code that reads the large file into an array of arrays (takes ages!) then checks consecutive rows of the field of interest to see if they decrease, if not print row to file else print row to new file.

I have some code, any suggestions to increase efficiency much appreciated!!

my(@ibdfile,@tmp) = ();
 open( INFILE, "< $infile" ) or die "Can't open $infile : $!";
 while( <INFILE> ) {
             next unless $. > 0;
             @tmp = split;
             push  @ibdfile, [@tmp];
             }
 
  my $chr=1;
  my $n = @ibdfile;
  for (my $j=0; $j < $n ; $j++){
               if ( $ibdfile[$j][3] >= $ibdfile[$j+1][3]){
                  my $outfile = "$chr chr.ibd";
                  open OUTFILE, ">>$outfile" or die "can't open '$outfile': $!";
                  print OUTFILE "@{$ibdfile[$j]}\n";
                  close OUTFILE;
                 }
            else {
            $chr++;
          }
   }

Well, if I had a sample of your data (and a little more time on my hands at the moment), I could probably give you a more definitive, code-driven answer, but here is what I would do (unicode wise).

I would work with two counters. One to append to the end of a general file name and one for comparison to the column. The one for the column I would initially set to 0 and when you grab the appropriate column in the first record, compare it against the variable (counter). If it is greater than it (assuming you start with zero and the first record starts with a number greater than zero) then you output it to the file. If it is smaller, then you create a new file, with the other counter's number appended to it and increment that other counter.
Additionally, after you have done the first comparison, you will want to set the counter for the column to whatever value was in the column so that can be compared.

Sorry, I tried to get this all down as clear as I could, although it may be a bit sketchy. Let me know if you understand it or not.

Regards,

Jeff

[KevinADC]

untested code:

my $chr = 0;
open( INFILE, "<", $infile) or die "Can't open $infile : $!";
open (my $OUTFILE, ">>", ++$chr . ' chr.ibd') or die "can't open outfile: $!";
my $first_line =  <INFILE>;
my $last = (split(/\s+/,$first_line))[3];
while( <INFILE> ) {
   my $next = (split(/\s+/))[3] );
   if ( $last >= $next ) {
      close $OUTFILE;
      open (my $OUTFILE, ">>", ++$chr . ' chr.ibd') or die "can't open outfile': $!";
   }
   print $OUTFILE;
   $last = $next; 
}

[KevinADC]

I still had this code in my perl IDE and noticed an error in line 7:

my $next = (split(/\s+/))[3] );

the last parenthesis needs to be removed, should be:

my $next = (split(/\s+/))[3];

[jagjot]

What you can do is

1. Read this entire file into an array.
2. use a loop to write the elements of the array to an output_file_var (where var is the var of the loop) till the loop encounters a zero , n then start the loop again by incrementing the var.
3. this way you will have multiple small files.
4. You have to take care to execute the loop as many number of times as there are 0s in the array which can easily be found.