By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
454,245 Members | 1,242 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 454,245 IT Pros & Developers. It's quick & easy.

using threads to create 75000 files

P: 2
Hi,

My requirement is to create 75000 files.

I have a perl script to create files , it is working fine for small number of files but the script is exiting while creating large number of files. For this I used threads. The following is the script I used. Please help me where the script is failing. I have been fighting with this since a week.

Expand|Select|Wrap|Line Numbers
  1. use threads;
  2.  
  3. &prepare_multi_data(1000,"C:\\Auto CIFS\\","dump",2048);
  4.  
  5. sub prepare_multi_data {
  6.     my $number_of_files = shift;
  7.     my $working_folder = shift;
  8.     my $file_type = shift;
  9.     my $size = shift;
  10.  
  11.     my $ctr;
  12.  
  13.     if (!&create_multi_files($number_of_files,$working_folder,$file_type, $size)){
  14.         return 0;
  15.     }else {
  16.         for ($ctr=0;$ctr<$number_of_files;$ctr++) {
  17.             push(@files,"$file_name$ctr");
  18.         }
  19.         return \@files;
  20.     }
  21. }
  22.  
  23. sub create_multi_files {
  24.     my $number_of_files = shift;
  25.     my $working_folder = shift;
  26.     my $file_type = shift;
  27.     my $size = shift;
  28.  
  29.     $file_name = "File-".time;
  30.     my ($ctr, $pid, @cpids, $loop);
  31.     for ($ctr=0;$ctr<$number_of_files;$ctr++) {
  32.  
  33.             if (!defined($pid = fork())) {
  34.                 return 0;
  35.             }elsif ($pid == 0) {
  36.                 if (!&create_file($working_folder, $file_type, $size, "$file_name$ctr")) {
  37.                             kill 9,@cpids;
  38.                             return 0;
  39.                     }
  40.                     exit 0;
  41.             }else {
  42.                     $cpids[$ctr] = $pid;
  43.             }
  44.     }
  45.  
  46.     for ($loop=0;$loop<$number_of_files;$loop++) {
  47.             waitpid($cpids[$loop],0);
  48.     }
  49.     return 1;
  50. }
  51.  
  52. sub create_file {
  53.     my $working_folder = shift;
  54.     my $file_type = shift;
  55.     my $size = shift;
  56.     my $file_name = shift;
  57.  
  58.     if (open(FH, ">$working_folder/$file_type/$file_name")){
  59.             while ($size>=1048576) {
  60.                 print FH ("0" x 1048576);
  61.                 $size = $size - 1048576;
  62.             }
  63.             print FH "0" x $size;
  64.         return 1;
  65.     }
  66.     else{
  67.         return 0;
  68.     }
  69.     close(FH);
  70.  
  71. }
  72.  
Thanx in advance.
Jan 7 '08 #1
Share this Question
Share on Google+
4 Replies


eWish
Expert 100+
P: 971
Please explain what you mean by failing? Do you get any error messages?

--Kevin
Jan 7 '08 #2

numberwhun
Expert Mod 2.5K+
P: 3,503
You haven't run out of disk space, have you? Also, do you have the "use strict" and "use warnings" pragmas in your code?

Regards,

Jeff
Jan 7 '08 #3

KevinADC
Expert 2.5K+
P: 4,059
You need to do some error checking here:

if (open(FH, ">$working_folder/$file_type/$file_name")){

and see why the files are failing. There does appear to be an extra slash in the path statement, after $working_folder, because you already had the trailing slash when you defined the variable: C:\\Auto CIFS\\

Also in the "create_file" sub, you are not closing the filehandle properly. You have two return() functions then the close() function. The return() functions never allow the sub to get to the close() function, so move the close(FH) up to just after the file is finished printing. Try something like this for that sub:

Expand|Select|Wrap|Line Numbers
  1. sub create_file {
  2.     my $working_folder = shift;
  3.     my $file_type = shift;
  4.     my $size = shift;
  5.     my $file_name = shift;
  6.  
  7.     open(FH, ">$working_folder$file_type/$file_name") or die "$working_folder$file_type/$file_name: $!";
  8.    while ($size>=100) {
  9.         print FH ("0" x 100);
  10.         $size = $size - 100;
  11.     }
  12.     print FH "0" x $size;
  13.     close(FH);
  14.     return 1;
  15. #    }
  16. #    else{
  17. #        return 0;
  18. #    }
  19.  
  20.  
  21. }
and see if you get any errors associated with opening a new file. Scale down the tests from 75000 until you start getting errors. Start with 1000 files and work your way up. You may also want to sleep() before the script opens a new file because you are using "time" in the file names:

$file_name = "File-".time;

so try adding: sleep(1); so at least one second passes between new file creations.
Jan 7 '08 #4

P: 2
Thank you , Thanks for the help .

Its working,
Jan 8 '08 #5

Post your reply

Sign in to post your reply or Sign up for a free account.