using threads to create 75000 files

Hi,

My requirement is to create 75000 files.

I have a perl script to create files , it is working fine for small number of files but the script is exiting while creating large number of files. For this I used threads. The following is the script I used. Please help me where the script is failing. I have been fighting with this since a week.

Expand|Select|Wrap|Line Numbers

 use threads;
 
&prepare_multi_data(1000,"C:\\Auto CIFS\\","dump",2048);
 
sub prepare_multi_data {

    my $number_of_files = shift;

    my $working_folder = shift;

    my $file_type = shift;

    my $size = shift;
 
    my $ctr;
 
    if (!&create_multi_files($number_of_files,$working_folder,$file_type, $size)){

        return 0;

    }else {

        for ($ctr=0;$ctr<$number_of_files;$ctr++) {

            push(@files,"$file_name$ctr");

        }

        return \@files;

    }

}
 
sub create_multi_files {

    my $number_of_files = shift;

    my $working_folder = shift;

    my $file_type = shift;

    my $size = shift;
 
    $file_name = "File-".time;

    my ($ctr, $pid, @cpids, $loop);

    for ($ctr=0;$ctr<$number_of_files;$ctr++) {
 
            if (!defined($pid = fork())) {

                return 0;

            }elsif ($pid == 0) {

                if (!&create_file($working_folder, $file_type, $size, "$file_name$ctr")) {

                            kill 9,@cpids;

                            return 0;

                    }

                    exit 0;

            }else {

                    $cpids[$ctr] = $pid;

            }

    }
 
    for ($loop=0;$loop<$number_of_files;$loop++) {

            waitpid($cpids[$loop],0);

    }

    return 1;

}
 
sub create_file {

    my $working_folder = shift;

    my $file_type = shift;

    my $size = shift;

    my $file_name = shift;
 
    if (open(FH, ">$working_folder/$file_type/$file_name")){

            while ($size>=1048576) {

                print FH ("0" x 1048576);

                $size = $size - 1048576;

            }

            print FH "0" x $size;

        return 1;

    }

    else{

        return 0;

    }

    close(FH);
 
}

Thanx in advance.

Jan 7 '08 #1

Subscribe Post Reply

1676

eWish

971

Expert 512MB

Please explain what you mean by failing? Do you get any error messages?

--Kevin

Jan 7 '08 #2

numberwhun

3,509

Expert Mod 2GB

You haven't run out of disk space, have you? Also, do you have the "use strict" and "use warnings" pragmas in your code?

Regards,

Jeff

Jan 7 '08 #3

KevinADC

4,059

Expert 2GB

You need to do some error checking here:

if (open(FH, ">$working_folder/$file_type/$file_name")){

and see why the files are failing. There does appear to be an extra slash in the path statement, after $working_folder, because you already had the trailing slash when you defined the variable: C:\\Auto CIFS\\

Also in the "create_file" sub, you are not closing the filehandle properly. You have two return() functions then the close() function. The return() functions never allow the sub to get to the close() function, so move the close(FH) up to just after the file is finished printing. Try something like this for that sub:

Expand|Select|Wrap|Line Numbers

 sub create_file {

    my $working_folder = shift;

    my $file_type = shift;

    my $size = shift;

    my $file_name = shift;
 
    open(FH, ">$working_folder$file_type/$file_name") or die "$working_folder$file_type/$file_name: $!";

   while ($size>=100) {

        print FH ("0" x 100);

        $size = $size - 100;

    }

    print FH "0" x $size;

    close(FH);

    return 1;

#    }

#    else{

#        return 0;

#    }
 
}

and see if you get any errors associated with opening a new file. Scale down the tests from 75000 until you start getting errors. Start with 1000 files and work your way up. You may also want to sleep() before the script opens a new file because you are using "time" in the file names:

$file_name = "File-".time;

so try adding: sleep(1); so at least one second passes between new file creations.

Jan 7 '08 #4

npankajk

Thank you , Thanks for the help .

Its working,

Jan 8 '08 #5

Similar topics

strange behavior when using 'read'

by: sstark | last post by:

Hi, I have a web/cgi script that allows users to upload a file to the web server. I want to only allow files up to a certain size, which is stored in $imageFileMaxSize (typically 75K). That part...

Perl

Most efficient way to process thousands of files using multiple threads (dealing with thread handling)

by: m | last post by:

Hello, I have an application that processes thousands of files each day. The filenames and various related file information is retrieved, related filenames are associate and placed in a linked...

C# / C Sharp

How to create files using threads at run time in VB.NET

by: Mamatha | last post by:

Hi friends, I have an application using mutithreads.In that application one threads writes the data in to a file. Another thread reads the data from same file. So some errors while at the time...

Visual Basic .NET

How do I keep parameters from being mixed up using Multithreading?

by: mrkbrndck | last post by:

Please see the code below as I am trying to use multithreading for copying files to a new location in a way that improves performance of the client windows application. The problem occurs when 2...

.NET Framework

How many threads is too many?

by: Alexander Walker | last post by:

Hello Is it inefficient to create an application that has many threads that individually may do a small amount of work over a given period of time as opposed to an application that has a...

C# / C Sharp

Threads & Memory

by: ano | last post by:

I create a method that copy files from one folder to another (see the ex. code). A new thread will create every time when user click the button. The button is enable only when the copy process was...

C# / C Sharp

Multiple Threads, Race Conditions, and Concurrent Processing

by: gnassar | last post by:

Hello, I've written to this group a couple times and would like to initially start by thanking those who reply to the posts. I seem to be having some issues that are out of my understanding at...

.NET Framework

os walk() and threads problems (os.walk are thread safe?)

by: Marcus Alves Grando | last post by:

Hello list, I have a strange problem with os.walk and threads in python script. I have one script that create some threads and consume Queue. For every value in Queue this script run os.walk()...

Python

access violation reading location when using WaitForMultipleObjects

by: Pinux | last post by:

Hi, I am writing a multi-threads encryption application. The idea of the code is to create a number of threads to encrypt files. I have a thread pool say the maximum threads is 10. If the number...

C / C++

Easy Steps to Fix "Canon Printer Won't Connect to WiFi Network"

by: taylorcarr | last post by:

A Canon printer is a smart device known for being advanced, efficient, and reliable. It is designed for home, office, and hybrid workspace use and can also be used for a variety of purposes. However,...

General

How to turn on java script in a villaon keypad mobile phone

by: Charles Arthur | last post by:

How do i turn on java script on a villaon, callus and itel keypad mobile phone

Java

Basic Javascript concepts

by: aa123db | last post by:

Variable and constants Use var or let for variables and const fror constants. Var foo ='bar'; Let foo ='bar';const baz ='bar'; Functions function $name$ ($parameters$) { } ...

Javascript

Merging data from multiple Excel files

by: ryjfgjl | last post by:

In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...

Data Management

Migrating Website to Cloud - Emmanuel Katto

by: emmanuelkatto | last post by:

Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel

General

Looking to do Android software development, any suggestions? Is flutter better?

by: nemocccc | last post by:

hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?

General

Is that possible of reading the .csv file in column wise and the column have different lengths ?

by: Sonnysonu | last post by:

This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...

C / C++

What is ONU?

by: marktang | last post by:

ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...

General

Changing the language in Windows 10

by: Hystou | last post by:

Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...

Windows Server