Hi,
My requirement is to create 75000 files.
I have a perl script to create files , it is working fine for small number of files but the script is exiting while creating large number of files. For this I used threads. The following is the script I used. Please help me where the script is failing. I have been fighting with this since a week. - use threads;
-
-
&prepare_multi_data(1000,"C:\\Auto CIFS\\","dump",2048);
-
-
sub prepare_multi_data {
-
my $number_of_files = shift;
-
my $working_folder = shift;
-
my $file_type = shift;
-
my $size = shift;
-
-
my $ctr;
-
-
if (!&create_multi_files($number_of_files,$working_folder,$file_type, $size)){
-
return 0;
-
}else {
-
for ($ctr=0;$ctr<$number_of_files;$ctr++) {
-
push(@files,"$file_name$ctr");
-
}
-
return \@files;
-
}
-
}
-
-
sub create_multi_files {
-
my $number_of_files = shift;
-
my $working_folder = shift;
-
my $file_type = shift;
-
my $size = shift;
-
-
$file_name = "File-".time;
-
my ($ctr, $pid, @cpids, $loop);
-
for ($ctr=0;$ctr<$number_of_files;$ctr++) {
-
-
if (!defined($pid = fork())) {
-
return 0;
-
}elsif ($pid == 0) {
-
if (!&create_file($working_folder, $file_type, $size, "$file_name$ctr")) {
-
kill 9,@cpids;
-
return 0;
-
}
-
exit 0;
-
}else {
-
$cpids[$ctr] = $pid;
-
}
-
}
-
-
for ($loop=0;$loop<$number_of_files;$loop++) {
-
waitpid($cpids[$loop],0);
-
}
-
return 1;
-
}
-
-
sub create_file {
-
my $working_folder = shift;
-
my $file_type = shift;
-
my $size = shift;
-
my $file_name = shift;
-
-
if (open(FH, ">$working_folder/$file_type/$file_name")){
-
while ($size>=1048576) {
-
print FH ("0" x 1048576);
-
$size = $size - 1048576;
-
}
-
print FH "0" x $size;
-
return 1;
-
}
-
else{
-
return 0;
-
}
-
close(FH);
-
-
}
-
Thanx in advance.
4  1700  eWish 971
Recognized Expert Contributor
Please explain what you mean by failing? Do you get any error messages?
--Kevin
numberwhun 3,509
Recognized Expert Moderator Specialist
You haven't run out of disk space, have you? Also, do you have the "use strict" and "use warnings" pragmas in your code?
Regards,
Jeff
 KevinADC 4,059
Recognized Expert Specialist
You need to do some error checking here: if (open(FH, ">$working_fold er/$file_type/$file_name")){
and see why the files are failing. There does appear to be an extra slash in the path statement, after $working_folder , because you already had the trailing slash when you defined the variable: C:\\Auto CIFS\\
Also in the "create_fil e" sub, you are not closing the filehandle properly. You have two return() functions then the close() function. The return() functions never allow the sub to get to the close() function, so move the close(FH) up to just after the file is finished printing. Try something like this for that sub: - sub create_file {
-
my $working_folder = shift;
-
my $file_type = shift;
-
my $size = shift;
-
my $file_name = shift;
-
-
open(FH, ">$working_folder$file_type/$file_name") or die "$working_folder$file_type/$file_name: $!";
-
while ($size>=100) {
-
print FH ("0" x 100);
-
$size = $size - 100;
-
}
-
print FH "0" x $size;
-
close(FH);
-
return 1;
-
# }
-
# else{
-
# return 0;
-
# }
-
-
-
}
and see if you get any errors associated with opening a new file. Scale down the tests from 75000 until you start getting errors. Start with 1000 files and work your way up. You may also want to sleep() before the script opens a new file because you are using "time" in the file names: $file_name = "File-".time;
so try adding: sleep(1); so at least one second passes between new file creations.
Thank you , Thanks for the help .
Its working,
Sign in to post your reply or Sign up for a free account.
Similar topics | |
by: sstark |
last post by:
Hi, I have a web/cgi script that allows users to upload a file to the
web server. I want to only allow files up to a certain size, which is
stored in $imageFileMaxSize (typically 75K). That part works correctly,
i.e. the script does not allow uploads larger than $imageFileMaxSize.
The problem is, when a user attempts to upload a file larger than
$imageFileMaxSize, the on_error routine is not called; instead the
script reloads from its...
|
by: m |
last post by:
Hello,
I have an application that processes thousands of files each day. The
filenames and various related file information is retrieved, related
filenames are associate and placed in a linked list within a single object,
which is then placed on a stack(This cuts down thread creation and deletions
roughly by a factor of 4). I create up to 12 threads, which then process a
single object off of the stack. I use a loop with a boolean...
|
by: Mamatha |
last post by:
Hi friends,
I have an application using mutithreads.In that
application one threads writes the data in to a file.
Another thread reads the data from same file.
So some errors while at the time of reading.
Now i want to create a files and threads at runtime to
avoid these errors.
How can i create mutiple files and threads for reading
data from that files at runtime.Is there any way?
|
by: mrkbrndck |
last post by:
Please see the code below as I am trying to use multithreading for copying
files to a new location in a way that improves performance of the client
windows application.
The problem occurs when 2 or more threads are created, the ImportOneFile
method attempts to add a previously added file. If I allow 4 maximum threads
and process 4 files, the last file is attempted 4 times and none of the other
files are added to the destination. If I...
|
by: Alexander Walker |
last post by:
Hello
Is it inefficient to create an application that has many threads that
individually may do a small amount of work over a given period of time as
opposed to an application that has a smaller number of threads that do a larger
amount of work over a given time period
here is an oversimplified example
application 1 has 4 threads
| | |
by: ano |
last post by:
I create a method that copy files from one folder to another (see the ex.
code).
A new thread will create every time when user click the button.
The button is enable only when the copy process was finished( ThreadState =
Stop )
My question is when ThreadState = Stop, the system will release the Memory
or not?
Is my code generated multiply threads?
|
by: gnassar |
last post by:
Hello,
I've written to this group a couple times and would like to initially
start by thanking those who reply to the posts. I seem to be having
some issues that are out of my understanding at the moment and
hopefully someone knowledgeable will be able to guide me the right way.
I have created a program that utilizes three threads minimum. The
initial gui thread, and two worker threads. I haven't set the worker
threads as background...
|
by: Marcus Alves Grando |
last post by:
Hello list,
I have a strange problem with os.walk and threads in python script. I
have one script that create some threads and consume Queue. For every
value in Queue this script run os.walk() and printing root dir. But if i
increase number of threads the result are inconsistent compared with one
thread.
For example, run this code plus sort with one thread and after run again
with ten threads and see diff(1).
|
by: Pinux |
last post by:
Hi,
I am writing a multi-threads encryption application. The idea of the code is to create a number of threads to encrypt files. I have a thread pool say the maximum threads is 10. If the number of tasks (number of files waiting to be encrypted) is larger than the thread pool size, I first create 10 new threads to execute the encryption function. Then I wait for any of the threads to be signaled in the thread pool using WaitForMultipleObjects...
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look !
Part I. Meaning of...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
| | |
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it.
Here is my compilation command:
g++-12 -std=c++20 -Wnarrowing bit_field.cpp
Here is the code in...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
|
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
| | |
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| |
