473,714 Members | 2,552 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Most efficient way to copy large volume of files

Using .NET 2.0 is it more efficient to copy files to a single folder versus
spreading them across multiple folders.
For instance if we have 100,000 files to be copied,

Do we copy all of them to a single folder called 'All Files'
[or]
Do we spread them out and copy them to multiple folders like
Folder 000 - Copy files from 0 to 1000
Folder 001 - Copy files from 1000 to 2000
Folder 002 - Copy files from 2000 to 2999
..
..
..
..
..
Folder 00x - Copy files from 99000 to 100,000
We find that the time taken to copy a file is not consistent when we do file
copies to a single folder. It varies from 0/1 milliseconds to
400+millisecond s. By changing the architecture to spread them across multiple
folders are we going to be efficient?
Jun 27 '08 #1
1 3894
Hi,

Regarding on the file copy scenario, are you copying large number of files
between two remote share folders in the same local network environment?

Generally, for large number of file(not very large size) copying between
local folders(or remote folders in local network), you can consider use
multi-threading(simul taneous copy multiple files) to improve performance
instead of copying files one by one. Also, when using multi-threading to
copy files concurently, you need to correctly record the files that are
being copied or already copied in your code.

As for copying to single folder or multiple folders, I think it mostly
depend on whether the folders are separated among different physical disk
drives. As we know, the most cose of file I/O operation is disk
reading/writing. If those folders are one the same physical disk/drive, it
has quite limited concurrent improvement space. Therefore, if possible,
you can consider separate the folders on two or more physical disk/drive so
as to further leverage I/O concurrency.

Sincerely,

Steven Cheng

Microsoft MSDN Online Support Lead
Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
ms****@microsof t.com.

=============== =============== =============== =====
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/subscripti...ult.aspx#notif
ications.

Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscripti...t/default.aspx.
=============== =============== =============== =====
This posting is provided "AS IS" with no warranties, and confers no rights.
--------------------
>From: =?Utf-8?B?UVNJRGV2ZWx vcGVy?= <QS**********@n ewsgroup.nospam >
Subject: Most efficient way to copy large volume of files
Date: Tue, 3 Jun 2008 08:43:01 -0700
>
Using .NET 2.0 is it more efficient to copy files to a single folder
versus
>spreading them across multiple folders.
For instance if we have 100,000 files to be copied,

Do we copy all of them to a single folder called 'All Files'
[or]
Do we spread them out and copy them to multiple folders like
Folder 000 - Copy files from 0 to 1000
Folder 001 - Copy files from 1000 to 2000
Folder 002 - Copy files from 2000 to 2999
.
.
.
.
.
Folder 00x - Copy files from 99000 to 100,000
We find that the time taken to copy a file is not consistent when we do
file
>copies to a single folder. It varies from 0/1 milliseconds to
400+millisecon ds. By changing the architecture to spread them across
multiple
>folders are we going to be efficient?
Jun 27 '08 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

21
2435
by: Chris S. | last post by:
I have a number of strings, containing wildcards (e.g. 'abc#e#' where # is anything), which I want to match with a test string (e.g 'abcdef'). What would be the best way for me to store my strings so lookup is as fast as possible?
5
5598
by: Louis LeBlanc | last post by:
Hey folks. I'm new to the list, and not quite what you'd call a DB Guru, so please be patient with me. I'm afraid the lead up here is a bit verbose . . . I am working on an application that uses very high volume DB transactions - in the order of tens of millions per day . . . Anyway, the current database which will remain nameless, but begins with O and rymes with debacle (sorta), has a problem with high volume work when it comes to...
0
1034
by: Dhananjay | last post by:
Hi All I am doing volume shadow copy of files and folders..I am able to make a backup of the files.During restore I need to use the xml file made during backup.But my restore is not working...dont know why? :-( I followed the steps that was explained in MSDN using IVssBackUpComponent but in vain... I also tried to change my direction my rewriting the program using
2
2133
by: 1388-2/HB | last post by:
On a W2K3 box running IIS 6 I have a web app that implements Forms Authentication mixed with AD (the login page autheticates users against AD & impersonates them with each page request). This means my process identity varies, and is not necessarily "ASPNET" or "IUSR...". My process identity for any given page request will be user1, user2, user3, etc. (there's a lot of them) This web app needs to copy files (Word documents) from our old...
21
2013
by: py_genetic | last post by:
Hello, I'm importing large text files of data using csv. I would like to add some more auto sensing abilities. I'm considing sampling the data file and doing some fuzzy logic scoring on the attributes (colls in a data base/ csv file, eg. height weight income etc.) to determine the most efficient 'type' to convert the attribute coll into for further processing and efficient storage... Example row from sampled file data: , ....]
3
5873
by: rubbishemail | last post by:
Hello, do you know of any way to copy locked / opened files under win xp? I know there is something like "Volume Shadow Copy" but I don't know how to use it. Maybe someone already has a python solution? Many thanks
1
4294
by: =?Utf-8?B?UmFkZW5rb19aZWM=?= | last post by:
I am using standard File.Copy(source,dest,true) method in C# and I have problem with copying large number of files. Here is my code: foreach (FileInfo file in files) { File.Copy(file.FullName,destPath+ "\\" + file.Name, true); } This code copies only 5 or 10 files but in "files" collection there is 60 files.
2
812
by: furytrader | last post by:
Does anyone know an efficient way to generate sound tones in Visual Basic 2005/2008 so that I can specify: - Pitch - Duration - Volume .... which uses the computer's soundcard (not the motherboard speaker).?
3
2606
by: MC | last post by:
I'm developing an application that needs a large (up to 4 GB) amount of temporary file space on a local (not network-attached) disk. What are some good tactics to use in a C# program to make sure the swap space is suitable? There can be some interaction with the user to choose the swap file location, but the more the program can find out about it, the better.
0
8801
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
9314
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
9174
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
9015
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
7953
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
4725
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
3158
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
2520
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
2110
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.