473,389 Members | 1,119 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,389 software developers and data experts.

Most efficient way to copy large volume of files

Using .NET 2.0 is it more efficient to copy files to a single folder versus
spreading them across multiple folders.
For instance if we have 100,000 files to be copied,

Do we copy all of them to a single folder called 'All Files'
[or]
Do we spread them out and copy them to multiple folders like
Folder 000 - Copy files from 0 to 1000
Folder 001 - Copy files from 1000 to 2000
Folder 002 - Copy files from 2000 to 2999
..
..
..
..
..
Folder 00x - Copy files from 99000 to 100,000
We find that the time taken to copy a file is not consistent when we do file
copies to a single folder. It varies from 0/1 milliseconds to
400+milliseconds. By changing the architecture to spread them across multiple
folders are we going to be efficient?
Jun 27 '08 #1
1 3844
Hi,

Regarding on the file copy scenario, are you copying large number of files
between two remote share folders in the same local network environment?

Generally, for large number of file(not very large size) copying between
local folders(or remote folders in local network), you can consider use
multi-threading(simultaneous copy multiple files) to improve performance
instead of copying files one by one. Also, when using multi-threading to
copy files concurently, you need to correctly record the files that are
being copied or already copied in your code.

As for copying to single folder or multiple folders, I think it mostly
depend on whether the folders are separated among different physical disk
drives. As we know, the most cose of file I/O operation is disk
reading/writing. If those folders are one the same physical disk/drive, it
has quite limited concurrent improvement space. Therefore, if possible,
you can consider separate the folders on two or more physical disk/drive so
as to further leverage I/O concurrency.

Sincerely,

Steven Cheng

Microsoft MSDN Online Support Lead
Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
ms****@microsoft.com.

==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/subscripti...ult.aspx#notif
ications.

Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscripti...t/default.aspx.
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.
--------------------
>From: =?Utf-8?B?UVNJRGV2ZWxvcGVy?= <QS**********@newsgroup.nospam>
Subject: Most efficient way to copy large volume of files
Date: Tue, 3 Jun 2008 08:43:01 -0700
>
Using .NET 2.0 is it more efficient to copy files to a single folder
versus
>spreading them across multiple folders.
For instance if we have 100,000 files to be copied,

Do we copy all of them to a single folder called 'All Files'
[or]
Do we spread them out and copy them to multiple folders like
Folder 000 - Copy files from 0 to 1000
Folder 001 - Copy files from 1000 to 2000
Folder 002 - Copy files from 2000 to 2999
.
.
.
.
.
Folder 00x - Copy files from 99000 to 100,000
We find that the time taken to copy a file is not consistent when we do
file
>copies to a single folder. It varies from 0/1 milliseconds to
400+milliseconds. By changing the architecture to spread them across
multiple
>folders are we going to be efficient?
Jun 27 '08 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

21
by: Chris S. | last post by:
I have a number of strings, containing wildcards (e.g. 'abc#e#' where # is anything), which I want to match with a test string (e.g 'abcdef'). What would be the best way for me to store my strings...
5
by: Louis LeBlanc | last post by:
Hey folks. I'm new to the list, and not quite what you'd call a DB Guru, so please be patient with me. I'm afraid the lead up here is a bit verbose . . . I am working on an application that...
0
by: Dhananjay | last post by:
Hi All I am doing volume shadow copy of files and folders..I am able to make a backup of the files.During restore I need to use the xml file made during backup.But my restore is not...
2
by: 1388-2/HB | last post by:
On a W2K3 box running IIS 6 I have a web app that implements Forms Authentication mixed with AD (the login page autheticates users against AD & impersonates them with each page request). This...
21
by: py_genetic | last post by:
Hello, I'm importing large text files of data using csv. I would like to add some more auto sensing abilities. I'm considing sampling the data file and doing some fuzzy logic scoring on the...
3
by: rubbishemail | last post by:
Hello, do you know of any way to copy locked / opened files under win xp? I know there is something like "Volume Shadow Copy" but I don't know how to use it. Maybe someone already has a python...
1
by: =?Utf-8?B?UmFkZW5rb19aZWM=?= | last post by:
I am using standard File.Copy(source,dest,true) method in C# and I have problem with copying large number of files. Here is my code: foreach (FileInfo file in files) {...
2
by: furytrader | last post by:
Does anyone know an efficient way to generate sound tones in Visual Basic 2005/2008 so that I can specify: - Pitch - Duration - Volume .... which uses the computer's soundcard (not the...
3
by: MC | last post by:
I'm developing an application that needs a large (up to 4 GB) amount of temporary file space on a local (not network-attached) disk. What are some good tactics to use in a C# program to make sure...
0
by: ryjfgjl | last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
0
by: ryjfgjl | last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
0
BarryA
by: BarryA | last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
0
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.