By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
435,030 Members | 1,997 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 435,030 IT Pros & Developers. It's quick & easy.

Most efficient way to copy large volume of files

P: n/a
Using .NET 2.0 is it more efficient to copy files to a single folder versus
spreading them across multiple folders.
For instance if we have 100,000 files to be copied,

Do we copy all of them to a single folder called 'All Files'
[or]
Do we spread them out and copy them to multiple folders like
Folder 000 - Copy files from 0 to 1000
Folder 001 - Copy files from 1000 to 2000
Folder 002 - Copy files from 2000 to 2999
..
..
..
..
..
Folder 00x - Copy files from 99000 to 100,000
We find that the time taken to copy a file is not consistent when we do file
copies to a single folder. It varies from 0/1 milliseconds to
400+milliseconds. By changing the architecture to spread them across multiple
folders are we going to be efficient?
Jun 27 '08 #1
Share this Question
Share on Google+
1 Reply


P: n/a
Hi,

Regarding on the file copy scenario, are you copying large number of files
between two remote share folders in the same local network environment?

Generally, for large number of file(not very large size) copying between
local folders(or remote folders in local network), you can consider use
multi-threading(simultaneous copy multiple files) to improve performance
instead of copying files one by one. Also, when using multi-threading to
copy files concurently, you need to correctly record the files that are
being copied or already copied in your code.

As for copying to single folder or multiple folders, I think it mostly
depend on whether the folders are separated among different physical disk
drives. As we know, the most cose of file I/O operation is disk
reading/writing. If those folders are one the same physical disk/drive, it
has quite limited concurrent improvement space. Therefore, if possible,
you can consider separate the folders on two or more physical disk/drive so
as to further leverage I/O concurrency.

Sincerely,

Steven Cheng

Microsoft MSDN Online Support Lead
Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
ms****@microsoft.com.

==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/subscripti...ult.aspx#notif
ications.

Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscripti...t/default.aspx.
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.
--------------------
>From: =?Utf-8?B?UVNJRGV2ZWxvcGVy?= <QS**********@newsgroup.nospam>
Subject: Most efficient way to copy large volume of files
Date: Tue, 3 Jun 2008 08:43:01 -0700
>
Using .NET 2.0 is it more efficient to copy files to a single folder
versus
>spreading them across multiple folders.
For instance if we have 100,000 files to be copied,

Do we copy all of them to a single folder called 'All Files'
[or]
Do we spread them out and copy them to multiple folders like
Folder 000 - Copy files from 0 to 1000
Folder 001 - Copy files from 1000 to 2000
Folder 002 - Copy files from 2000 to 2999
.
.
.
.
.
Folder 00x - Copy files from 99000 to 100,000
We find that the time taken to copy a file is not consistent when we do
file
>copies to a single folder. It varies from 0/1 milliseconds to
400+milliseconds. By changing the architecture to spread them across
multiple
>folders are we going to be efficient?
Jun 27 '08 #2

This discussion thread is closed

Replies have been disabled for this discussion.