473,322 Members | 1,379 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,322 software developers and data experts.

Need a fast way(c#) to get directory of files over a WAN using selection criteria

Help!!

I need to interrogate about 150,000 files across a WAN and pull in ONLY
the files less than 5 days old and larger than 10k bytes. I'm currently
using Directory.GetFiles and it's taking hours!!

Anyone have any great ideas? Any help will be greatly appreciated.

Thanks
Greg

Nov 17 '05 #1
6 2716
Can you post the code. Maybe we can see something.

--
William Stacey [MVP]

<ga******@wmcdirect.com> wrote in message
news:11**********************@g47g2000cwa.googlegr oups.com...
Help!!

I need to interrogate about 150,000 files across a WAN and pull in ONLY
the files less than 5 days old and larger than 10k bytes. I'm currently
using Directory.GetFiles and it's taking hours!!

Anyone have any great ideas? Any help will be greatly appreciated.

Thanks
Greg

Nov 17 '05 #2
Another way would be to create a small console app on the server side that
does the search and outputs a txt file with the filenames that match your
search. Launch this program using psexec or something from the client and
wait for it to finish then copy that file local and parse the file names to
string[]. Then download each of those files. That should run much faster.

--
William Stacey [MVP]

"William Stacey [MVP]" <st*****@mvps.org> wrote in message
news:uh**************@tk2msftngp13.phx.gbl...
Can you post the code. Maybe we can see something.

--
William Stacey [MVP]

<ga******@wmcdirect.com> wrote in message
news:11**********************@g47g2000cwa.googlegr oups.com...
Help!!

I need to interrogate about 150,000 files across a WAN and pull in ONLY
the files less than 5 days old and larger than 10k bytes. I'm currently
using Directory.GetFiles and it's taking hours!!

Anyone have any great ideas? Any help will be greatly appreciated.

Thanks
Greg


Nov 17 '05 #3
Relevant part of code:

minimumFilesize =
ConfigurationSettings.AppSettings["minimum_filesize"];
numberDays = ConfigurationSettings.AppSettings["number_days"];
try
{
DisplayMsg("Getting Directory info for... " + sDir);
foreach (string f in Directory.GetFiles(sDir, "*.tif"))
{
txtViewed.Text = Convert.ToString(Convert.ToInt32(txtViewed.Text)
+ 1);
FileInfo fi = new FileInfo(f);
//textBox1.Text = ("File size of mscdrun.cab: {0}" +
fi.Length.ToString());

if (fi.CreationTime >
DateTime.Now.AddDays(-Convert.ToInt32(numberDays)))
{
if (fi.Length > Convert.ToInt32(minimumFilesize))
{

Nov 17 '05 #4
Unfortunately - I have only read access to these remote servers.

I like the idea if I could just put a little program out there to run
time-scheduled and create a nice little text file of all the qualifying
Tif files. If I can't get anything else to work this may have to be
done.

Thanks
Greg

Nov 17 '05 #5
ga******@wmcdirect.com wrote:
Relevant part of code:

minimumFilesize =
ConfigurationSettings.AppSettings["minimum_filesize"];
numberDays = ConfigurationSettings.AppSettings["number_days"];
try
{
DisplayMsg("Getting Directory info for... " + sDir);
foreach (string f in Directory.GetFiles(sDir, "*.tif"))
{
txtViewed.Text = Convert.ToString(Convert.ToInt32(txtViewed.Text)
+ 1);
FileInfo fi = new FileInfo(f);
//textBox1.Text = ("File size of mscdrun.cab: {0}" +
fi.Length.ToString());

if (fi.CreationTime >
DateTime.Now.AddDays(-Convert.ToInt32(numberDays)))
{
if (fi.Length > Convert.ToInt32(minimumFilesize))
{


I would pre-calculate some of those values, instead of doing it
over and over again:
- store the cutoff-date as DateTime (once)
- pre-convert mimimumFilesize to int (once)
- use a local int to store number of files read, and convert *that* to string for the textbox
or, if you know how many files to check, use a progressbar instead of a textbox,
might be more efficient (writing to screen is expensive)

This might shave off some time, but I doubt it will gain you "hours".

Hans Kesting
Nov 17 '05 #6

<ga******@wmcdirect.com> wrote in message
news:11*********************@z14g2000cwz.googlegro ups.com...
Unfortunately - I have only read access to these remote servers.

I like the idea if I could just put a little program out there to run
time-scheduled and create a nice little text file of all the qualifying
Tif files. If I can't get anything else to work this may have to be
done.

Thanks
Greg


If those boxes are open for WMI queries, you could use System.Management and
WMI's class "cim_datafile"

for instance following query returns all files that have a size > 5000000
bytes from d:\developertools that have been modified since June 5th, 2005

select * from cim_datafile where drive='d:' and path='\\developertools\\'
and filesize > 5000000
and lastmodified > ""20050605000000.000000+000"

When you connect to the remote box, the query is executed by the remote WMI
service and the result is send back to the client when done, running such
query on a remote box is just as fast as on a local box, for instance it
only takes less than 5 seconds to query a complete 40GB disk using next
query.
select * from cim_datafile where drive='d:' and filesize > 5000000
and lastmodified > ""20010605000000.000000+000"
Willy.

Nov 17 '05 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

14
by: Christopher Subich | last post by:
As a hobby project, I'm writing a MUD client -- this scratches an itch, and is also a good excuse to become familiar with the Python language. I have a conceptual handle on most of the...
9
by: netpurpose | last post by:
I need to extract data from this table to find the lowest prices of each product as of today. The product will be listed/grouped by the name only, discarding the product code - I use...
19
by: James Fortune | last post by:
I have a lot of respect for David Fenton and Allen Browne, but I don't understand why people who know how to write code to completely replace a front end do not write something that will automate...
1
by: BigJay | last post by:
I am 1. trying to have a combobox used as a selector to display records in a subform. and not sure on how to get selected info into subform.the combo is populated but can not get subform updated...
9
by: TPS | last post by:
I have a virtual directory where all posted files are stored. The ASP app does not have rights to the share on the other server where the vir dir resides. What is the best way to give the asp...
2
by: Steve | last post by:
I have a FileSystemWatcher watching a directory where I will add and update files. When it detects a change, I search through all the files and update my list of files in the UI. This works fine....
5
by: Col | last post by:
I have a query with criteria that comes from a combo box on a form. I'd like to set the query up so the user can either limit the criteria to the selection in the combo box or the user can pull all...
2
by: engwar1 | last post by:
I'm a .Net newbie and have started writing a Windows Forms application to assist me in choosing files/directories to move from one drive to another. Basically what I want is something like the...
4
by: Robert | last post by:
Greetings New to Visual Studio 2008. Always us VB6.0 SP5. I am trying to copy files from A to B. I select a directory, and copy files. Works ok, but it does not create the directory first. ...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.