471,325 Members | 1,282 Online
Bytes | Software Development & Data Engineering Community
Post +

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 471,325 software developers and data experts.

Sorting data in fixed file format

I'm looking for a way to sort text files consisting of fixed file
format. The files are big, typically over 10 million records and they
consist of about 100 fields with the record being over 600 bytes in
length. I need to
sort on a combination of 8 of the fields. Has anyobe attempted a to sort
data in this way using C#? Performance and scalability are the main
factors - the input data I
have is likely to grow probably up to 40 million records. I want to
avoid loading data into a database just to sort it.

Joe

*** Sent via Developersdex http://www.developersdex.com ***
Dec 16 '05 #1
5 3817
Hi,

IMO your best (if not unique) solution is to put everything in a database,
sort it and then export it. With this amount of data there is no a better
solution.

Importing/Exporting is very easy using DTS packages. let me know if you
need code for it
--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation

"booksnore" <sa******@plandatamgmt.com> wrote in message
news:Oo**************@tk2msftngp13.phx.gbl...
I'm looking for a way to sort text files consisting of fixed file
format. The files are big, typically over 10 million records and they
consist of about 100 fields with the record being over 600 bytes in
length. I need to
sort on a combination of 8 of the fields. Has anyobe attempted a to sort
data in this way using C#? Performance and scalability are the main
factors - the input data I
have is likely to grow probably up to 40 million records. I want to
avoid loading data into a database just to sort it.

Joe

*** Sent via Developersdex http://www.developersdex.com ***

Dec 16 '05 #2
You have over 10 million records in a fixed file format, and you don't want
to put it into a database? Dude, that's what databases are *for*. Fixed file
formats are fine for storing static data. But filtering, selecting,
ordering, that's all stuff for databases.

--
HTH,

Kevin Spencer
Microsoft MVP
..Net Developer
You can lead a fish to a bicycle,
but it takes a very long time,
and the bicycle has to *want* to change.

"booksnore" <sa******@plandatamgmt.com> wrote in message
news:Oo**************@tk2msftngp13.phx.gbl...
I'm looking for a way to sort text files consisting of fixed file
format. The files are big, typically over 10 million records and they
consist of about 100 fields with the record being over 600 bytes in
length. I need to
sort on a combination of 8 of the fields. Has anyobe attempted a to sort
data in this way using C#? Performance and scalability are the main
factors - the input data I
have is likely to grow probably up to 40 million records. I want to
avoid loading data into a database just to sort it.

Joe

*** Sent via Developersdex http://www.developersdex.com ***

Dec 16 '05 #3
Joe
Thanks Ignacio, what sort of code is it that you have? I would be
interested in taking a look.
regards
Joe

Joe

--
Sent via .NET Newsgroups
http://www.dotnetnewsgroups.com
Dec 16 '05 #4
Hi,

this is what I use, I create a DTS from enterprise manager , where the data
is comnig from and where to put it, then select "create a file" or something
similar in the DTS wizard, it does create a .dts file this is the one you
will use later

Here is the code, note that I change the datasource from the code, you can
do a similar thing with the destination:
Also you need to add a reference to DTS COM library

using DTS;
using System.Data;
using System.Data.SqlClient;

void RunPackage( string packSource, string packName, string dataSource)
{
try
{
Package2Class package = new Package2Class();
object pVarPersistStgOfHost = null;

// if you need to load from file
package.LoadFromStorageFile(
packSource,
null,
null,
null,
packName,
ref pVarPersistStgOfHost);

/*
package.LoadFromSQLServer(
"",
null,
null,
DTSSQLServerStorageFlags.DTSSQLStgFlag_UseTrustedC onnection,
null,
null,
null,
"Test Import Package",
ref pVarPersistStgOfHost);
*/

package._Package_Connections.Item(1).DataSource = dataSource;
package.Execute();
package.UnInitialize();

// force Release() on COM object
//
System.Runtime.InteropServices.Marshal.ReleaseComO bject(package);
package = null;
}
catch(System.Runtime.InteropServices.COMException e)
{
Console.WriteLine("COMException {0}", e.ErrorCode.ToString() );
Console.WriteLine("{0}", e.Message);
Console.WriteLine("{0}", e.Source);
Console.WriteLine("Stack dump\n{0}\n", e.StackTrace);
Console.ReadLine();
}
catch(System.Exception e)
{
Console.WriteLine("Exception");
Console.WriteLine("{0}", e.Message);
Console.WriteLine("{0}", e.Source);
Console.WriteLine("Stack dump\n{0}\n", e.StackTrace);

Console.ReadLine();
}
}

--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation

"Joe" <bo*******@netscape.net> wrote in message
news:uw**************@TK2MSFTNGP09.phx.gbl...
Thanks Ignacio, what sort of code is it that you have? I would be
interested in taking a look.
regards
Joe

Joe

--
Sent via .NET Newsgroups
http://www.dotnetnewsgroups.com

Dec 16 '05 #5
I want to sort the data in the file - that's it nothing else. Unix has a
sort command, I wondered if there was an equivalent in Windows.

Joe

*** Sent via Developersdex http://www.developersdex.com ***
Dec 17 '05 #6

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

7 posts views Thread by Federico G. Babelis | last post: by
40 posts views Thread by Elijah Bailey | last post: by
22 posts views Thread by mike | last post: by
8 posts views Thread by nidhog | last post: by
7 posts views Thread by Kamal | last post: by
5 posts views Thread by jrod11 | last post: by
reply views Thread by rosydwin | last post: by

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.