473,583 Members | 2,873 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Sorting data in fixed file format

I'm looking for a way to sort text files consisting of fixed file
format. The files are big, typically over 10 million records and they
consist of about 100 fields with the record being over 600 bytes in
length. I need to
sort on a combination of 8 of the fields. Has anyobe attempted a to sort
data in this way using C#? Performance and scalability are the main
factors - the input data I
have is likely to grow probably up to 40 million records. I want to
avoid loading data into a database just to sort it.

Joe

*** Sent via Developersdex http://www.developersdex.com ***
Dec 16 '05 #1
5 3920
Hi,

IMO your best (if not unique) solution is to put everything in a database,
sort it and then export it. With this amount of data there is no a better
solution.

Importing/Exporting is very easy using DTS packages. let me know if you
need code for it
--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation

"booksnore" <sa******@pland atamgmt.com> wrote in message
news:Oo******** ******@tk2msftn gp13.phx.gbl...
I'm looking for a way to sort text files consisting of fixed file
format. The files are big, typically over 10 million records and they
consist of about 100 fields with the record being over 600 bytes in
length. I need to
sort on a combination of 8 of the fields. Has anyobe attempted a to sort
data in this way using C#? Performance and scalability are the main
factors - the input data I
have is likely to grow probably up to 40 million records. I want to
avoid loading data into a database just to sort it.

Joe

*** Sent via Developersdex http://www.developersdex.com ***

Dec 16 '05 #2
You have over 10 million records in a fixed file format, and you don't want
to put it into a database? Dude, that's what databases are *for*. Fixed file
formats are fine for storing static data. But filtering, selecting,
ordering, that's all stuff for databases.

--
HTH,

Kevin Spencer
Microsoft MVP
..Net Developer
You can lead a fish to a bicycle,
but it takes a very long time,
and the bicycle has to *want* to change.

"booksnore" <sa******@pland atamgmt.com> wrote in message
news:Oo******** ******@tk2msftn gp13.phx.gbl...
I'm looking for a way to sort text files consisting of fixed file
format. The files are big, typically over 10 million records and they
consist of about 100 fields with the record being over 600 bytes in
length. I need to
sort on a combination of 8 of the fields. Has anyobe attempted a to sort
data in this way using C#? Performance and scalability are the main
factors - the input data I
have is likely to grow probably up to 40 million records. I want to
avoid loading data into a database just to sort it.

Joe

*** Sent via Developersdex http://www.developersdex.com ***

Dec 16 '05 #3
Joe
Thanks Ignacio, what sort of code is it that you have? I would be
interested in taking a look.
regards
Joe

Joe

--
Sent via .NET Newsgroups
http://www.dotnetnewsgroups.com
Dec 16 '05 #4
Hi,

this is what I use, I create a DTS from enterprise manager , where the data
is comnig from and where to put it, then select "create a file" or something
similar in the DTS wizard, it does create a .dts file this is the one you
will use later

Here is the code, note that I change the datasource from the code, you can
do a similar thing with the destination:
Also you need to add a reference to DTS COM library

using DTS;
using System.Data;
using System.Data.Sql Client;

void RunPackage( string packSource, string packName, string dataSource)
{
try
{
Package2Class package = new Package2Class() ;
object pVarPersistStgO fHost = null;

// if you need to load from file
package.LoadFro mStorageFile(
packSource,
null,
null,
null,
packName,
ref pVarPersistStgO fHost);

/*
package.LoadFro mSQLServer(
"",
null,
null,
DTSSQLServerSto rageFlags.DTSSQ LStgFlag_UseTru stedConnection,
null,
null,
null,
"Test Import Package",
ref pVarPersistStgO fHost);
*/

package._Packag e_Connections.I tem(1).DataSour ce = dataSource;
package.Execute ();
package.UnIniti alize();

// force Release() on COM object
//
System.Runtime. InteropServices .Marshal.Releas eComObject(pack age);
package = null;
}
catch(System.Ru ntime.InteropSe rvices.COMExcep tion e)
{
Console.WriteLi ne("COMExceptio n {0}", e.ErrorCode.ToS tring() );
Console.WriteLi ne("{0}", e.Message);
Console.WriteLi ne("{0}", e.Source);
Console.WriteLi ne("Stack dump\n{0}\n", e.StackTrace);
Console.ReadLin e();
}
catch(System.Ex ception e)
{
Console.WriteLi ne("Exception") ;
Console.WriteLi ne("{0}", e.Message);
Console.WriteLi ne("{0}", e.Source);
Console.WriteLi ne("Stack dump\n{0}\n", e.StackTrace);

Console.ReadLin e();
}
}

--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation

"Joe" <bo*******@nets cape.net> wrote in message
news:uw******** ******@TK2MSFTN GP09.phx.gbl...
Thanks Ignacio, what sort of code is it that you have? I would be
interested in taking a look.
regards
Joe

Joe

--
Sent via .NET Newsgroups
http://www.dotnetnewsgroups.com

Dec 16 '05 #5
I want to sort the data in the file - that's it nothing else. Unix has a
sort command, I wondered if there was an equivalent in Windows.

Joe

*** Sent via Developersdex http://www.developersdex.com ***
Dec 17 '05 #6

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

7
3253
by: Federico G. Babelis | last post by:
Hi All: I have this line of code, but the syntax check in VB.NET 2003 and also in VB.NET 2005 Beta 2 shows as unknown: Dim local4 As Byte Fixed(local4 = AddressOf dest(offset)) CType(local4, Short) = CType(src, Short)
40
4245
by: Elijah Bailey | last post by:
I want to sort a set of records using STL's sort() function, but dont see an easy way to do it. I have a char *data; which has size mn bytes where m is size of the record and n is the number of records. Both these numbers are known
17
3671
by: Matt Kruse | last post by:
I'm looking for the best JS/CSS solution to add functionality to tables. The only browser which needs to be supported is IE5.5+, but no activeX can be used. to be able to do: - Fixed header row - Data sorting (preferrably intelligently determining data type based on content) - If possible, locking the first column from scrolling
22
4132
by: mike | last post by:
If I had a date in the format "01-Jan-05" it does not sort properly with my sort routine: function compareDate(a,b) { var date_a = new Date(a); var date_b = new Date(b); if (date_a < date_b) { return -1; } else
8
2254
by: nidhog | last post by:
Hello guys, I made a script that extracts strings from a binary file. It works. My next problem is sorting those strings. Output is like: ---- snip ---- 200501221530
7
4808
by: Kamal | last post by:
Hello all, I have a very simple html table with collapsible rows and sorting capabilities. The collapsible row is hidden with css rule (display:none). When one clicks in the left of the expandable row, the hidden row is made visible with css. The problem is when i sort the rows, the hidden rows get sorted as well which i don't want and want...
1
7179
KevinADC
by: KevinADC | last post by:
Introduction In part one we discussed the default sort function. In part two we will discuss more advanced techniques you can use to sort data. Some of the techniques might introduce unfamiliar methods or syntax to a less experienced perl coder. I will post links to online resources you can read if necessary. Experienced perl coders might find...
3
7322
KevinADC
by: KevinADC | last post by:
If you are entirely unfamiliar with using Perl to sort data, read the "Sorting Data with Perl - Part One and Two" articles before reading this article. Beginning Perl coders may find this article uses unfamiliar terms and syntax. Intermediate and advanced Perl coders should find this article useful. The object of the article is to inform the...
5
4922
by: jrod11 | last post by:
hi, I found a jquery html table sorting code i have implemented. I am trying to figure out how to edit how many colums there are, but every time i remove code that I think controls how many colums there are, it crashes. There are currently 6 columns, and I only want 4. How do I remove the last two (discount and date)? Here is a link:...
0
7888
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main...
0
8314
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
1
7922
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For...
0
6571
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then...
1
5689
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes...
0
3811
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in...
0
3836
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
2317
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
0
1147
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.