473,788 Members | 3,068 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Dealing with CSV

I have to import a CSV file into my system and I am currently spliting the
rows and then the columns and inserting that data into my objects. This is
very slow, does anyone have another way of doing this?

Here is my psuedo code

//Reload DetailIDs

char[] Newline = System.Environm ent.NewLine.ToC harArray();

string[] Rows = strData.Split(N ewline);

int MaxRows = Rows.GetUpperBo und(0);
//Loop through each row and parse out first column

for (i = 0; i <= MaxRows ; i++)

{

string ColumnData = Rows[i].Trim();

if (ColumnData.Len gth > 0)

{

string[] Cols = ColumnData.Spli t(',');

//Get each Value

for(int tmpCounter = ColValIndex; tmpCounter <= ColMax; tmpCounter++)

{

//Do something

}

}

}
Nov 15 '05 #1
3 1578
HI,
Weee that is a huge way to solve the csv file? what
would you do if you had around 60,000 records and above.
What I would suggest is why dont you put the csv file into
a dataset and then do the required functioning from
dataset than taking individual row and column.

CSV - > DataSet --> do the reqd changes..

Cheers,
Pradeep M M
-----Original Message-----
I have to import a CSV file into my system and I am currently spliting therows and then the columns and inserting that data into my objects. This isvery slow, does anyone have another way of doing this?

Here is my psuedo code

//Reload DetailIDs

char[] Newline = System.Environm ent.NewLine.ToC harArray();

string[] Rows = strData.Split(N ewline);

int MaxRows = Rows.GetUpperBo und(0);
//Loop through each row and parse out first column

for (i = 0; i <= MaxRows ; i++)

{

string ColumnData = Rows[i].Trim();

if (ColumnData.Len gth > 0)

{

string[] Cols = ColumnData.Spli t(',');

//Get each Value

for(int tmpCounter = ColValIndex; tmpCounter <= ColMax; tmpCounter++)
{

//Do something

}

}

}
.

Nov 15 '05 #2
Ok. Does anyone now how I can import a CSV to a dataset. The problem with
my XML files is that it is a CSV file inside of XML
Here is an example:
<XMLIntegration >9/10/2003 10:23:23 AM

<Schedule COLUMNS="Statio nName, Date_time, Item1, Item2, Item3, Item4,
Item5">

Columbia 1, 9/5/2003 0:00:00 AM, 164, 48, 0, 1, 1

Columbia 1, 9/5/2003 0:30:00 AM, 164, 48, 1, 1, 1

Columbia 1, 9/5/2003 1:00:00 AM, 164, 48, 2, 1, 1

Columbia 1, 9/5/2003 1:30:00 AM, 164, 48, 3, 1, 1

Columbia 1, 9/5/2003 2:00:00 AM, 164, 48, 4, 1, 1

Columbia 1, 9/5/2003 2:30:00 AM, 164, 48, 5, 1, 1

Columbia 1, 9/5/2003 3:00:00 AM, 164, 48, 6, 1, 1

</Schedule>

</XMLIntegration>

"Pradeep M M" <pr*******@hotm ail.com> wrote in message
news:07******** *************** *****@phx.gbl.. .
HI,
Weee that is a huge way to solve the csv file? what
would you do if you had around 60,000 records and above.
What I would suggest is why dont you put the csv file into
a dataset and then do the required functioning from
dataset than taking individual row and column.

CSV - > DataSet --> do the reqd changes..

Cheers,
Pradeep M M
-----Original Message-----
I have to import a CSV file into my system and I am

currently spliting the
rows and then the columns and inserting that data into my

objects. This is
very slow, does anyone have another way of doing this?

Here is my psuedo code

//Reload DetailIDs

char[] Newline = System.Environm ent.NewLine.ToC harArray();

string[] Rows = strData.Split(N ewline);

int MaxRows = Rows.GetUpperBo und(0);
//Loop through each row and parse out first column

for (i = 0; i <= MaxRows ; i++)

{

string ColumnData = Rows[i].Trim();

if (ColumnData.Len gth > 0)

{

string[] Cols = ColumnData.Spli t(',');

//Get each Value

for(int tmpCounter = ColValIndex; tmpCounter <= ColMax;

tmpCounter++)

{

//Do something

}

}

}
.

Nov 15 '05 #3
MFRASER,

I think there was an ODBC driver for text files that you could have used
for this. You can use the classes in the System.Data.Odb c namespace to
access this data by saving the CSV portion to a temp file, then reading what
you need from the temp file through the ODBC data provider.

Hope this helps.
--
- Nicholas Paldino [.NET/C# MVP]
- mv*@spam.guard. caspershouse.co m

"MFRASER" <mf*****@henwoo denergy.com> wrote in message
news:ev******** ******@TK2MSFTN GP12.phx.gbl...
Ok. Does anyone now how I can import a CSV to a dataset. The problem with my XML files is that it is a CSV file inside of XML
Here is an example:
<XMLIntegration >9/10/2003 10:23:23 AM

<Schedule COLUMNS="Statio nName, Date_time, Item1, Item2, Item3, Item4,
Item5">

Columbia 1, 9/5/2003 0:00:00 AM, 164, 48, 0, 1, 1

Columbia 1, 9/5/2003 0:30:00 AM, 164, 48, 1, 1, 1

Columbia 1, 9/5/2003 1:00:00 AM, 164, 48, 2, 1, 1

Columbia 1, 9/5/2003 1:30:00 AM, 164, 48, 3, 1, 1

Columbia 1, 9/5/2003 2:00:00 AM, 164, 48, 4, 1, 1

Columbia 1, 9/5/2003 2:30:00 AM, 164, 48, 5, 1, 1

Columbia 1, 9/5/2003 3:00:00 AM, 164, 48, 6, 1, 1

</Schedule>

</XMLIntegration>

"Pradeep M M" <pr*******@hotm ail.com> wrote in message
news:07******** *************** *****@phx.gbl.. .
HI,
Weee that is a huge way to solve the csv file? what
would you do if you had around 60,000 records and above.
What I would suggest is why dont you put the csv file into
a dataset and then do the required functioning from
dataset than taking individual row and column.

CSV - > DataSet --> do the reqd changes..

Cheers,
Pradeep M M
-----Original Message-----
I have to import a CSV file into my system and I am

currently spliting the
rows and then the columns and inserting that data into my

objects. This is
very slow, does anyone have another way of doing this?

Here is my psuedo code

//Reload DetailIDs

char[] Newline = System.Environm ent.NewLine.ToC harArray();

string[] Rows = strData.Split(N ewline);

int MaxRows = Rows.GetUpperBo und(0);
//Loop through each row and parse out first column

for (i = 0; i <= MaxRows ; i++)

{

string ColumnData = Rows[i].Trim();

if (ColumnData.Len gth > 0)

{

string[] Cols = ColumnData.Spli t(',');

//Get each Value

for(int tmpCounter = ColValIndex; tmpCounter <= ColMax;

tmpCounter++)

{

//Do something

}

}

}
.


Nov 15 '05 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
2226
by: Fazer | last post by:
Hello, I have the following code: #!/usr/local/bin/python import cgi import Cookie C = Cookie.SimpleCookie() C = "f"
3
2742
by: Matt Gerrans | last post by:
Is there an ideal approach to dealing with the FutureWarning about getting the hex value of a negative int? I'm using zlib.crc32() which can return a negative 32-bit int value, so for example print '%x' % zlib.crc32( buffer ) generates the FutureWarning. I know I could jump through hoops to coerce the int into a long that will have the same CRC, but is there some particular recomendation for this (I looked at PEP 237, but there wasn't...
5
1645
by: Tom Willis | last post by:
How are the expert pythoneers dealing with config files? Is there anything similair to .net's config files or java's .properties? A quick search on google didn't return anything that looked useful, and I almost would expect to see some module that would be for dealing with config information. I can think of at least one way to do it, but I'm sure there are shortcomings I can't see yet, and I'd rather use something someone
44
3789
by: flyingfred0 | last post by:
A small software team (developers, leads and even the manager when he's had time) has been using (wx)Python/PostgreSQL for over 2 years and developed a successful 1.0 release of a client/server product. A marketing/product manager has brought in additional management and "architecture" experts to propose moving the entire thing to a Java (application server) platform for the next release. They want a "scalable, enterprise solution"...
1
2423
by: news.microsoft.com | last post by:
Hello group, My goal is to attach an image over another image. Top image should be transparent so the back image is visible through the top one. Bellow is a test code in VB.NET. You need to create a form, a button (Button1) and an Picture Box (picBox). "watermark.jpg" is any jpg image, "beispiel.tif" is a multipage TIFF image with the white background, which will turn to transparent. The most important thing here is that the TIFF image...
2
2320
by: malcolm | last post by:
Hello, We have a robust (.NET 1.1 c# winforms) client-server application that utilizes many typed DataSets, typed DataTables and typed DataRows. Our application is a series of windows and popup windows where you can edit information and data, nothing out of the ordinary. I estimate we have something like 50 to 100 tables and/or views in our database each of which map to one strongly typed DataTable. Now we have some odd 20 to 30 typed...
1
1674
by: Igor Kramarsich - EDIT | last post by:
I'm having problems dealing with dates in my queries/SQL code. I need a search digit date. In form define date field without format I receive results. But I go to define date filed with short date format I receive error. In my regional settings are in the format "dd.mm.yyyy". What I need to do?
16
2045
by: pereges | last post by:
ok so i have written a program in C where I am dealing with huge data(millions and lots of iterations involved) and for some reason the screen tends to freeze and I get no output every time I execute it. However, I have tried to reduce the amount of data and the program runs fine. What could possibly be done to resolve this ?
2
2924
by: none | last post by:
All, I was wondering is their a way of dealing with subStatusCode's with in the webconfig file? I unfortunately don't have access to anything other then webconfig and qould perfer to set in code rater then my webhost's control panel. I've even tried: <customErrors mode="On" defaultRedirect="/pages/error.aspx" />
0
9964
AmberJain
by: AmberJain | last post by:
Windows Autorun FAQs: Programs dealing with autoruns Linked from the Original article- "Windows Autorun FAQs: Description". Que: Can you list programs that help me to view/modify the autoruns on my Windows PC? Ans: Other than Sysinternals Autoruns, there are only a few programs which are good enough to be used when dealing with autoruns. For more information about Sysinternals Autoruns, read Que-7 in Windows Autorun FAQs: Description....
0
9655
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
10363
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
10172
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
9964
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
1
7517
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
5535
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
4069
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
3670
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
2894
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.