473,397 Members | 1,985 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,397 software developers and data experts.

how to read a large data file

Hello,...

I am new in c programming and I have to do some data analysis of some files where I have a header a 5 columns of data.
I want to reads the file, delete the wrong lines and wrong lines and save it in a new file.
e.g. header2 data start at 00:00:00 09-09-07
10000 2000e-4 20000e-4 2000e-4 20000e-4
..............

wrong files: 10001 65000 65000 65000 65000
or 10001 2000e-4 3000e-4 2000e-4 100002 2000e-4

I hope that can help me .

cheers
juan
Sep 5 '07 #1
4 2249
dmjpro
2,476 2GB
Hello,...

I am new in c programming and I have to do some data analysis of some files where I have a header a 5 columns of data.
I want to reads the file, delete the wrong lines and wrong lines and save it in a new file.
e.g. header2 data start at 00:00:00 09-09-07
10000 2000e-4 20000e-4 2000e-4 20000e-4
..............

wrong files: 10001 65000 65000 65000 65000
or 10001 2000e-4 3000e-4 2000e-4 100002 2000e-4

I hope that can help me .

cheers
juan

Welcome to TSDN.
How much you tried?
Show me that.

Kind regards,
Dmjpro.
Sep 5 '07 #2
ganeshp
38
first of all
how does the right line differ from wrong line
what is the indication in the line ; that lets you know that it is a wrong line ?

please answer the above question this will help me/others help you sovle the problem
Sep 5 '07 #3
first of all
how does the right line differ from wrong line
what is the indication in the line ; that lets you know that it is a wrong line ?

please answer the above question this will help me/others help you sovle the problem
Hi,
To determine the wrong file I have to set the number of values per line,
I mean, if I define that the right file has 5 values per column, a wrong file is when I have more than 5 values per line. Once we have defined the wrong line I delete it.

I hope this can help.
Sep 8 '07 #4
ganeshp
38
Hi,
To determine the wrong file I have to set the number of values per line,
I mean, if I define that the right file has 5 values per column, a wrong file is when I have more than 5 values per line. Once we have defined the wrong line I delete it.

I hope this can help.
Yes it is a good idea do it.
Have you solved it or do you still have problems?
Sep 18 '07 #5

Sign in to post your reply or Sign up for a free account.

Similar topics

11
by: Sebastian Krause | last post by:
Hello, I tried to read in some large ascii files (200MB-2GB) in Python using scipy.io.read_array, but it did not work as I expected. The whole idea was to find a fast Python routine to read in...
3
by: Johnny | last post by:
Hello all, I have a 1GB XML file that I need to read once a day and I would like to get feedback to find out what is the most efficient way to go about reading this file. The application reading...
4
by: pisscot | last post by:
Lind Apr 13, 6:32 am show options Newsgroups: comp.lang.c From: piss...@gmail.com (Lind) - Find messages by this author Date: 13 Apr 2005 06:32:15 -0700 Local: Wed,Apr 13 2005 6:32 am...
9
by: sweety | last post by:
Dear All, How to encrypt a C data file and make binary file and then have to read a bin file at run time and decrypt the file and have to read the data. Any help to achive this pls. Would be...
3
by: james.burrows | last post by:
I'm looking for a quick way to read lines of data from a text file into memory. Each line will contain e.g. a book entry: - id,name,price,category,author,description I could have over 20000 of...
6
by: comp.lang.php | last post by:
if (!function_exists('bigfile')) { /** * Works like file() in PHP except that it will work more efficiently with very large files * * @access public * @param mixed $fullFilePath * @return...
12
by: Sean Davis | last post by:
I am working on a simple script to read from one database (oracle) and write to another (postgresql). I retrieve the data from oracle in chunks and drop the data to postgresql continuously. The...
3
by: =?Utf-8?B?ZGF2aWQ=?= | last post by:
I try to follow Steve's paper to build a database, and store a small text file into SQL Server database and retrieve it later. Only difference between my table and Steve's table is that I use NTEXT...
2
by: Kevin Ar18 | last post by:
I posted this on the forum, but nobody seems to know the solution: http://python-forum.org/py/viewtopic.php?t=5230 I have a zip file that is several GB in size, and one of the files inside of it...
9
by: vineeth | last post by:
Hello all, I have come across a weird problem, I need to determine the amount of bytes read from a file, but couldn't figure it out , My program does this : __ file = open("somefile") data =...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
BarryA
by: BarryA | last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
0
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
0
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
0
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
0
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.