473,409 Members | 2,057 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,409 software developers and data experts.

Parsing huge data

440 256MB
Hi ,

I have to parse a huge data of 3GB and store it in the dictnoray.After storing the data ,I need to do lot of calculations and write to the output file.The application is working for 500 MB file data,but for 3GB file it throwing Out of Memory message on the Python Editor.

Could you please help me with the best approach of data handling and traversing.

Thanks
PSB
Oct 23 '09 #1
1 1618
bvdet
2,851 Expert Mod 2GB
Please show us what you were doing codewise when you received the memory error. Will you end up with a dictionary of 3 GB? Consider storing the data in a database such as MySQL.

BV
Oct 23 '09 #2

Sign in to post your reply or Sign up for a free account.

Similar topics

0
by: marco | last post by:
I'm trying to parse a xml bookmarkpage with php. I found a very useful example script about how you can parse a xml document with php. The scriptworks really smooth. The xml test document (See...
3
by: Girish | last post by:
Hi All, I have written a component(ATL COM) that wraps Xerces C++ parser. I am firing necessary events for each of the notifications that I have handled for the Content and Error handler. The...
4
by: ralphNOSPAM | last post by:
Is there a function or otherwise some way to pull out the target text within an XML tag? For example, in the XML tag below, I want to pull out 'CALIFORNIA'. ...
6
by: Daniel Walzenbach | last post by:
Hi, I have a web application which sometimes throws an “out of memory” exception. To get an idea what happens I traced some values using performance monitor and got the following values (for...
3
by: jkugler | last post by:
Hello, I am trying to store huge amounts of data from xml files and put them into a MySQL database. The xml files all are in this format: <?xml version="1.0" encoding="UTF-8"?>...
5
by: mailtogops | last post by:
Hi All, I am involved in one project which tends to collect news information published on selected, known web sites inthe format of HTML, RSS, etc and sortlist them and create a bookmark on our...
3
by: CptDondo | last post by:
I am working on an embedded system. The entire configuration for the system is stored in an XML file, which is pretty long. It takes about 3 seconds to open the file using domxml_open_file. ...
3
by: toton | last post by:
Hi, I have some ascii files, which are having some formatted text. I want to read some section only from the total file. For that what I am doing is indexing the sections (denoted by .START in...
6
by: John Rogers | last post by:
Can someone show me how to parse a string to find a specific value? <b><a id="wt2500_WC_xc2500_GVB_drtl00_WQR_xt400_G" href="/WW/XZ/LinkToDetailsList.asp">Details List Filers</a></b> That is...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
0
BarryA
by: BarryA | last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
0
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
0
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development projectplanning, coding, testing,...
0
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.