473,769 Members | 1,674 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Need speed increase while reading large chunk of data

2 New Member
The discussion about the title already available in this forum. But i want to know how to copy the 2d array content or from a buffer into vector.
or is there any other way to read data into vector.
Nov 22 '06 #1
2 1883
horace1
1,510 Recognized Expert Top Contributor
The discussion about the title already available in this forum. But i want to know how to copy the 2d array content or from a buffer into vector.
or is there any other way to read data into vector.
what do you mean by a buffer? is the data in
1. a file to be read and put into a vector - if so is the file contents text or binary?
2. in an array[][] to be copied into a vector

what is the data? would you require the resultant vector to be 1D or 2D, etc? give an example
Nov 22 '06 #2
anandaraj
2 New Member
what do you mean by a buffer? is the data in
1. a file to be read and put into a vector - if so is the file contents text or binary?
2. in an array[][] to be copied into a vector

what is the data? would you require the resultant vector to be 1D or 2D, etc? give an example
i am reading log data.It is a bulk one nearly 1GB.I need to put all in memory for later process.I can read all with in 2 min.But when i am using the vector to store the value(read from the file) it takes more than 10 min.I want to reduce the time.same query already posted in this forum,where the following is adviced.First we need to read all the data into buffer then copy it into vector.I want to try this thing.so i want to know how to copy the data from the buffer into vector with out using push back.or else is there any other alternative to reduce the time which is required to read the log data from the file.The resultant vector should to be a structure pointer.

Thanks
Nov 29 '06 #3

Sign in to post your reply or Sign up for a free account.

Similar topics

14
1742
by: ^ | last post by:
Hi group, I've become interested in Python a while ago and just converted a simple perl script to python. The script is very simple, it generates a list of found virusses from some maillog files for further processing. I've found that there's a huge difference in execution time for the scripts, in favor of perl and I can't pinpoint what's going wrong; perl runs: 0.07 real 0.05 user 0.01 sys 0.07 real 0.05 user ...
8
2905
by: Darsant | last post by:
I'm currently reading 1-n number of binary files, each with 3 different arrays of floats containing about 10,000 values a piece for a total of about 30,000 values per file. I'm looking for a way to load them all into memory. I've tried using vector pushback with reserving, but it was horribly slow. The current method I am using is upon opening the file and reading the number of values, resizing the vectors (I have 3, one for each data...
23
2643
by: Mark Dickinson | last post by:
I have a simple 192-line Python script that begins with the line: dummy0 = 47 The script runs in less than 2.5 seconds. The variable dummy0 is never referenced again, directly or indirectly, by the rest of the script. Here's the surprise: if I remove or comment out this first line, the script takes more than 15 seconds to run. So it appears that adding a redundant line produces a spectacular six-fold increase in speed!
14
3719
by: Ron Johnson | last post by:
Hi, While on the topic of "need for in-place upgrades", I got to think- ing how the pg_restore could be speeded up. Am I wrong in saying that in the current pg_restore, all of the indexes are created in serial? How about this new, multi-threaded way of doing the pg_restore: 0. On the command line, you specify how many threads you want.
4
12803
by: Erpman | last post by:
I am trying to access the data with in a wav file. I am testing with very small files in order to keep the code simple to start with. Basically, im writing the entire wav file to a byte using a fileStream. The problem is that when reading back the data word by word and printing the results on a general form, once the data chunk header has been read, i.e, "data" and "chunk size", all the rest of the bytes in the array have the value...
6
3609
by: Lee Harr | last post by:
I have a database where I remove the schema public. When I try to use the createlang script, it fails like this ... >createdb foo CREATE DATABASE >psql foo -c "select version()" version --------------------------------------------------------------------- PostgreSQL 7.4.1 on i386-portbld-freebsd4.9, compiled by GCC 2.95.4 (1 row)
3
500
by: Froggy / Froggy Corp. | last post by:
First thx for your quick answer :) iostat dont return bad load average. I never grow up to 1Mb/s and the IDE drive can go faster. I really can't change the server, its a locative server which i own (only me on the server, no cooperative, but i dont have physically access to it). So the configuration is ... not the best :). And at this time, i dont have money for server-hosting.
2
1829
by: TheMadHatter | last post by:
Yello, Quick Q: What is the fastest way of getting data from a file? Does the though put of the file increase if I collect a great big chunk of it? Eg say 6meg? or does that even help? Does a bufferedstream work better if I am just going to open a file and read it?
18
1958
by: HYRY | last post by:
I want to join two mono wave file to a stereo wave file by only using the default python module. Here is my program, but it is much slower than the C version, so how can I increase the speed? I think the problem is at line #1, #2, #3. import wave import array lfile = wave.open(lfilename) rfile = wave.open(rfilename)
0
9423
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
10045
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
9994
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
9863
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
8870
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
6673
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5447
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
3958
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
3561
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.