473,696 Members | 1,779 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Large Amount of Data

I need to process large amount of data. The data structure fits well
in a dictionary but the amount is large - close to or more than the size
of physical memory. I wonder what will happen if I try to load the data
into a dictionary. Will Python use swap memory or will it fail?

Thanks.
May 25 '07 #1
16 2574
On May 25, 10:50 am, "Jack" <nos...@invalid .comwrote:
I need to process large amount of data. The data structure fits well
in a dictionary but the amount is large - close to or more than the size
of physical memory. I wonder what will happen if I try to load the data
into a dictionary. Will Python use swap memory or will it fail?

Thanks.
The OS will take care of memory swapping. It might get slow, but I
don't think it should fail.

Matt

May 25 '07 #2
In <H8************ *************** ***@comcast.com >, Jack wrote:
I need to process large amount of data. The data structure fits well
in a dictionary but the amount is large - close to or more than the size
of physical memory. I wonder what will happen if I try to load the data
into a dictionary. Will Python use swap memory or will it fail?
What about putting the data into a database? If the keys are strings the
`shelve` module might be a solution.

Ciao,
Marc 'BlackJack' Rintsch
May 25 '07 #3
Thanks for the replies!

Database will be too slow for what I want to do.

"Marc 'BlackJack' Rintsch" <bj****@gmx.net wrote in message
news:pa******** *************** *****@gmx.net.. .
In <H8************ *************** ***@comcast.com >, Jack wrote:
>I need to process large amount of data. The data structure fits well
in a dictionary but the amount is large - close to or more than the size
of physical memory. I wonder what will happen if I try to load the data
into a dictionary. Will Python use swap memory or will it fail?

What about putting the data into a database? If the keys are strings the
`shelve` module might be a solution.

Ciao,
Marc 'BlackJack' Rintsch

May 25 '07 #4
Jack wrote:
Thanks for the replies!

Database will be too slow for what I want to do.

"Marc 'BlackJack' Rintsch" <bj****@gmx.net wrote in message
news:pa******** *************** *****@gmx.net.. .
>In <H8************ *************** ***@comcast.com >, Jack wrote:
>>I need to process large amount of data. The data structure fits well
in a dictionary but the amount is large - close to or more than the size
of physical memory. I wonder what will happen if I try to load the data
into a dictionary. Will Python use swap memory or will it fail?
What about putting the data into a database? If the keys are strings the
`shelve` module might be a solution.

Ciao,
Marc 'BlackJack' Rintsch

Purchase more memory. It is REALLY cheap these days.

-Larry
May 25 '07 #5
On 5/25/07, Jack <no****@invalid .comwrote:
I need to process large amount of data. The data structure fits well
in a dictionary but the amount is large - close to or more than the size
of physical memory. I wonder what will happen if I try to load the data
into a dictionary. Will Python use swap memory or will it fail?

Thanks.
--
http://mail.python.org/mailman/listinfo/python-list
Could you process it in chunks, instead of reading in all the data at once?
May 26 '07 #6
Larry Bates wrote:
Jack wrote:
>Thanks for the replies!

Database will be too slow for what I want to do.

"Marc 'BlackJack' Rintsch" <bj****@gmx.net wrote in message
news:pa******* *************** ******@gmx.net. ..
>>In <H8************ *************** ***@comcast.com >, Jack wrote:

I need to process large amount of data. The data structure fits well
in a dictionary but the amount is large - close to or more than the size
of physical memory. I wonder what will happen if I try to load the data
into a dictionary. Will Python use swap memory or will it fail?
What about putting the data into a database? If the keys are strings the
`shelve` module might be a solution.

Ciao,
Marc 'BlackJack' Rintsch
Purchase more memory. It is REALLY cheap these days.
Not a solution at all. What about if amount of data exceed architecture
memory limits? i.e. 4Gb at 32bit.

Better solution is to use database for data storage/processing

--
Vyacheslav Maslov
May 26 '07 #7
Jack wrote:
I need to process large amount of data. The data structure fits well
in a dictionary but the amount is large - close to or more than the size
of physical memory. I wonder what will happen if I try to load the data
into a dictionary. Will Python use swap memory or will it fail?

Thanks.
What are you trying to do? At one extreme, you're implementing something
like a search engine that needs gigabytes of bitmaps to do joins fast as
hundreds of thousands of users hit the server, and need to talk seriously
about 64-bit address space machines. At the other, you have no idea how
to either use a database or do sequential processing. Tell us more.

John Nagle
May 26 '07 #8
I suppose I can but it won't be very efficient. I can have a smaller
hashtable,
and process those that are in the hashtable and save the ones that are not
in the hash table for another round of processing. But chunked hashtable
won't work that well because you don't know if they exist in other chunks.
In order to do this, I'll need to have a rule to partition the data into
chunks.
So this is more work in general.

"kaens" <ap************ ***@gmail.comwr ote in message
news:ma******** *************** *************** *@python.org...
On 5/25/07, Jack <no****@invalid .comwrote:
>I need to process large amount of data. The data structure fits well
in a dictionary but the amount is large - close to or more than the size
of physical memory. I wonder what will happen if I try to load the data
into a dictionary. Will Python use swap memory or will it fail?

Thanks.
--
http://mail.python.org/mailman/listinfo/python-list

Could you process it in chunks, instead of reading in all the data at
once?

May 26 '07 #9
If swap memery can not handle this efficiently, I may need to partition
data to multiple servers and use RPC to communicate.

"Dennis Lee Bieber" <wl*****@ix.net com.comwrote in message
news:YY******** **********@news read1.news.pas. earthlink.net.. .
On Fri, 25 May 2007 11:11:28 -0700, "Jack" <no****@invalid .com>
declaimed the following in comp.lang.pytho n:
>Thanks for the replies!

Database will be too slow for what I want to do.
Slower than having every process on the computer potentially slowed
down due to page swapping (and, for really huge data, still running the
risk of exceeding the single-process address space)?
--
Wulfraed Dennis Lee Bieber KD6MOG
wl*****@ix.netc om.com wu******@bestia ria.com
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: we******@bestia ria.com)
HTTP://www.bestiaria.com/

May 26 '07 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
2733
by: steve | last post by:
Hi, I have researched but have not found a good solution to this problem. I am importing large amounts of data (over 50 Meg) into a new mysql db that I set up. I use >mysql dbname < importfile.txt But I keep getting timeouts and errors due to the data being too large. I know that since if I break the imported data into multiple chuncks (by importing a few tables at a time) then everything works.
1
1625
by: Robert May | last post by:
Hi, I am trying to execute some code compiled by g++ on Linux and have found that after some time, the program allocates a huge amount of swap space (250MB on my machine which has 512MB physical) and (700MB on another server with 1GB physical RAM). I have used vmstat to trend the amount of swap and observed that the memory is not being "thrashed" and there is simply a large amount of data that has been swapped out. This still slows...
5
2182
by: Mike | last post by:
This is a general question on the best way to import a large amount of data to a MS-SQL DB. I can have the data in just about any format I need to, I just don't know how to import the data. I some experience with SQL but not much. There is about 1500 to 2000 lines of data. I am looking for the best way to get this amount of data in on a monthly basis. Any help is greatly thanked!!
11
4126
by: Macca | last post by:
Hi, I'm writing an application that will pass a large amount of data between classes/functions. In C++ it was more efficient to send a pointer to the object, e.g structure rather than passing the actual structure itself. Is this true of C# also?
3
1241
by: kamran | last post by:
Hi, I have a web service that may return a very large amount of data. I want that data to return in chunks, like first return 10% of data than return the next 10% and so on, until all is finished. How are other people doing this? I have come up with the following way: While inside the webmethod, fetch the data from the db and store it in a file called "someguid.data" and return the guid. The client will receive a guid and request again,...
7
10820
by: =?Utf-8?B?TW9iaWxlTWFu?= | last post by:
Hello everyone: I am looking for everyone's thoughts on moving large amounts (actually, not very large, but large enough that I'm throwing exceptions using the default configurations). We're doing a proof-of-concept on WCF whereby we have a Windows form client and a Server. Our server is a middle-tier that interfaces with our SQL 05 database server.
4
1926
by: bcomeara | last post by:
I am writing a program which needs to include a large amount of data. Basically, the data are p values for different possible outcomes from trials with different number of observations (the p values are necessarily based on slow simulations rather than on a standard function, so I estimated them once and want the program to include this information). Currently, I have this stored as a vector of vectors of varying sizes (first vector is...
22
3657
by: Jesse Burns | last post by:
I'm about to start working on my first large scale site (in my opinion) that will hopefully have 1000+ users a day. ok, this isn't on the google/facebook scale, but it's going to be have more hits than just family and friends. Either way, I'm planning on this site blowing up once I have enough of a feature set, so I'm concerned about performance and scalability in the long run. I've worked for a software company, but I've never...
0
2357
by: jcatubay | last post by:
I have a function that returns a list more than 200000 objects and the object has 37 fields. I added the wcf as a web reference so i dont have to add any configuration item in my web apps config file. WCF works when I only have small amount of data. SInce I did'nt add the config file, how can I allow the wcf to send large amount of data. It sems to be timing out. When I debugged my wcf I'm getting this error "An error occurred while receiving...
0
8656
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
9140
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
9005
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
8869
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
1
6512
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
5848
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
4351
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
2
2304
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
1988
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.