473,804 Members | 2,140 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

unicode memory usage

We have an application which involves storing a lot of strings in RAM. It
would be most convenient to use Unicode strings, but I am wary of doubling
memory usage. My fear is based on the idea that unicode strings may take two
bytes per character in order to accomodate non-ascii characters.

But I don't know whether that's actually how Python strings work internally.

So, my question: Do unicode strings in Python take substantially more memory
than classic python strings or not, assuming the strings are generally 99%
ASCII characters (but not 100%)?
--Gary

--
Putting http://wecanstopspam.org in your email helps it pass through
overzealous spam filters.

Gary Robinson
CEO
Transpose, LLC
gr*******@trans pose.com
207-942-3463
http://www.transpose.com
http://radio.weblogs.com/0101454

Jul 18 '05 #1
1 2113
Gary Robinson wrote:
But I don't know whether that's actually how Python strings work internally.
Python Unicode objects use normally 2 bytes per character, unless Python
is built in UCS-4 mode, in which case they use 4 bytes per character.
So, my question: Do unicode strings in Python take substantially more memory
than classic python strings or not, assuming the strings are generally 99%
ASCII characters (but not 100%)?


Yes; you can expect that 99% of the storage for characters are null
bytes, then. Whether this is substantial depends on the total amount of
storage that you need for string objects, compared to the storage needed
for other things, or the storage available.

Regards,
Martin

Jul 18 '05 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

23
25964
by: Hallvard B Furuseth | last post by:
Has someone got a Python routine or module which converts Unicode strings to lowercase (or uppercase)? What I actually need to do is to compare a number of strings in a case-insensitive manner, so I assume it's simplest to convert to lower/upper first. Possibly all strings will be from the latin-1 character set, so I could convert to 8-bit latin-1, map to lowercase, and convert back, but that seems rather cumbersome.
5
6094
by: Justice | last post by:
Currently I'm doing some experimenting with the XMLHTTP object in Javascript. Now, the XMLHttp object is asynchronous (at least in this case), and the following code causes a significant memory loss even though I seem to be allocaitng everything; help would be *vastly* appreciated. What am I doing wrong here? I thought I was doing everything correctly (setting things to null, for example) but none of the memory seems to get replaced. ...
2
460
by: tomvr | last post by:
Hello I have noticed some 'weird' memory usage in a vb.net windows app The situation is as follows I have an app (heavy on images) with 2 forms (actually there are more forms and on starting the app I load some things into memory for global use of the app but I'll use only 2 starting forms to explain the situation) situation 1 start app with form 1 (72mb memory usage), show form 2 and hide form 1 (89 mb memory usage
6
3279
by: Tom | last post by:
We have a VERY simple .NET C# Form Application, that has about a 23MB Memory Footprint. It starts a window runs a process and does a regular expression. I have done a GC.Collect to make sure that, no memory is lying around. GC reports only 84k of allocations. Starting 5-10 of this apps is going to start taking a considerable amount of memory. Is there a way to reduce this? Tom
3
4154
by: Ian Taite | last post by:
Hello, I'm exploring why one of my C# .NET apps has "high" memory usage, and whether I can reduce the memory usage. I have an app that wakes up and processes text files into a database periodically. What happens, is that the app reads the contents of a text file line by line into an ArrayList. Each element of the ArrayList is a string representing a record from the file. The ArrayList is then processed, and the arraylist goes out of...
20
4252
by: Philip Carnstam | last post by:
How come .Net applications use so much memory? Every application I compile uses at least 10 MB of memory, even the ones consisting of only a form and nothing else. If I minimize them though the memory usage drops to a couple hundred KB. Why? Is there anything I should to to prevent this? I have compiled in release and deactivated all forms of debugging, I think! Thanks, Philip
5
2078
by: Tamir Khason | last post by:
I have a program will really big embedded text resources. Because of internationalization I have to save the embedded text in UTF-8, but it more then triple bigger then the original file. The last problem is the compiled file size, the REALLY problem is the memory amount used by the program, because of the embedded unicode file. It loads the values of the file into some hashes. With ASCII the program takes about 17k of RAM in runtime With...
13
3316
by: Tomás | last post by:
Let's start off with: class Nation { public: virtual const char* GetName() const = 0; } class Norway : public Nation { public: virtual const char* GetName() const
1
2051
by: Jean-Paul Calderone | last post by:
On Tue, 22 Apr 2008 14:54:37 -0700 (PDT), yzghan@gmail.com wrote: The test doesn't demonstrate any leaks. It does demonstrate that memory usage can remain at or near peak memory usage even after the objects for which that memory was allocated are no longer live in the process. This is only a leak if peak memory goes up again each time you create any new objects. Try repeated allocations of a large dictionary and observe how memory...
0
9715
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
10603
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
10353
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
9176
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
6869
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5536
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
5675
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
4314
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
3836
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.