473,320 Members | 1,863 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

How much memory used by a name

Hello,

I would like to know if there is a way to know how much memory (bytes,
kilobytes, megabytes, etc) a name is using.

More specifically, I have this list of strings that I want to write to
a file as lines.
This list grows througout the script execution, and toward the end,
the file is written.

However I would like to know how much memory, before writing to the
file, is this list using. Is it possible at all?
Thanks
Bernard
Feb 14 '07 #1
7 1232
Bernard Lebel wrote:
Hello,

I would like to know if there is a way to know how much memory (bytes,
kilobytes, megabytes, etc) a name is using.

More specifically, I have this list of strings that I want to write to
a file as lines.
This list grows througout the script execution, and toward the end,
the file is written.

However I would like to know how much memory, before writing to the
file, is this list using. Is it possible at all?
How about summing up the individual string lengths?

total = sum(len(s) for s in my_list)
Diez
Feb 14 '07 #2
Bernard Lebel a écrit :
Hello,

I would like to know if there is a way to know how much memory (bytes,
kilobytes, megabytes, etc) a name is using.

More specifically, I have this list of strings that I want to write to
a file as lines.
This list grows througout the script execution, and toward the end,
the file is written.
Do you really need to first grow the list then write it ?

Feb 14 '07 #3
Diez: thanks, I will try that. However isn't sum() returning an
integer that here would represent the number of elements?
Bruno: good question. We're talking about text files that can have
300,000 lines, if not more. Currently, the way I have coded the file
writing, every line calls for a write() to the file object, which in
turns write to the text file. The file is on the network.

This is taking a long time, and I'm looking for ways to speed up this
process. I though that keeping the list in memory and dropping to the
file at the very end could be a possible approach.
Bernard


On 2/14/07, Bruno Desthuilliers <bd*****************@free.quelquepart.frwrote:
Bernard Lebel a écrit :
Hello,

I would like to know if there is a way to know how much memory (bytes,
kilobytes, megabytes, etc) a name is using.

More specifically, I have this list of strings that I want to write to
a file as lines.
This list grows througout the script execution, and toward the end,
the file is written.

Do you really need to first grow the list then write it ?

--
http://mail.python.org/mailman/listinfo/python-list
Feb 14 '07 #4
Bernard Lebel a écrit :
Diez: thanks, I will try that. However isn't sum() returning an
integer that here would represent the number of elements?
Nope, it will return the sum of the length of the lines in the list. The
long way to write it is:

total = 0
for line in thelist:
total += len(line)

>
Bruno: good question. We're talking about text files that can have
300,000 lines, if not more. Currently, the way I have coded the file
writing, every line calls for a write() to the file object,
Seems sensible so far...
The file is on the network.
Mmm... Let's guess : it's taking too much time ?-)
This is taking a long time,
(You know what ? I cheated)
and I'm looking for ways to speed up this
process. I though that keeping the list in memory and dropping to the
file at the very end could be a possible approach.
OTOH, if the list grows too big, you may end up swapping (ok, it would
need a very huge list). A "mixed" solution may be to wrap the file in a
"buffered" writer that only perform a real write when it's full. This
would avoid effective i/o on each line while keeping memory usage
reasonable. Another one would be async I/O, but I don't know if and how
it could be done in Python (never had to manage such a problem myself).

My 2 cents...
Feb 14 '07 #5
On Feb 15, 11:08 am, Bruno Desthuilliers
<bdesth.quelquech...@free.quelquepart.frwrote:
Bernard Lebel a écrit :
Diez: thanks, I will try that. However isn't sum() returning an
integer that here would represent the number of elements?

Nope, it will return the sum of the length of the lines in the list. The
long way to write it is:

total = 0
for line in thelist:
total += len(line)
Bruno: good question. We're talking about text files that can have
300,000 lines, if not more. Currently, the way I have coded the file
writing, every line calls for a write() to the file object,

Seems sensible so far...
The file is on the network.

Mmm... Let's guess : it's taking too much time ?-)
This is taking a long time,

(You know what ? I cheated)
and I'm looking for ways to speed up this
process. I though that keeping the list in memory and dropping to the
file at the very end could be a possible approach.

OTOH, if the list grows too big, you may end up swapping (ok, it would
need a very huge list). A "mixed" solution may be to wrap the file in a
"buffered" writer that only perform a real write when it's full. This
would avoid effective i/o on each line while keeping memory usage
reasonable. Another one would be async I/O, but I don't know if and how
it could be done in Python (never had to manage such a problem myself).

My 2 cents...
What i can suggest is to use threads (for async I/O). One approach
would be
to split the script into two, so you have your main thread generating
the
strings, then you use another thread (which has a Queue object) that
blocks
on the get method of the Queue, once it gets the string it then writes
it to
the file. Your main thread generates strings and keeps on adding this
to the
Queue of the second thread. How much of a speed advantage this will
provide
i do not know.

Email me ff you require assistance. I would be more than welcome to
help.

Cheers

Feb 15 '07 #6
Bernard Lebel wrote:
Bruno: good question. We're talking about text files that can have
300,000 lines, if not more. Currently, the way I have coded the file
writing, every line calls for a write() to the file object, which in
turns write to the text file. The file is on the network.
assuming an average line length of 30 (for program code) to 60-80
characters (for human text), that's no more than 12-24 megabytes of
data. few modern computers should have any trouble holding that in
memory.

just build the list in memory, and use a single "writelines" call to write
everything to disk.

(alternatively, try write("".join(data)). that'll use twice as much memory,
but may be a little bit faster)
This is taking a long time, and I'm looking for ways to speed up this
process. I though that keeping the list in memory and dropping to the
file at the very end could be a possible approach.
chances are that you're already I/O bound, though...

</F>

Feb 15 '07 #7
On Feb 14, 9:41 pm, "Bernard Lebel" <3dbern...@gmail.comwrote:
This is taking a long time, and I'm looking for ways to speed up this
process. I though that keeping the list in memory and dropping to the
file at the very end could be a possible approach.
It seems, that you're trying to reinvent wheel. Why don't you just use
g'old mmap()?
http://docs.python.org/lib/module-mmap.html
Feb 15 '07 #8

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

9
by: Mike P | last post by:
I know everything about reference counting and making sure you don't have large objects lying around. I have also profiled my app with multiple tools. I know about the fact GC collects memory but...
7
by: boss_bhat | last post by:
Hi all , I am beginner to C programming. I have a defined astructure like the following, and i am using aliases for the different data types in the structure, typedef struct _NAME_INFO {...
5
by: SStory | last post by:
Hi all, I really needed to get the icons associated with each file that I want to show in a listview. I used the follow modified code sniplets found on the internet. I have left in...
4
by: Sean Shanny | last post by:
To all, Running into an out of memory error on our data warehouse server. This occurs only with our data from the 'September' section of a large fact table. The exact same query running over...
0
by: Antonio Arauzo Azofra | last post by:
Hello everybody, Probably, this is being too demanding for Python, but it may be useful to unimport modules to work with dynamic code (though not the best, one example is ). In fact, it is...
2
by: Mike | last post by:
Hi, I am new to C and having problems with the following program. Basically I am trying to read some files, loading data structures into memory for latter searching. I am trying to use structres...
9
by: weidongtom | last post by:
Hi, I've written the code that follows, and I use the function add_word(), it seems to work fine *before* increase_arrays() is called that uses realloc() to allocate more memory to words. But...
39
by: Ravi | last post by:
Can you all please suggest a program which tell us the range of memry addresses occupied by the given c program?
3
by: crazy420fingers | last post by:
I'm running a python program that simulates a wireless network protocol for a certain number of "frames" (measure of time). I've observed the following: 1. The memory consumption of the program...
6
by: Mahendra | last post by:
I have two cases - 1. When I have a pointer A pointing to a heap memory - What happens when I dereference the pointer A using free(A). It deallocated the heap memory the pointer was pointing...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.