473,289 Members | 2,091 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,289 software developers and data experts.

Python threads and memory usage

Hi,

I'm writing client-server application in Python. It's monitoring
system, where server listen and waits for TCP connections, and every
connection takes own thread. Every thread puts data from clients to
Queue and exits. Then there is one DB loader thread, which loads all
data from Queue to MySQL DB.

I observed, that every thread reserved some memory, and after exit
thread doesn't freed it. When i leaved my server working for 3 days,
then it takes 15% of 512MB memory (during that time about 15000
threads were created and stopped). When server starts it only takes
about 1% of memory.

I know that I can made client which connects once and keep this
session with server thread all time (it should resolve my problems),
but I'm curious if this is normal or it is something like memory leak.
Or maybe I'm doing something wrong :)

If You need I can paste all my code.

Regards
Jun 27 '08 #1
2 5255
On May 30, 9:16*am, Dennis Lee Bieber <wlfr...@ix.netcom.comwrote:
On Thu, 29 May 2008 12:01:30 -0700 (PDT), Mike <mszpad...@gmail.com>
declaimed the following in comp.lang.python:
I observed, that every thread reserved some memory, and after exit
thread doesn't freed it. When i leaved my server working for 3 days,
then it takes 15% of 512MB memory (during that time about 15000
threads were created and stopped). When server starts it only takes
about 1% of memory.

* * * * Do you have any outstanding references to the threads? If so, have
you performed a .join() with the thread? Until you join it, the thread
state (thread local objects/"variables") are probably being held for
access from outside the thread.
--
* * * * Wulfraed * * * *Dennis Lee Bieber * * * * * * * KD6MOG
* * * * wlfr...@ix.netcom.com * * * * * * wulfr...@bestiaria.com
* * * * * * * * HTTP://wlfraed.home.netcom.com/
* * * * (Bestiaria Support Staff: * * * * * * * web-a...@bestiaria.com)
* * * * * * * * HTTP://www.bestiaria.com/
I'm joining threads only during my program exit. I'll try to do what
You suggest.

THX
Jun 27 '08 #2
On May 30, 9:42*am, Mike <mszpad...@gmail.comwrote:
On May 30, 9:16*am, Dennis Lee Bieber <wlfr...@ix.netcom.comwrote:
On Thu, 29 May 2008 12:01:30 -0700 (PDT), Mike <mszpad...@gmail.com>
declaimed the following in comp.lang.python:
I observed, that every thread reserved some memory, and after exit
thread doesn't freed it. When i leaved my server working for 3 days,
then it takes 15% of 512MB memory (during that time about 15000
threads were created and stopped). When server starts it only takes
about 1% of memory.
* * * * Do you have any outstanding references to the threads? If so, have
you performed a .join() with the thread? Until you join it, the thread
state (thread local objects/"variables") are probably being held for
access from outside the thread.
--
* * * * Wulfraed * * * *Dennis Lee Bieber * * * * * * * KD6MOG
* * * * wlfr...@ix.netcom.com * * * * * * wulfr...@bestiaria.com
* * * * * * * * HTTP://wlfraed.home.netcom.com/
* * * * (Bestiaria Support Staff: * * * * * * * web-a...@bestiaria.com)
* * * * * * * * HTTP://www.bestiaria.com/

I'm joining threads only during my program exit. I'll try to do what
You suggest.

THX
It helped. Now all threads are added to thread list and every some
period of time I'm checking which threads are alive (enumerate), and
joining all which aren't. Now [memory usage is still on 1% :D:D:D

Thanks
Jun 27 '08 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

10
by: Andrew Dalke | last post by:
Is there an author index for the new version of the Python cookbook? As a contributor I got my comp version delivered today and my ego wanted some gratification. I couldn't find my entries. ...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: marcoviolo | last post by:
Dear all, I would like to implement on my worksheet an vlookup dynamic , that consider a change of pivot excel via win32com, from an external excel (without open it) and save the new file into a...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.