Massive unit test vs MySQL

[Richard Wesley]

I don't know if this is really a Python question of a MySQL question,
but i am hopen that there is enough overlap that someone can help me ;-)

I have a unit test suite for our server that loads a clean database
image for many of the tests. I use

p = os.popen('mysql -u uid -ppassword mydatabase', 'w')
p.write(sql_commands)

to load the database.

While the tests are running, I find that mysql has a growing number of
"Sleeping" threads in the 'show processlist' table. Towards the end of
the test suite, the database loads start to fail with a MySQL error of
"ERROR 1040: Too many connections".

So my question would be, is there any way to shut down the child process
so that MySQL notices and cleans up after itself before it runs out of
threads?

TIA,

--

- rmgw

<http://www.trustedmedianetworks.com/>

----------------------------------------------------------------------------
Richard Wesley Trusted Media Networks, Inc.

"If we're not careful, we could have a farce on our hands!"
- Tom Stoppard, _On The Razzle_

[Stephan Schaumann]

"Richard Wesley" <ha******@trustedmedianetworks.com> schrieb:

I don't know if this is really a Python question of a MySQL question,
but i am hopen that there is enough overlap that someone can help me ;-)

I have a unit test suite for our server that loads a clean database
image for many of the tests. I use

p = os.popen('mysql -u uid -ppassword mydatabase', 'w')
p.write(sql_commands)
Do you p.close() when you are finished? Otherwise the connection opend by
mysql will stay open.
to load the database.

While the tests are running, I find that mysql has a growing number of
"Sleeping" threads in the 'show processlist' table. Towards the end of
the test suite, the database loads start to fail with a MySQL error of
"ERROR 1040: Too many connections".

So my question would be, is there any way to shut down the child process
so that MySQL notices and cleans up after itself before it runs out of
threads?

[Neil Padgen]

On Thursday 04 September 2003 20:35, Richard Wesley wrote:

I have a unit test suite for our server that loads a clean database
image for many of the tests. I use

p = os.popen('mysql -u uid -ppassword mydatabase', 'w')
p.write(sql_commands)
Your pipe is still open at this point, and therefore you have a
connection to the database open.
So my question would be, is there any way to shut down the child
process so that MySQL notices and cleans up after itself before it
runs out of threads?

p.close()

-- Neil

[Richard Wesley]

In article <bj**********@nntp0.reith.bbc.co.uk>,
Neil Padgen <ne*********@mon.bbc.co.uk> wrote:

On Thursday 04 September 2003 20:35, Richard Wesley wrote:

I have a unit test suite for our server that loads a clean database
image for many of the tests. I use

p = os.popen('mysql -u uid -ppassword mydatabase', 'w')
p.write(sql_commands)

Your pipe is still open at this point, and therefore you have a
connection to the database open.

So my question would be, is there any way to shut down the child
process so that MySQL notices and cleans up after itself before it
runs out of threads?

p.close()

Yeah, I tried this, but it had no effect.

The odd thing is that the process list cleans up _immediately_ when the
script terminates. Maybe this is some sort of gc problem?

--

- rmgw

<http://www.trustedmedianetworks.com/>

----------------------------------------------------------------------------
Richard Wesley Trusted Media Networks, Inc.

"You're confusing boredom with motivation."
- Sherman in "Sherman's Lagoon"

[Richard Wesley]

In article <ha****************************@spectator.sj.sys.u s.xo.net>,
Richard Wesley <ha******@trustedmedianetworks.com> wrote:

In article <bj**********@nntp0.reith.bbc.co.uk>,
Neil Padgen <ne*********@mon.bbc.co.uk> wrote:

On Monday 08 September 2003 22:28, Richard Wesley wrote:

In article <bj**********@nntp0.reith.bbc.co.uk>,
Neil Padgen <ne*********@mon.bbc.co.uk> wrote:

I doubt it - it is generated by mysqldump. And there is no problem
from the command line.

I use exactly the same approach to set up my unit tests, but I have a
different way to get the mysqldump data into the database:

os.system('mysql database_name < dumpfile')

Sadly, this did not work, nor did garbage collection (gc.collect()).

For some reason, mysql is holding onto a whole lot of connections from
my script until the script terminates.

OK, I'm an idiot. It had nothing to do with the reload script. The
connections were from the TestCase subclass I have that instantiates a
connection management object. It appears that the unittest module does
not dispose of the objects it instantiates until the end of the run, so
you need to REALLY clean up in the tearDown method. I discovered this
by making a persistent connection for the reload operation and noticing
that the number of connections still increased.

Thanks to all who responded.

--
Best regards,

Richard Wesley
Co-President, Electric Fish, Inc.
<http://www.electricfish.com/>
(v) [phone]x210
(f) [phone]
(h) [phone]
(m) [phone]

[Neil Padgen]

>>>>> "Richard" == Richard Wesley <ha******@trustedmedianetworks.com> writes:

Richard> OK, I'm an idiot. It had nothing to do with the reload
Richard> script. The connections were from the TestCase subclass
Richard> I have that instantiates a connection management object.
Richard> It appears that the unittest module does not dispose of
Richard> the objects it instantiates until the end of the run, so
Richard> you need to REALLY clean up in the tearDown method. I
Richard> discovered this by making a persistent connection for the
Richard> reload operation and noticing that the number of
Richard> connections still increased.

Doh! I should have thought of this, as I've had exactly the same
problem in the past.

Thanks for posting your solution.

-- Neil