Caching in memory for Apache

[Simon Johnson]

Dear All,

I have decided to take the big plunge and drop the Microsoft platform
and use Mod_Python and Apache in it's place.

I've never used Linux before this project so it's a really big learning
curve, but so far it's going well. I've managed to create some simple
pages using the basic of Python.

One of the questions I have is: How do you cache stuff in memory? Say
I've got a bunch of stuff I need to create from a set of database rows.
Obviously, I do not want to consult the database each time I have a
request for my page. Is there anyway in Mod_python for me to store
values inside memory so I can look it up on the next GET request?

Simon.

[Harlin Seritt]

Pickle (http://www.network-theory.co.uk/docs/pytut/tut_58.html) comes
to mind when anyone mentions wanting to save a value, method, object,
etc without writing to storage (file or db). Is it possible for you to
post some code? There may be an alternative algorithm so you can avoid
writing to "memory."

Harlin Seritt

[Paul Boddie]

Simon Johnson wrote:

I have decided to take the big plunge and drop the Microsoft platform
and use Mod_Python and Apache in it's place.
Congratulations! ;-)

[...]
One of the questions I have is: How do you cache stuff in memory? Say
I've got a bunch of stuff I need to create from a set of database rows.
Obviously, I do not want to consult the database each time I have a
request for my page. Is there anyway in Mod_python for me to store
values inside memory so I can look it up on the next GET request?

I suppose that you probably need to use sessions in your application in
order to achieve what you're aiming for. If you're not familiar with
the concept of a session in the context of Web applications, sessions
are just things which associate a particular user or client with
persistent information stored on the server: when sending responses
(ie. Web pages) to users/clients, you include a token which those
users/clients then submit back to you with each request; you use these
tokens to look up any information that you've stored for quick
retrieval; the method of transmitting tokens is most often a "cookie"
which is automatically sent back by browsers unless configured
otherwise.

I know that mod_python does support sessions, including in-memory
sessions, as do a number of different frameworks. It should be fairly
convenient to take database rows and store them in a session, and in
most cases the framework (eg. mod_python) will manage the low-level
details (such as the sending of cookies, and so on). Sometimes, there
are restrictions on what the persistent information in sessions can
actually look like: some implementations use things like pickle, and
you can run into "unpickleable" objects; some implementations recommend
that you only store things like strings or text.

Whether sessions are ultimately the "right" answer rests upon a number
of technical considerations, and some people have stated that they
believe them to be generally inappropriate. However, they're probably
the most immediately appropriate thing for you to use in this case.

Paul

[Adam DePrince]

On Fri, 2006-03-24 at 06:56 -0800, Simon Johnson wrote:

Dear All,

I have decided to take the big plunge and drop the Microsoft platform
and use Mod_Python and Apache in it's place.

I've never used Linux before this project so it's a really big learning
curve, but so far it's going well. I've managed to create some simple
pages using the basic of Python.

One of the questions I have is: How do you cache stuff in memory? Say
I've got a bunch of stuff I need to create from a set of database rows.
Obviously, I do not want to consult the database each time I have a
Why not hit the database each time? That's what it is there for. What
you are trying to do is called premature optimization.

You are facing a huge learning curve. My recommendation is to build it
first with close adherence to the KISS principal. After its built, take
it for a spin. If it is too slow, then tune it. Think towards
implementation simplicity, not speed. You will probably find that
whatever you build will be fast enough when you are finished.

request for my page. Is there anyway in Mod_python for me to store
values inside memory so I can look it up on the next GET request?

Your database really is the best place for that.

Good luck - Adam DePrince

[grahamd]

Simon Johnson wrote:

Dear All,

I have decided to take the big plunge and drop the Microsoft platform
and use Mod_Python and Apache in it's place.

I've never used Linux before this project so it's a really big learning
curve, but so far it's going well. I've managed to create some simple
pages using the basic of Python.

One of the questions I have is: How do you cache stuff in memory? Say
I've got a bunch of stuff I need to create from a set of database rows.
Obviously, I do not want to consult the database each time I have a
request for my page. Is there anyway in Mod_python for me to store
values inside memory so I can look it up on the next GET request?

It all depends a bit on what you want to cache and whether the data is
specific to one users interaction with a page, or whether the data is
able to be used across requests from multiple users. This will dictate
whether you can use a global data cached within modules used to
generate
the page, whether you can use sessions, shared memory or whether you
have no choice but to always go back to the database.

As others have pointed out, don't get trapped in doing premature
optimisation. More often than not what you may think will be a bottle
neck will not and any problems if they even exist will be elsewhere.

In respect of mod_python, it is also vitally important to understand
the
process/interpreter module under which Python runs within the context
of Apache. A starting point for getting information about this is:

http://www.dscpl.com.au/articles/modpython-004.html

If you do decide that sessions may be useful to what you are doing, as
pointed out by others, there can be limitations as to what data can be
stored. For limitations in respect of sessions in mod_python have a
look
at:

http://www.dscpl.com.au/articles/modpython-005.html

Finally, make sure you use the latest version of mod_python available.
Avoid using mod_python 2.7.X or 3.1.X as there are lots of known issues
and problems. Not all problems are solved in 3.2.X version, but good
progress has been made on getting rid of many issues. The main area
which can still cause problems, especially when trying to cache data in
the Apache process, is that of module loading and reloading. For a run
down on issues in that area see:

http://www.dscpl.com.au/articles/modpython-003.html

A lot of work has already been done on solving these issues for 3.3.X
version of mod_python.

BTW, the best place to ask questions about mod_python is the mod_python
mailing list. Thus, get yourself subscribed to it. Details of the
mailing list
are on the mod_python web site.

Graham