Memory-resident message queue

[quigstah]

From: qu******@gmail. com - view profile
Date: Wed, Apr 5 2006 12:56 am
Email: "quigs...@gmail .com" <quigs...@gmail .com>
Groups: comp.unix.progr ammer
Not yet rated
Rating:
show options

Reply | Reply to Author | Forward | Print | Individual Message | Show
original | Remove | Report Abuse | Find messages by this author

Hey Folks!

Here's the scenario: I'm trying to rapidly develop some C code that
interfaces with some embedded Python. I need custom message queues, to
pass data between the C and Python.

I'm looking for something more high-level than IPC shared memory.
Berkeley DB seems a very viable option, but I'm concerned that it can't
be kept completely memory resident (as I'm expecting millions of tiny
messages being passed back and forth, and I want to avoid high disk i/o
if possible).

The messages are to be strings of some (possibly dynamic) length. Some
metadata will have to be associated with each message, including a
unique integer id, and a few other minor things.

I've considered allocating a big block of memory and making some custom
API to it with an xml-rpc protocol. I'm much more inclined to use
something that is already out there, of course, especially since I'm
racing to get this code out the door. I also heard of Prevaylor
(Java-based prevalence layer), which is an interesting concept I hadn't
previously heard of.

Any thoughts or ideas? I'm open to anything, and I want to thank you
in advance for your time and consideration.

Regards,
John Quigley
https://chicagolug.org/~jquigley/

[Nick Keighley]

qu******@gmail. com wrote:

<snip header stuff>

Here's the scenario: I'm trying to rapidly develop some C code that
interfaces with some embedded Python. I need custom message queues, to
pass data between the C and Python.
this is really off-topic to comp.lang.c I've added comp.programmin g to
the groups posted to
I'm looking for something more high-level than IPC shared memory.
Berkeley DB seems a very viable option, but I'm concerned that it can't
be kept completely memory resident (as I'm expecting millions of tiny
messages being passed back and forth, and I want to avoid high disk i/o
if possible).
how big can your queue get? If you are holding millions of entries in
the
queue it may get quite large.
The messages are to be strings of some (possibly dynamic) length. Some
metadata will have to be associated with each message, including a
unique integer id, and a few other minor things.

I've considered allocating a big block of memory and making some custom
API to it with an xml-rpc protocol. I'm much more inclined to use
something that is already out there, of course, especially since I'm
racing to get this code out the door. I also heard of Prevaylor
(Java-based prevalence layer), which is an interesting concept I hadn't
previously heard of.

Any thoughts or ideas? I'm open to anything, and I want to thank you
in advance for your time and consideration.

I'm rather a fan of roll-your own. XML-RPC seems a bit heavy weight to
me
but I don't know your application. I'd be thinking linked list or
cyclic buffer
in IPC. The dynamic strings sound a bit messy (is there an upper limit)
but not totally undoable. I'm sure the XP will be telling you to write
a test.
I'd ignore the shared memory bit for the moment and implment a simple
queing
system. Presumably there are libraries out there to do this sort of
thing.
Google?

In the past I've used linked lists of blocks of memory. Reasonable
compromise
between flexibility and simple memory management.
--
Nick Keighley

[Ben Pfaff]

"qu******@gmail .com" <qu******@gmail .com> writes:

Berkeley DB seems a very viable option, but I'm concerned that it can't
be kept completely memory resident (as I'm expecting millions of tiny
messages being passed back and forth, and I want to avoid high disk i/o
if possible).

Berkeley DB has a memory-only mode.
--
"doe not call up Any that you can not put downe."
--H. P. Lovecraft

[Joe Estock]

Nick Keighley wrote:

qu******@gmail. com wrote:

<snip header stuff>

Here's the scenario: I'm trying to rapidly develop some C code that
interfaces with some embedded Python. I need custom message queues, to
pass data between the C and Python.
this is really off-topic to comp.lang.c I've added comp.programmin g to
the groups posted to

I'm looking for something more high-level than IPC shared memory.
Berkeley DB seems a very viable option, but I'm concerned that it can't
be kept completely memory resident (as I'm expecting millions of tiny
messages being passed back and forth, and I want to avoid high disk i/o
if possible).

how big can your queue get? If you are holding millions of entries in
the
queue it may get quite large.

The messages are to be strings of some (possibly dynamic) length. Some
metadata will have to be associated with each message, including a
unique integer id, and a few other minor things.

I've considered allocating a big block of memory and making some custom
API to it with an xml-rpc protocol. I'm much more inclined to use
something that is already out there, of course, especially since I'm
racing to get this code out the door. I also heard of Prevaylor
(Java-based prevalence layer), which is an interesting concept I hadn't
previously heard of.

Any thoughts or ideas? I'm open to anything, and I want to thank you
in advance for your time and consideration.

[snip]

In the past I've used linked lists of blocks of memory. Reasonable
compromise
between flexibility and simple memory management.

I'd recommend a doubly-linked list for the simple fact that traversing
it would be a little less cpu-intensive than starting from the top of
the list and looping through untill you find the right entry. Since the
OP is using an integral type to uniquely identify the message you could
store them in sequential order from smallest to largest which would make
it faster to search through them.

Just my two cents worth.

Joe

[Nick Keighley]

Joe Estock wrote:

Nick Keighley wrote:

qu******@gmail. com wrote:

Here's the scenario: I'm trying to rapidly develop some C code that
interfaces with some embedded Python. I need custom message queues, to
pass data between the C and Python.

this is really off-topic to comp.lang.c I've added comp.programmin g to
the groups posted to

I'm looking for something more high-level than IPC shared memory.
Berkeley DB seems a very viable option, but I'm concerned that it can't
be kept completely memory resident (as I'm expecting millions of tiny
messages being passed back and forth, and I want to avoid high disk i/o
if possible).

how big can your queue get? If you are holding millions of entries in
the queue it may get quite large.

The messages are to be strings of some (possibly dynamic) length. Some
metadata will have to be associated with each message, including a
unique integer id, and a few other minor things.

I've considered allocating a big block of memory and making some custom
API to it with an xml-rpc protocol. I'm much more inclined to use
something that is already out there, of course, especially since I'm
racing to get this code out the door. I also heard of Prevaylor
(Java-based prevalence layer), which is an interesting concept I hadn't
previously heard of.

Any thoughts or ideas? I'm open to anything, and I want to thank you
in advance for your time and consideration.

In the past I've used linked lists of blocks of memory. Reasonable
compromise between flexibility and simple memory management.

I'd recommend a doubly-linked list for the simple fact that traversing
it would be a little less cpu-intensive than starting from the top of
the list and looping through untill you find the right entry. Since the
OP is using an integral type to uniquely identify the message you could
store them in sequential order from smallest to largest which would make
it faster to search through them.

Just my two cents worth.

yes but if its a true FIFO queue he can simply add to the tail and
remove from
the head. Just two pointers and double linking doesn't do much for you.
In m experience it's more usual to process messages in order of arrival
unless there are some special priority messages. Processing in order of
msg type sounds... odd.
--
Nick Keighley