persistent fifo queue class

[David Bear]

I'm looking to see if there are any examples or prewritting fifo queue
classes. I know this is a broad topic. I'm looking to implement a simple
application where a web server enqueue and pickle using a local socket on
to a 'queue server' -- and then I will have another application dequeue the
pickles again using a local socket.

--
David Bear
-- let me buy your intellectual property, I want to own your thoughts --

[Diez B. Roggisch]

David Bear schrieb:

I'm looking to see if there are any examples or prewritting fifo queue
classes. I know this is a broad topic. I'm looking to implement a simple
application where a web server enqueue and pickle using a local socket on
to a 'queue server' -- and then I will have another application dequeue the
pickles again using a local socket.

Why don't you use a DB for that? If you want pickles, use a blob
column. But all the rest - a defined protocol, stable server,
transactions - you get for free.

Diez

[David Bear]

Diez B. Roggisch wrote:

David Bear schrieb:

>I'm looking to see if there are any examples or prewritting fifo queue
classes. I know this is a broad topic. I'm looking to implement a simple
application where a web server enqueue and pickle using a local socket on
to a 'queue server' -- and then I will have another application dequeue
the pickles again using a local socket.

Why don't you use a DB for that? If you want pickles, use a blob
column. But all the rest - a defined protocol, stable server,
transactions - you get for free.

Diez

Thanks for the suggestion. I did think of this. Indeed the final destination
of the data is in a db. However, the postsgresql server is on a separate
box. It will be connected via a private lan. I was worried that possible
network disruptions would cause either the web application to hang -- or
data to just get lost. I was thinking along the lines is a message queue
architecture, where the web app would push data directly onto a queue --
and then a worker app would dequeue the data and handle it by sending it to
the db server or otherwise.
--
David Bear
-- let me buy your intellectual property, I want to own your thoughts --

[Hendrik van Rooyen]

"David Bear" <da....be**@asu.eduwrote:

Diez B. Roggisch wrote:

Why don't you use a DB for that? If you want pickles, use a blob
column. But all the rest - a defined protocol, stable server,
transactions - you get for free.

Diez

Thanks for the suggestion. I did think of this. Indeed the final destination
of the data is in a db. However, the postsgresql server is on a separate
box. It will be connected via a private lan. I was worried that possible
network disruptions would cause either the web application to hang -- or
data to just get lost. I was thinking along the lines is a message queue
architecture, where the web app would push data directly onto a queue --
and then a worker app would dequeue the data and handle it by sending it to
the db server or otherwise.

Have you looked at Pyro?

I think it could help to tie the boxes on the local LAN together,
with a 'Pusher' on the first box and a 'Popper' on the box where the
final DB lives. And Diez' DB could provide the persistence on the first
box.
The 'push' and 'pop' above are a bit of a misnomer, as Pyro will allow
you to have a 'puller' on the final DB box to front end the DB.
I think a "double DB" architecture like that will be very robust as
Diez' DB can provide a "transactions queued, in process, and
completed" log.

- Hendrik

[Diez B. Roggisch]

David Bear schrieb:

Diez B. Roggisch wrote:

>David Bear schrieb:

>>I'm looking to see if there are any examples or prewritting fifo queue
classes. I know this is a broad topic. I'm looking to implement a simple
application where a web server enqueue and pickle using a local socket on
to a 'queue server' -- and then I will have another application dequeue
the pickles again using a local socket.

Why don't you use a DB for that? If you want pickles, use a blob
column. But all the rest - a defined protocol, stable server,
transactions - you get for free.

Diez

Thanks for the suggestion. I did think of this. Indeed the final destination
of the data is in a db. However, the postsgresql server is on a separate
box. It will be connected via a private lan. I was worried that possible
network disruptions would cause either the web application to hang -- or
data to just get lost. I was thinking along the lines is a message queue
architecture, where the web app would push data directly onto a queue --
and then a worker app would dequeue the data and handle it by sending it to
the db server or otherwise.

Well, if your DB connection is flaky, data will be corrupted - or lost.
In any case. What good does it do for you to have the messages then? And
what if the Harddisk the file sits on is corrupt? And don't forget the
added complexity of a home-brewn solution with all its possible errors -
much more likely to cause data loss.
Diez