473,703 Members | 2,334 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

simultaneous multiple requests to very simple database

I have an application where I need a very simple database, effectively a
very large dictionary. The very large dictionary must be accessed from
multiple processes simultaneously. I need to be able to lock records
within the very large dictionary when records are written to. Estimated
number of records will be in the ballpark of 50,000 to 100,000 in his
early phase and 10 times that in the future. Each record will run about
100 to 150 bytes.

speed is not a huge concern although I must complete processing in less
than 90 seconds. The longer the delay however the greater number of
processes must be running parallel in order to keep the throughput up.
It's the usual trade-off we have all come to know and love.

it is not necessary for the dictionary to persist beyond the life of the
parent process although I have another project coming up in which this
would be a good idea.

at this point, I know they will be some kind souls suggesting various
SQL solutions. While I appreciate the idea, unfortunately I do not have
time to puzzle out yet another component. Someday I will figure it out
because I really liked what I see with SQL lite but unfortunately, today
is not that day (unless they will give me their work, home and cell
phone numbers so I can call when I am stuck. ;-)

So the solutions that come to mind are some form of dictionary in shared
memory with locking semaphore scoreboard or a multithreaded process
containing a single database (Python native dictionary, metakit, gdbm??)
and have all of my processes speak to it using xmlrpc which leaves me
with the question of how to make a multithreaded server using stock xmlrpc.

so feedback and pointers to information would be most welcome. I'm
still exploring the idea so I am open to any and all suggestions (except
maybe SQL :-)

---eric

Jul 18 '05 #1
10 2424
"Eric S. Johansson" <es*@harvee.org > wrote in message
news:ma******** *************** *************** @python.org...
<snip>
at this point, I know they will be some kind souls suggesting various
SQL solutions. While I appreciate the idea, unfortunately I do not have
time to puzzle out yet another component. Someday I will figure it out
because I really liked what I see with SQL lite but unfortunately, today
is not that day (unless they will give me their work, home and cell
phone numbers so I can call when I am stuck. ;-)

<snip>

Forgive me if this reply sounds a bit glib. But I do mean it without malice.

Do you seriously expect to write your own (database) solution and that this
will save you time and effort over learning an existing (SQL) solution?

Because -
If you are seeking to "save time" on "puzzles", you are certainly going
about it the wrong way.

Best of luck
Thomas Bartkus
Jul 18 '05 #2
Thomas Bartkus wrote:
"Eric S. Johansson" <es*@harvee.org > wrote in message
news:ma******** *************** *************** @python.org...
<snip>
at this point, I know they will be some kind souls suggesting various
SQL solutions. While I appreciate the idea, unfortunately I do not have
time to puzzle out yet another component. Someday I will figure it out
because I really liked what I see with SQL lite but unfortunately, today
is not that day (unless they will give me their work, home and cell
phone numbers so I can call when I am stuck. ;-)
<snip>

Forgive me if this reply sounds a bit glib. But I do mean it without malice.


understood and taken in that spirit.
Do you seriously expect to write your own (database) solution and that this
will save you time and effort over learning an existing (SQL) solution?

Because -
If you are seeking to "save time" on "puzzles", you are certainly going
about it the wrong way.


one thing I learned a long time ago was to respect the nagging voice in
the back of my head that says "there is something wrong". Right now
with databases, that voice is not nagging but screaming. So I made my
query to try and prove that intuition wrong. So far, that has not happened.

When I look at databases, I see a bunch of very good solutions that are
either overly complex or heavyweight on one hand and very nice and
simple but unable to deal with concurrency on the other. two sets of
point solutions that try to stretch themselves and the developers to fit
other application contexts.

99.9 percent of what I do (and I suspect this could be true for others)
could be satisfied by a slightly enhanced super dictionary with a record
level locking. but, the database world does not fit this model. It has
a great deal more complication then what is frequently necessary.

If I ever find the time, I will try to build such a beast probably
around Metakit. The only reason for reluctance is that I have spent too
many hours tracking down concurrency problems at the OS level way to
many years ago and so I do not create multithreaded applications lightly.

so in conclusion, my only reason for querying was to see if I was
missing a solution. So far, I have not found any work using because
they add orders of magnitude more complexity than simple dbm with file
locking. Obviously, the simple solution has horrible performance right
now I need simplicity implementation.

thanks for your commentary.

---eric

Jul 18 '05 #3
On Tue, 18 Jan 2005 17:33:26 -0500, Eric S. Johansson wrote:
When I look at databases, I see a bunch of very good solutions that are
either overly complex or heavyweight on one hand and very nice and simple
but unable to deal with concurrency on the other. two sets of point
solutions that try to stretch themselves and the developers to fit other
application contexts.


Have you considerded SQLite/pySQLite ?

--
Ricardo

Jul 18 '05 #4
On Tue, 18 Jan 2005 17:33:26 -0500, Eric S. Johansson <es*@harvee.org > wrote:
so in conclusion, my only reason for querying was to see if I was
missing a solution. So far, I have not found any work using because
they add orders of magnitude more complexity than simple dbm with file
locking. Obviously, the simple solution has horrible performance right
now I need simplicity implementation.

thanks for your commentary.


Maybe you can just get the best of both worlds.

Have a look at SQLObject. You can ignore the fact that underneath the
SQLObject there's a postgres (or mysql, or whatever) database, and get
OO based persistance.

SQLObject is crippled in that there are degrees of freedom that SQL
gives you that SQLObject takes away/makes hard to use, but what you're
trying to do, and what most people actually do with databases, can be
easily wrapped around with a simple, pythonic wrapper.

It even has a .createTable() function for those times when you don't
even want to log into the database.

Regards,
Stephen Thorne.
Jul 18 '05 #5
Ricardo Bugalho wrote:
On Tue, 18 Jan 2005 17:33:26 -0500, Eric S. Johansson wrote:

When I look at databases, I see a bunch of very good solutions that are
either overly complex or heavyweight on one hand and very nice and simple
but unable to deal with concurrency on the other. two sets of point
solutions that try to stretch themselves and the developers to fit other
application contexts.

Have you considerded SQLite/pySQLite ?


yep and apparently it won't work

http://www.sqlite.org/faq.html#q7

if I had record level locking, the code would do a very common pattern like:

if record present:
Lock record
modify record
release lock
else:
create record atomically (actual method TBB)

if I read their opinion correctly, the SQL lite folks are wrong in that
only the applications need massive concurrency. Small applications need
significant to massive concurrency for very tiny windows on very little
data.

but I do appreciate the pointer.

Jul 18 '05 #6
"Eric S. Johansson" <es*@harvee.org > wrote in message
news:ma******** *************** *************** @python.org...
<snip>
99.9 percent of what I do (and I suspect this could be true for others)
could be satisfied by a slightly enhanced super dictionary with a record
level locking.
BUT - Did you not mention! : Estimated number of records will be in the ballpark of 50,000 to 100,000 in his early phase and 10 times that in the future. Each record will run about 100 to 150 bytes. ..
And The very large dictionary must be accessed from
multiple processes simultaneously
And I need to be able to lock records
within the very large dictionary when records are written to
And although I must complete processing in less than 90 seconds.


And - the hole in the bottom of the hull -
all of the above using "a slightly enhanced super dictionary".

*Super* dictionary??? *Slightly* enhanced???
Have you attempted any feasability tests? Are you running a Cray?

There are many database systems available, and Python (probably) has free
bindings to every one of them. Whichever one might choose, it would add
simplicity, not complexity to what you are attempting. The problems you
mention are precisely those that databases are meant to solve. The only
tough (impossible?) requirement you have is that you don't want to use one.

When you write that "super dictionary", be sure to post code!
I could use one of those myself.
Thomas Bartkus
Jul 18 '05 #7
Thomas Bartkus wrote:
When you write that "super dictionary", be sure to post code!
I could use one of those myself.


hmmm it looks like you have just flung down the gauntlet of "put up or
quityerwhinging ". I need to get the crude implementation done first but
I think I can do it if I can find a good XMLRPC multithreading framework.

---eric

Jul 18 '05 #8
On Tue, 18 Jan 2005 11:26:46 -0500, Eric S. Johansson wrote:
So the solutions that come to mind are some form of dictionary in shared
memory with locking semaphore scoreboard or a multithreaded process
containing a single database (Python native dictionary, metakit, gdbm??)
and have all of my processes speak to it using xmlrpc which leaves me
with the question of how to make a multithreaded server using stock
xmlrpc.


Another solution might be to store the records as files in a directory,
and use file locking to control access to the files (careful over NFS!).

You might also consider berkeley db, which is a simple database to add to
an application, (and which I believe supports locks), but I must admit I'm
not a fan of the library.

I assume that the bottleneck is processing the records, otherwise this all
seems a bit academic.

Jeremy

Jul 18 '05 #9

Just learned of this today, so I don't know enough details to judge
its suitability for you:

Durus
http://www.mems-exchange.org/software/durus/

It does not do locking, but alleges to be compact and easy to
understand, so perhaps you could modify it to meet your needs,
or find some other way to handle that requirement.

-Tom

--

To respond by email, replace "somewhere" with "astro" in the
return address.
Jul 18 '05 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

16
7512
by: noah | last post by:
Does PHP have a feature to associate Cookie sessions with a persistent database connection that will allow a single transaction across multiple HTTP requests? Here is how I imagine my process: I have an series of interactive HTML forms. The user begins a Cookie session. A database connection is opened and a transaction is begun. After the user goes through any number of pages where they update the database they finish on a page where...
1
2446
by: slugger | last post by:
Hope this is not OT: I am running into some strange things whenever my ASP pages send out simultaneous requests to another ASP page which in turn gains access to a MySQL database using a DSNless connection string. Whenever I run a page that just puts out a single request things seem to go fine, but if I activate some Flash components that reside on my ASP pages that put out requests to ASP pages which in turn try to retrieve data from...
9
23078
by: Abhishek Srivastava | last post by:
Hello All, In IIS 6.0 We have a concept of worker processes and application pools. As I understand it, we can have multiple worker process per appliction pool. Each worker process is dedicated to a pool. If I assign only one application to a applicaton pool and have multiple worker processes assigned to that pool. Will my application be processed by many worker processes?
12
2311
by: Dan V. | last post by:
Since an ASP.NET/ADO.NET website is run on the server by a single "asp_net worker process", therefore doesn't that mean that even 50 simultaneous human users of the website would appear to the database backend as a single user. Therefore, wouldn't switching our Access (actually Jet 4.0 OLEDB) database backend on our ASP.NET/ADO.NET website to an MSDE backend be an advantageous change to make while NOT limiting the number of simultaneous web...
1
5168
by: googlegroups | last post by:
Hello everyone, I need a new web server for our existing website and I don't want subscribe to the whole IIS way of doing things. I thought I would implement a custom web server using the new HttpListener interface to HTTP.SYS. However, I found after playing with HttpListener that it seems to completely block all handling of incoming requests from the same IP address while a response to that IP address is outstanding. For
2
3838
by: dmagliola | last post by:
Hello all, I'm experiencing a problem with ASP.Net for which I can't find a reasonable explanation, or any information. I'm currently developing an application that, through AJAX, asks the server for updated information to show, and can also send information to the server. To do this, I have an XMLHttpRequest that queries an Asynchronous HTTP
35
9354
by: keerthyragavendran | last post by:
hi i'm downloading a single file using multiple threads... how can i specify a particular range of bytes alone from a single large file... for example say if i need only bytes ranging from 500000 to 3200000 of a file whose size is say 20MB... how do i request a download which starts directly at 500000th byte... thank u cheers
9
1986
by: David | last post by:
With a non-server app there is one instance of the program running and one user 'using' it at a time. With this scenario I'm pretty comfortable with variable scope and lifetime. With a server app there is one instance of the program running but several simultaneous clients connecting to and 'using' it. When I think about this I'm wondering what this may add to what needs to be considered for scope and lifetime... is a scenario created where...
21
2787
by: mark | last post by:
Hello, I want to create a php scraper that will get some information from e.g. 5 sites simultaneously. I tried the following script: http://www.phpied.com/simultaneuos-http-requests-in-php-with-curl/ Everything works fine, but what I want is simultaneuos (something to multithread, when these 5 websites will be loaded not one after another, but by using different sockets) scraper. In addition I would like to display the results as soon...
0
8669
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
9121
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
8962
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
7867
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
6588
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
5922
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
4432
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
4687
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
2
2450
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.