473,326 Members | 2,111 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,326 software developers and data experts.

client/server design and advice

I recently completed the general guidelines for a future project that I
would like to start developing...but I've sort of hit a wall with
respect to how to design it. In short, I want to run through
approximately 5gigs of financial data, all of which is stored in a
large number of text files. Now as far as formatting and data
integrity...I would go through and ensure that each file had the
required setup so thats not really the issue. The problem I am having
is with respect to speed.

The languages I knew the best when coming into this project includes
c++ and php. However, I then thought about how long it would take one
PC to iterate through everything and figured it would probably take a
significant amount of time. As such, I started looking into various
languages and python caught my interest the most due to its power and
what seems to be ease of use. I was going to initially just use python
as a means of creating various indicators (i.e. calculations that would
be performed on the data in the file)...however I am leaning towards
moving to python entirely mostly due to its gui support.

First off, i was wondering if this is a reasonable setup: The entire
process would involve a server which manages which pc is processing
which set of data (which may be a given text file or the like), and a
client application which i would run on a few pc's locally when they
aren't in use. I would have a database (sqlite) holding all calculated
data of significance. Each client will basically login/connect with
the server, request a time interval (i.e. does anything need processed?
if so what data should i look at), and then it would update its status
with the server which would place a lock on that data set.

One thing i was wondering is if it would be worth it to use c++ for the
actual iteration through the text file or should i simply use python?
While i'm sure that c++ would be faster i am not entirely sure its
worth the headache if its not going to save me significant processing
time. Another thing is...if i was going to work with python instead of
c++, would it be worth it to import all of the data into an sqlite
database before hand (for speed issues)?

Lastly, as far as the networking goes, i have seen posts and such about
something called Pyro (http://pyro.sourceforge.net) and wondered if
that was worth looking into for the client/server interaction.

I apologize if any of these questions are more lower level, this is
simply the first client/server application ive created and am doing so
in a language ive never used before ;)

Thanks for the help
-Tony

Dec 1 '06 #1
5 1215
First off, i was wondering if this is a reasonable setup: The entire
process would involve a server which manages which pc is processing
which set of data (which may be a given text file or the like), and a
client application which i would run on a few pc's locally when they
aren't in use. I would have a database (sqlite) holding all calculated
data of significance. Each client will basically login/connect with
the server, request a time interval (i.e. does anything need processed?
if so what data should i look at), and then it would update its status
with the server which would place a lock on that data set.
Don't use sqlite, use a "real" RDBMS. sqlite is cool, but not really suited
for large amounts of data, and the concurrent access aspects that are dealt
with with an RDBMS for free are not to be underestimated.
One thing i was wondering is if it would be worth it to use c++ for the
actual iteration through the text file or should i simply use python?
While i'm sure that c++ would be faster i am not entirely sure its
worth the headache if its not going to save me significant processing
time. Another thing is...if i was going to work with python instead of
c++, would it be worth it to import all of the data into an sqlite
database before hand (for speed issues)?
I'd be putting them in the DB, yes.
Lastly, as far as the networking goes, i have seen posts and such about
something called Pyro (http://pyro.sourceforge.net) and wondered if
that was worth looking into for the client/server interaction.
Pyro rocks for that.

Diez
Dec 1 '06 #2
Don't use sqlite, use a "real" RDBMS. sqlite is cool, but not really suited
for large amounts of data, and the concurrent access aspects that are dealt
with with an RDBMS for free are not to be underestimated.
Would PostgreSQL be suitable in this situation? I hadn't even thought
about the possible problems that could arise with concurrency but i do
recall it being an issue the last time i worked with sqlite. I have
also looked into mysql given my extensive experience with it...however
postgresql seems to be faster from what ive read. Either way i'll work
on writing something to convert and insert the data so that it can
process while im working on the gui and client/server apps.
Pyro rocks for that.
Awesome, ill look into it in greater detail and will most likely use
it. Given what ive seen so far it looks like it will make the
client/server interface fairly easy to write.

Now...if only i could master python gui programming and development ;)
I'm not entirely sure which gui lib im going to end up using, but as of
now im leaning more towards tkinter as i know it will work where i need
and it seems to be one of the more documented. Ive looked at used
wxpython a little but had trouble figure out a few minor things while
playing around with it initially. I've also thought about pygtk
although I haven't taken the time to play with it quite yet as i
assumed it was primarily for linux (id be running the majority of these
on windows pcs).

Thanks for the suggestions :)
Tony

Dec 1 '06 #3
TonyM wrote:
Lastly, as far as the networking goes, i have seen posts and such about
something called Pyro (http://pyro.sourceforge.net) and wondered if
that was worth looking into for the client/server interaction.
I'm currently busy with a new version of Pyro (3.6) and it already
includes a new 'distributed computing' example, where there is
a single dispatcher service and one or more 'worker' clients.
The clients request work 'packets' from the dispatcher and
process them in parallel.
Maybe this is a good starting point of your system?
Current code is available from Pyro's CVS repository.

--Irmen
Dec 1 '06 #4
TonyM wrote:
<snip>
Pyro rocks for that.

Awesome, ill look into it in greater detail and will most likely use
it. Given what ive seen so far it looks like it will make the
client/server interface fairly easy to write.
Correction: not "fairly easy" - make that "incredibly easy". Even
Micky likes it. :=)
Now...if only i could master python gui programming and development ;)
I'm not entirely sure which gui lib im going to end up using, but as of
now im leaning more towards tkinter as i know it will work where i need
and it seems to be one of the more documented. Ive looked at used
wxpython a little but had trouble figure out a few minor things while
playing around with it initially. I've also thought about pygtk
although I haven't taken the time to play with it quite yet as i
assumed it was primarily for linux (id be running the majority of these
on windows pcs).
You would short change yourself if you don't check out the other
packages such as Pythoncard, and Dabo.

The other thing I recommend for large scale applications:

http://www-128.ibm.com/developerwork.../l-pythrd.html

Thanks for the suggestions :)
Tony
Dec 1 '06 #5
On 1 Dec 2006 10:37:39 -0800, John Henry <jo**********@hotmail.comwrote:
Now...if only i could master python gui programming and development ;)
You would short change yourself if you don't check out the other
packages such as Pythoncard, and Dabo.
FWIW, Dabo has all of the database connectivity stuff built-in. With
PythonCard, you have to roll your own.

--

# p.d.
Dec 1 '06 #6

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

7
by: rdh | last post by:
Hi all, I am in process of developing a Server in C++ supporting multiple protocols. The server will be exposing various functionalities, and the clients can communicate over any of the...
8
by: Uttam | last post by:
Hello, I am currently in the process of developing an application in a pure desktop world using Access 2000. I am intending to convert this pure desktop application into a Client Server...
67
by: Mike MacSween | last post by:
I've got a SQL Server database. Nearly finished. It's going to go on a single non networked machine. One day somebody might get access to it over ADSL (probably TS), but for now it's a single user...
2
by: Brian | last post by:
NOTE ALSO POSTED IN microsoft.public.dotnet.framework.aspnet.buildingcontrols I have solved most of my Server Control Collection property issues. I wrote an HTML page that describes all of the...
2
by: Randy Cox | last post by:
I've been tasked with developing an application that will have a server program providing portions of data to be processed by multiple client programs running on separate systems across a possible...
6
by: Simon Harvey | last post by:
Hi everyone, We have a need to make a Windows Forms (2.0) client application that will be installed on our clients site. The data that the application uses needs to be centrally available to a...
11
by: Jeff | last post by:
Hello everyone. I've searched through the archives here, and it seems that questions similar to this one have come up in the past, but I was hoping that I could pick your Pythonic brains a bit. ...
12
by: bullockbefriending bard | last post by:
I am a complete ignoramus and newbie when it comes to designing and coding networked clients (or servers for that matter). I have a copy of Goerzen (Foundations of Python Network Programming) and...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.