473,465 Members | 1,946 Online
Bytes | Software Development & Data Engineering Community
Create Post

Home Posts Topics Members FAQ

Python - what is the fastest database ?

H!,

I'm testing things with Python with databases.
But I have one big question.

What is the 'fastest' database for the internet in combination with
Python ?

- with +/- 15 GB data.
- fast internet SELECT query's.

Python use bsddb but could he handle 15 GB fast ?

A other question is:
How is it possible that google (super big database) is super fast?
What type database do they use / software ?

Thanks Very Much,
GC-Martijn

Jul 18 '05 #1
5 4167
PA

On Feb 28, 2005, at 13:10, ma*****@gamecreators.nl wrote:
What type database do they use / software ?


Hmmm... they don't use a "database" in the traditional sense of the
term.

http://www-db.stanford.edu/~backrub/google.html

Cheers

--
PA, Onnay Equitursay
http://alt.textdrive.com/

Jul 18 '05 #2
PA wrote:
On Feb 28, 2005, at 13:10, ma*****@gamecreators.nl wrote:
What type database do they use / software ?


Hmmm... they don't use a "database" in the traditional sense of the term.

http://www-db.stanford.edu/~backrub/google.html


Section "4.3 Crawling the Web" in that discusses the role Python
plays in Google, at least in the form it (Google) had when this
paper was written.

-Peter
Jul 18 '05 #3
It depends on what you mean by database.

If you want really fast I/O, try pytables.
"PyTables is a hierarchical database package designed to efficiently
manage very large amounts of data."

http://pytables.sourceforge.net/html/WelcomePage.html

some more comments from the webpage:

# High performance I/O: On modern systems, and for large amounts of
data, tables and array objects can be read and written at a speed only
limited by the performance of the underlying I/O subsystem. Moreover,
if your data is compressible, even faster than your I/O maximum
throughput (!).

# Support of files bigger than 2 GB: So that you won't be limited if
you want to deal with very large datasets. In fact, PyTables support
full 64-bit file addressing even on 32-bit platforms (provided that the
underlying filesystem does so too, of course).

# Architecture-independent: PyTables has been carefully coded (as HDF5
itself) with little-endian/big-endian byte orderings issues in mind .
So, you can write a file in a big-endian machine (like a Sparc or MIPS)
and read it in other little-endian (like Intel or Alpha) without
problems.

# Portability: PyTables has been ported to many architectures, namely
Linux, Windows, MacOSX, FreeBSD, Solaris, IRIX and probably works in
many more. Moreover, it runs just fine also in 64 bit plaforms (like
AMD64, Intel64, UltraSparc or MIPS RXX000 processors).

Jul 18 '05 #4

<ma*****@gamecreators.nl> wrote in message
news:11**********************@g14g2000cwa.googlegr oups.com...
How is it possible that google (super big database) is super fast?

What type database do they use / software ?


On the hardware side, Google's secret is massively parallel cluster
computing, coupled with proprietary software for splitting tasks and
joining results. They have perhaps 200,000 CPUs. A query might be given
to hundreds of them for a fraction of a second.

Terry J. Reedy

Jul 18 '05 #5
km
Hi all,

Google has specially designed file system 'Goolgle File System' too.

KM
-------------------------------------------------------------------------
On Mon, Feb 28, 2005 at 01:43:53PM -0500, Terry Reedy wrote:

<ma*****@gamecreators.nl> wrote in message
news:11**********************@g14g2000cwa.googlegr oups.com...
How is it possible that google (super big database) is super fast?

What type database do they use / software ?


On the hardware side, Google's secret is massively parallel cluster
computing, coupled with proprietary software for splitting tasks and
joining results. They have perhaps 200,000 CPUs. A query might be given
to hundreds of them for a fraction of a second.

Terry J. Reedy

--

Jul 18 '05 #6

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: stephenlee | last post by:
I want to develop a simple Python program to edit source code that is persisted in a database (memo fields). The code to be edited is iHTML and SQL and I want syntax colouring and good basic...
137
by: Philippe C. Martin | last post by:
I apologize in advance for launching this post but I might get enlightment somehow (PS: I am _very_ agnostic ;-). - 1) I do not consider my intelligence/education above average - 2) I am very...
12
by: vpr | last post by:
Hi All I want to build an Website using Apache / Python and MySQL. I dont want to spend to much time hacking html. I'm looking for some recommendations e.g. should I be using mod_python ?...
10
by: A.M | last post by:
Hi, I am having difficulty with shell scripting in Python. I use the following command to run a DOS command and put the return value in a Python variable:
48
by: northband | last post by:
Hi, I am interested in re-writing my website in Python vs PHP but have a few questions. Here are my specs, please advise as to which configuration would be best: 1.Dell Poweredge Server, w/IIS,...
2
by: GinTon | last post by:
EyeDB is a free ODBMS based on the ODMG 3 specification with programming interfaces for C++ and Java. It is very powerfull, mature, safe and stable. In fact, it was developed in 1992 for the Genome...
3
by: Harry Haller | last post by:
What is the fastest way to search a client-side database? I have about 60-65 kb of data downloaded to the client which is present in 3 dynamically created list boxes. The boxes are filled from 3...
1
by: Harry Haller | last post by:
What is the fastest way to search a client-side database? I have about 60-65 kb of data downloaded to the client which is present in 3 dynamically created list boxes. The boxes are filled from 3...
23
by: Python Maniac | last post by:
I am new to Python however I would like some feedback from those who know more about Python than I do at this time. def scrambleLine(line): s = '' for c in line: s += chr(ord(c) | 0x80)...
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
1
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
0
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...
0
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing,...
0
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and...
0
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
0
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated ...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.