473,569 Members | 2,721 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Database suggestion

Hi.

I work with recursive array trees of 10-20 levels in PHP. So far I
have been using serialize() to store the arrays, generating files 5+
MB large, which take 10-15 seconds to unserialize and about tenfold of
RAM! I've been adviced to use MySQL (relational db's), but are there
any other options beside that and var_export/include? Something that
works in a similar way as MySQL when adding a new element without
loading the whole database itself...

Thanks!
Dec 5 '07 #1
27 1663
Mikhail Kovalev wrote:
I've been adviced to use MySQL (relational db's), but are there any
other options beside that and var_export/include? Something that works
in a similar way as MySQL when adding a new element without loading the
whole database itself...
SQLite, PostgreSQL, Oracle, Microsoft SQL Server, ...

--
Toby A Inkster BSc (Hons) ARCS
[Geek of HTML/SQL/Perl/PHP/Python/Apache/Linux]
[OS: Linux 2.6.17.14-mm-desktop-9mdvsmp, up 10 days, 18:31.]

Sharing Music with Apple iTunes
http://tobyinkster.co.uk/blog/2007/1...tunes-sharing/
Dec 5 '07 #2
I mean, I don't mind using MySQL in itself, it's just that I don't
want use relational db's.
I don't know if it is possible to predefine the structures I'm using
in MySQL, because the number of levels varies etc.
Here is an example recording a single sequence of 6 elements:

How do I put this in a db?

array (
0 =>
array (
0 =2,
1 =>
array (
1 =>
array (
0 =1,
1 =>
array (
),
),
2 =>
array (
0 =1,
1 =>
array (
3 =>
array (
0 =1,
1 =>
array (
4 =>
array (
0 =1,
1 =>
array (
3 =>
array (
0 =1,
1 =>
array (
1 =>
array (
0 =1,
1 =>
array (
),
),
),
),
),
),
),
),
),
),
),
),
1 =>
array (
0 =2,
1 =>
array (
),
),
2 =>
array (
0 =1,
1 =>
array (
3 =>
array (
0 =1,
1 =>
array (
4 =>
array (
0 =1,
1 =>
array (
3 =>
array (
0 =1,
1 =>
array (
1 =>
array (
0 =1,
1 =>
array (
),
),
),
),
),
),
),
),
),
),
3 =>
array (
0 =2,
1 =>
array (
4 =>
array (
0 =1,
1 =>
array (
3 =>
array (
0 =1,
1 =>
array (
1 =>
array (
0 =1,
1 =>
array (
),
),
),
),
),
),
1 =>
array (
0 =1,
1 =>
array (
),
),
),
),
4 =>
array (
0 =1,
1 =>
array (
3 =>
array (
0 =1,
1 =>
array (
1 =>
array (
0 =1,
1 =>
array (
),
),
),
),
),
),
)


On 5 Des, 12:42, Toby A Inkster <usenet200...@t obyinkster.co.u k>
wrote:
Mikhail Kovalev wrote:
I've been adviced to use MySQL (relational db's), but are there any
other options beside that and var_export/include? Something that works
in a similar way as MySQL when adding a new element without loading the
whole database itself...

SQLite, PostgreSQL, Oracle, Microsoft SQL Server, ...

--
Toby A Inkster BSc (Hons) ARCS
[Geek of HTML/SQL/Perl/PHP/Python/Apache/Linux]
[OS: Linux 2.6.17.14-mm-desktop-9mdvsmp, up 10 days, 18:31.]

Sharing Music with Apple iTunes
http://tobyinkster.co.uk/blog/2007/1...tunes-sharing/
Dec 5 '07 #3
Mikhail Kovalev wrote:
Hi.

I work with recursive array trees of 10-20 levels in PHP. So far I
have been using serialize() to store the arrays, generating files 5+
MB large, which take 10-15 seconds to unserialize and about tenfold of
RAM! I've been adviced to use MySQL (relational db's), but are there
any other options beside that and var_export/include? Something that
works in a similar way as MySQL when adding a new element without
loading the whole database itself...

Thanks!
What do you mean by "without loading the whole database itself"?

--
=============== ===
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
js*******@attgl obal.net
=============== ===

Dec 5 '07 #4
I unserialize an 8MB file, which contains the db array, and put the
new element there (then serialize the whole thing again and store)

On 5 Des, 13:28, Jerry Stuckle <jstuck...@attg lobal.netwrote:
Mikhail Kovalev wrote:
Hi.
I work with recursive array trees of 10-20 levels in PHP. So far I
have been using serialize() to store the arrays, generating files 5+
MB large, which take 10-15 seconds to unserialize and about tenfold of
RAM! I've been adviced to use MySQL (relational db's), but are there
any other options beside that and var_export/include? Something that
works in a similar way as MySQL when adding a new element without
loading the whole database itself...
Thanks!

What do you mean by "without loading the whole database itself"?

--
=============== ===
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstuck...@attgl obal.net
=============== ===
Dec 5 '07 #5
On 5 Des, 14:02, The Natural Philosopher <a...@b.cwrot e:
Mikhail Kovalev wrote:
I unserialize an 8MB file, which contains the db array, and put the
new element there (then serialize the whole thing again and store)

I am far from understanding what you are actually doing, but it sounds
like you are willing to trade disk usage for speed and memory usage.
Yep, you nailed it right on through, my friend

My question is, will I be able to access and store values by
addresses?
Such as here:
$database[33][NEXT][1221][NEXT][12][COUNT] += 1; for example

Each element contains a COUNT value and an array of childs (NEXT),
which can be non-empty, empty or non-existent (when i check if it
exists)
Looks like this at top level:

Array(
433 =Array(
COUNT =2121,
NEXT =Array( //
23 =Array( count&next etc ...)
433 =...
etc...
)
),
1012 =Array( count & next.....)
.....
)
Dec 5 '07 #6
My question is, will I be able to access and store values by
addresses?
Such as here:
$database[33][NEXT][1221][NEXT][12][COUNT] += 1; for example

Each element contains a COUNT value and an array of childs (NEXT),
which can be non-empty, empty or non-existent (when i check if it
exists)
Maybe you should tell us what it is exactly that you are storing,
cause it seems to me that you are over complicating things. What is it
you are counting and why do you currently group it in multi-level
arrays?

Marijn
Dec 5 '07 #7
On 5 Des, 14:11, "C. (http://symcbean.blogsp ot.com/)"
<colin.mckin... @gmail.comwrote :
On 5 Dec, 11:13, Mikhail Kovalev <mikhail_kova.. .@mail.ruwrote:
Hi.
I work with recursive array trees of 10-20 levels in PHP. So far I
have been using serialize() to store the arrays, generating files 5+
MB large, which take 10-15 seconds to unserialize and about tenfold of
RAM! I've been adviced to use MySQL (relational db's), but are there
any other options beside that and var_export/include? Something that
works in a similar way as MySQL when adding a new element without
loading the whole database itself...
Thanks!

Any relational DB can model a hierarchical data structure - at the
simplest level:

node_id unique identifier, not null
parent_node null
payload whatever

But getting the threaded structure is a bit tricky -

SELECT parent.node_id as Parent_node_id, parent.payload as
parent_payload,
child.node_id as child_node_id, child.payload as child_payload
FROM mytable parent, mytable child
WHERE child.parent_no de=parent.node_ id
AND parent.node_id= $start_here

.. i.e. by explicitly declaring aliases for each level in the tree you
can get the sub-nodes, but when you don't know how deep the tree is,
its not possible to return the whole structure in a single query using
standard SQL. Oracle allows you to compose queries returning
representations of hierarchical data (have a google for 'connect by' &
'start with') but thats rather non-standard.
I don't know how deep the tree is, but I don't need to return the
whole structure in a single query. Alas, I don't know the nature of
MySQL

At this moment everything I do with my structure happens by calling
addresses
See my reply to The Natural Philosopher.
Here's an example with one node at top level and it's childs:

112 (10)
225 (3)
930 (1)
11 (5)

$array[112][COUNT] = 10;
$array[112][NEXT][225][COUNT] = 3;
$array[112][NEXT][225][NEXT][930][COUNT] = 1;
$array[112][NEXT][225][NEXT][930][NEXT] = array();
$array[112][NEXT][11][COUNT] = 5;
$array[112][NEXT][11][NEXT] = array();

An address is an array, f ex array(112, 225, 930)

I then use the following mechanism:

// retrieves a node with its two values as an array(COUNT, NEXT)
// NEXT contain further childs, but i dont use them
// array_1 is the whole structure, and array_2 is the address
function function_7($arr ay_1, $array_2) { return eval('return
$array_1'.funct ion_8($array_2) .';'); }

// array(112, 225, 930) becomes "[112][NEXT][225][NEXT][930]"
function function_8($arr ay_1) { return implode('[NEXT]',
array_map('func tion_9', $array_1)); }

function function_9($int eger_1) { return '['.$integer_1.']'; }

// saves a value array_3 to an address array_2 in structure array_1
function function_10($ar ray_1, $array_2, $array_3)
{
eval('$array_1' .function_8($ar ray_2).' = $array_3;');
return $array_1;
}
Dec 5 '07 #8
On 5 Des, 14:52, Marijn <marijn.huizend v...@gmail.comw rote:
My question is, will I be able to access and store values by
addresses?
Such as here:
$database[33][NEXT][1221][NEXT][12][COUNT] += 1; for example
Each element contains a COUNT value and an array of childs (NEXT),
which can be non-empty, empty or non-existent (when i check if it
exists)

Maybe you should tell us what it is exactly that you are storing,
cause it seems to me that you are over complicating things. What is it
you are counting and why do you currently group it in multi-level
arrays?

Marijn
I'm storing some statistics on Markov chains of characters in words
Dec 5 '07 #9
On 5 Dec, 13:56, Mikhail Kovalev <mikhail_kova.. .@mail.ruwrote:
Alas, I don't know the nature of
MySQL
This has nothing to do with MySQL per se and everything to do with
understanding data storage models in general.

Here is some good reading on the subject.
http://del.icio.us/Captain_Paralytic/hierarchical
Dec 5 '07 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

5
9118
by: FoxRunner | last post by:
Hi folks, I got a script which restores a database. It works fine if it is running in my Query Analyzer. It fails when I put this script in an automated schedule using the SQL agent. This is my script
6
2311
by: teedilo | last post by:
We have an application with a SQL Server 2000 back end that is fairly database intensive -- lots of fairly frequent queries, inserts, updates -- the gamut. The application does not make use of performance hogs like cursors, but I know there are lots of ways the application could be made more efficient database-wise. The server code is...
13
7460
by: Manuel Lopez | last post by:
I have a puzzling form timer problem that I didn't experience prior to Access 2003 (though I'm not sure access 2003 is to blame). Here's the situation: a computer has two access 2003 databases on it, a frontend and a backend. Case 1: If vba code on the frontend updates many rows (360,000) on the backend, a form's timer event (from the...
1
2003
by: IkyL234 | last post by:
I'm using Access2000. I have just designed a database which seems to be operating very slow on a network. There are currently only a few records in it. Should I be compacting it now before it gets busy with usage ? The database has OLE bounded fields where pictures are linked to a network folder where the pictures are kept. Will this linking...
4
5576
by: Martin Pritchard | last post by:
Hi, I'm working on a project that historically contains around 40 enums. In the database various fields refer to the int values of these enums, but of course ref integrity is not enofrced and when looking at the db we can't tell what the value in a field represents. The other problem is that our enums are currently all stored in a single...
35
4817
by: Terry Jolly | last post by:
Web Solution Goal: Have a global database connection Why: (There will be 30+ tables, represented by 30+ classes) I only want to reference the database connection once. I put the connection string in the web.config. I created a class with a static database connection and the class opens and closes the database.
14
1429
by: @sh | last post by:
Guys, We're in the midst of building a new site and have some decisions to make RE the meta data, i.e. Title, Keywords and Description. We need to allow other non development staff to update such fields, there will be an administration panel available to do this. Seen as the site is very popular and receives many hits we're looking very...
4
2271
by: onecorp | last post by:
I have a SQL table comprised of 31 columns. The first column is simply an id column, the next 30 columns are labelled ,.... The numerical columns have a tinyint type and the data stored is either 1 or null. I wish to count the number of times a one appears in one column simultaneously with another column: eg count the number of times 1...
17
2542
by: Flic | last post by:
Is this possible? I know about a bit about ODBC and found how to import an ODBC database stored on the computer, however I am after accessing an SQL database stored on a webserver. I'd like to keep it up to date, but that could probably be done with a macro. At the moment I'm just considoring possible options, so just need to know if it...
10
2945
by: C# Beginner | last post by:
Hi there, I'm currently trying to create a MS access file at runtime, but I stumble into some problems. 1. When I use Datatype adUnsignedInt I get an error (invalid type). 2. Which datatype must I use to create an autonumbering field? 3. Which datatype must I use to create an OLE-object field? I'm currently using Microsoft ActiveX Data...
0
7695
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main...
0
7922
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
0
8119
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
1
7668
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For...
0
6281
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then...
1
5509
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes...
0
5218
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert...
1
2111
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
1
1209
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.