Working with Large vs Small databases

[Salad]

Every now and then I see ads that state something like "Experience with
Large Databases ...multi-gig...blah-de-blah" And I have to laugh.
What's the difference between a large or small database? A table is a
table, a record is a record, a field is a field. All you are doing is
manipulating data in tables. I wouldn't think it'd make much difference
in working with a table with 10 records or a billion records...they' re
nothing more than rows and columns.

That's like saying "Must be able to work with large amounts of money"
Whether an app works with 10s, 1000s, 1000000s, or 1000000000s of
dollars, it's nothing more than a value...the same concepts work with
all of them...it simply needs to be accurate.

Or am I wrong? Does size really matter?

[Rick Brandt]

Salad wrote:

Every now and then I see ads that state something like "Experience
with Large Databases ...multi-gig...blah-de-blah" And I have to
laugh. What's the difference between a large or small database? A
table is a table, a record is a record, a field is a field. All you
are doing is manipulating data in tables. I wouldn't think it'd make
much difference in working with a table with 10 records or a billion
records...they' re nothing more than rows and columns.

That's like saying "Must be able to work with large amounts of money"
Whether an app works with 10s, 1000s, 1000000s, or 1000000000s of
dollars, it's nothing more than a value...the same concepts work with
all of them...it simply needs to be accurate.

Or am I wrong? Does size really matter?

It certainly does. One can get away with doing LOTS of things wrong in
terms of inefficiencies with small databases. If you hire someone who has
worked exclusively on smaller projects there's a higher chance that they
have not yet learned all of the "best practices" for databases (particularly
keeping network traffic to a minimum). Someone who has a lot of experience
with large databases will already have learned how to make db apps as
efficient as possible.

--
I don't check the Email account attached
to this message. Send instead to...
RBrandt at Hunter dot com

[Kevin Nechodom]

--
Kevin Nechodom
University of Utah Hospital and Clinics
Kevin dit Nechodom ack hsc dit utah dit edu
"Call me paranoid, but I think you are reading what I'm
writing!"

Salad<oi*@vineg ar.com> 6/16/2005 6:40:23 AM >>>

Every now and then I see ads that state something like
"Experience with
Large Databases ...multi-gig...blah-de-blah" And I have

tolaugh.
What's the difference between a large or small database? Atable is a
table, a record is a record, a field is a field. All you
are doing is
manipulating data in tables. I wouldn't think it'd make
much difference
in working with a table with 10 records or a billion
records...they 're
nothing more than rows and columns.

That's like saying "Must be able to work with large amountsof money"
Whether an app works with 10s, 1000s, 1000000s, or
1000000000s of
dollars, it's nothing more than a value...the same conceptswork with
all of them...it simply needs to be accurate.

Or am I wrong? Does size really matter?

From http://www.kettering.edu/~jhuggins/humor/theory.html:

"In theory the difference between practice and theory is due
to practical considerations that theorists find it
impractical to fit into their theories.

"In practice, theory uses the practice of theorising about
practical matters, while not noticing that the theoretical
method practically distorts the theory beyond application to
practice.

"Theoretica lly then the practical facts are that the theory
is in practice good for predicting what happens in theory,
but impractical as a theory with direct implications for
practice, except where theory states that the practice is
sufficiently close to the theory to make any difference for
all practical purposes theoreticaly zero.

"In practice this does not happen very often."

In other words, "Yes, it makes a difference."

Small databases don't require the same degree of care,
because the effects of inefficiencies are likely to be
minor. Put those same problems into a large database, and
you can wind up being cursed by your users.

[Salad]

Kevin Nechodom wrote:

Or am I wrong? Does size really matter?

In other words, "Yes, it makes a difference."

Small databases don't require the same degree of care,
because the effects of inefficiencies are likely to be
minor. Put those same problems into a large database, and
you can wind up being cursed by your users.

I tend to disagree with both you and Rick. There are people that are
good at what they do, and those that aren't so good. I'm sure you've
worked on a project and you wonder if the previous programmer was brain
dead.

If one is competent, I should think it doesn't matter. All you are
doing with a database is manipulating data.

I doubt I'd get a job and someone would say "Salad, we need you to work
on ABC database. It's got a billion records and is 250 gigabytes in
size." and I quiver and quake and stutter..."But sir...I've only worked
on a database that was less than a gig and had less than a million
records!" and then rush out of the room, sobbing and crying as I make it
to the restroom where I retch my stomache dry in humiliation and fear.
I'd instead say "Cool, where's my cubicle? I want to look over the
tables and their structures."

[Rick Brandt]

Salad wrote:

Kevin Nechodom wrote:

Or am I wrong? Does size really matter?

In other words, "Yes, it makes a difference."

Small databases don't require the same degree of care,
because the effects of inefficiencies are likely to be
minor. Put those same problems into a large database, and
you can wind up being cursed by your users.

I tend to disagree with both you and Rick. There are people that are
good at what they do, and those that aren't so good. I'm sure you've
worked on a project and you wonder if the previous programmer was
brain dead.

If one is competent, I should think it doesn't matter. All you are
doing with a database is manipulating data.

I doubt I'd get a job and someone would say "Salad, we need you to
work on ABC database. It's got a billion records and is 250
gigabytes in size." and I quiver and quake and stutter..."But
sir...I've only worked on a database that was less than a gig and had
less than a million records!" and then rush out of the room, sobbing
and crying as I make it to the restroom where I retch my stomache dry
in humiliation and fear. I'd instead say "Cool, where's my cubicle?
I want to look over the tables and their structures."

Different context. Of course competent is competent, but the OP was
speaking of "ads" where the person hiring (or considering hiring) is dealing
with an unknown entity (the developer posting the ad). Knowing whether that
person has worked on large databases is a valid piece of information to
have.

This does not automatically disqualify a person who has not worked on large
databases, but given two candidates who are otherwise on equal footing the
one that has worked on large databases is a safer bet that the one who has
not.

--
I don't check the Email account attached
to this message. Send instead to...
RBrandt at Hunter dot com

[Trevor Best]

Salad wrote:

Kevin Nechodom wrote:

Or am I wrong? Does size really matter?

Yes
If one is competent, I should think it doesn't matter. All you are
doing with a database is manipulating data.
But how much data and how efficiently? Inefficiencies in a small
database application aren't going to show up and make you look stupid.
I doubt I'd get a job and someone would say "Salad, we need you to work
on ABC database. It's got a billion records and is 250 gigabytes in
size." and I quiver and quake and stutter..."But sir...I've only worked
on a database that was less than a gig and had less than a million
records!" and then rush out of the room, sobbing and crying as I make it
to the restroom where I retch my stomache dry in humiliation and fear.
I'd instead say "Cool, where's my cubicle? I want to look over the
tables and their structures."

If you open a recordset and that contained 1 record, from a table that
contained say 10 records, how well do you think a .FindFirst operation
will perform?

Open a recordset that contains a million records and do a .findfirst,
what worked well in the small database doesn't work so well now.

Open a recordset (1 record) based on a table that contains many
thousands of records and perform the same findfirst and find out that
Access thinks it's cleverer than you and bypasses your original criteria
and starts pumping a gazillion prepared stored procedure calls at the
server and this happens http://www.besty.org.uk/memory.htm

Although I haven't been able to reproduce that exceptional feature since
A2K was first thrust upon us so it looks like a bug that has been fixed
but you have to think that the guys at MS originally tested this
functionality on a small database rather than a large one.

BTW Don't expect Merrill Lynch to hand you a large portfolio to manage
based on your ability to manage your houskeeping money.

--
[OO=00=OO]

[David W. Fenton]

"Rick Brandt" <ri*********@ho tmail.com> wrote in
news:8r******** ********@newssv r30.news.prodig y.com:

Of course competent is competent, but the OP was
speaking of "ads" where the person hiring (or considering hiring)
is dealing with an unknown entity (the developer posting the ad).
Knowing whether that person has worked on large databases is a
valid piece of information to have.

What exactly does it tell you, absent a clear agreement on the
meaning of "worked on large databases"?

I could design an extremely simple application that would operate on
a database with millions of records.

Or a very complex application that would operate on a database with
only a few thousand records.

To me, the former is trivial, while the latter may reflect extremely
advanced design skills.

In terms of schema, what matters, it seems to me, is not number of
records, but complexity of the schema. That is, if the data
represents a large number of interrelated entities, then there will
be lots of normalized tables, related to each other, and these will
make the data structure quite complex.

Likewise, a relatively simple data structure could support an
application that was quite complex, such as a scheduling app. I
would expect that my TiVo's database is not overly complex, but the
algorithm that resolves conflicts in my Season Pass list (my list of
recurring programs that I want automatically recorded) is, of
necessity, going to be fairly complex.

So, I'd say that what matters is not size, but how complex the app
was, and what kinds of tasks it performed. If I wanted to implement
a scheduling application, I'd want to hire someone with experience
in writing scheduling apps.

So, I just think the "large vs. small" criterion is pretty
meaningless, absent some pretty extensive qualification and
explication.

--
David W. Fenton http://www.bway.net/~dfenton
dfenton at bway dot net http://www.bway.net/~dfassoc

[David W. Fenton]

Trevor Best <no****@besty.o rg.uk> wrote in
news:42******** *************** @news.zen.co.uk :

Open a recordset (1 record) based on a table that contains many
thousands of records and perform the same findfirst and find out
that Access thinks it's cleverer than you and bypasses your
original criteria and starts pumping a gazillion prepared stored
procedure calls at the server and this happens
http://www.besty.org.uk/memory.htm

Using ADO, for instance?

--
David W. Fenton http://www.bway.net/~dfenton
dfenton at bway dot net http://www.bway.net/~dfassoc

[Bri]

Kevin Nechodom wrote:

Or am I wrong? Does size really matter?

"In practice this does not happen very often."

In other words, "Yes, it makes a difference."

Small databases don't require the same degree of care,
because the effects of inefficiencies are likely to be
minor. Put those same problems into a large database, and
you can wind up being cursed by your users.

Add a network with even a medium-ish Access BE and you can see these
inefficiencies add up even sooner. I develope on a standalone PC and I
am often surprised at how some process that takes seconds while I'm
testing it suddenly takes minutes in the network environment.

--
Bri

[Salad]

Trevor Best wrote:

Salad wrote:

Kevin Nechodom wrote:

Or am I wrong? Does size really matter?

Yes

If one is competent, I should think it doesn't matter. All you are
doing with a database is manipulating data.

But how much data and how efficiently? Inefficiencies in a small
database application aren't going to show up and make you look stupid.

I doubt I'd get a job and someone would say "Salad, we need you to
work on ABC database. It's got a billion records and is 250 gigabytes
in size." and I quiver and quake and stutter..."But sir...I've only
worked on a database that was less than a gig and had less than a
million records!" and then rush out of the room, sobbing and crying as
I make it to the restroom where I retch my stomache dry in humiliation
and fear. I'd instead say "Cool, where's my cubicle? I want to look
over the tables and their structures."

If you open a recordset and that contained 1 record, from a table that
contained say 10 records, how well do you think a .FindFirst operation
will perform?

Open a recordset that contains a million records and do a .findfirst,
what worked well in the small database doesn't work so well now.

Open a recordset (1 record) based on a table that contains many
thousands of records and perform the same findfirst and find out that
Access thinks it's cleverer than you and bypasses your original criteria
and starts pumping a gazillion prepared stored procedure calls at the
server and this happens http://www.besty.org.uk/memory.htm

Although I haven't been able to reproduce that exceptional feature since
A2K was first thrust upon us so it looks like a bug that has been fixed
but you have to think that the guys at MS originally tested this
functionality on a small database rather than a large one.

BTW Don't expect Merrill Lynch to hand you a large portfolio to manage
based on your ability to manage your houskeeping money.

I agree somewhat with what you say since keeping track of portfolios is
a bit different than managing a howhold budget. However, if the
programmer were a designer of QuickBooks I would not circular file the
resume. A person that could design QuickBooks would need to be very
detail oriented and used to handling complexities.