Working with Large vs Small databases

[Salad]

Every now and then I see ads that state something like "Experience with
Large Databases ...multi-gig...blah-de-blah" And I have to laugh.
What's the difference between a large or small database? A table is a
table, a record is a record, a field is a field. All you are doing is
manipulating data in tables. I wouldn't think it'd make much difference
in working with a table with 10 records or a billion records...they're
nothing more than rows and columns.

That's like saying "Must be able to work with large amounts of money"
Whether an app works with 10s, 1000s, 1000000s, or 1000000000s of
dollars, it's nothing more than a value...the same concepts work with
all of them...it simply needs to be accurate.

Or am I wrong? Does size really matter?

[Rick Brandt]

Salad wrote:

Every now and then I see ads that state something like "Experience
with Large Databases ...multi-gig...blah-de-blah" And I have to
laugh. What's the difference between a large or small database? A
table is a table, a record is a record, a field is a field. All you
are doing is manipulating data in tables. I wouldn't think it'd make
much difference in working with a table with 10 records or a billion
records...they're nothing more than rows and columns.

That's like saying "Must be able to work with large amounts of money"
Whether an app works with 10s, 1000s, 1000000s, or 1000000000s of
dollars, it's nothing more than a value...the same concepts work with
all of them...it simply needs to be accurate.

Or am I wrong? Does size really matter?

It certainly does. One can get away with doing LOTS of things wrong in
terms of inefficiencies with small databases. If you hire someone who has
worked exclusively on smaller projects there's a higher chance that they
have not yet learned all of the "best practices" for databases (particularly
keeping network traffic to a minimum). Someone who has a lot of experience
with large databases will already have learned how to make db apps as
efficient as possible.

--
I don't check the Email account attached
to this message. Send instead to...
RBrandt at Hunter dot com

[Kevin Nechodom]

--
Kevin Nechodom
University of Utah Hospital and Clinics
Kevin dit Nechodom ack hsc dit utah dit edu
"Call me paranoid, but I think you are reading what I'm
writing!"

Salad<oi*@vinegar.com> 6/16/2005 6:40:23 AM >>>

Every now and then I see ads that state something like
"Experience with
Large Databases ...multi-gig...blah-de-blah" And I have

tolaugh.
What's the difference between a large or small database? Atable is a
table, a record is a record, a field is a field. All you
are doing is
manipulating data in tables. I wouldn't think it'd make
much difference
in working with a table with 10 records or a billion
records...they're
nothing more than rows and columns.

That's like saying "Must be able to work with large amountsof money"
Whether an app works with 10s, 1000s, 1000000s, or
1000000000s of
dollars, it's nothing more than a value...the same conceptswork with
all of them...it simply needs to be accurate.

Or am I wrong? Does size really matter?

From http://www.kettering.edu/~jhuggins/humor/theory.html:

"In theory the difference between practice and theory is due
to practical considerations that theorists find it
impractical to fit into their theories.

"In practice, theory uses the practice of theorising about
practical matters, while not noticing that the theoretical
method practically distorts the theory beyond application to
practice.

"Theoretically then the practical facts are that the theory
is in practice good for predicting what happens in theory,
but impractical as a theory with direct implications for
practice, except where theory states that the practice is
sufficiently close to the theory to make any difference for
all practical purposes theoreticaly zero.

"In practice this does not happen very often."

In other words, "Yes, it makes a difference."

Small databases don't require the same degree of care,
because the effects of inefficiencies are likely to be
minor. Put those same problems into a large database, and
you can wind up being cursed by your users.

[Salad]

Kevin Nechodom wrote:

Or am I wrong? Does size really matter?

In other words, "Yes, it makes a difference."

Small databases don't require the same degree of care,
because the effects of inefficiencies are likely to be
minor. Put those same problems into a large database, and
you can wind up being cursed by your users.

I tend to disagree with both you and Rick. There are people that are
good at what they do, and those that aren't so good. I'm sure you've
worked on a project and you wonder if the previous programmer was brain
dead.

If one is competent, I should think it doesn't matter. All you are
doing with a database is manipulating data.

I doubt I'd get a job and someone would say "Salad, we need you to work
on ABC database. It's got a billion records and is 250 gigabytes in
size." and I quiver and quake and stutter..."But sir...I've only worked
on a database that was less than a gig and had less than a million
records!" and then rush out of the room, sobbing and crying as I make it
to the restroom where I retch my stomache dry in humiliation and fear.
I'd instead say "Cool, where's my cubicle? I want to look over the
tables and their structures."

[Rick Brandt]

Salad wrote:

Kevin Nechodom wrote:

Or am I wrong? Does size really matter?

In other words, "Yes, it makes a difference."

Small databases don't require the same degree of care,
because the effects of inefficiencies are likely to be
minor. Put those same problems into a large database, and
you can wind up being cursed by your users.

I tend to disagree with both you and Rick. There are people that are
good at what they do, and those that aren't so good. I'm sure you've
worked on a project and you wonder if the previous programmer was
brain dead.

If one is competent, I should think it doesn't matter. All you are
doing with a database is manipulating data.

I doubt I'd get a job and someone would say "Salad, we need you to
work on ABC database. It's got a billion records and is 250
gigabytes in size." and I quiver and quake and stutter..."But
sir...I've only worked on a database that was less than a gig and had
less than a million records!" and then rush out of the room, sobbing
and crying as I make it to the restroom where I retch my stomache dry
in humiliation and fear. I'd instead say "Cool, where's my cubicle?
I want to look over the tables and their structures."

Different context. Of course competent is competent, but the OP was
speaking of "ads" where the person hiring (or considering hiring) is dealing
with an unknown entity (the developer posting the ad). Knowing whether that
person has worked on large databases is a valid piece of information to
have.

This does not automatically disqualify a person who has not worked on large
databases, but given two candidates who are otherwise on equal footing the
one that has worked on large databases is a safer bet that the one who has
not.

--
I don't check the Email account attached
to this message. Send instead to...
RBrandt at Hunter dot com

[Trevor Best]

Salad wrote:

Kevin Nechodom wrote:

Or am I wrong? Does size really matter?

Yes
If one is competent, I should think it doesn't matter. All you are
doing with a database is manipulating data.
But how much data and how efficiently? Inefficiencies in a small
database application aren't going to show up and make you look stupid.
I doubt I'd get a job and someone would say "Salad, we need you to work
on ABC database. It's got a billion records and is 250 gigabytes in
size." and I quiver and quake and stutter..."But sir...I've only worked
on a database that was less than a gig and had less than a million
records!" and then rush out of the room, sobbing and crying as I make it
to the restroom where I retch my stomache dry in humiliation and fear.
I'd instead say "Cool, where's my cubicle? I want to look over the
tables and their structures."

If you open a recordset and that contained 1 record, from a table that
contained say 10 records, how well do you think a .FindFirst operation
will perform?

Open a recordset that contains a million records and do a .findfirst,
what worked well in the small database doesn't work so well now.

Open a recordset (1 record) based on a table that contains many
thousands of records and perform the same findfirst and find out that
Access thinks it's cleverer than you and bypasses your original criteria
and starts pumping a gazillion prepared stored procedure calls at the
server and this happens http://www.besty.org.uk/memory.htm

Although I haven't been able to reproduce that exceptional feature since
A2K was first thrust upon us so it looks like a bug that has been fixed
but you have to think that the guys at MS originally tested this
functionality on a small database rather than a large one.

BTW Don't expect Merrill Lynch to hand you a large portfolio to manage
based on your ability to manage your houskeeping money.

--
[OO=00=OO]

[David W. Fenton]

"Rick Brandt" <ri*********@hotmail.com> wrote in
news:8r****************@newssvr30.news.prodigy.com :

Of course competent is competent, but the OP was
speaking of "ads" where the person hiring (or considering hiring)
is dealing with an unknown entity (the developer posting the ad).
Knowing whether that person has worked on large databases is a
valid piece of information to have.

What exactly does it tell you, absent a clear agreement on the
meaning of "worked on large databases"?

I could design an extremely simple application that would operate on
a database with millions of records.

Or a very complex application that would operate on a database with
only a few thousand records.

To me, the former is trivial, while the latter may reflect extremely
advanced design skills.

In terms of schema, what matters, it seems to me, is not number of
records, but complexity of the schema. That is, if the data
represents a large number of interrelated entities, then there will
be lots of normalized tables, related to each other, and these will
make the data structure quite complex.

Likewise, a relatively simple data structure could support an
application that was quite complex, such as a scheduling app. I
would expect that my TiVo's database is not overly complex, but the
algorithm that resolves conflicts in my Season Pass list (my list of
recurring programs that I want automatically recorded) is, of
necessity, going to be fairly complex.

So, I'd say that what matters is not size, but how complex the app
was, and what kinds of tasks it performed. If I wanted to implement
a scheduling application, I'd want to hire someone with experience
in writing scheduling apps.

So, I just think the "large vs. small" criterion is pretty
meaningless, absent some pretty extensive qualification and
explication.

--
David W. Fenton http://www.bway.net/~dfenton
dfenton at bway dot net http://www.bway.net/~dfassoc

[David W. Fenton]

Trevor Best <no****@besty.org.uk> wrote in
news:42***********************@news.zen.co.uk:

Open a recordset (1 record) based on a table that contains many
thousands of records and perform the same findfirst and find out
that Access thinks it's cleverer than you and bypasses your
original criteria and starts pumping a gazillion prepared stored
procedure calls at the server and this happens
http://www.besty.org.uk/memory.htm

Using ADO, for instance?

--
David W. Fenton http://www.bway.net/~dfenton
dfenton at bway dot net http://www.bway.net/~dfassoc

[Bri]

Kevin Nechodom wrote:

Or am I wrong? Does size really matter?

"In practice this does not happen very often."

In other words, "Yes, it makes a difference."

Small databases don't require the same degree of care,
because the effects of inefficiencies are likely to be
minor. Put those same problems into a large database, and
you can wind up being cursed by your users.

Add a network with even a medium-ish Access BE and you can see these
inefficiencies add up even sooner. I develope on a standalone PC and I
am often surprised at how some process that takes seconds while I'm
testing it suddenly takes minutes in the network environment.

--
Bri

[Salad]

Trevor Best wrote:

Salad wrote:

Kevin Nechodom wrote:

Or am I wrong? Does size really matter?

Yes

If one is competent, I should think it doesn't matter. All you are
doing with a database is manipulating data.

But how much data and how efficiently? Inefficiencies in a small
database application aren't going to show up and make you look stupid.

I doubt I'd get a job and someone would say "Salad, we need you to
work on ABC database. It's got a billion records and is 250 gigabytes
in size." and I quiver and quake and stutter..."But sir...I've only
worked on a database that was less than a gig and had less than a
million records!" and then rush out of the room, sobbing and crying as
I make it to the restroom where I retch my stomache dry in humiliation
and fear. I'd instead say "Cool, where's my cubicle? I want to look
over the tables and their structures."

If you open a recordset and that contained 1 record, from a table that
contained say 10 records, how well do you think a .FindFirst operation
will perform?

Open a recordset that contains a million records and do a .findfirst,
what worked well in the small database doesn't work so well now.

Open a recordset (1 record) based on a table that contains many
thousands of records and perform the same findfirst and find out that
Access thinks it's cleverer than you and bypasses your original criteria
and starts pumping a gazillion prepared stored procedure calls at the
server and this happens http://www.besty.org.uk/memory.htm

Although I haven't been able to reproduce that exceptional feature since
A2K was first thrust upon us so it looks like a bug that has been fixed
but you have to think that the guys at MS originally tested this
functionality on a small database rather than a large one.

BTW Don't expect Merrill Lynch to hand you a large portfolio to manage
based on your ability to manage your houskeeping money.

I agree somewhat with what you say since keeping track of portfolios is
a bit different than managing a howhold budget. However, if the
programmer were a designer of QuickBooks I would not circular file the
resume. A person that could design QuickBooks would need to be very
detail oriented and used to handling complexities.

[Rick Brandt]

Salad wrote:

I agree somewhat with what you say since keeping track of portfolios
is a bit different than managing a howhold budget. However, if the
programmer were a designer of QuickBooks I would not circular file the
resume. A person that could design QuickBooks would need to be very
detail oriented and used to handling complexities.

I think you and David are somewhat missing the original point. The question
that was asked is if there is any value in knowing that a developer has
worked on large databases. I think the answer to that is yes as there is a
certain skill-set that is developed (or at least more thoroughly tested)
when working against larger sets of data. This in no way means that anyone
who has not worked on TB sized databases is not worthy of consideration. It
is just "one bullet in the magazine".

I also don't think that one needs to be talking TB size when using the term
"large database". In my view any database that causes noticeable
performance problems when the entire thing is dragged across a network is
sufficient to hone the skills that I am talking about.

--
I don't check the Email account attached
to this message. Send instead to...
RBrandt at Hunter dot com

[Salad]

David W. Fenton wrote:

"Rick Brandt" <ri*********@hotmail.com> wrote in
news:8r****************@newssvr30.news.prodigy.com :

Of course competent is competent, but the OP was
speaking of "ads" where the person hiring (or considering hiring)
is dealing with an unknown entity (the developer posting the ad).
Knowing whether that person has worked on large databases is a
valid piece of information to have.

What exactly does it tell you, absent a clear agreement on the
meaning of "worked on large databases"?

I could design an extremely simple application that would operate on
a database with millions of records.

Or a very complex application that would operate on a database with
only a few thousand records.

To me, the former is trivial, while the latter may reflect extremely
advanced design skills.

In terms of schema, what matters, it seems to me, is not number of
records, but complexity of the schema. That is, if the data
represents a large number of interrelated entities, then there will
be lots of normalized tables, related to each other, and these will
make the data structure quite complex.

Likewise, a relatively simple data structure could support an
application that was quite complex, such as a scheduling app. I
would expect that my TiVo's database is not overly complex, but the
algorithm that resolves conflicts in my Season Pass list (my list of
recurring programs that I want automatically recorded) is, of
necessity, going to be fairly complex.

So, I'd say that what matters is not size, but how complex the app
was, and what kinds of tasks it performed. If I wanted to implement
a scheduling application, I'd want to hire someone with experience
in writing scheduling apps.

So, I just think the "large vs. small" criterion is pretty
meaningless, absent some pretty extensive qualification and
explication.

You described my point succinctly. Large numbers of records would seem
to be meaningless. It is the structures, uses of indexes,
relationships, and complexities to solve the problems associated with
application that are the crux of the matter.

[Rick Brandt]

Salad wrote:

You described my point succinctly. Large numbers of records would
seem to be meaningless. It is the structures, uses of indexes,
relationships, and complexities to solve the problems associated with
application that are the crux of the matter.

Yes, but our point was that you can do most of things badly and get away
with it on small databases to a larger degree than you can on larger ones.

--
I don't check the Email account attached
to this message. Send instead to...
RBrandt at Hunter dot com

[Trevor Best]

David W. Fenton wrote:

Trevor Best <no****@besty.org.uk> wrote in
news:42***********************@news.zen.co.uk:

Open a recordset (1 record) based on a table that contains many
thousands of records and perform the same findfirst and find out
that Access thinks it's cleverer than you and bypasses your
original criteria and starts pumping a gazillion prepared stored
procedure calls at the server and this happens
http://www.besty.org.uk/memory.htm

Using ADO, for instance?

May do but that example was DAO.

--
[OO=00=OO]

[David W. Fenton]

"Rick Brandt" <ri*********@hotmail.com> wrote in
news:Al*****************@newssvr12.news.prodigy.co m:

Salad wrote:

I agree somewhat with what you say since keeping track of
portfolios is a bit different than managing a howhold budget.
However, if the programmer were a designer of QuickBooks I would
not circular file the resume. A person that could design
QuickBooks would need to be very detail oriented and used to
handling complexities.
I think you and David are somewhat missing the original point.
The question that was asked is if there is any value in knowing
that a developer has worked on large databases. I think the
answer to that is yes as there is a certain skill-set that is
developed (or at least more thoroughly tested) when working
against larger sets of data. . . .

No, no, and NO.

What matters is the *definition* of the term LARGE. You seem to take
for granted the idea of LARGE NUMBER OF RECORDS IN A COMPLEX DATA
STRUCTURE, while to me LARGE simply implies a lot of data, which
could be very simple, a single table, even.

Indeed, I had a project last summer where I wrote a reporting system
for a data collection system that was reporting data from electronic
sensors in generators in a power plant. The single data table in the
application had only about 10 fields, but the number of records was
quite large, as it was sampling a once a minute from 12 different
generators.

Large number of records, but, hell, how could anyone screw up this
one?
. . . This in no way means that anyone who
has not worked on TB sized databases is not worthy of
consideration. It is just "one bullet in the magazine".
Without knowing something about the actual systems the person has
worked on, knowing the applicant has "experience with large
databases" tells you exactly nothing of any use whatsoever.
I also don't think that one needs to be talking TB size when using
the term "large database". In my view any database that causes
noticeable performance problems when the entire thing is dragged
across a network is sufficient to hone the skills that I am
talking about.

So, in other words, your definition of LARGE is not even limited to
LARGE NUMBER OF RECORDS.

There are so many qualifications to your argument that I don't see
why you can't recognize that your case completely collapses under
the weight of all the qualifications you've added to the original
point.

--
David W. Fenton http://www.bway.net/~dfenton
dfenton at bway dot net http://www.bway.net/~dfassoc

[Rick Brandt]

"David W. Fenton" <dX********@bway.net.invalid> wrote in message
news:Xn**********************************@24.168.1 28.74...

No, no, and NO.

What matters is the *definition* of the term LARGE. You seem to take
for granted the idea of LARGE NUMBER OF RECORDS IN A COMPLEX DATA
STRUCTURE, while to me LARGE simply implies a lot of data, which
could be very simple, a single table, even.
Many (not all) of the issues that come up when working with large databases
simply require a large database, not necessarily a complex one.
Indeed, I had a project last summer where I wrote a reporting system
for a data collection system that was reporting data from electronic
sensors in generators in a power plant. The single data table in the
application had only about 10 fields, but the number of records was
quite large, as it was sampling a once a minute from 12 different
generators.

Large number of records, but, hell, how could anyone screw up this
one?
Oh I don't know... Use of domain aggregate functions in loops and/or queries, no
proper indexing, the use of the built in "Find" tool, Numerous functions where
you open a Recordset and immediately issue a MoveLast to determine the count.
All things that would be no problem at all on a small database.
Without knowing something about the actual systems the person has
worked on, knowing the applicant has "experience with large
databases" tells you exactly nothing of any use whatsoever.
And without knowing those things knowing that they have worked on multi-user
apps or not tells you exactly nothing. We're talking about "position wanted" ad
here. It goes wihtou saying that anything claimed is useless without further
digging during the interview. Should the ad then be blank?
So, in other words, your definition of LARGE is not even limited to
LARGE NUMBER OF RECORDS.

My definition of large is enough data to cause bad practices to be a performance
problem. That will be a much different point on a networked app versus a local
app and between a multi-user app versus a single user or server-only app.

--
I don't check the Email account attached
to this message. Send instead to...
RBrandt at Hunter dot com

[David W. Fenton]

"Rick Brandt" <ri*********@hotmail.com> wrote in
news:Nr*****************@newssvr11.news.prodigy.co m:

"David W. Fenton" <dX********@bway.net.invalid> wrote in message
news:Xn**********************************@24.168.1 28.74...

No, no, and NO.

What matters is the *definition* of the term LARGE. You seem to
take for granted the idea of LARGE NUMBER OF RECORDS IN A COMPLEX
DATA STRUCTURE, while to me LARGE simply implies a lot of data,
which could be very simple, a single table, even.

Many (not all) of the issues that come up when working with large
databases simply require a large database, not necessarily a
complex one.

Indeed, I had a project last summer where I wrote a reporting
system for a data collection system that was reporting data from
electronic sensors in generators in a power plant. The single
data table in the application had only about 10 fields, but the
number of records was quite large, as it was sampling a once a
minute from 12 different generators.

Large number of records, but, hell, how could anyone screw up
this one?

Oh I don't know... Use of domain aggregate functions in loops
and/or queries, no proper indexing, the use of the built in "Find"
tool, Numerous functions where you open a Recordset and
immediately issue a MoveLast to determine the count. All things
that would be no problem at all on a small database.

Er, it was a *reporting* application. That means it had reports and
NOTHING ELSE, i.e., NO RECORD NAVIGATION, NO RECORDSETS, NO LOOPS.

And the schema was not my responsibility -- that was designed by
someone else (and it was properly done, BTW).

Tell me again exactly where an inexperienced developer could have
gone wrong on this one?

Without knowing something about the actual systems the person has
worked on, knowing the applicant has "experience with large
databases" tells you exactly nothing of any use whatsoever.

And without knowing those things knowing that they have worked on
multi-user apps or not tells you exactly nothing. We're talking
about "position wanted" ad here. It goes wihtou saying that
anything claimed is useless without further digging during the
interview. Should the ad then be blank?

No, but "large databases" is meaningless and tells you nothing about
an applicant.

Which is the point of discussion here.

So, in other words, your definition of LARGE is not even limited
to LARGE NUMBER OF RECORDS.

My definition of large is enough data to cause bad practices to be
a performance problem. That will be a much different point on a
networked app versus a local app and between a multi-user app
versus a single user or server-only app.

Again, you seem to be using "LARGE DATABASE" to stand for all sorts
of things that are not inherent in the meaning of the words you're
using.

Which is the reason why we're disputing you, because you're saying
one thing and meaning another.

--
David W. Fenton http://www.bway.net/~dfenton
dfenton at bway dot net http://www.bway.net/~dfassoc

[corey lawson]

Salad wrote:

Kevin Nechodom wrote:

Or am I wrong? Does size really matter?
In other words, "Yes, it makes a difference."

Small databases don't require the same degree of care,
because the effects of inefficiencies are likely to be
minor. Put those same problems into a large database, and
you can wind up being cursed by your users.

I tend to disagree with both you and Rick. There are people that are
good at what they do, and those that aren't so good. I'm sure you've
worked on a project and you wonder if the previous programmer was brain
dead.

If one is competent, I should think it doesn't matter. All you are
doing with a database is manipulating data.

No, it comes down to things like, "how come this query takes 15 minutes
on this huge table?", and knowing ways to make it work faster.

The noob will just think, "it's SQL. I can write the query better!", but
someone experienced with making large DBs usable should have a few more
tricks up his sleeve.

In Access, one is limited really to properly adding indexes. In SQL
Server 7.0/2000/+, Oracle 8i+, etc., you get into "vertical table
partitions", i.e., sensibly splitting the table up into separate pieces,
joining them together with a UNION query and writing an INSTEAD OF
trigger on the view to make it insertable/updatable, and setting things
up appropriately, so that the probability of your search criteria hits
the smallest, "hottest" dataset and returns quickly (OK, in Oracle you
just partition the table in its DDL, you don't have to create the UNION
query, instead-of triggers, etc. It's rather magical), things like that.
Then, you can store each chunk of data on a separate tablespace, which
you can then put on a separate physical drive, which in the case of
SCSI, means you could very well be doing parallelized data queries, not
only to the query engine, but physically (at the drive controller) as
well...

Also, you don't want to set things up like this until you need them,
because they can become a PITA to administer, or may actually make
things worse for smaller datasets because of the increased overhead.
I doubt I'd get a job and someone would say "Salad, we need you to work
on ABC database. It's got a billion records and is 250 gigabytes in
size." and I quiver and quake and stutter..."But sir...I've only worked
on a database that was less than a gig and had less than a million
records!" and then rush out of the room, sobbing and crying as I make it
to the restroom where I retch my stomache dry in humiliation and fear.
I'd instead say "Cool, where's my cubicle? I want to look over the
tables and their structures."

Then, there's backups and restores of such huge datasets, but that's a
different issue... because it will not look good to the boss for you to
simplisticly say, "but, Sir, we can only back up 100 GB at a time!"

A billion record table probably will be wayyyy more than 250 GB in size,
btw, because that leaves about 250 bytes per record at the max, which
just typically isn't very much at all..

[corey lawson]

David W. Fenton wrote:

Oh I don't know... Use of domain aggregate functions in loops
and/or queries, no proper indexing, the use of the built in "Find"
tool, Numerous functions where you open a Recordset and
immediately issue a MoveLast to determine the count. All things
that would be no problem at all on a small database.

Er, it was a *reporting* application. That means it had reports and
NOTHING ELSE, i.e., NO RECORD NAVIGATION, NO RECORDSETS, NO LOOPS.

And the schema was not my responsibility -- that was designed by
someone else (and it was properly done, BTW).

Tell me again exactly where an inexperienced developer could have
gone wrong on this one?

Oh, this kind of stuff happens all the time in reporting tools like
Crystal Reports. Access also *seems* simple in its reporting, but
hopefully the developer would be able to ask for a few things to help
make certain parts of the reporting work better on large datasets, such
as prebuilt summary tables, stored procedures, etc., (or access to the
data warehouse...), knowing that in Access queries being clever in using
VBA functions on data generally results in the entire dataset being
pulled into Access to apply that function, even if it's just on an
output field, and stuff like that.

[David W. Fenton]

corey lawson <co**********@ayeteatea.net> wrote in
news:11*************@corp.supernews.com:

Then, there's backups and restores of such huge datasets, but
that's a different issue... because it will not look good to the
boss for you to simplisticly say, "but, Sir, we can only back up
100 GB at a time!"

A billion record table probably will be wayyyy more than 250 GB in
size, btw, because that leaves about 250 bytes per record at the
max, which just typically isn't very much at all..

Seems to me there's a great deal of confusion in this discussion
about the definition of "large databases." You're describing a
scenario in which the database application developer will absolutely
have ZILCH to do with the actual administration of the database,
because once a database is as large as you described, there's going
to be a DBA administering it.

The developer's job then becomes just *working* with the large
database.

The things you've described are all design and administration
issues.

While it's essential that a DBA know those things, it's only
necessary for an application developer to know how to write
effective apps to run against such large dbs.

--
David W. Fenton http://www.bway.net/~dfenton
dfenton at bway dot net http://www.bway.net/~dfassoc

[David W. Fenton]

corey lawson <co**********@ayeteatea.net> wrote in
news:11*************@corp.supernews.com:

David W. Fenton wrote:

Oh I don't know... Use of domain aggregate functions in loops
and/or queries, no proper indexing, the use of the built in
"Find" tool, Numerous functions where you open a Recordset and
immediately issue a MoveLast to determine the count. All things
that would be no problem at all on a small database.

Er, it was a *reporting* application. That means it had reports
and NOTHING ELSE, i.e., NO RECORD NAVIGATION, NO RECORDSETS, NO
LOOPS.

And the schema was not my responsibility -- that was designed by
someone else (and it was properly done, BTW).

Tell me again exactly where an inexperienced developer could have
gone wrong on this one?

Oh, this kind of stuff happens all the time in reporting tools
like Crystal Reports. Access also *seems* simple in its reporting,
but hopefully the developer would be able to ask for a few things
to help make certain parts of the reporting work better on large
datasets, such as prebuilt summary tables, stored procedures,
etc., (or access to the data warehouse...), knowing that in Access
queries being clever in using VBA functions on data generally
results in the entire dataset being pulled into Access to apply
that function, even if it's just on an output field, and stuff
like that.

The data is never stored in anything other than a Jet database.

So, again, please explain again to me exactly how a developer could
go wrong?

I'm not making up this example. It was a real project that I did
last summer.

--
David W. Fenton http://www.bway.net/~dfenton
dfenton at bway dot net http://www.bway.net/~dfassoc

[rkc]

David W. Fenton wrote:

While it's essential that a DBA know those things, it's only
necessary for an application developer to know how to write
effective apps to run against such large dbs.

Couple that with the fact that the DBA probably isn't going
to let the developers any where near the actual tables it's
only up to the developer(s) to work efficiently with what
they're given to work with.

[Rick Brandt]

"rkc" <rk*@rochester.yabba.dabba.do.rr.bomb> wrote in message
news:Js*******************@twister.nyroc.rr.com...

David W. Fenton wrote:

While it's essential that a DBA know those things, it's only
necessary for an application developer to know how to write
effective apps to run against such large dbs.

Couple that with the fact that the DBA probably isn't going
to let the developers any where near the actual tables it's
only up to the developer(s) to work efficiently with what
they're given to work with.

That depends on the structure of the organization. Nearly all of my apps
include the creation of all of the tables, views, stored procedures, and indexes
on the server. Our "admin" people are only responsible for backups and keeping
the hardware/network running. We have no "DBA" per-se.

--
I don't check the Email account attached
to this message. Send instead to...
RBrandt at Hunter dot com

[jimfortune]

Bri wrote:

Add a network with even a medium-ish Access BE and you can see these
inefficiencies add up even sooner. I develope on a standalone PC and I
am often surprised at how some process that takes seconds while I'm
testing it suddenly takes minutes in the network environment.

--
Bri

You just need to use a 333 MHz processor on your standalone PC :-)!

James A. Fortune

[Bri]

ji********@compumarc.com wrote:

Bri wrote:

Add a network with even a medium-ish Access BE and you can see these
inefficiencies add up even sooner. I develope on a standalone PC and I
am often surprised at how some process that takes seconds while I'm
testing it suddenly takes minutes in the network environment.

--
Bri

You just need to use a 333 MHz processor on your standalone PC :-)!

James A. Fortune

I've tried that too, on a P1-120 even (test machine). The network still
slows things down more then it does. . . well slows down differently.

--
Bri