Linux ready for high-volume databases?

[Ron Johnson]

On Mon, 2003-08-25 at 16:28, Gregory S. Williamson wrote:

One of our sysads sent this link ... wondering if there is any comment on it from the world of actual users of linux and a database.

<http://story.news.yaho o.com/news?tmpl=story &cid=1738&ncid= 738&e=9&u=/zd/20030825/tc_zd/55311>

"Weak points include lack of available tools, ease of use and ease
of installation"

Sounds like he needs point-and-drool tools...

On the other hand, could even a beefy Linux 2.4 *today* system handle
a 24x7 500GB db that must process 6-8M OLTP-style transactions per
day, while also getting hit by report queries?

Don't think of this as a troll, because I really don't know, even
though I do know that MVS, OpenVMS & Solaris can. (I won't even
ask about toys like Windows and FreeBSD.)

--
-----------------------------------------------------------------
Ron Johnson, Jr. ro***********@c ox.net
Jefferson, LA USA

"Knowledge should be free for all."
Harcourt Fenton Mudd, Star Trek:TOS, "I, Mudd"
---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddres sHere" to ma*******@postg resql.org)

[Vivek Khera]

>>>>> "RJ" == Ron Johnson <ro***********@ cox.net> writes:

RJ> Don't think of this as a troll, because I really don't know, even
RJ> though I do know that MVS, OpenVMS & Solaris can. (I won't even
RJ> ask about toys like Windows and FreeBSD.)

Well, you must be smoking something funny if you think FreeBSD is a
'toy' to be lumped in with windows....

I run a 24x7x365 db on FreeBSD which has *never* crashed in the 3
years it has been in production. Only downtime was the upgrade from
PG 7.1 to 7.2 and once for a switchover from RAID5 to RAID10. I *may*
have a few minutes of down time shortly when updating from PG 7.2 to
7.4 on a new box since I'm saturating the disk I/O bandwidth on the
old box. The eRServer software will be doing the data migration on
the live db so I don't have significant down time.
The DB is currently about 27Mb on disk (including indexes) and
processes several million inserts and updates daily, and a few million
deletes once every two weeks.
--
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Vivek Khera, Ph.D. Khera Communications, Inc.
Internet: kh***@kciLink.c om Rockville, MD [phone]
AIM: vivekkhera Y!: vivek_khera http://www.khera.org/~vivek/

---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match

[Ron Johnson]

On Tue, 2003-08-26 at 10:00, Dennis Gearon wrote:

Free BSD may be a toy, maybe not, but it runs more of the webhosting
domains than any other OS.
That was supposed to be a joke. Putting FreeBSD in the same class
with Winblows is a prima facia absurdity, no matter how you cut it...
Ron Johnson wrote:

On Mon, 2003-08-25 at 16:28, Gregory S. Williamson wrote:
[snip]though I do know that MVS, OpenVMS & Solaris can. (I won't even
ask about toys like Windows and FreeBSD.)

--
-----------------------------------------------------------------
Ron Johnson, Jr. ro***********@c ox.net
Jefferson, LA USA

"Millions of Chinese speak Chinese, and it's not hereditary..."
Dr. Dean Edell
---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ma*******@postg resql.org

[Al Hulaton]

After seeing this article yesterday, I did a bit of research. One _big_ reason
why Sourceforge/VA/OSDN is moving over to IBM/Webshere/DB2 from PostgreSQL is
the resulting product will be jointly marketed by Sourceforge and IBM's
zillions of sales people. So not only will they get a shiny, new db, but
backend revenue.

"The companies will jointly market and sell the software as part of the
commercial agreement. "-- 4th paragraph, last sentence.
http://www.eweek.com/print_article/0...a=30025,00.asp

"In a separate announcement today, VA Software announced a significant
commercial agreement with IBM focused on the joint marketing and sales of the
next generation of SourceForgeâ„¢ Enterprise Edition." -- 7th paragram from
their press release at
http://www.vasoftware.com/news/press.php/2002/1070.html

Perhaps the PostgreSQL team bidding for the job, if any were even consulted,
didn't frame the project as IBM did -- a product joint venture. It's a good
tactic and I don't blame Sourceforge one bit for the opportunity.

The decision wasn't entirely technical so I don't see this as a loss for
PostgreSQL. DB2 isn't a slouch db by any means but not many companies will be
able to bargain with IBM as Sourceforge did. If you're a retailer in Topeka
with 3 locations, I doubt IBM would give you the same attention or joint
marketing deal they gave Sourceforge. DB2 ain't cheap.

--
Best,
Al Hulaton | Sr. Account Engineer | Command Prompt, Inc.
503.222.2783 | ah******@comman dprompt.com
Home of Mammoth PostgreSQL and 'Practical PostgreSQL'
Managed PostgreSQL, Linux services and consulting
Read and Search O'Reilly's 'Practical PostgreSQL' at
http://www.commandprompt.com
---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddres sHere" to ma*******@postg resql.org)

[Andrew Sullivan]

On Tue, Aug 26, 2003 at 09:15:14AM -0700, Al Hulaton wrote:

Perhaps the PostgreSQL team bidding for the job, if any were even consulted,
didn't frame the project as IBM did -- a product joint venture. It's a good
tactic and I don't blame Sourceforge one bit for the opportunity.

Well, since the main point was to get some $$ into the company, bucks
which IBM has and PostgreSQL doesn't, it's not too surprising that
the PostgreSQL team didn't win. The move to DB2 was apparently a
quid pro quo for the cash.

A

--
----
Andrew Sullivan 204-4141 Yonge Street
Liberty RMS Toronto, Ontario Canada
<an****@liberty rms.info> M2P 2A8
[phone] x110
---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faqs/FAQ.html

[scott.marlowe]

On Tue, 26 Aug 2003, Al Hulaton wrote:

After seeing this article yesterday, I did a bit of research. One _big_ reason
why Sourceforge/VA/OSDN is moving over to IBM/Webshere/DB2 from PostgreSQL is
the resulting product will be jointly marketed by Sourceforge and IBM's
zillions of sales people. So not only will they get a shiny, new db, but
backend revenue.

"The companies will jointly market and sell the software as part of the
commercial agreement. "-- 4th paragraph, last sentence.
http://www.eweek.com/print_article/0...a=30025,00.asp

"In a separate announcement today, VA Software announced a significant
commercial agreement with IBM focused on the joint marketing and sales of the
next generation of SourceForgeâ„¢ Enterprise Edition." -- 7th paragram from
their press release at
http://www.vasoftware.com/news/press.php/2002/1070.html

Perhaps the PostgreSQL team bidding for the job, if any were even consulted,
didn't frame the project as IBM did -- a product joint venture. It's a good
tactic and I don't blame Sourceforge one bit for the opportunity.

The decision wasn't entirely technical so I don't see this as a loss for
PostgreSQL. DB2 isn't a slouch db by any means but not many companies will be
able to bargain with IBM as Sourceforge did. If you're a retailer in Topeka
with 3 locations, I doubt IBM would give you the same attention or joint
marketing deal they gave Sourceforge. DB2 ain't cheap.

Actually, I remember quite clearly the incredibly bad performance of
sourceforge's search engine for the better part of a year after switching
out postgresql for db2. It had been quite snappy, and I could enter
database or some other keyword and have a page display in ~2 seconds or
less. For the first three months or so after the switch, most searchs
simply timed out to PHP's default 30 seconds. Even when they got it
working better, it only had maybe 1/10th or less of the keywords indexed
that they had had in postgresql (i.e. words like index or email weren't
being indexed. :-)

It was probably at least 9 months later that the search engine was finally
back to being usable, and another 3 or 4 before it was about as good as
postgresql. And we're talking an older version (I believe it was 7.1) of
postgresql as well.

The switch to db2 was driven by partnering business needs, not by poor
performance of postgresql.
---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend

[Greg Stark]

Vivek Khera <kh***@kcilink. com> writes:

I run a 24x7x365 db on FreeBSD which has *never* crashed in the 3
years it has been in production. Only downtime was the upgrade from
PG 7.1 to 7.2 and once for a switchover from RAID5 to RAID10.
I would be interested to know what backup strategy you use for this. Without
online backups this means that if you had crashed you would have lost data up
to the last pg_dump you took? Had you done tests to see how long it would have
taken to restore from the pg_dump?

Online backups with archived transaction logs are the next big killer feature
(the last one remaining?) for 24x7 operation I think.
The DB is currently about 27Mb on disk (including indexes) and
processes several million inserts and updates daily, and a few million
deletes once every two weeks.

Oh, it's a really small database. That helps a lot with the backup problems of
24x7 operation. Still I would be interested.
--
greg
---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings

[Christopher Browne]

After a long battle with technology,kh** *@kcilink.com (Vivek Khera), an earthling, wrote:

>> "RJ" == Ron Johnson <ro***********@ cox.net> writes:

RJ> Don't think of this as a troll, because I really don't know, even
RJ> though I do know that MVS, OpenVMS & Solaris can. (I won't even
RJ> ask about toys like Windows and FreeBSD.)

Well, you must be smoking something funny if you think FreeBSD is a
'toy' to be lumped in with windows....

I suspect your irony-meter didn't get activated when it was supposed
to.

Please keep in mind that to the sorts of people that read and believe
and act on the source article, any system that doesn't have a vendor
to "certify" its fitness for database use is manifestly a "toy" that
only fools and Englishmen would use for any purpose that was the
slightest bit important.

Your injection of technical fact into the matter just confuses the
matter for people that prefer to get their "technical expertise" from
some white-paper-writer at the Gartner Group.

And there is a very slight bit of genuine technical reality to this,
too. People _assert_ that there are technical reasons to prefer
FreeBSD over other systems, but it is difficult to get forcibly past
the anecdotal evidence. The fact that you have had a system running,
apparently quite successfully, for a while, is not a _proof_ that
FreeBSD is more or less satisfactory than other OSes for the purpose.
It is merely an anecdote.

Unfortunately, we seldom see _anything_ better than anecdotes. People
report anecdotes that they heard that someone lost data to ext2.
Others report anecdotes that they have had good results with one
filesystem or another or one OS or another.

When there are problems, there isn't a good "certifiabl e" (or
'statistically significant') way of evaluating whether the faults
resulted from:
a) A PG bug
b) An OS filesystem bug
c) An OS device driver bug
d) Bad disk controller
e) Bad disk drive

It's quite easy for these to feed into one another so that a severe
problem combines together a tragedy of errors. (Been there :-(.) Is
there a way to "certify" that the composition of your particular
hardware with FreeBSD with PostgreSQL can't lead to tragedy? I'd
think not.

There's some pathos in with that irony...
--
http://cbbrowne.com/info/wp.html
Rules of the Evil Overlord #41. "Once my power is secure, I will
destroy all those pesky time-travel devices."
<http://www.eviloverlor d.com/>

[Jonathan Bartlett]

> Online backups with archived transaction logs are the next big killer feature

(the last one remaining?) for 24x7 operation I think.

I believe this is at least theoretically possible using Linux device layer
tricks. Using network block devices, you can have a network RAID1, with
the transaction logs living over NFS. Never tried this, but it seems all
the tools are there.

---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend

[Dennis Gearon]

With the low cost of disks, it might be a good idea to just copy to
disks, that one can put back in.

Greg Stark wrote:

Vivek Khera <kh***@kcilink. com> writes:

I run a 24x7x365 db on FreeBSD which has *never* crashed in the 3
years it has been in production. Only downtime was the upgrade from
PG 7.1 to 7.2 and once for a switchover from RAID5 to RAID10.

I would be interested to know what backup strategy you use for this. Without
online backups this means that if you had crashed you would have lost data up
to the last pg_dump you took? Had you done tests to see how long it would have
taken to restore from the pg_dump?

Online backups with archived transaction logs are the next big killer feature
(the last one remaining?) for 24x7 operation I think.

The DB is currently about 27Mb on disk (including indexes) and
processes several million inserts and updates daily, and a few million
deletes once every two weeks.

Oh, it's a really small database. That helps a lot with the backup problems of
24x7 operation. Still I would be interested.

---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ma*******@postg resql.org