473,836 Members | 2,102 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Recomended FS

Hi

I'm upgrading the DB sever hardware and also the Linux OS.

My Questions are:

1. What is the preferred FS to go with ? EXT3, Reiseref, JFS, XFS ? ( speed,
efficiency )
2. What is the most importent part in the Hardware ? fast HD, alot of mem,
or maybe strong cpu ?

Thanks in Advance

--------------------------
Canaan Surfing Ltd.
Internet Service Providers
Ben-Nes Michael - Manager
Tel: 972-4-6991122
Fax: 972-4-6990098
http://www.canaan.net.il
--------------------------
---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend

Nov 12 '05
55 5188
Here are some recent benchmarks on different Linux filesystems. As with
any benchmarks, take what you will from the numbers.

Note the Summary section, and then the detailed benchmark numbers (if
you have a stomach for huge tables of pure numbers :)

http://fsbench.netnation.com/

scott.marlowe wrote:
On Mon, 20 Oct 2003, Ben-Nes Michael wrote:

----- Original Message -----
From: "Nick Burrett" <ni**@dsvr.ne t>
To: "Ben-Nes Michael" <mi**@canaan.co .il>
Cc: "postgresql " <pg***********@ postgresql.org>
Sent: Monday, October 20, 2003 2:54 PM
Subject: Re: [GENERAL] Recomended FS

>>But still the greatest question is what FS to put on ?
>>
>>I heard Reiesref can handle small files very quickly.
>
>Switchin g from ext3 to reiserfs for our name servers reduced the time
>taken to load 110,000 zones from 45 minutes to 5 minutes.
>
>However for a database, I don't think you can really factor this type of
>stuff into the equation. The performance benefits you get from
>differen t filesystem types are going to be small compared to the
>modificati ons that you can make to your database structure, queries and
>applicatio ns. The actual algorithms used in processing the data will be
>much slower than the time taken to fetch the data off disk.
So you say the FS has no real speed impact on the SB ?

In my pg data folder i have 2367 files, some big some small.

I'm saying: don't expect your DB performance to come on leaps and bounds
just because you changed to a different filesystem format. If you've
got speed problems then it might help to look elsewhere first.


I dont expect miracles :)
but still i have to choose one,so why shouldnt i choose the one which best
fit ?

I agree. I also think that the top of that logic develoment tree you
should ask yourself the first question of

"Is it ok that if the machine should suffer sudden catastrophic shutdown
due to any reason that I would have a corrupted database and would be
willing to reinitdb/restore from scratch?"

While I agree that in many instances this is acceptable, in
many it is not. If you may need it one day, SCSI is so much faster than
IDE when you turn off IDE's write cache that you now have a machine 1/2
as fast when you're on the IDE machine.

I pitted two systems against each other.

Machine A: < Clone of our current production box
Dual PIII-750MHz
1.5 Gig PC133 memory
dual 18 gig 10Krpm USCSI 160 drives

Maching B: < New machines intended to replace production box
Dual PIV Xeons-2.4GHz
2 Gig 400MHz memory
dual 80 gig 7200 RPM UDMA 133 drives

With two configs (all fresh 'initdb --locale=C'):
and postgresql.conf : wal_sync_method = open_sync, buffers = 4000.

Config 1:
/db on one partition (on IDE this always had write cache on.)
/pg_xlog on another (write cache on or off (W0/W1))

Config 2:
everything on /db/ which is a RAID-1 (both with write cache on or off on
W0/W1 on IDE) Allowed the software RAID-1 to replicate on both machines
before starting the tests.

With two possible IDE settings:

W0: Write cache off
W1: Write cache on

Note that W1 does not guarantee data integrity if power is lost while a
transaction is in progress (i.e. it's like running with fsync=false all
the time)

I ran pgbench -i -s 5 then pgbench -c 5 -t 1000 several times to
settle the machine, then ran pgbench -c 5 -t 1000 three times and chose
the median result of those three.

MachineA Config1:
141 tps

MachineB Config1 W0:
60 tps

MachineB Config1 W1:
112 tps

MachineA Config2:
101 tps

MachineB Config2 W0:
44 tps

MachineB Config2 W1:
135 tps

Just some numbers someone might find useful. I'll try to test both setups
in the same box later on if I get a chance. But it would seem that RAID
is performing better. I've tested all these configurations with the "pull
the plug" test. The SCSI survives in both configurations, while the IDE
will only survive uncorrupted when Write cache is off (W0).
---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend


---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

Nov 12 '05 #41
On Fri, 24 Oct 2003, Michael Teter wrote:
Here are some recent benchmarks on different Linux filesystems. As with
any benchmarks, take what you will from the numbers.

Note the Summary section, and then the detailed benchmark numbers (if
you have a stomach for huge tables of pure numbers :)

http://fsbench.netnation.com/


Right, but NONE of the benchmarks I've seen have been with IDE drives with
their cache disabled, which is the only way to make them reliable under
postgresql should something bad happen. but thanks for the benchmarks,
I'll look them over.
---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

Nov 12 '05 #42
On Friday 24 October 2003 16:23, scott.marlowe wrote:
Right, but NONE of the benchmarks I've seen have been with IDE drives with
their cache disabled, which is the only way to make them reliable under
postgresql should something bad happen. but thanks for the benchmarks,
I'll look them over.


I don't recall seeing anyone explain how to disable caching on a drive in this
thread. Did I miss that? 'Would be useful. I'm running a 3Ware mirror of 2
IDE drives.

Scott

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

Nov 12 '05 #43
Got to going this today, after a small delay due to the arrival of new
disks,

So the system is 2x700Mhz PIII, 512 Mb, Promise TX2000, 2x40G ATA-133
Maxtor Diamond+8 .
The relevent software is Freebsd 4.8 and Postgresql 7.4 Beta 2.

Two runs of 'pgbench -c 50 -t 1000000 -s 10 bench' with a power cord
removal after about 2 minutes were performed, one with hw.ata.wc = 1
(write cache enabled) and other with hw.ata.wc = 0 (disabled).

In *both* cases the Pg server survived - i.e it came up, performed
automatic recovery. Subsequent 'vacuum full' and further runs of pgbench
completed with no issues.

I would conclude that it not *always* the case that power failure
renders the database unuseable.

I have just noticed a similar posting from Scott were he finds the cache
enabled case has an dead database after power failure. It seems that
it's a question of how *likely* is it that the database will survive/not
survive a power failure...

The other interesting possibility is that Freebsd with soft updates
helped things remain salvageable in the cache enabled case (as some
writes *must* be lost at power off in this case)....

regards

Mark

scott.marlowe wrote:

OK, but here's the real test. As the postgres user, run 'pgbench -i',
then after that runs, run 'pgbench -c 50 -t 1000000'. While it's running
and settled (pg aux|grep postgres|wc -l should show a number of ~54 or
so.) pull the plug. Wait for the hard drives to spin down, then plug it
back in and power it one. With SCSI you will still have a coherent
database.

---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend

Nov 12 '05 #44
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sun, 26 Oct 2003 16:24:17 +1300, Mark Kirkwood wrote:
I would conclude that it not *always* the case that power failure
renders the database unuseable.

I have just noticed a similar posting from Scott were he finds the cache
enabled case has a dead database after power failure.

Other posts have noted that SCSI never fails under this condition. Apparently SCSI
drives sense an impending power loss and flush the cache before power completely
disappears. Speed *and* reliability. Hm.
Of course, anyone serious about a server would have it backed up with a UPS and
appropriate software to shut the system down during an extended power outage. This just
leaves people tripping over the power cords or maliciously pulling the plugs.
- --
jimoe at sohnen-moe dot com
pgp/gpg public key: http://www.keyserver.net/en/
-----BEGIN PGP SIGNATURE-----
Version: PGPfreeware 5.0 OS/2 for non-commercial use
Comment: PGP 5.0 for OS/2
Charset: cp850

wj8DBQE/m2PQsxxMki0foKo RAjsOAJ0ed1MV8F cWcALoxIJk66wn4 0EEvwCfVTPB
n/rxejkV2upgeZmoy 3yipes=
=fDes
-----END PGP SIGNATURE-----

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faqs/FAQ.html

Nov 12 '05 #45
On Sat, Oct 25, 2003 at 11:04:00PM -0700, James Moe wrote:
Other posts have noted that SCSI never fails under this condition. Apparently SCSI
drives sense an impending power loss and flush the cache before power completely
disappears. Speed *and* reliability. Hm.
I understood it differently. Postgresql has WAL to deal with this situation.
This issue that it only works as long as the drive doesn't lie about which
blocks have been written and which are merely in cache. Apparently IDE disks
lie and SCSI disks don't. It may be a protocol thing.

The other alternative is battery backed memory. i.e. keep the blocks in
memory hoping that power will return to the drive before it fails. Some RAID
cards do this.

Another thing is that 3ware RAID controllers stick a SCSI interface in
front of the IDE drives, so perhaps it has more scope to deal with this
issue.

Remember, when power fails the first thing that happens is the system
cancels any DMA tranfer in progress as memory is the part most sensative to
power fluctuations.
Of course, anyone serious about a server would have it backed up with aUPS and
appropriate software to shut the system down during an extended power outage. This just
leaves people tripping over the power cords or maliciously pulling the plugs.
If you start adding up the points of failure it's quite a lot. But you
should be able to proof the system against even malicious tampering.
--
Martijn van Oosterhout <kl*****@svana. org> http://svana.org/kleptog/ "All that is needed for the forces of evil to triumph is for enough good
men to do nothing." - Edmond Burke
"The penalty good people pay for not being interested in politics is to be
governed by people worse than themselves." - Plato


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE/m2g5Y5Twig3Ge+Y RApUAAKCgdeohxv/jn49jGHW4dJdsvI sTNQCgomWz
kuOH+216afo/LCes5lcTSmw=
=3AWJ
-----END PGP SIGNATURE-----

Nov 12 '05 #46
Don't forget that the power supply can fail too, so its not all about UPS,
and cords.

--------------------------
Canaan Surfing Ltd.
Internet Service Providers
Ben-Nes Michael - Manager
Tel: 972-4-6991122
Fax: 972-4-6990098
http://www.canaan.net.il
--------------------------
----- Original Message -----
From: "James Moe" <ji***@sohnen-moe.com>
To: "Postgresql General Mail List" <pg***********@ postgresql.org>
Sent: Sunday, October 26, 2003 8:04 AM
Subject: Re: [GENERAL] Recomended FS

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sun, 26 Oct 2003 16:24:17 +1300, Mark Kirkwood wrote:
I would conclude that it not *always* the case that power failure
renders the database unuseable.

I have just noticed a similar posting from Scott were he finds the cache
enabled case has a dead database after power failure.
Other posts have noted that SCSI never fails under this condition.

Apparently SCSI drives sense an impending power loss and flush the cache before power completely disappears. Speed *and* reliability. Hm.
Of course, anyone serious about a server would have it backed up with a UPS and appropriate software to shut the system down during an extended power outage. This just leaves people tripping over the power cords or maliciously pulling the plugs.

- --
jimoe at sohnen-moe dot com
pgp/gpg public key: http://www.keyserver.net/en/
-----BEGIN PGP SIGNATURE-----
Version: PGPfreeware 5.0 OS/2 for non-commercial use
Comment: PGP 5.0 for OS/2
Charset: cp850

wj8DBQE/m2PQsxxMki0foKo RAjsOAJ0ed1MV8F cWcALoxIJk66wn4 0EEvwCfVTPB
n/rxejkV2upgeZmoy 3yipes=
=fDes
-----END PGP SIGNATURE-----

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faqs/FAQ.html

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faqs/FAQ.html

Nov 12 '05 #47
En un mensaje anterior, Scott Chapman escribió:
I don't recall seeing anyone explain how to disable caching on a drive in this
thread. Did I miss that? 'Would be useful. I'm running a 3Ware mirror of 2
IDE drives.


In FreeBSD, add "hw.ata.wc= 0" to /boot/loader.conf.

Regards.

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

Nov 12 '05 #48
On Sat, 25 Oct 2003, James Moe wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sun, 26 Oct 2003 16:24:17 +1300, Mark Kirkwood wrote:
I would conclude that it not *always* the case that power failure
renders the database unuseable.

I have just noticed a similar posting from Scott were he finds the cache
enabled case has a dead database after power failure.
Other posts have noted that SCSI never fails under this condition. Apparently SCSI
drives sense an impending power loss and flush the cache before power completely
disappears. Speed *and* reliability. Hm.


Actually, it would appear that the SCSI drives simply don't lie about
fsync. I.e. when they tell the OS that they wrote the data, they wrote
the data. Some of them may have caching flushing with lying about fsync
built in, but the performance looks more like just good fsyncing to me.
It's all a guess without examining the microcode though... :-)
Of course, anyone serious about a server would have it backed up with a UPS and
appropriate software to shut the system down during an extended power outage. This just
leaves people tripping over the power cords or maliciously pulling the plugs.


Or a CPU frying, or a power supply dying, or a motherboard failure, or a
kernel panic, or any number of other possibilities. Admittedly, the first
line of defense is always good backups, but it's nice knowing that if one
of my CPUs fry, I can pull it, put in the terminator / replacement, and my
whole machine will likely come back up.

But anyone serious about a server will also likely be running on SCSI as
well as on a UPS. We use a hosting center with 3 UPS and a Diesel
generator, and we still managed to lose power about a year ago when one
UPS went haywire, browned out the circuits of the other two, and the
diesel generator's switch burnt out. Millions of dollars worth of UPS /
high reliability equipment, and a $50 switch brought it all down.
---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ma*******@postg resql.org

Nov 12 '05 #49
On Sun, 26 Oct 2003, Mark Kirkwood wrote:
Got to going this today, after a small delay due to the arrival of new
disks,

So the system is 2x700Mhz PIII, 512 Mb, Promise TX2000, 2x40G ATA-133
Maxtor Diamond+8 .
The relevent software is Freebsd 4.8 and Postgresql 7.4 Beta 2.

Two runs of 'pgbench -c 50 -t 1000000 -s 10 bench' with a power cord
removal after about 2 minutes were performed, one with hw.ata.wc = 1
(write cache enabled) and other with hw.ata.wc = 0 (disabled).

In *both* cases the Pg server survived - i.e it came up, performed
automatic recovery. Subsequent 'vacuum full' and further runs of pgbench
completed with no issues.
Sweet. It may be that the promise is turning off the cache, or that the
new generation of IDE drives is finally reporting fsync correctly. Was
there a performance difference in the set with write cache on or off?
I would conclude that it not *always* the case that power failure
renders the database unuseable.
But it usually is if write cache is enabled.
I have just noticed a similar posting from Scott were he finds the cache
enabled case has an dead database after power failure. It seems that
it's a question of how *likely* is it that the database will survive/not
survive a power failure...

The other interesting possibility is that Freebsd with soft updates
helped things remain salvageable in the cache enabled case (as some
writes *must* be lost at power off in this case)....


Free BSD may be the reason here. If it's softupdates are ordered in the
right way, it may be that even with write caching on, the drives "do the
right thing" under BSD. Time to get out my 5.0 disks and start playing
with my test server. Thanks for the test!
---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

Nov 12 '05 #50

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
1449
by: Brad Brad | last post by:
Hi, I'm looking at buying a new production server, as soon as i mentioned mysql the Dell salesmen started pushing 3-4Gb of ram, i'm not sure if this is excessive though. The OpenBSD server is 2.8Ghz and may have as many as 230 mysql sessions with 14 queries a second, the rest will be sleeping (ftp sessions maintain connection). The db directory is 80mb total, and the server is currently handling 14 requests/s with all queries being...
0
9810
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
9656
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
10821
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
1
10575
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
9358
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
7774
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
6975
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5642
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
5812
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.