473,854 Members | 1,830 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Recomended FS

Hi

I'm upgrading the DB sever hardware and also the Linux OS.

My Questions are:

1. What is the preferred FS to go with ? EXT3, Reiseref, JFS, XFS ? ( speed,
efficiency )
2. What is the most importent part in the Hardware ? fast HD, alot of mem,
or maybe strong cpu ?

Thanks in Advance

--------------------------
Canaan Surfing Ltd.
Internet Service Providers
Ben-Nes Michael - Manager
Tel: 972-4-6991122
Fax: 972-4-6990098
http://www.canaan.net.il
--------------------------
---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend

Nov 12 '05
55 5192
I have found this as well -

I have a nice simple example of a program that loops and occasionally
writes a block to a file.
On a 2 cpu machine, running 2 of these processes in parallel takes twice
as long as running just 1 process!
However if I comment out the IO, then 2 processes takes the same elapsed
time as 1.

My conclusion is there exists some sort of "big" lock on access to the
ATA array.

I believe that 3ware have a non blocking implementation of ATA RAID -
I intend to sell the Promise and obtain a 3ware in the next month of so
and test this out.

regards

Mark

Andrew Sullivan wrote:
That said, even the 2-way RAID became almost uselessly slow when
multiple queries were running -- indeed, dramatically slower than a
plain single IDE drive.

---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddres sHere" to ma*******@postg resql.org)

Nov 12 '05 #31

scott.marlowe wrote:

OK, but here's the real test. As the postgres user, run 'pgbench -i',
then after that runs, run 'pgbench -c 50 -t 1000000'. While it's running
and settled (pg aux|grep postgres|wc -l should show a number of ~54 or
so.) pull the plug. Wait for the hard drives to spin down, then plug it
back in and power it one. With SCSI you will still have a coherent
database.

Agreed in principle - pgbench is the most interesting test... for this
mailing list anyway :-).
However s = 1 makes a tiny database that fits into the file buffer cache
on most machines, which is not a very realistic situation.

e.g. the Dell gets tps = 250 for s = 1 c = 5 t = 1000. This number
looks great but its not too much to do with IO....

I am happier about s = 10 - 50 for machines with 512+ Mb of RAM.

From memory the Dell gets tps = 36 for s = 10 c = 5 t = 100000. This
result seems more believable!

If you want a coherent database on IDE drives under postgresql you will
need to issue this command: 'hdparm -W0 /dev/hdx' where x is the letter of
the drives under the RAID array to turn off write caching. This will slow
them to a crawl on writes.

I should have said that I was using Freebsd 4.8 with write caching off.
The question of whether the disk *actually* turned it off is the
significant issue, so yes, "use with care" should preface any comments
about IDE usage!

best wishes

Mark
---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ma*******@postg resql.org

Nov 12 '05 #32
I believe that 3ware have a non blocking implementation of ATA RAID -
I intend to sell the Promise and obtain a 3ware in the next month of
so and test this out.

I use 3Ware exclusively for my ATA-RAID solutions. The nice thing about
them is that
they are REAL hardware RAID and the use the SCSI layer within Linux so
you address
them as a standard SCSI device.

Also their support is in the kernel... no wierd, experimental patching.

On a Dual 2000 Athlon MP I was able to sustain 50MB/sec over large
copys (4+ gigs). Very, Very happy with them.

Sincerely,

Joshua Drake

regards

Mark

Andrew Sullivan wrote:
That said, even the 2-way RAID became almost uselessly slow when
multiple queries were running -- indeed, dramatically slower than a
plain single IDE drive.

---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddres sHere" to ma*******@postg resql.org)


--
Command Prompt, Inc., home of Mammoth PostgreSQL - S/ODBC and S/JDBC
Postgresql support, programming shared hosting and dedicated hosting.
+1-503-222-2783 - jd@commandpromp t.com - http://www.commandprompt.com
Editor-N-Chief - PostgreSQl.Org - http://www.postgresql.org

---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddres sHere" to ma*******@postg resql.org)

Nov 12 '05 #33


Mark Kirkwood wrote:
I should have said that I was using Freebsd 4.8 with write caching off.


write caching *on* - I got myself confused about what the value "1"
means....
---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match

Nov 12 '05 #34
On Thu, 23 Oct 2003, Mark Kirkwood wrote:

scott.marlowe wrote:

OK, but here's the real test. As the postgres user, run 'pgbench -i',
then after that runs, run 'pgbench -c 50 -t 1000000'. While it's running
and settled (pg aux|grep postgres|wc -l should show a number of ~54 or
so.) pull the plug. Wait for the hard drives to spin down, then plug it
back in and power it one. With SCSI you will still have a coherent
database.

Agreed in principle - pgbench is the most interesting test... for this
mailing list anyway :-).
However s = 1 makes a tiny database that fits into the file buffer cache
on most machines, which is not a very realistic situation.

e.g. the Dell gets tps = 250 for s = 1 c = 5 t = 1000. This number
looks great but its not too much to do with IO....

I am happier about s = 10 - 50 for machines with 512+ Mb of RAM.

From memory the Dell gets tps = 36 for s = 10 c = 5 t = 100000. This
result seems more believable!


You missed my point there. I wasn't CARING what kind of numbers you get
back at all. My point was that if you place the database under fairly
high transactional load, and pull the plug, is the database still coherent
when it comes back up.

I generally test with -s10 through -s50, but for this test it makes no
difference I can see, i.e. if the thing is gonna get scrammed at -s50,
it'll get scrammed at -s1 as well, and take less time to test.
If you want a coherent database on IDE drives under postgresql you will
need to issue this command: 'hdparm -W0 /dev/hdx' where x is the letter of
the drives under the RAID array to turn off write caching. This will slow
them to a crawl on writes.

I should have said that I was using Freebsd 4.8 with write caching off.
The question of whether the disk *actually* turned it off is the
significant issue, so yes, "use with care" should preface any comments
about IDE usage!


-- NOTE in a correction Mark stated that caching was on, not off --

Assuming that the caching was on, I'm betting your database won't survive
a power plug pull in the middle of transactions like the test I put up
above.
---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend

Nov 12 '05 #35
On Wed, 22 Oct 2003, Joshua D. Drake wrote:
I believe that 3ware have a non blocking implementation of ATA RAID -
I intend to sell the Promise and obtain a 3ware in the next month of
so and test this out.

I use 3Ware exclusively for my ATA-RAID solutions. The nice thing about
them is that
they are REAL hardware RAID and the use the SCSI layer within Linux so
you address
them as a standard SCSI device.

Also their support is in the kernel... no wierd, experimental patching.

On a Dual 2000 Athlon MP I was able to sustain 50MB/sec over large
copys (4+ gigs). Very, Very happy with them.


Do they survive the power plug pulling test I was talking about elsewhere
in this thread?
---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ma*******@postg resql.org

Nov 12 '05 #36
Its worth checking - isn't it ?

I appeciate that you may have performed such tests previously - but as
hardware and software evolve its often worth repeating such tests (goes
away to do the suggested one tonight).

Note that I am not trying to argue away the issue about write caching -
it *has* to increase the risk of database corruption following a power
failure, however if your backups are regular and reliable this may be a
risk worth taking to achieve acceptable performance at a low price.

regards

Mark
scott.marlowe wrote:

Assuming that the caching was on, I'm betting your database won't survive
a power plug pull in the middle of transactions like the test I put up
above.

---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings

Nov 12 '05 #37
Mark Kirkwood wrote:
Its worth checking - isn't it ?

I appeciate that you may have performed such tests previously - but as
hardware and software evolve its often worth repeating such tests (goes
away to do the suggested one tonight).

Note that I am not trying to argue away the issue about write caching -
it *has* to increase the risk of database corruption following a power
failure, however if your backups are regular and reliable this may be a
risk worth taking to achieve acceptable performance at a low price.


Sure, but how many people are taking that risk and not knowing it!

--
Bruce Momjian | http://candle.pha.pa.us
pg***@candle.ph a.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match

Nov 12 '05 #38
I suspect almost everyone using IDE drives -

We the "consumers" of this technology need to demand that the vendors:

1. Be honest about these limitations / bugs
2. Work to fix obvious bugs - e.g. drives lying about write cache status
need to have their behaviour changed as soon as possible.

In the meantime I guess all we can do is try to understand the issue and
raise awareness

regards

Mark

Bruce Momjian wrote:
Mark Kirkwood wrote:

Its worth checking - isn't it ?

I appeciate that you may have performed such tests previously - but as
hardware and software evolve its often worth repeating such tests (goes
away to do the suggested one tonight).

Note that I am not trying to argue away the issue about write caching -
it *has* to increase the risk of database corruption following a power
failure, however if your backups are regular and reliable this may be a
risk worth taking to achieve acceptable performance at a low price.


Sure, but how many people are taking that risk and not knowing it!

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

Nov 12 '05 #39
On Mon, 20 Oct 2003, Ben-Nes Michael wrote:
----- Original Message -----
From: "Nick Burrett" <ni**@dsvr.ne t>
To: "Ben-Nes Michael" <mi**@canaan.co .il>
Cc: "postgresql " <pg***********@ postgresql.org>
Sent: Monday, October 20, 2003 2:54 PM
Subject: Re: [GENERAL] Recomended FS
>>But still the greatest question is what FS to put on ?
>>
>>I heard Reiesref can handle small files very quickly.
>
>Switching from ext3 to reiserfs for our name servers reduced the time
>taken to load 110,000 zones from 45 minutes to 5 minutes.
>
>However for a database, I don't think you can really factor this type of
>stuff into the equation. The performance benefits you get from
>different filesystem types are going to be small compared to the
>modificatio ns that you can make to your database structure, queries and
>application s. The actual algorithms used in processing the data will be
>much slower than the time taken to fetch the data off disk.
So you say the FS has no real speed impact on the SB ?

In my pg data folder i have 2367 files, some big some small.


I'm saying: don't expect your DB performance to come on leaps and bounds
just because you changed to a different filesystem format. If you've
got speed problems then it might help to look elsewhere first.

I dont expect miracles :)
but still i have to choose one,so why shouldnt i choose the one which best
fit ?


I agree. I also think that the top of that logic develoment tree you
should ask yourself the first question of

"Is it ok that if the machine should suffer sudden catastrophic shutdown
due to any reason that I would have a corrupted database and would be
willing to reinitdb/restore from scratch?"

While I agree that in many instances this is acceptable, in
many it is not. If you may need it one day, SCSI is so much faster than
IDE when you turn off IDE's write cache that you now have a machine 1/2
as fast when you're on the IDE machine.

I pitted two systems against each other.

Machine A: < Clone of our current production box
Dual PIII-750MHz
1.5 Gig PC133 memory
dual 18 gig 10Krpm USCSI 160 drives

Maching B: < New machines intended to replace production box
Dual PIV Xeons-2.4GHz
2 Gig 400MHz memory
dual 80 gig 7200 RPM UDMA 133 drives

With two configs (all fresh 'initdb --locale=C'):
and postgresql.conf : wal_sync_method = open_sync, buffers = 4000.

Config 1:
/db on one partition (on IDE this always had write cache on.)
/pg_xlog on another (write cache on or off (W0/W1))

Config 2:
everything on /db/ which is a RAID-1 (both with write cache on or off on
W0/W1 on IDE) Allowed the software RAID-1 to replicate on both machines
before starting the tests.

With two possible IDE settings:

W0: Write cache off
W1: Write cache on

Note that W1 does not guarantee data integrity if power is lost while a
transaction is in progress (i.e. it's like running with fsync=false all
the time)

I ran pgbench -i -s 5 then pgbench -c 5 -t 1000 several times to
settle the machine, then ran pgbench -c 5 -t 1000 three times and chose
the median result of those three.

MachineA Config1:
141 tps

MachineB Config1 W0:
60 tps

MachineB Config1 W1:
112 tps

MachineA Config2:
101 tps

MachineB Config2 W0:
44 tps

MachineB Config2 W1:
135 tps

Just some numbers someone might find useful. I'll try to test both setups
in the same box later on if I get a chance. But it would seem that RAID
is performing better. I've tested all these configurations with the "pull
the plug" test. The SCSI survives in both configurations, while the IDE
will only survive uncorrupted when Write cache is off (W0).
---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend

Nov 12 '05 #40

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
1449
by: Brad Brad | last post by:
Hi, I'm looking at buying a new production server, as soon as i mentioned mysql the Dell salesmen started pushing 3-4Gb of ram, i'm not sure if this is excessive though. The OpenBSD server is 2.8Ghz and may have as many as 230 mysql sessions with 14 queries a second, the rest will be sleeping (ftp sessions maintain connection). The db directory is 80mb total, and the server is currently handling 14 requests/s with all queries being...
0
9903
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
9752
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
10692
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
10373
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
1
7921
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
7083
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5753
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
5946
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
2
4167
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.