473,594 Members | 2,752 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

SCSI vs. IDE performance test

http://hardware.devchannel.org/hardw...&tid=38&tid=49

--
-----------------------------------------------------------------
Ron Johnson, Jr. ro***********@c ox.net
Jefferson, LA USA

I can't make you have an abortion, but you can *make* me pay
child support for 18 years? However, if I want the child (and
all the expenses that entails) for the *rest*of*my*lif e*, and you
don't want it for 9 months, tough luck???
---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ma*******@postg resql.org

Nov 12 '05 #1
36 3943
The SCSI improvement over IDE seems overrated in the test. I would have
expected at most a 30% improvement. Other reviews seem to point out that IDE
performs just as well or better.

See Tom's hardware:
http://www20.tomshardware.com/storag...305/index.html

Stephen
"Ron Johnson" <ro***********@ cox.net> wrote in message
news:1066837102 .12532.176.came l@haggis...
http://hardware.devchannel.org/hardw...9.shtml?tid=20
&tid=38&tid= 49
--
-----------------------------------------------------------------
Ron Johnson, Jr. ro***********@c ox.net
Jefferson, LA USA

I can't make you have an abortion, but you can *make* me pay
child support for 18 years? However, if I want the child (and
all the expenses that entails) for the *rest*of*my*lif e*, and you
don't want it for 9 months, tough luck???
---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ma*******@postg resql.org

Nov 12 '05 #2
On Wed, 2003-10-22 at 11:01, Stephen wrote:
The SCSI improvement over IDE seems overrated in the test. I would have
expected at most a 30% improvement. Other reviews seem to point out that IDE
performs just as well or better.

See Tom's hardware:
http://www20.tomshardware.com/storag...305/index.html
When TCQ becomes a reality in IDE drives, they'll have a fighting
chance, but the slower seek times and rotational speeds will still
do them in.

Also, does an 8MB cache *really* make that much of a difference?
After all, it can only cache 0.0067% of a 120GB drive, and 0.00267%
of the new 300GB disks.

Speaking of which, that 300GB HDD sounds like a dream for near-
line storage, and even for nightly backups, if it is ever put in
SBB-type packaging.
http://www20.tomshardware.com/storag...008/index.html
Imagine a scheme where you rapidly pg_dump to the 300GB drive,
then, at leisure, tar the dump file to tape. Stripe a few together,
and keep a month of backups on-line for quick recovery, along with
the tape archives, in case the stripeset gets wasted, too.
"Ron Johnson" <ro***********@ cox.net> wrote in message
news:1066837102 .12532.176.came l@haggis...

http://hardware.devchannel.org/hardw...9.shtml?tid=20
&tid=38&tid= 49


--
-----------------------------------------------------------------
Ron Johnson, Jr. ro***********@c ox.net
Jefferson, LA USA

"Adventure is a sign of incompetence"
Stephanson, great polar explorer
---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

Nov 12 '05 #3
I just ran some benchmarks against a 10K SCSI drive and 7200 RPM IDE
drive here:

http://fsbench.netnation.com/

The results vary quite a bit, and it seems the file system you use
can make a huge difference.

SCSI is obviously faster, but a 20% performance gain for 5x the cost is
only worth it for a very small percentage of people, I would think.

On Wed, 2003-10-22 at 09:01, Stephen wrote:
The SCSI improvement over IDE seems overrated in the test. I would have
expected at most a 30% improvement. Other reviews seem to point out that IDE
performs just as well or better.

See Tom's hardware:
http://www20.tomshardware.com/storag...305/index.html

Stephen
"Ron Johnson" <ro***********@ cox.net> wrote in message
news:1066837102 .12532.176.came l@haggis...

http://hardware.devchannel.org/hardw...9.shtml?tid=20
&tid=38&tid= 49

--
-----------------------------------------------------------------
Ron Johnson, Jr. ro***********@c ox.net
Jefferson, LA USA

I can't make you have an abortion, but you can *make* me pay
child support for 18 years? However, if I want the child (and
all the expenses that entails) for the *rest*of*my*lif e*, and you
don't want it for 9 months, tough luck???
---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ma*******@postg resql.org


---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match

--
Best Regards,

Mike Benoit

---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match

Nov 12 '05 #4
Mike Benoit wrote:
I just ran some benchmarks against a 10K SCSI drive and 7200 RPM IDE
drive here:

http://fsbench.netnation.com/

The results vary quite a bit, and it seems the file system you use
can make a huge difference.

SCSI is obviously faster, but a 20% performance gain for 5x the cost is
only worth it for a very small percentage of people, I would think.


Did you turn off the IDE write cache? If not, the SCSI drive is
reliable in case of OS failure, while the IDE is not.

--
Bruce Momjian | http://candle.pha.pa.us
pg***@candle.ph a.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

Nov 12 '05 #5
It seems to me file system journaling should fix the whole problem by giving
you a record of what was actually commited to disk and what was not. I must
not understand journaling correctly. Can anyone explain to me how
journaling works.

----- Original Message -----
From: "Bruce Momjian" <pg***@candle.p ha.pa.us>
To: <mi***@netnatio n.com>
Cc: "Stephen" <jl*****@xxxxxx .com>; <pg***********@ postgresql.org>
Sent: Monday, October 27, 2003 12:14 PM
Subject: Re: [GENERAL] SCSI vs. IDE performance test

Mike Benoit wrote:
I just ran some benchmarks against a 10K SCSI drive and 7200 RPM IDE
drive here:

http://fsbench.netnation.com/

The results vary quite a bit, and it seems the file system you use
can make a huge difference.

SCSI is obviously faster, but a 20% performance gain for 5x the cost is
only worth it for a very small percentage of people, I would think.
Did you turn off the IDE write cache? If not, the SCSI drive is
reliable in case of OS failure, while the IDE is not.

--
Bruce Momjian | http://candle.pha.pa.us
pg***@candle.ph a.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania

19073
---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match

Nov 12 '05 #6
"Rick Gigger" <ri**@alpinenet working.com> writes:
It seems to me file system journaling should fix the whole problem by giving
you a record of what was actually commited to disk and what was not.


Nope, a journaling FS has exactly the same problem Postgres does
(because the underlying "WAL" concept is the same: write the log entries
before you change the files they describe). If the drive lies about
write order, the FS can be screwed just as badly. Now the FS code might
have a low-level way to force write order that Postgres doesn't have
access to ... but simply uttering the magic incantation "journaling file
system" will not make this problem disappear.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faqs/FAQ.html

Nov 12 '05 #7
ahhh. "lies about write order" is the phrase that I was looking for. That
seemed to make sense but I didn't know if I could go directly from "lying
about fsync" to that. Obviously I don't understand exactly what fsync is
doing. I assume this means that if you were to turn fsync off you would get
considerably better performance but introduce the possibility of corrupting
the files in your database.

Thank you. This makes a lot more sense now.

----- Original Message -----
From: "Tom Lane" <tg*@sss.pgh.pa .us>
To: "Rick Gigger" <ri**@alpinenet working.com>
Cc: <pg***********@ postgresql.org>
Sent: Monday, October 27, 2003 3:39 PM
Subject: Re: [GENERAL] SCSI vs. IDE performance test

"Rick Gigger" <ri**@alpinenet working.com> writes:
It seems to me file system journaling should fix the whole problem by giving you a record of what was actually commited to disk and what was not.


Nope, a journaling FS has exactly the same problem Postgres does
(because the underlying "WAL" concept is the same: write the log entries
before you change the files they describe). If the drive lies about
write order, the FS can be screwed just as badly. Now the FS code might
have a low-level way to force write order that Postgres doesn't have
access to ... but simply uttering the magic incantation "journaling file
system" will not make this problem disappear.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faqs/FAQ.html

---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend

Nov 12 '05 #8
Tom, this discussion brings up something that's been bugging me about the
recommendations for getting more performance out of PG.. in particular the
one that suggests you put your WAL files on a different physical drive from
the database.

Consider the following scenario:
Database on drive1
WAL on drive2

1. PG write of some sort occurs.
2. PG writes out the WAL.
3. PG writes out the data.
4. PG updates the WAL to reflect data actually written.
5. System crashes/reboots/whatever.

With the DB and the WAL on different drives, it seems possible to me that
drive2 could've fsync()'d or otherwise properly written all of the data
out, but drive1 could have failed somewhere along the way and not actually
written the data to the DB.

The next time PG is brought up, the WAL would indicate the transaction, as
it were, was a success.. but the data wouldn't actually be there.

In the case of using only one drive, the rollback (from a FS perspective)
couldn't possibly occur in such a way as to leave step 4 as a success, but
step 3 as a failure -- worst case, the data would be written out but the
WAL wouldn't have been updated (rolled back say by the FS) and thus PG will
roll back the data itself, or use whatever mechanism it uses to insure data
integrity is consistent with the WAL.

Am I smoking something here or is this a real, if rare in practice, risk
that occurs when you have the WAL on a different drive than the data is on?
At 17:39 10/27/2003, Tom Lane wrote:
"Rick Gigger" <ri**@alpinenet working.com> writes:
It seems to me file system journaling should fix the whole problem by

giving
you a record of what was actually commited to disk and what was not.


Nope, a journaling FS has exactly the same problem Postgres does
(because the underlying "WAL" concept is the same: write the log entries
before you change the files they describe). If the drive lies about
write order, the FS can be screwed just as badly. Now the FS code might
have a low-level way to force write order that Postgres doesn't have
access to ... but simply uttering the magic incantation "journaling file
system" will not make this problem disappear.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faqs/FAQ.html

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

Nov 12 '05 #9
"Rick Gigger" <ri**@alpinenet working.com> writes:
ahhh. "lies about write order" is the phrase that I was looking for. That
seemed to make sense but I didn't know if I could go directly from "lying
about fsync" to that. Obviously I don't understand exactly what fsync is
doing.


What we actually care about is write order: WAL entries have to hit the
platter before the corresponding data-file changes do. Unfortunately we
have no portable means of expressing that exact constraint to the
kernel. We use fsync() (or related constructs) instead: issue the WAL
writes, fsync the WAL file, then issue the data-file writes. This
constrains the write ordering more than is really needed, but it's the
best we can do in a portable Unix application.

The problem is that the kernel thinks fsync is done when the disk drive
reports the writes are complete. When we say a drive lies about this,
we mean it accepts a sector of data into its on-board RAM and then
immediately claims write-complete, when in reality the data hasn't hit
the platter yet and will be lost if power dies before the drive gets
around to writing it.

So we can have a scenario where we think WAL is down to disk and go
ahead with issuing data-file writes. These will also be shoved over to
the drive and stored in its on-board RAM. Now the drive has multiple
sectors pending write in its buffers. If it chooses to write these in
some order other than the order they were given to it, it could write
the data file updates to disk first. If power drops *now*, we lose,
because the data files are inconsistent and there's no WAL entry to tell
us to fix it.

Got it? It's really the combination of "lie about write completion" and
"write pending sectors out of order" that can mess things up.

The reason IDE drives have to do this for reasonable performance is that
the IDE interface is single-threaded: you can only have one read or
write in process at a time, from the point of view of the
kernel-to-drive interface. But in order to schedule reads and writes in
a way that makes sense physically (minimizes seeks), the drive has to
have multiple read and write requests pending that it can pick and
choose from. The only possibility to do that in the IDE world is to
let a write "complete" in interface terms before it's really done ...
that is, lie.

The reason SCSI drives do *not* do this is that the SCSI interface is
logically multi-threaded: you can have multiple reads or writes pending
at once. When you want to write on a SCSI drive, you send over a
command that says "write this data at this sector". Sometime later the
drive sends back a status report "yessir boss, I done did that write".
Similarly, a read consists of a command "read this sector", followed
sometime later by a response that delivers the requested data. But you
can send other commands to read or write other sectors meanwhile, and
the drive is free to reorder them to suit its convenience. So in the
SCSI world, there is no need for the drive to lie in order to do its own
read/write scheduling. The kernel knows the truth about whether a given
sector has hit disk, and so it won't conclude that the WAL file has been
completely fsync'd until it really is all down to the platter.

This is also why SCSI disks shine on the read side when you have lots of
processes doing reads: in an IDE drive, there is no way for the drive to
satisfy read requests in any order but the one they're issued in. If the
kernel guesses wrong about the best ordering for a set of read requests,
then everybody waits for the seeks needed to get the earlier processes'
data. A SCSI drive can fetch the "nearest" data first, and then that
requester is freed to make progress in the CPU while the other guys wait
for their longer seeks. There's no win here with a single active user
process (since it probably wants specific data in a specific order), but
it's a huge win if lots of processes are making unrelated read requests.

Clear now?

(In a previous lifetime I wrote SCSI disk driver code ...)

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

Nov 12 '05 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
2580
by: Randell D. | last post by:
Folks, A ng poster recently questioned their usage/creation of arrays and their correct syntax. I got the idea to performance test from a recent (excellent) PHP Tutorial article that was in Linux Format magazine (which dealt with performance). The original poster had a reply from someone who had said using $testArray was better (and proper) than $testArray I *had* believed this to be correct... but... against my better judgement, I
1
2238
by: Marc H. | last post by:
Hello, I recently converted one of my perl scripts to python. What the script does is simply search a lot of big mail files (~40MB) to retrieve specific emails. I simply converted the script line by line to python, keeping the algorithms & functions as they were in perl (no optimization). The purpose was mainly to learn python and see the differences with perl. Now, once the converted script was finished, I was amazed to find that
12
9184
by: Dave Theese | last post by:
Hello all, I'm in a poition of trying to justify use of the STL from a performance perspective. As a starting point, can anyone cite any benchmarks comparing vectors to plain old statically-declared arrays? I'll also be looking at the other STL containers later... Also, I'd appreciate any comments anyone has on the suitability of the little program below for comparing vectors and arrays. One obvious shortcoming is that it uses...
8
2041
by: Sebastian Werner | last post by:
Howdy, I currently develop the javascript toolkit qooxdoo (http://qooxdoo.sourceforge.net), some of you heard it already. We have discovered a slowdown on Internet Explorers performance when creating objects with some data and store them in a global object registry. It take some time to get this example extracted from our codebase. The attached file (take a look at it please) shows the exact problem. The time for each object increases...
13
4116
by: Bern McCarty | last post by:
I have run an experiment to try to learn some things about floating point performance in managed C++. I am using Visual Studio 2003. I was hoping to get a feel for whether or not it would make sense to punch out from managed code to native code (I was using IJW) in order to do some amount of floating point work and, if so, what that certain amount of floating point work was approximately. To attempt to do this I made a program that...
2
2255
by: Mark | last post by:
Can anyone recommend some resources to learn how to issue scsi commands using vb.net? Thanks in advance, Mark
4
1244
by: Rob R. Ainscough | last post by:
Seriously, could VS 2005 be any slower? I've got 4GB RAM, dual core AMD CPU FX60, two 10K rpm SATA drives, and 7800GTX video card and VS 2005 is still horribly SLOW!! It is the most frustrating interface I've ever had to wait for (even with dynamic help turned off). Premiere 2.0 Pro, Ulead, Photoshop all fly along on my system, but VS 2005 brings it to a crawl!!
12
2110
by: Bob Cramer | last post by:
I'm a developer using XP Pro for developing business apps with Visual Studio.NET and SQL Server. My current workstation is a few years old and uses a single IDE drive. I'm looking to build a new workstation. I plan to NOT install Vista and continue using XP Pro for the foreseeable future. My question: What would you recommend for the hard drive(s) on my new workstation? I ask because several years ago I saw a number of friends have...
0
7936
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
7874
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
1
7997
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
8227
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
6646
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
5738
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
5402
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
1
2383
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
0
1203
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.