473,230 Members | 1,555 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,230 software developers and data experts.

File writing performance

Lu
Hi there,

I got a program to write data to a randomly accessed file (the program
moves file pointer to a certain position of the file according the
current "keyword" and then writes data). It compiles and runs in Win2K
and various Unix and Linux systems. The main program is in Fortran,
calling a data writing library writen in C++.

The weird thing is, it takes much longer elapse time (wall-clock time)
than CPU time (e.g., 20 min vs. 2 min) in IBM-AIX (AIX4.3), while in
all other systems, these two kinds of time are almost the same (e.g.
2min). The extra waiting in IBM machine does happen during file
pointer locating and data writing stage. In all cases, this program is
the only runing application during testing.

Do you have any idea what may cause the long waiting time in IBM
machine? It created a big performance headache for me now.

Thanks for any suggestion.

Regards,
Lu
Jul 19 '05 #1
8 3295
Lu wrote:
Hi there,

I got a program to write data to a randomly accessed file (the program
moves file pointer to a certain position of the file according the
current "keyword" and then writes data). It compiles and runs in Win2K
and various Unix and Linux systems. The main program is in Fortran,
calling a data writing library writen in C++.

The weird thing is, it takes much longer elapse time (wall-clock time)
than CPU time (e.g., 20 min vs. 2 min) in IBM-AIX (AIX4.3), while in
all other systems, these two kinds of time are almost the same (e.g.
2min). The extra waiting in IBM machine does happen during file
pointer locating and data writing stage. In all cases, this program is
the only runing application during testing.

Do you have any idea what may cause the long waiting time in IBM
machine? It created a big performance headache for me now.

Thanks for any suggestion.

Regards,
Lu

sounds like seismic :-)

BTW, I think this question is better for comp.unix.programmer, this is a
newsgroup for std C++ questions, not OS systems calls.

~S

Jul 19 '05 #2
In comp.lang.fortran Lu <ql*@lgc.com> wrote:
The weird thing is, it takes much longer elapse time (wall-clock time)
than CPU time (e.g., 20 min vs. 2 min) in IBM-AIX (AIX4.3), while in
all other systems, these two kinds of time are almost the same (e.g.
2min). The extra waiting in IBM machine does happen during file
pointer locating and data writing stage. In all cases, this program is
the only runing application during testing. Do you have any idea what may cause the long waiting time in IBM
machine? It created a big performance headache for me now.


I/O should not generally use a lot of CPU time, so there is nothing wrong
with wall time being much longer than CPU time. It's just slow I/O. Are
you doing the I/O over NFS? How old and slow are the hard disks? It
could be an AIX problem, but I don't use it so can't help there - that's
for the AIX folks to help with.

Regards,
Daniel.
------------------------------------------------------------------------
Dr. Daniel Grimwood Department of Chemistry
Email : re****@theochem.uwa.edu.au The University of Western Australia
Phone : +61 8 93808563 35 Stirling Highway
Fax : +61 8 93801005 Crawley WA 6009
Jul 19 '05 #3
Lu,

you statement sounds as if AIX is much slower than any other machine. Is
that true or do you mean it completed in the same time but with less CPU?

Regards Rolf
Jul 19 '05 #4
Lu
ql*@lgc.com (Lu) wrote in message news:<89**************************@posting.google. com>...

Thanks for all your response.

All tests wrote the data to local hard disks to avoid network traffic.
I tested in AIX, SGI, Linux and W2K, the CPU times are roughly the
same (2- 4 mins), only AIX got a long extra waiting time. The IBM-AIX
machines were bought about two years ago, so the hard disk should be
good enough.

If writing to a sequentialy accessed file, the extra waiting time is
gone.

I also tried to run the executable in a SGI machine and write
(randomly accessed) data file to a remote IBM hard disk (through NFS),
or run the executable in a IBM machine and write data file to a remote
SGI hard disk, the long waiting time also occured. But if I run the
executable in SGI machine and wrote to a remote SGI hard disk (through
NFS), there is no extra waiting time.

Thanks,
Lu
Jul 19 '05 #5
> Thanks for all your response.

All tests wrote the data to local hard disks to avoid network traffic.
I tested in AIX, SGI, Linux and W2K, the CPU times are roughly the
same (2- 4 mins), only AIX got a long extra waiting time. The IBM-AIX
machines were bought about two years ago, so the hard disk should be
good enough.

If writing to a sequentialy accessed file, the extra waiting time is
gone.

I also tried to run the executable in a SGI machine and write
(randomly accessed) data file to a remote IBM hard disk (through NFS),
or run the executable in a IBM machine and write data file to a remote
SGI hard disk, the long waiting time also occured. But if I run the
executable in SGI machine and wrote to a remote SGI hard disk (through
NFS), there is no extra waiting time.

Thanks,
Lu


1. This is not a DB2 problem.

2. This is not a problem with an IBM machine running AIX that is correctly
configured. Otherwise, IBM would not be able to sell any RS/6000 machines if
it were an inherent problem with the platform.
Jul 19 '05 #6

"Lu" <ql*@lgc.com> wrote in message
news:89**************************@posting.google.c om...
ql*@lgc.com (Lu) wrote in message news:<89**************************@posting.google. com>...
Thanks for all your response.

All tests wrote the data to local hard disks to avoid network traffic.
I tested in AIX, SGI, Linux and W2K, the CPU times are roughly the
same (2- 4 mins), only AIX got a long extra waiting time. The IBM-AIX
machines were bought about two years ago, so the hard disk should be
good enough.

If writing to a sequentialy accessed file, the extra waiting time is
gone.


It might be that, in the case of direct access (random access) files each
write is flushed (fflush in C) to the disk, where normal sequential writes
go through the library and OS buffers and disk cache. That might be true
even if the random access file is sequentially accessed.

-- glen
Jul 19 '05 #7
Hi
Did you check the size of the file that you were writing to.

It would not be the first time that someone ( myself included) tried to fill
a 1,3 file with squigabytes of data. Pick is so brilliant it will cope
unfortunately it will all be in overflow.

Peter McMurray
Jul 19 '05 #8
See in comp.unix.aix about vmtune and numclust, maxrandwrt

vmtune is part of bos.adt.samples

The current default settings in AIX 4.3.3 ML11 is -c 0 -W 0

Example of test program with seeks and writes to timex and see how
changing the parms affects the elapsed time.

#include <stdio.h>
#include <fcntl.h>
#include <signal.h>
#include <sys/ioctl.h>
#define NWRITES 3000
extern int errno;
main(argc,argv)
int argc;
char *argv[];
{ int fd1,j,i=0;
int arg1;
char ar_1[1024000];
if(argc==1)
fd1=open("tmp0",O_WRONLY|O_TRUNC|O_CREAT);
if(fd1==-1) {printf("\nError %d on open %s \n",errno,ar_1);
exit(1);
}
for(i=0;i<1024;i++) ar_1[i]='1';
i=0;
for(i=0;i<NWRITES;i++)
{lseek(fd1,i*1024,SEEK_SET);
j=write(fd1,ar_1,1024);
if(j==-1) {printf("\n Error %d on write ",errno);break;
}
lseek(fd1,NWRITES*1024+i*1024,SEEK_SET);
j=write(fd1,ar_1,1024);
if(j==-1) {printf("\n Error %d on write ",errno);break;
}
}
printf("\nTotal writes is: %d\n",i);
}

Bernard Dhooghe


ql*@lgc.com (Lu) wrote in message news:<89**************************@posting.google. com>...
Hi there,

I got a program to write data to a randomly accessed file (the program
moves file pointer to a certain position of the file according the
current "keyword" and then writes data). It compiles and runs in Win2K
and various Unix and Linux systems. The main program is in Fortran,
calling a data writing library writen in C++.

The weird thing is, it takes much longer elapse time (wall-clock time)
than CPU time (e.g., 20 min vs. 2 min) in IBM-AIX (AIX4.3), while in
all other systems, these two kinds of time are almost the same (e.g.
2min). The extra waiting in IBM machine does happen during file
pointer locating and data writing stage. In all cases, this program is
the only runing application during testing.

Do you have any idea what may cause the long waiting time in IBM
machine? It created a big performance headache for me now.

Thanks for any suggestion.

Regards,
Lu

Jul 19 '05 #9

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

6
by: Sebastian Kemi | last post by:
How should a write a class to a file? Would this example work: object *myobject = 0; tfile.write(reinterpret_cast<char *>(myobject), sizeof(*object)); / sebek
3
by: ishekar | last post by:
Hi, I have an application where i want to write data to a file, the data is being sent from an external source. I know the total size of the data and then i retrieve the data in small segments...
8
by: Lu | last post by:
Hi there, I got a program to write data to a randomly accessed file (the program moves file pointer to a certain position of the file according the current "keyword" and then writes data). It...
19
by: wetherbean | last post by:
Hi group..I am writing a playlist management protocol where I have a file that holds all the playlists and a file that holds all the songs....before a playlist is created I need to check to see if...
3
by: Mike Kelly | last post by:
Hi. I've built a page using standard ASP.NET 2.0 features and when I upload a large file (>20MB) to our intranet server, I get a paltry 100KB/s on our 100Mb/s LAN. Simply copying the file, I get...
10
by: m00nm0nkey | last post by:
Hello. I am trying to split a file with 334,386 lines into seperate files of 50,000 each. This is the code i am running: Dim intFragmentRawIndex As Integer Dim swRawDataFile As StreamWriter...
1
by: laredotornado | last post by:
Hi, I'm using PHP 4.4.4 on Apache 2 on Fedora Core 5. PHP was installed using Apache's apxs and the php library was installed to /usr/local/php. However, when I set my "error_reporting"...
4
by: Dameon | last post by:
Hi All, I have a process where I'd like to search the contents of a file(in a dir) for all occurences (or the count of) of a given string. My goal is to focus more on performance, as some of the...
12
by: glennanthonyb | last post by:
Hi The company I work for has finally woken up to data security on our field laptops. I'm writing something in C# that will allow remote deletion of sensitive data and I don't believe...
3
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 3 Jan 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). For other local times, please check World Time Buddy In...
0
by: jianzs | last post by:
Introduction Cloud-native applications are conventionally identified as those designed and nurtured on cloud infrastructure. Such applications, rooted in cloud technologies, skillfully benefit from...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: fareedcanada | last post by:
Hello I am trying to split number on their count. suppose i have 121314151617 (12cnt) then number should be split like 12,13,14,15,16,17 and if 11314151617 (11cnt) then should be split like...
0
by: stefan129 | last post by:
Hey forum members, I'm exploring options for SSL certificates for multiple domains. Has anyone had experience with multi-domain SSL certificates? Any recommendations on reliable providers or specific...
0
Git
by: egorbl4 | last post by:
Скачал я git, хотел начать настройку, а там вылезло вот это Что это? Что мне с этим делать? ...
1
by: davi5007 | last post by:
Hi, Basically, I am trying to automate a field named TraceabilityNo into a web page from an access form. I've got the serial held in the variable strSearchString. How can I get this into the...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.