File map performance

[George2]

Hello everyone,
I have several physical file and I want to use file map
(MapViewOfFileEx) to map the file into memory to improve performance.
Each file is about several hundred M bytes. All the memory mapped
files are kept open during my application.

The mapping is successful, but the strange thing is,

1. the performance to access the files which are opened at first is
very fast;
2. the performance to access the files which are opened later is
slower and slower (the performance to access the 10th file is very
bad).

Any ideas to improve performance?
thanks in advance,
George

[Victor Bazarov]

George2 wrote:

I have several physical file and I want to use file map
(MapViewOfFileEx) to map the file into memory to improve performance.

....whatever "file map" is...

Each file is about several hundred M bytes. All the memory mapped
files are kept open during my application.

The mapping is successful, but the strange thing is,

1. the performance to access the files which are opened at first is
very fast;
2. the performance to access the files which are opened later is
slower and slower (the performance to access the 10th file is very
bad).

Any ideas to improve performance?

You need to ask in the newsgroup where "file map" is on topic. It is
not a feature of the language, it's a feature of your OS.

V
--
Please remove capital 'A's when replying by e-mail
I do not respond to top-posted replies, please don't ask

[Jim Langston]

George2 wrote:

Hello everyone,
I have several physical file and I want to use file map
(MapViewOfFileEx) to map the file into memory to improve performance.
Each file is about several hundred M bytes. All the memory mapped
files are kept open during my application.

The mapping is successful, but the strange thing is,

1. the performance to access the files which are opened at first is
very fast;
2. the performance to access the files which are opened later is
slower and slower (the performance to access the 10th file is very
bad).

Any ideas to improve performance?

It sounds like you are running into disk swapping and/or thrashing issues.
Physical memory is limited (1GB 2GB is common) and is shared amoung
programs, the operating system, etc... As you request memory from the OS it
allocates the memory. :The more memory you request the more that gets
swapped to disk.

So basically what it sounds like is happening, you load a file into memory,
and keep loading files. The memory starts to get swapped to disk. As you
go back and read the memory the OS has to read it from the disk swap file
and write other memory to the swap file for what it's replacing. So now
instead of saving time from reading a file once, it has to write and read.

Several hundred MB of many files is too much to keep in physical memory at
one time. I'm thinking it may actually be faster to read the files as
needed counting on the OS's own file buffering to reduce physical disk I/O.

--
Jim Langston
ta*******@rocketmail.com

[Rahul]

On Jan 2, 9:06 pm, George2 <[e-mail]:

Hello everyone,

I have several physical file and I want to use file map
(MapViewOfFileEx) to map the file into memory to improve performance.
Each file is about several hundred M bytes. All the memory mapped
files are kept open during my application.

The mapping is successful, but the strange thing is,

1. the performance to access the files which are opened at first is
very fast;
2. the performance to access the files which are opened later is
slower and slower (the performance to access the 10th file is very
bad).

Any ideas to improve performance?

thanks in advance,
George

Are you using mmap() and does your operating system support virtual
memory?
What is your objective and what kind of application are you developing?

[Rahul]

>

Several hundred MB of many files is too much to keep in physical memory at
one time. I'm thinking it may actually be faster to read the files as
needed counting on the OS's own file buffering to reduce physical disk I/O.

--
Jim Langston
[e-mail]

I really wonder how OS's own file buffering would improve performance
for the OP's scenario...
after all, it too would need to load the contents of the file into the
memory right? So i would imagine the same performance issues with OS's
mechanisms too...

may be the op's application can't read from a file but instead only
from the memory with the help of a pointer to the buffer...
I have faced some third party applications, which accepts data only in
a buffer and not in a file... i used mmap() in those case...

[Jim Langston]

Rahul wrote:

>Several hundred MB of many files is too much to keep in physical
memory at one time. I'm thinking it may actually be faster to read
the files as needed counting on the OS's own file buffering to
reduce physical disk I/O.

--
Jim Langston
[e-mail]

I really wonder how OS's own file buffering would improve performance
for the OP's scenario...
after all, it too would need to load the contents of the file into the
memory right? So i would imagine the same performance issues with OS's
mechanisms too...

may be the op's application can't read from a file but instead only
from the memory with the help of a pointer to the buffer...
I have faced some third party applications, which accepts data only in
a buffer and not in a file... i used mmap() in those case...

Because then the OS would only have to buffer it once. Not the OS buffering
it, and him buffering it as is the current case. It depends on how well the
OS buffers files.

--
Jim Langston
ta*******@rocketmail.com

[Rahul]

On Jan 2, 10:47 pm, "Jim Langston" <[e-mail]:

Rahul wrote:

Several hundred MB of many files is too much to keep in physical
memory at one time. I'm thinking it may actually be faster to read
the files as needed counting on the OS's own file buffering to
reduce physical disk I/O.

--
Jim Langston
[e-mail]

I really wonder how OS's own file buffering would improve performance
for the OP's scenario...
after all, it too would need to load the contents of the file into the
memory right? So i would imagine the same performance issues with OS's
mechanisms too...

may be the op's application can't read from a file but instead only
from the memory with the help of a pointer to the buffer...
I have faced some third party applications, which accepts data only in
a buffer and not in a file... i used mmap() in those case...

Because then the OS would only have to buffer it once. Not the OS buffering
it, and him buffering it as is the current case. It depends on how well the
OS buffers files.

--
Jim Langston
[e-mail]

Nothing to comment until the OP tells about his implementation...
But i'm sure mmap() is a single buffer operation just like file
operation...