473,585 Members | 2,496 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

File map performance

Hello everyone,
I have several physical file and I want to use file map
(MapViewOfFileE x) to map the file into memory to improve performance.
Each file is about several hundred M bytes. All the memory mapped
files are kept open during my application.

The mapping is successful, but the strange thing is,

1. the performance to access the files which are opened at first is
very fast;
2. the performance to access the files which are opened later is
slower and slower (the performance to access the 10th file is very
bad).

Any ideas to improve performance?
thanks in advance,
George
Jan 2 '08 #1
6 3156
George2 wrote:
I have several physical file and I want to use file map
(MapViewOfFileE x) to map the file into memory to improve performance.
....whatever "file map" is...
Each file is about several hundred M bytes. All the memory mapped
files are kept open during my application.

The mapping is successful, but the strange thing is,

1. the performance to access the files which are opened at first is
very fast;
2. the performance to access the files which are opened later is
slower and slower (the performance to access the 10th file is very
bad).

Any ideas to improve performance?
You need to ask in the newsgroup where "file map" is on topic. It is
not a feature of the language, it's a feature of your OS.

V
--
Please remove capital 'A's when replying by e-mail
I do not respond to top-posted replies, please don't ask
Jan 2 '08 #2
George2 wrote:
Hello everyone,
I have several physical file and I want to use file map
(MapViewOfFileE x) to map the file into memory to improve performance.
Each file is about several hundred M bytes. All the memory mapped
files are kept open during my application.

The mapping is successful, but the strange thing is,

1. the performance to access the files which are opened at first is
very fast;
2. the performance to access the files which are opened later is
slower and slower (the performance to access the 10th file is very
bad).

Any ideas to improve performance?
It sounds like you are running into disk swapping and/or thrashing issues.
Physical memory is limited (1GB 2GB is common) and is shared amoung
programs, the operating system, etc... As you request memory from the OS it
allocates the memory. :The more memory you request the more that gets
swapped to disk.

So basically what it sounds like is happening, you load a file into memory,
and keep loading files. The memory starts to get swapped to disk. As you
go back and read the memory the OS has to read it from the disk swap file
and write other memory to the swap file for what it's replacing. So now
instead of saving time from reading a file once, it has to write and read.

Several hundred MB of many files is too much to keep in physical memory at
one time. I'm thinking it may actually be faster to read the files as
needed counting on the OS's own file buffering to reduce physical disk I/O.

--
Jim Langston
ta*******@rocke tmail.com
Jan 2 '08 #3
On Jan 2, 9:06 pm, George2 <george4acade.. .@yahoo.comwrot e:
Hello everyone,

I have several physical file and I want to use file map
(MapViewOfFileE x) to map the file into memory to improve performance.
Each file is about several hundred M bytes. All the memory mapped
files are kept open during my application.

The mapping is successful, but the strange thing is,

1. the performance to access the files which are opened at first is
very fast;
2. the performance to access the files which are opened later is
slower and slower (the performance to access the 10th file is very
bad).

Any ideas to improve performance?

thanks in advance,
George
Are you using mmap() and does your operating system support virtual
memory?
What is your objective and what kind of application are you developing?
Jan 2 '08 #4
>
Several hundred MB of many files is too much to keep in physical memory at
one time. I'm thinking it may actually be faster to read the files as
needed counting on the OS's own file buffering to reduce physical disk I/O.

--
Jim Langston
tazmas...@rocke tmail.com
I really wonder how OS's own file buffering would improve performance
for the OP's scenario...
after all, it too would need to load the contents of the file into the
memory right? So i would imagine the same performance issues with OS's
mechanisms too...

may be the op's application can't read from a file but instead only
from the memory with the help of a pointer to the buffer...
I have faced some third party applications, which accepts data only in
a buffer and not in a file... i used mmap() in those case...
Jan 2 '08 #5
Rahul wrote:
>Several hundred MB of many files is too much to keep in physical
memory at one time. I'm thinking it may actually be faster to read
the files as needed counting on the OS's own file buffering to
reduce physical disk I/O.

--
Jim Langston
tazmas...@rock etmail.com

I really wonder how OS's own file buffering would improve performance
for the OP's scenario...
after all, it too would need to load the contents of the file into the
memory right? So i would imagine the same performance issues with OS's
mechanisms too...

may be the op's application can't read from a file but instead only
from the memory with the help of a pointer to the buffer...
I have faced some third party applications, which accepts data only in
a buffer and not in a file... i used mmap() in those case...
Because then the OS would only have to buffer it once. Not the OS buffering
it, and him buffering it as is the current case. It depends on how well the
OS buffers files.

--
Jim Langston
ta*******@rocke tmail.com
Jan 2 '08 #6
On Jan 2, 10:47 pm, "Jim Langston" <tazmas...@rock etmail.comwrote :
Rahul wrote:
Several hundred MB of many files is too much to keep in physical
memory at one time. I'm thinking it may actually be faster to read
the files as needed counting on the OS's own file buffering to
reduce physical disk I/O.
--
Jim Langston
tazmas...@rocke tmail.com
I really wonder how OS's own file buffering would improve performance
for the OP's scenario...
after all, it too would need to load the contents of the file into the
memory right? So i would imagine the same performance issues with OS's
mechanisms too...
may be the op's application can't read from a file but instead only
from the memory with the help of a pointer to the buffer...
I have faced some third party applications, which accepts data only in
a buffer and not in a file... i used mmap() in those case...

Because then the OS would only have to buffer it once. Not the OS buffering
it, and him buffering it as is the current case. It depends on how well the
OS buffers files.

--
Jim Langston
tazmas...@rocke tmail.com
Nothing to comment until the OP tells about his implementation. ..
But i'm sure mmap() is a single buffer operation just like file
operation...
Jan 2 '08 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

8
1665
by: Skwerl | last post by:
Hi guys. I've written code to embed an ICC profile in a TIFF image, and I think my IO operations are slowing things down. It is taking about a second to embed each tag in 7-meg TIFF files. Doesn't sound too bad until you try doing it to 500 files. Basically what I am doing is this: 1. Read header from image, add 12-byte tag to header,...
5
2968
by: Knackeback | last post by:
task: - read/parse CSV file code snippet: string key,line; typedef tokenizer<char_separator<char> > tokenizer; tokenizer tok(string(""), sep); while ( getline(f, line) ){ ++lineNo; tok.assign(line, sep);
0
1454
by: Maxim | last post by:
I am wondering if anyone could suggest come performance improvements for processing a very large file. THe processing taking place here is on 30-50MB chunks of the file. Performance is extremely important here. + I initialise a StringBuilder object with the result of the function System.Text.Encoding.GetString to convert the byte input...
1
1196
by: jm | last post by:
Those of you who use Visual Studio, do you find a big performance difference between the single file model and the web form / class behind form model? I am trying to justify purchasing Visual Studio ..NET and want the "extended features" (compiling the .net page) to be one of the reasons. Thanks for any links, guidance, and help.
3
1984
by: Darrel | last post by:
I've been working on a fairly simple CMS for a fairly small site. Each page is a DB query to grab the content for the page. I also have some ancillary content that isn't specifically associated with one specific page. For instance, the page footer, which is the same on every page. The authors can edit this content which is then stored in...
5
7842
by: Raj | last post by:
What is the purpose of file system caching while creating a tablespace? Memory on the test server gets used up pretty quickly after a user executes a complex query(database is already activated), after some investgation i found out that most of it being consumed by filesystem caching... thanks to Liam and Phil Sherman for their valuable...
26
1435
by: Bill | last post by:
Every now and then when I go on my form in the IDE the form goes blank and the computer grinds away for 10 or 20 seconds and then everything reaapears. What exactly is the computer doing and is there a way to prevent this from happening
1
1894
by: Dan R. | last post by:
I'm using the following to open and print values from a .dat file but I can't figure out how to sort the results. I think I need to use either the array_multisort() or krsort() functions. If so, how would I do that? <html> <body> <pre> <?php
6
5154
by: =?Utf-8?B?ZGF2aWQ=?= | last post by:
The same error messages full out the application event log. --- "The open procedure for service "ASP.NET" in DLL "C:\WINNT\Microsoft.NET\Framework\v2.0.50727\aspnet_perf.dll" failed. Performance data for this service will not be available. Status code returned is data DWORD 0. Data (words): 0000: 80004005 ---
7
26380
by: cesco | last post by:
Hi, I have a file containing four columns of data separated by tabs (\t) and I'd like to read a specific column from it (say the third). Is there any simple way to do this in Python? I've found quite interesting the linecache module but unfortunately that is (to my knowledge) only working on lines, not columns. Any suggestion?
0
7908
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main...
0
7836
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language...
0
8336
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
1
7950
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For...
0
6606
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then...
1
5710
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes...
0
5389
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert...
1
2343
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
1
1447
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.