473,320 Members | 1,845 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

large array is not garbage collected

I am reading a 160 mB file in my application. I want to access the file as
shorts. FileStream->Read wants to access the file as Byte so I do the
following

1) Read the filesize
2) Create a Byte[] array of the appropriate size
3) Read the file
4) Create a (global) nogc array of the same size and copy the data from the
Byte array to the gc array
5) Delete the Byte[] array

as follows:

FileStream *myFile = new FileStream(pPath, FileMode::Open,
FileAccess::Read);
__int64 myFileSize = myFile->get_Length();
Byte[] myFileData = new Byte[(int) myFileSize];
myFile->Read(myFileData, 0, (int) myFileSize);
int x = myFileData->Length;
pFileData = new unsigned char [(int) myFileSize];
for (int ii = 0; ii < myFileSize; ii++)
pFileData[ii] = myFileData[ii];
delete myFileData;
myFile->Close();

The code for delete myFileData sets the pointer to 0. Presumeably, this
will cause the garbage collector to reclaim the memory. Using the task
manager, I observe that the Mem Usage for my process increases by twice the
file size (over 300 mB) and never decreases (e.g. the Byte array is never
garbage collected). I have tried adding a GC::Collect() call after the
delete myFileData but nothing happens.

Any suggestions?

Howard Weiss
Nov 17 '05 #1
6 1537
Correction: The 4th line is

Byte myFileData[] = new Byte[(int)myFileSize];

"Howard Weiss" <hp*****@comcast.net> wrote in message
news:uJ*************@TK2MSFTNGP11.phx.gbl...
I am reading a 160 mB file in my application. I want to access the file as
shorts. FileStream->Read wants to access the file as Byte so I do the
following

1) Read the filesize
2) Create a Byte[] array of the appropriate size
3) Read the file
4) Create a (global) nogc array of the same size and copy the data from the Byte array to the gc array
5) Delete the Byte[] array

as follows:

FileStream *myFile = new FileStream(pPath, FileMode::Open,
FileAccess::Read);
__int64 myFileSize = myFile->get_Length();
Byte[] myFileData = new Byte[(int) myFileSize];
myFile->Read(myFileData, 0, (int) myFileSize);
int x = myFileData->Length;
pFileData = new unsigned char [(int) myFileSize];
for (int ii = 0; ii < myFileSize; ii++)
pFileData[ii] = myFileData[ii];
delete myFileData;
myFile->Close();

The code for delete myFileData sets the pointer to 0. Presumeably, this
will cause the garbage collector to reclaim the memory. Using the task
manager, I observe that the Mem Usage for my process increases by twice the file size (over 300 mB) and never decreases (e.g. the Byte array is never
garbage collected). I have tried adding a GC::Collect() call after the
delete myFileData but nothing happens.

Any suggestions?

Howard Weiss

Nov 17 '05 #2

Howard,

I'm no expert but it looks to me like you have two arrays of size
myFileSize and I was under the
impression that the CLR did not bother releasing memory back to the OS
unless it was explicitly
required.

I'm sorry I haven't answered your question but seeing the 160Mb file
prompted me to ask why you are not mapping a view of the file instead of
loading it into arrays?

Ross.

"Howard Weiss" wrote
I am reading a 160 mB file in my application. I want to access the file as shorts. FileStream->Read wants to access the file as Byte so I do the
following

1) Read the filesize
2) Create a Byte[] array of the appropriate size
3) Read the file
4) Create a (global) nogc array of the same size and copy the data from

the
Byte array to the gc array
5) Delete the Byte[] array

as follows:

FileStream *myFile = new FileStream(pPath, FileMode::Open,
FileAccess::Read);
__int64 myFileSize = myFile->get_Length();
Byte[] myFileData = new Byte[(int) myFileSize];
myFile->Read(myFileData, 0, (int) myFileSize);
int x = myFileData->Length;
pFileData = new unsigned char [(int) myFileSize];
for (int ii = 0; ii < myFileSize; ii++)
pFileData[ii] = myFileData[ii];
delete myFileData;
myFile->Close();

The code for delete myFileData sets the pointer to 0. Presumeably, this
will cause the garbage collector to reclaim the memory. Using the task
manager, I observe that the Mem Usage for my process increases by twice

the
file size (over 300 mB) and never decreases (e.g. the Byte array is never garbage collected). I have tried adding a GC::Collect() call after the
delete myFileData but nothing happens.

Any suggestions?

Howard Weiss


Nov 17 '05 #3
Ross -

I agree that it is somewhat short sighted to have 2 full copies of the array
in memory at the same time. Nonetheless, I would have expected to see my
memory usage increase by twice the file size and then collapse back to 1x
the file size.

Please point me to an appropriate class to map a view of the data

Thanks

Howard Weiss

"Ross Jones" <ro**@community.nospam> wrote in message
news:%2****************@TK2MSFTNGP10.phx.gbl...

Howard,

I'm no expert but it looks to me like you have two arrays of size
myFileSize and I was under the
impression that the CLR did not bother releasing memory back to the OS
unless it was explicitly
required.

I'm sorry I haven't answered your question but seeing the 160Mb file
prompted me to ask why you are not mapping a view of the file instead of
loading it into arrays?

Ross.

"Howard Weiss" wrote
I am reading a 160 mB file in my application. I want to access the file
as
shorts. FileStream->Read wants to access the file as Byte so I do the
following

1) Read the filesize
2) Create a Byte[] array of the appropriate size
3) Read the file
4) Create a (global) nogc array of the same size and copy the data
from
the
Byte array to the gc array
5) Delete the Byte[] array

as follows:

FileStream *myFile = new FileStream(pPath, FileMode::Open,
FileAccess::Read);
__int64 myFileSize = myFile->get_Length();
Byte[] myFileData = new Byte[(int) myFileSize];
myFile->Read(myFileData, 0, (int) myFileSize);
int x = myFileData->Length;
pFileData = new unsigned char [(int) myFileSize];
for (int ii = 0; ii < myFileSize; ii++)
pFileData[ii] = myFileData[ii];
delete myFileData;
myFile->Close();

The code for delete myFileData sets the pointer to 0. Presumeably,
this will cause the garbage collector to reclaim the memory. Using the task manager, I observe that the Mem Usage for my process increases by twice the
file size (over 300 mB) and never decreases (e.g. the Byte array is

never garbage collected). I have tried adding a GC::Collect() call after

the delete myFileData but nothing happens.

Any suggestions?

Howard Weiss



Nov 17 '05 #4
Ah. Um. I don't think there is a managed class provided by Microsoft to do
this (although I could be wrong).
It might be best to wrap some P/Invoke calls in a wrapper class and then use
the Win32 MapViewOfFile to access the file/memory in chunks.

http://msdn.microsoft.com/library/de...viewoffile.asp

I agree with the expected behaviour you saw, and I was disappointed when I
saw C# doesn't do it either. But better minds than mine tell me that it's
better that way :)

Ross.

"Howard Weiss" <hp*****@comcast.net> wrote in message
news:uN**************@TK2MSFTNGP11.phx.gbl...
Ross -

I agree that it is somewhat short sighted to have 2 full copies of the array in memory at the same time. Nonetheless, I would have expected to see my
memory usage increase by twice the file size and then collapse back to 1x
the file size.

Please point me to an appropriate class to map a view of the data

Thanks

Howard Weiss

"Ross Jones" <ro**@community.nospam> wrote in message
news:%2****************@TK2MSFTNGP10.phx.gbl...

Howard,

I'm no expert but it looks to me like you have two arrays of size
myFileSize and I was under the
impression that the CLR did not bother releasing memory back to the OS
unless it was explicitly
required.

I'm sorry I haven't answered your question but seeing the 160Mb file
prompted me to ask why you are not mapping a view of the file instead of
loading it into arrays?

Ross.

"Howard Weiss" wrote
> I am reading a 160 mB file in my application. I want to access the file
as
> shorts. FileStream->Read wants to access the file as Byte so I do the > following
>
> 1) Read the filesize
> 2) Create a Byte[] array of the appropriate size
> 3) Read the file
> 4) Create a (global) nogc array of the same size and copy the data

from the
> Byte array to the gc array
> 5) Delete the Byte[] array
>
> as follows:
>
> FileStream *myFile = new FileStream(pPath, FileMode::Open,
> FileAccess::Read);
> __int64 myFileSize = myFile->get_Length();
> Byte[] myFileData = new Byte[(int) myFileSize];
> myFile->Read(myFileData, 0, (int) myFileSize);
> int x = myFileData->Length;
> pFileData = new unsigned char [(int) myFileSize];
> for (int ii = 0; ii < myFileSize; ii++)
> pFileData[ii] = myFileData[ii];
> delete myFileData;
> myFile->Close();
>
> The code for delete myFileData sets the pointer to 0. Presumeably, this > will cause the garbage collector to reclaim the memory. Using the task > manager, I observe that the Mem Usage for my process increases by twice the
> file size (over 300 mB) and never decreases (e.g. the Byte array is

never
> garbage collected). I have tried adding a GC::Collect() call after the > delete myFileData but nothing happens.
>
> Any suggestions?
>
> Howard Weiss
>
>



Nov 17 '05 #5
> Any suggestions?
make sure you use the release build config

ben
Nov 17 '05 #6
Hello,
Byte myFileData[] = new Byte[(int)myFileSize];


Did you mean:

Byte[] myFileData = __gc new Byte[(int) myFileSize];

--
Vladimir Nesterovsky
e-mail: vl******@nesterovsky-bros.com
home: www.nesterovsky-bros.com
Nov 17 '05 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
by: Scott Robinson | last post by:
I have been having trouble with the garbage collector and sockets. Unfortunately, google keeps telling me that the problem is the garbage collector ignoring dead (closed?) sockets instead of...
3
by: WinstonSmith | last post by:
Hello everyone, I got a problem about GC when creating large fields (some MB), set reference to null and call GC.Collect. Not all virtual mem is released. Situation improved in .net 1.1 but not...
3
by: Bryan | last post by:
I've been messing around with a C++ application on Xbox, and have been encountering problems with my objects getting garbage collected when they go out of scope, but before I'm actually done using...
2
by: C P | last post by:
I'm coming from Delphi where I have to explicitly create and destroy instances of objects. I've been working through a C#/ASP.NET book, and many of the examples repeat the same SqlConnection,...
2
by: Sarfraz Hooda | last post by:
Hi, I have created an array of Objects in a collection. I was wondering is there a way to destroy the array to free up the space in the memory ? or they are automatically destroyed and garbagge...
5
by: Razzie | last post by:
Hi all, A question from someone on a website got me thinking about this, and I wondered if anyone could explain this. A System.Threading.Timer object is garbage collected if it has no...
6
by: massimo s. | last post by:
Hi, Python 2.4, Kubuntu 6.06. I'm no professional programmer (I am a ph.d. student in biophysics) but I have a fair knowledge of Python. I have a for loop that looks like the following : ...
56
by: Johnny E. Jensen | last post by:
Hellow I'am not sure what to think about the Garbage Collector. I have a Class OutlookObject, It have two private variables. Private Microsoft.Office.Interop.Outlook.Application _Application =...
5
by: Jon | last post by:
If I allocate an array, eg: double x = new double; then use the array, then at some later point, allocate it again, eg: double x = new double; does the garbage collector know that the...
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.