By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
434,894 Members | 2,185 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 434,894 IT Pros & Developers. It's quick & easy.

Maximum Size of Byte Array

P: n/a
Hi all,

I'm getting an OutOfMemoryException when I initialize a byte array in
C# like this:

Byte[] test = new Byte[420000000];

I'm using ASP.NET 2.0. In ASP.Net 1.1 it works fine. So what am I doing
wrong?

What is the maximum length of a byte array?

Can somebody help me? Thanks in advance!

--
Gerrit Horeis

Software Developer
CI-Gate Development & Consulting GmbH
http://www.ci-gate.de
http://www.xira.de
http://www.bitbauer.de

Sep 26 '06 #1
Share this Question
Share on Google+
23 Replies


P: n/a
Gerrit <Ge***********@web.dewrote:
I'm getting an OutOfMemoryException when I initialize a byte array in
C# like this:

Byte[] test = new Byte[420000000];

I'm using ASP.NET 2.0. In ASP.Net 1.1 it works fine. So what am I doing
wrong?

What is the maximum length of a byte array?

Can somebody help me? Thanks in advance!
My guess is that ASP.NET 2.0 has some extra constraints in terms of how
much memory your app is allowed to use, to prevent wayward apps from
killing the machine. If I may say so, that's a fairly extreme use of
memory...

I can't say I know much about ASP.NET configuration - perhaps someone
in the ASP.NET group could help you there - but my gut feeling is
that's where the problem is.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Sep 26 '06 #2

P: n/a
Thank you very much!

The strange on this behavior is that it sometimes occurs and sometimes
not.
But I'll try my luck in a ASP.NET group.

Thank you so far!

--
Gerrit Horeis

Software Developer
CI-Gate Development & Consulting GmbH
http://www.ci-gate.de
http://www.xira.de
http://www.bitbauer.de

Jon schrieb:
Gerrit <Ge***********@web.dewrote:
I'm getting an OutOfMemoryException when I initialize a byte array in
C# like this:

Byte[] test = new Byte[420000000];

I'm using ASP.NET 2.0. In ASP.Net 1.1 it works fine. So what am I doing
wrong?

What is the maximum length of a byte array?

Can somebody help me? Thanks in advance!

My guess is that ASP.NET 2.0 has some extra constraints in terms of how
much memory your app is allowed to use, to prevent wayward apps from
killing the machine. If I may say so, that's a fairly extreme use of
memory...

I can't say I know much about ASP.NET configuration - perhaps someone
in the ASP.NET group could help you there - but my gut feeling is
that's where the problem is.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Sep 26 '06 #3

P: n/a
The strange on this behavior is that it sometimes occurs and sometimes not.
I guess it could also be some fragmentation of heaps / virtual address space.

=Get a 64-Bit system ;-) {but even 64-Bit .NET 2.0 has some 32-Bit limitations }

http://blogs.msdn.com/tess/archive/2...25/496898.aspx

http://msdn.microsoft.com/library/en...nitor_perf.asp
--
Thomas Scheidegger - MVP .NET - 'NETMaster'
http://www.cetus-links.org/oo_dotnet.html - http://dnetmaster.net/
Sep 26 '06 #4

P: n/a

"Gerrit" <Ge***********@web.dewrote in message
news:11**********************@h48g2000cwc.googlegr oups.com...
| Hi all,
|
| I'm getting an OutOfMemoryException when I initialize a byte array in
| C# like this:
|
| Byte[] test = new Byte[420000000];
|
| I'm using ASP.NET 2.0. In ASP.Net 1.1 it works fine. So what am I doing
| wrong?
|
| What is the maximum length of a byte array?
|
| Can somebody help me? Thanks in advance!
|
| --
| Gerrit Horeis
|
| Software Developer
| CI-Gate Development & Consulting GmbH
| http://www.ci-gate.de
| http://www.xira.de
| http://www.bitbauer.de
|

You should never assume that any portion of memory is available, even a
simple object allocation may fail. Now, as for your array of bytes,
theoretically the maximum size is ~2 billion bytes, but you will never be
able to allocate that size from a 32 bit process. Same goes for your 420
Mbytes array, allocating this from a 32 bit process will certainly lead to
OOM failures, as a result of heap fragmentation.
There is no guarantee that such an amount of contiguous memory space will be
available, certainly not when allocated in long running applications like
services (asp.net). For asp.net there is also a configuration parameter
which limits the total size of the process memory to a certain percentages
of the available virtual memory, while this is something to watch for, the
only solution to your problem, redesign your application in order to reduce
your memory consumption (this is a memory hog which doesn't scale at all),
or move to a 64 bit OS.

Willy.

Sep 26 '06 #5

P: n/a
Hi,
"Gerrit" <Ge***********@web.dewrote in message
news:11**********************@h48g2000cwc.googlegr oups.com...
Hi all,

I'm getting an OutOfMemoryException when I initialize a byte array in
C# like this:

Byte[] test = new Byte[420000000];

I'm using ASP.NET 2.0. In ASP.Net 1.1 it works fine. So what am I doing
wrong?

What is the maximum length of a byte array?

Why you need such a big array in the first place?

I bet it depends of either (or both) IIS and the framework

--
--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation
Sep 26 '06 #6

P: n/a

"Ignacio Machin ( .NET/ C# MVP )" <ignacio.machin AT dot.state.fl.uswrote
in message news:%2****************@TK2MSFTNGP03.phx.gbl...
| Hi,
|
|
| "Gerrit" <Ge***********@web.dewrote in message
| news:11**********************@h48g2000cwc.googlegr oups.com...
| Hi all,
| >
| I'm getting an OutOfMemoryException when I initialize a byte array in
| C# like this:
| >
| Byte[] test = new Byte[420000000];
| >
| I'm using ASP.NET 2.0. In ASP.Net 1.1 it works fine. So what am I doing
| wrong?
| >
| What is the maximum length of a byte array?
|
|
| Why you need such a big array in the first place?
|
| I bet it depends of either (or both) IIS and the framework
|
Hmm... IIS has nothing to do with this, ASP.NET runs in it's own process,
the framework only limit's the byte array size (just like any other object)
to ~2 GByte.
It's the way you allocate large contiguous blocks of memory from your
available virtual address space on a 32 bit OS which predicts the amount of
memory you can allocate. No single process, unless very carefully designed
and tested, will run for long before it encounters OOM exceptions when you
start allocating large objects or huge array's at wild. That's why asp.net
(the hosting process) provides the possibility to restart an AD and or the
worker process when a configurable memory threshold has been reached.

Willy.

Sep 26 '06 #7

P: n/a
Thanks for your answer!

I want to upload an image to a web application and therefore i need to
allocate this memory to fill the byte array with image data. Then I
need to do a resizing operation.

Ignacio Machin ( .NET/ C# MVP ) schrieb:
Hi,
"Gerrit" <Ge***********@web.dewrote in message
news:11**********************@h48g2000cwc.googlegr oups.com...
Hi all,

I'm getting an OutOfMemoryException when I initialize a byte array in
C# like this:

Byte[] test = new Byte[420000000];

I'm using ASP.NET 2.0. In ASP.Net 1.1 it works fine. So what am I doing
wrong?

What is the maximum length of a byte array?


Why you need such a big array in the first place?

I bet it depends of either (or both) IIS and the framework

--
--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation
Sep 26 '06 #8

P: n/a
I think I'll try your advice at home where I have a 64Bit system but
with WinXP 32Bit OS installed. Do I also need to install a 64Bit OS to
try that ? (would be a task for the weekend ;-))
Thomas Scheidegger [MVP] schrieb:
The strange on this behavior is that it sometimes occurs and sometimes not.


I guess it could also be some fragmentation of heaps / virtual address space.

=Get a 64-Bit system ;-) {but even 64-Bit .NET 2.0 has some 32-Bit limitations }

http://blogs.msdn.com/tess/archive/2...25/496898.aspx

http://msdn.microsoft.com/library/en...nitor_perf.asp
--
Thomas Scheidegger - MVP .NET - 'NETMaster'
http://www.cetus-links.org/oo_dotnet.html - http://dnetmaster.net/
Sep 26 '06 #9

P: n/a
>I think I'll try your advice at home where I have a 64Bit system but
with WinXP 32Bit OS installed. Do I also need to install a 64Bit OS to
try that ? (would be a task for the weekend ;-))

yes both, 64-Bit Windows (XP/2003/[Vista])
and the 64-Bit .NET Runtime.
For x64:
http://www.microsoft.com/downloads/d...B-40E78D788B00

And compile for 'AnyCpu' target (or x64).

And maybe switching ASP.NET to 64-Bit
http://support.microsoft.com/kb/894435/en-us


--
Thomas Scheidegger - MVP .NET - 'NETMaster'
http://www.cetus-links.org/oo_dotnet.html - http://dnetmaster.net/
Sep 26 '06 #10

P: n/a
and the 64-Bit .NET Runtime.
for array limitations
http://blogs.msdn.com/joshwil/archiv...10/450202.aspx

--
Thomas Scheidegger - MVP .NET - 'NETMaster'
http://www.cetus-links.org/oo_dotnet.html - http://dnetmaster.net/
Sep 26 '06 #11

P: n/a
Gerrit <Ge***********@web.dewrote:
I want to upload an image to a web application and therefore i need to
allocate this memory to fill the byte array with image data. Then I
need to do a resizing operation.
Is your image really 420MB large? That's really pretty massive. Can you
not do it in pieces? If not, perhaps you could launch a different
process to do the loading and resizing. That process could write the
result to disk, and then you could stream it to the web application.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Sep 26 '06 #12

P: n/a
Hi Jon,

yes this pictures are really up to 1GB. My method to shrink the images
works fine until a (decompressed) FileSize of 350 MB is reached.

If I do it in pieces, I also need to decompress the Picture on harddisk
as a bmp file and then read it into memory in small pieces. But to
decompress such a large picture with an .Net service or something i
will have to load it into memory anyway, which would cause a OOM
exception.

Do I have to use unmanaged code? Or is there a Windows API or anything
else I could use?

--
Gerrit Horeis

Software Developer
CI-Gate Development & Consulting GmbH
http://www.ci-gate.de
http://www.xira.de
http://www.bitbauer.de
Jon schrieb:
Gerrit <Ge***********@web.dewrote:
I want to upload an image to a web application and therefore i need to
allocate this memory to fill the byte array with image data. Then I
need to do a resizing operation.

Is your image really 420MB large? That's really pretty massive. Can you
not do it in pieces? If not, perhaps you could launch a different
process to do the loading and resizing. That process could write the
result to disk, and then you could stream it to the web application.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Sep 27 '06 #13

P: n/a
As been said before in this thread, you won't be able to handle such huge
arrays on a 32 bit platform, not using unmanaged code or anything else.
You can't rely on the fact that such a huge free chunk of (CONTIGUOUS)
memory will ever be available, and if it's available at some point in time,
chances are that it's no longer available some time later as a result of
heap fragmentation.
A 32 bit process has a virtual memory space of max. ~GAB (or ~GAB for /GET
RAM tuned system), this space is shared by things like code modules, data
files, private heaps, process heaps the thread stacks etc...
.. After process creation and loading the initial modules (DLL's) this VIM
space is already fragmented, such that only ~1.GAB is available in one large
chunk (this figure is for the asp.net worker process).
But there is something else what scares me, when you are talking about
images and bitmaps in asp.net, does it mean that you are using
System.Drawing classes? If that's the case you should keep in mind that this
namespace is not supported (for good reasons) in asp nor in any other
service based application.

Willy.
"Gerrit" <Ge***********@web.dewrote in message
news:11**********************@b28g2000cwb.googlegr oups.com...
| Hi Jon,
|
| yes this pictures are really up to 1GB. My method to shrink the images
| works fine until a (decompressed) FileSize of 350 MB is reached.
|
| If I do it in pieces, I also need to decompress the Picture on harddisk
| as a bmp file and then read it into memory in small pieces. But to
| decompress such a large picture with an .Net service or something i
| will have to load it into memory anyway, which would cause a OOM
| exception.
|
| Do I have to use unmanaged code? Or is there a Windows API or anything
| else I could use?
|
| --
| Gerrit Horeis
|
| Software Developer
| CI-Gate Development & Consulting GmbH
| http://www.ci-gate.de
| http://www.xira.de
| http://www.bitbauer.de
|
|
| Jon schrieb:
|
| Gerrit <Ge***********@web.dewrote:
| I want to upload an image to a web application and therefore i need to
| allocate this memory to fill the byte array with image data. Then I
| need to do a resizing operation.
| >
| Is your image really 420MB large? That's really pretty massive. Can you
| not do it in pieces? If not, perhaps you could launch a different
| process to do the loading and resizing. That process could write the
| result to disk, and then you could stream it to the web application.
| >
| --
| Jon Skeet - <sk***@pobox.com>
| http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
| If replying to the group, please do not mail me too
|
Sep 27 '06 #14

P: n/a
Hi,

You will not be able to handle such a big image in memory. You would have to
devise a way to use it in parts.
Also select carefully your tools, I don't think System.Drawing can handle
such a big image, you would be better using DirectX or even better image
manipulation code written in unmanaged code.
--
--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation

"Gerrit" <Ge***********@web.dewrote in message
news:11**********************@b28g2000cwb.googlegr oups.com...
Hi Jon,

yes this pictures are really up to 1GB. My method to shrink the images
works fine until a (decompressed) FileSize of 350 MB is reached.

If I do it in pieces, I also need to decompress the Picture on harddisk
as a bmp file and then read it into memory in small pieces. But to
decompress such a large picture with an .Net service or something i
will have to load it into memory anyway, which would cause a OOM
exception.

Do I have to use unmanaged code? Or is there a Windows API or anything
else I could use?

--
Gerrit Horeis

Software Developer
CI-Gate Development & Consulting GmbH
http://www.ci-gate.de
http://www.xira.de
http://www.bitbauer.de
Jon schrieb:
>Gerrit <Ge***********@web.dewrote:
I want to upload an image to a web application and therefore i need to
allocate this memory to fill the byte array with image data. Then I
need to do a resizing operation.

Is your image really 420MB large? That's really pretty massive. Can you
not do it in pieces? If not, perhaps you could launch a different
process to do the loading and resizing. That process could write the
result to disk, and then you could stream it to the web application.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

Sep 27 '06 #15

P: n/a
But I'm not able to handle a jpeg image (for example) in parts. If I
would do so, I would have load the whole image into memory to get parts
of it (because of compression)

I've decided to take a look on DirectX. Perhaps it has a method that
can handle jpeg images. If I use unmanaged code, I would have to write
the jpeg decompression algorithm on my own :-( This would be too hard
for me and too time intensive...

Thanks for all your help!

--
Gerrit Horeis

Software Developer
CI-Gate Development & Consulting GmbH
http://www.ci-gate.de
http://www.xira.de
http://www.bitbauer.de

Ignacio Machin ( .NET/ C# MVP ) schrieb:
Hi,

You will not be able to handle such a big image in memory. You would have to
devise a way to use it in parts.
Also select carefully your tools, I don't think System.Drawing can handle
such a big image, you would be better using DirectX or even better image
manipulation code written in unmanaged code.
--
--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation

"Gerrit" <Ge***********@web.dewrote in message
news:11**********************@b28g2000cwb.googlegr oups.com...
Hi Jon,

yes this pictures are really up to 1GB. My method to shrink the images
works fine until a (decompressed) FileSize of 350 MB is reached.

If I do it in pieces, I also need to decompress the Picture on harddisk
as a bmp file and then read it into memory in small pieces. But to
decompress such a large picture with an .Net service or something i
will have to load it into memory anyway, which would cause a OOM
exception.

Do I have to use unmanaged code? Or is there a Windows API or anything
else I could use?

--
Gerrit Horeis

Software Developer
CI-Gate Development & Consulting GmbH
http://www.ci-gate.de
http://www.xira.de
http://www.bitbauer.de
Jon schrieb:
Gerrit <Ge***********@web.dewrote:
I want to upload an image to a web application and therefore i need to
allocate this memory to fill the byte array with image data. Then I
need to do a resizing operation.

Is your image really 420MB large? That's really pretty massive. Can you
not do it in pieces? If not, perhaps you could launch a different
process to do the loading and resizing. That process could write the
result to disk, and then you could stream it to the web application.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Sep 28 '06 #16

P: n/a
Once again, you won't be able to handle such huge images (1GB) on Windows
32bit, whatever you try - managed code and GDI+, unmanaged code and GDI+ or
DirectX ... you name it - it will fail sooner than later, there is simply no
space in a process to accommodate such huge arrays, note I said arrays,
using System.Drawing(GDI+) you need memory space for the compressed raw
image (unmanaged heap), you need space in the managed heap for the image
data and you need space for the uncompressed image. Probably you want to
handle multiple images, so, even if it works for the first couple of images
(size dependent) if will certainly fail for one the following due to memory
fragmentation.

What I would suggest is that you write a very basic console application that
handles one image, run this in an unmanaged debugger like windbg.exe and
issue the !address -summary command. You will see that the largest free
region is ~1.1GB even before a single image is loaded.

Another point you seem to ignore, is the fact that this all doesn't work in
asp.net, you can't use GDI+ (System.Drawing) or DirectX in a web
application.
Willy.

"Gerrit" <Ge***********@web.dewrote in message
news:11**********************@h48g2000cwc.googlegr oups.com...
| But I'm not able to handle a jpeg image (for example) in parts. If I
| would do so, I would have load the whole image into memory to get parts
| of it (because of compression)
|
| I've decided to take a look on DirectX. Perhaps it has a method that
| can handle jpeg images. If I use unmanaged code, I would have to write
| the jpeg decompression algorithm on my own :-( This would be too hard
| for me and too time intensive...
|
| Thanks for all your help!
|
| --
| Gerrit Horeis
|
| Software Developer
| CI-Gate Development & Consulting GmbH
| http://www.ci-gate.de
| http://www.xira.de
| http://www.bitbauer.de
|
| Ignacio Machin ( .NET/ C# MVP ) schrieb:
|
| Hi,
| >
| You will not be able to handle such a big image in memory. You would
have to
| devise a way to use it in parts.
| Also select carefully your tools, I don't think System.Drawing can
handle
| such a big image, you would be better using DirectX or even better image
| manipulation code written in unmanaged code.
| >
| >
| --
| --
| Ignacio Machin,
| ignacio.machin AT dot.state.fl.us
| Florida Department Of Transportation
| >
| "Gerrit" <Ge***********@web.dewrote in message
| news:11**********************@b28g2000cwb.googlegr oups.com...
| Hi Jon,
|
| yes this pictures are really up to 1GB. My method to shrink the images
| works fine until a (decompressed) FileSize of 350 MB is reached.
|
| If I do it in pieces, I also need to decompress the Picture on
harddisk
| as a bmp file and then read it into memory in small pieces. But to
| decompress such a large picture with an .Net service or something i
| will have to load it into memory anyway, which would cause a OOM
| exception.
|
| Do I have to use unmanaged code? Or is there a Windows API or anything
| else I could use?
|
| --
| Gerrit Horeis
|
| Software Developer
| CI-Gate Development & Consulting GmbH
| http://www.ci-gate.de
| http://www.xira.de
| http://www.bitbauer.de
|
|
| Jon schrieb:
|
| Gerrit <Ge***********@web.dewrote:
| I want to upload an image to a web application and therefore i need
to
| allocate this memory to fill the byte array with image data. Then I
| need to do a resizing operation.
| >
| Is your image really 420MB large? That's really pretty massive. Can
you
| not do it in pieces? If not, perhaps you could launch a different
| process to do the loading and resizing. That process could write the
| result to disk, and then you could stream it to the web application.
| >
| --
| Jon Skeet - <sk***@pobox.com>
| http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
| If replying to the group, please do not mail me too
|
|
Sep 28 '06 #17

P: n/a
Hi,

It's been a while since I last saw the jpeg compression algorith but I'm
almost sure you can do it, it won't be efficient though. you would have to
discard parts of the image as you move forward (maybe saving chunk of it)

Again, 1 GB image is BIG, how many memory do you have to start with? And if
it's 1 GB in jpeg it would be mucho bigger in BMP !!!

And finally this is not task for a asp.net app at all.
What do you want to do with these images in the first place?
--
--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation
"Gerrit" <Ge***********@web.dewrote in message
news:11**********************@h48g2000cwc.googlegr oups.com...
But I'm not able to handle a jpeg image (for example) in parts. If I
would do so, I would have load the whole image into memory to get parts
of it (because of compression)

I've decided to take a look on DirectX. Perhaps it has a method that
can handle jpeg images. If I use unmanaged code, I would have to write
the jpeg decompression algorithm on my own :-( This would be too hard
for me and too time intensive...

Thanks for all your help!

--
Gerrit Horeis

Software Developer
CI-Gate Development & Consulting GmbH
http://www.ci-gate.de
http://www.xira.de
http://www.bitbauer.de

Ignacio Machin ( .NET/ C# MVP ) schrieb:
>Hi,

You will not be able to handle such a big image in memory. You would have
to
devise a way to use it in parts.
Also select carefully your tools, I don't think System.Drawing can handle
such a big image, you would be better using DirectX or even better image
manipulation code written in unmanaged code.
--
--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation

"Gerrit" <Ge***********@web.dewrote in message
news:11**********************@b28g2000cwb.googleg roups.com...
Hi Jon,

yes this pictures are really up to 1GB. My method to shrink the images
works fine until a (decompressed) FileSize of 350 MB is reached.

If I do it in pieces, I also need to decompress the Picture on harddisk
as a bmp file and then read it into memory in small pieces. But to
decompress such a large picture with an .Net service or something i
will have to load it into memory anyway, which would cause a OOM
exception.

Do I have to use unmanaged code? Or is there a Windows API or anything
else I could use?

--
Gerrit Horeis

Software Developer
CI-Gate Development & Consulting GmbH
http://www.ci-gate.de
http://www.xira.de
http://www.bitbauer.de
Jon schrieb:

Gerrit <Ge***********@web.dewrote:
I want to upload an image to a web application and therefore i need
to
allocate this memory to fill the byte array with image data. Then I
need to do a resizing operation.

Is your image really 420MB large? That's really pretty massive. Can
you
not do it in pieces? If not, perhaps you could launch a different
process to do the loading and resizing. That process could write the
result to disk, and then you could stream it to the web application.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

Sep 28 '06 #18

P: n/a
Ok I will use a service for the conversation in future, that's clear
now.
Its for an image management application. You can (or should be able to)
upload and download images up to 1GB in size (as spezified by the
customer). Supported filetypes should be every "usual" type you know
(therefore saving jpeg as bitmap is only a small part of all conversion
problems I'm running into. what is with png, Tiff, etc...they all can
be compressed)

Ok, Images 500MB are very uncommon even in my case, and when somebody
wants to upload such an image, only bitmaps and tiffs would be
allowed.(By the way: does anybody know a way to determine filetype and
size BEFORE uploading?)

I've found a C++ dll which I would like to give a try to. Its called
FreeImage
http://freeimage.sourceforge.net/

Another thing would be to setup a Windows XP 64 Bit OS this
weekend...perhaps this helps too...

--
Gerrit Horeis

Software Developer
CI-Gate Development & Consulting GmbH
http://www.ci-gate.de
http://www.xira.de
http://www.bitbauer.de
Ignacio Machin ( .NET/ C# MVP ) schrieb:
Hi,

It's been a while since I last saw the jpeg compression algorith but I'm
almost sure you can do it, it won't be efficient though. you would have to
discard parts of the image as you move forward (maybe saving chunk of it)

Again, 1 GB image is BIG, how many memory do you have to start with? And if
it's 1 GB in jpeg it would be mucho bigger in BMP !!!

And finally this is not task for a asp.net app at all.
What do you want to do with these images in the first place?
--
--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation
"Gerrit" <Ge***********@web.dewrote in message
news:11**********************@h48g2000cwc.googlegr oups.com...
But I'm not able to handle a jpeg image (for example) in parts. If I
would do so, I would have load the whole image into memory to get parts
of it (because of compression)

I've decided to take a look on DirectX. Perhaps it has a method that
can handle jpeg images. If I use unmanaged code, I would have to write
the jpeg decompression algorithm on my own :-( This would be too hard
for me and too time intensive...

Thanks for all your help!

--
Gerrit Horeis

Software Developer
CI-Gate Development & Consulting GmbH
http://www.ci-gate.de
http://www.xira.de
http://www.bitbauer.de

Ignacio Machin ( .NET/ C# MVP ) schrieb:
Hi,

You will not be able to handle such a big image in memory. You would have
to
devise a way to use it in parts.
Also select carefully your tools, I don't think System.Drawing can handle
such a big image, you would be better using DirectX or even better image
manipulation code written in unmanaged code.
--
--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation

"Gerrit" <Ge***********@web.dewrote in message
news:11**********************@b28g2000cwb.googlegr oups.com...
Hi Jon,

yes this pictures are really up to 1GB. My method to shrink the images
works fine until a (decompressed) FileSize of 350 MB is reached.

If I do it in pieces, I also need to decompress the Picture on harddisk
as a bmp file and then read it into memory in small pieces. But to
decompress such a large picture with an .Net service or something i
will have to load it into memory anyway, which would cause a OOM
exception.

Do I have to use unmanaged code? Or is there a Windows API or anything
else I could use?

--
Gerrit Horeis

Software Developer
CI-Gate Development & Consulting GmbH
http://www.ci-gate.de
http://www.xira.de
http://www.bitbauer.de
Jon schrieb:

Gerrit <Ge***********@web.dewrote:
I want to upload an image to a web application and therefore i need
to
allocate this memory to fill the byte array with image data. Then I
need to do a resizing operation.

Is your image really 420MB large? That's really pretty massive. Can
you
not do it in pieces? If not, perhaps you could launch a different
process to do the loading and resizing. That process could write the
result to disk, and then you could stream it to the web application.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Sep 29 '06 #19

P: n/a

Setting Up a Windows XP 64 works really better with large images! I was
able to load a a 400 MB tif File into memory (on my 1GB RAM, 2800+ AMD
PC).

Gerrit schrieb:
Ok I will use a service for the conversation in future, that's clear
now.
Its for an image management application. You can (or should be able to)
upload and download images up to 1GB in size (as spezified by the
customer). Supported filetypes should be every "usual" type you know
(therefore saving jpeg as bitmap is only a small part of all conversion
problems I'm running into. what is with png, Tiff, etc...they all can
be compressed)

Ok, Images 500MB are very uncommon even in my case, and when somebody
wants to upload such an image, only bitmaps and tiffs would be
allowed.(By the way: does anybody know a way to determine filetype and
size BEFORE uploading?)

I've found a C++ dll which I would like to give a try to. Its called
FreeImage
http://freeimage.sourceforge.net/

Another thing would be to setup a Windows XP 64 Bit OS this
weekend...perhaps this helps too...

--
Gerrit Horeis

Software Developer
CI-Gate Development & Consulting GmbH
http://www.ci-gate.de
http://www.xira.de
http://www.bitbauer.de
Ignacio Machin ( .NET/ C# MVP ) schrieb:
Hi,

It's been a while since I last saw the jpeg compression algorith but I'm
almost sure you can do it, it won't be efficient though. you would have to
discard parts of the image as you move forward (maybe saving chunk of it)

Again, 1 GB image is BIG, how many memory do you have to start with? And if
it's 1 GB in jpeg it would be mucho bigger in BMP !!!

And finally this is not task for a asp.net app at all.
What do you want to do with these images in the first place?
--
--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation
"Gerrit" <Ge***********@web.dewrote in message
news:11**********************@h48g2000cwc.googlegr oups.com...
But I'm not able to handle a jpeg image (for example) in parts. If I
would do so, I would have load the whole image into memory to get parts
of it (because of compression)
>
I've decided to take a look on DirectX. Perhaps it has a method that
can handle jpeg images. If I use unmanaged code, I would have to write
the jpeg decompression algorithm on my own :-( This would be too hard
for me and too time intensive...
>
Thanks for all your help!
>
--
Gerrit Horeis
>
Software Developer
CI-Gate Development & Consulting GmbH
http://www.ci-gate.de
http://www.xira.de
http://www.bitbauer.de
>
Ignacio Machin ( .NET/ C# MVP ) schrieb:
>
>Hi,
>>
>You will not be able to handle such a big image in memory. You would have
>to
>devise a way to use it in parts.
>Also select carefully your tools, I don't think System.Drawing can handle
>such a big image, you would be better using DirectX or even better image
>manipulation code written in unmanaged code.
>>
>>
>--
>--
>Ignacio Machin,
>ignacio.machin AT dot.state.fl.us
>Florida Department Of Transportation
>>
>"Gerrit" <Ge***********@web.dewrote in message
>news:11**********************@b28g2000cwb.googleg roups.com...
Hi Jon,
>
yes this pictures are really up to 1GB. My method to shrink the images
works fine until a (decompressed) FileSize of 350 MB is reached.
>
If I do it in pieces, I also need to decompress the Picture on harddisk
as a bmp file and then read it into memory in small pieces. But to
decompress such a large picture with an .Net service or something i
will have to load it into memory anyway, which would cause a OOM
exception.
>
Do I have to use unmanaged code? Or is there a Windows API or anything
else I could use?
>
--
Gerrit Horeis
>
Software Developer
CI-Gate Development & Consulting GmbH
http://www.ci-gate.de
http://www.xira.de
http://www.bitbauer.de
>
>
Jon schrieb:
>
>Gerrit <Ge***********@web.dewrote:
I want to upload an image to a web application and therefore i need
to
allocate this memory to fill the byte array with image data. Then I
need to do a resizing operation.
>>
>Is your image really 420MB large? That's really pretty massive. Can
>you
>not do it in pieces? If not, perhaps you could launch a different
>process to do the loading and resizing. That process could write the
>result to disk, and then you could stream it to the web application.
>>
>--
>Jon Skeet - <sk***@pobox.com>
>http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
>If replying to the group, please do not mail me too
>
>
Oct 13 '06 #20

P: n/a

"Gerrit" <Ge***********@web.dewrote in message
news:11**********************@f16g2000cwb.googlegr oups.com...
|
| Setting Up a Windows XP 64 works really better with large images! I was
| able to load a a 400 MB tif File into memory (on my 1GB RAM, 2800+ AMD
| PC).
|

Really better?? running X64 on a 1GB system is not a real solution either,
sure you have a larger virtual address space, but you don't have the
necessary RAM to accomodate the memory you are allocating, that means that
you will suffer from extreme paging. Moving to 64 bit systems makes little
sense unless you have at least 4GB of RAM.
Willy.

| Gerrit schrieb:
|
| Ok I will use a service for the conversation in future, that's clear
| now.
| >
| >
| Its for an image management application. You can (or should be able to)
| upload and download images up to 1GB in size (as spezified by the
| customer). Supported filetypes should be every "usual" type you know
| (therefore saving jpeg as bitmap is only a small part of all conversion
| problems I'm running into. what is with png, Tiff, etc...they all can
| be compressed)
| >
| Ok, Images 500MB are very uncommon even in my case, and when somebody
| wants to upload such an image, only bitmaps and tiffs would be
| allowed.(By the way: does anybody know a way to determine filetype and
| size BEFORE uploading?)
| >
| I've found a C++ dll which I would like to give a try to. Its called
| FreeImage
| http://freeimage.sourceforge.net/
| >
| Another thing would be to setup a Windows XP 64 Bit OS this
| weekend...perhaps this helps too...
| >
| --
| Gerrit Horeis
| >
| Software Developer
| CI-Gate Development & Consulting GmbH
| http://www.ci-gate.de
| http://www.xira.de
| http://www.bitbauer.de
| >
| >
| Ignacio Machin ( .NET/ C# MVP ) schrieb:
| >
| Hi,
|
| It's been a while since I last saw the jpeg compression algorith but
I'm
| almost sure you can do it, it won't be efficient though. you would
have to
| discard parts of the image as you move forward (maybe saving chunk of
it)
|
| Again, 1 GB image is BIG, how many memory do you have to start with?
And if
| it's 1 GB in jpeg it would be mucho bigger in BMP !!!
|
| And finally this is not task for a asp.net app at all.
|
|
| What do you want to do with these images in the first place?
|
|
| --
| --
| Ignacio Machin,
| ignacio.machin AT dot.state.fl.us
| Florida Department Of Transportation
|
|
| "Gerrit" <Ge***********@web.dewrote in message
| news:11**********************@h48g2000cwc.googlegr oups.com...
| But I'm not able to handle a jpeg image (for example) in parts. If I
| would do so, I would have load the whole image into memory to get
parts
| of it (because of compression)
| >
| I've decided to take a look on DirectX. Perhaps it has a method that
| can handle jpeg images. If I use unmanaged code, I would have to
write
| the jpeg decompression algorithm on my own :-( This would be too
hard
| for me and too time intensive...
| >
| Thanks for all your help!
| >
| --
| Gerrit Horeis
| >
| Software Developer
| CI-Gate Development & Consulting GmbH
| http://www.ci-gate.de
| http://www.xira.de
| http://www.bitbauer.de
| >
| Ignacio Machin ( .NET/ C# MVP ) schrieb:
| >
| >Hi,
| >>
| >You will not be able to handle such a big image in memory. You
would have
| >to
| >devise a way to use it in parts.
| >Also select carefully your tools, I don't think System.Drawing can
handle
| >such a big image, you would be better using DirectX or even better
image
| >manipulation code written in unmanaged code.
| >>
| >>
| >--
| >--
| >Ignacio Machin,
| >ignacio.machin AT dot.state.fl.us
| >Florida Department Of Transportation
| >>
| >"Gerrit" <Ge***********@web.dewrote in message
| >news:11**********************@b28g2000cwb.googleg roups.com...
| Hi Jon,
| >
| yes this pictures are really up to 1GB. My method to shrink the
images
| works fine until a (decompressed) FileSize of 350 MB is reached.
| >
| If I do it in pieces, I also need to decompress the Picture on
harddisk
| as a bmp file and then read it into memory in small pieces. But
to
| decompress such a large picture with an .Net service or something
i
| will have to load it into memory anyway, which would cause a OOM
| exception.
| >
| Do I have to use unmanaged code? Or is there a Windows API or
anything
| else I could use?
| >
| --
| Gerrit Horeis
| >
| Software Developer
| CI-Gate Development & Consulting GmbH
| http://www.ci-gate.de
| http://www.xira.de
| http://www.bitbauer.de
| >
| >
| Jon schrieb:
| >
| >Gerrit <Ge***********@web.dewrote:
| I want to upload an image to a web application and therefore i
need
| to
| allocate this memory to fill the byte array with image data.
Then I
| need to do a resizing operation.
| >>
| >Is your image really 420MB large? That's really pretty massive.
Can
| >you
| >not do it in pieces? If not, perhaps you could launch a
different
| >process to do the loading and resizing. That process could write
the
| >result to disk, and then you could stream it to the web
application.
| >>
| >--
| >Jon Skeet - <sk***@pobox.com>
| >http://www.pobox.com/~skeet Blog:
http://www.msmvps.com/jon.skeet
| >If replying to the group, please do not mail me too
| >
| >
|
Oct 13 '06 #21

P: n/a
Willy Denoyette [MVP] wrote:
Really better?? running X64 on a 1GB system is not a real solution either,
sure you have a larger virtual address space, but you don't have the
necessary RAM to accomodate the memory you are allocating, that means that
you will suffer from extreme paging. Moving to 64 bit systems makes little
sense unless you have at least 4GB of RAM.
Sometimes just being able to do it is enough even though
performance is not good.

In some special cases it is possible to use the 64 bit
memory with much less RAM efficiently.

Arne
Oct 17 '06 #22

P: n/a
"Arne Vajhøj" <ar**@vajhoej.dkwrote in message
news:oyWYg.22302$2g4.20089@dukeread09...
Willy Denoyette [MVP] wrote:
>Really better?? running X64 on a 1GB system is not a real solution
either, sure you have a larger virtual address space, but you don't have
the necessary RAM to accomodate the memory you are allocating, that means
that you will suffer from extreme paging. Moving to 64 bit systems makes
little sense unless you have at least 4GB of RAM.

Sometimes just being able to do it is enough even though
performance is not good.

In some special cases it is possible to use the 64 bit
memory with much less RAM efficiently.
Agreed!

One classic example of this (ironically, since there's another off-topic
long thread here about this very thing) is memory mapped files. The file
system supports files larger than the 32-bit OS can map at once. The 64-bit
OS can simply map a whole file into the virtual address space, even if
larger than what the 32-bit OS can map. Physical RAM usage can be quite
low.

Likewise a very large array. Yes, page file usage will be quite large, but
then one ought to expect that on a 64-bit OS (and have enough hard disk
space to support it). As long as usage of the array is "friendly" (doesn't
try to access the whole thing all at once over and over again), swapping
need not be excessive, even if physical RAM is only 1GB.

What causes swapping is access of a large amount of data over a short period
of time. This can happen with or without 64-bit addressing, and 64-bit
addressing can happen with or without this kind of access.

Using 32-bit addressing made plenty of sense even when we only had 32MB of
RAM...I see no reason 64-bit addressing can't be useful when one has only
1GB of RAM.

Pete
Oct 17 '06 #23

P: n/a

"Arne Vajhøj" <ar**@vajhoej.dkwrote in message
news:oyWYg.22302$2g4.20089@dukeread09...
| Willy Denoyette [MVP] wrote:
| Really better?? running X64 on a 1GB system is not a real solution
either,
| sure you have a larger virtual address space, but you don't have the
| necessary RAM to accomodate the memory you are allocating, that means
that
| you will suffer from extreme paging. Moving to 64 bit systems makes
little
| sense unless you have at least 4GB of RAM.
|
| Sometimes just being able to do it is enough even though
| performance is not good.
|
| In some special cases it is possible to use the 64 bit
| memory with much less RAM efficiently.
|
In some cases some fair amount of paging can be acceptable, but in this
particular case I'm pretty sure it's not. The OP's initial question was
about loading up to 1GB large images in an ASP.NET application, and loading
an image really means loading the data into the managed heap, so you will
need this amount of physical memory anyway. I just tried to load a huge gif
file on a X64 bit system (not running asp.net) for which I reduced the
available RAM to 1GB, and the system starts swapping when loading a gif file
of ~400MB, loading a gif file of 800MB starts thrashing and the system
becomes almost unusable.
Also, don't forget that the memory footprint is somewhat larger for a 64 bit
OS/application than a 32 bit OS/application, this means that less RAM is
available to begin with.

Willy.


Oct 17 '06 #24

This discussion thread is closed

Replies have been disabled for this discussion.