ReadElementContentAsXXX

[B]

I am using the XMLReader to read an XML document with a large image embedded
inside of it. By large, I mean +150MB. When I use the
ReadElementContentAsBase64 method to read the element content, the page file
usage peaks and eventually the system throws a System.OutOfMemoryException.
Does anybody have a suggestion on how to remedy this type of problem?

Thanks,
B

[Zafar Abbas]

Thanks for reporting the issue.
Could you share the XML data and code to reproduce this issue?

-Zafar Abbas

"B" <[e-mail] in message
news:92**********************************@microsof t.com...

>I am using the XMLReader to read an XML document with a large image
embedded
inside of it. By large, I mean +150MB. When I use the
ReadElementContentAsBase64 method to read the element content, the page
file
usage peaks and eventually the system throws a
System.OutOfMemoryException.
Does anybody have a suggestion on how to remedy this type of problem?

Thanks,
B

[B]

Thanks for responding.

The code below is using the ReadElementContentAsString method and converts
it to a byte.

---- CODE ----
Dim strBinaryContent As String = inDocumentReader.ReadElementContentAsString()
DocumentBinaryContent = Convert.FromBase64String(strBinaryContent)
---- CODE -----

The XML data is just an image placed in the file as binary.

Using the ReadElementContentAsBase64 rendered the same outcome. Either way,
the Page File jumps to around 1.6GB of usage.

"Zafar Abbas" wrote:

Thanks for reporting the issue.
Could you share the XML data and code to reproduce this issue?

-Zafar Abbas

"B" <[e-mail] in message
news:92**********************************@microsof t.com...

I am using the XMLReader to read an XML document with a large image
embedded
inside of it. By large, I mean +150MB. When I use the
ReadElementContentAsBase64 method to read the element content, the page
file
usage peaks and eventually the system throws a
System.OutOfMemoryException.
Does anybody have a suggestion on how to remedy this type of problem?

Thanks,
B

[Zafar Abbas]

You are reading the data as a string which is definitely the cause of this.
We need to allocate that string in UTF-16, so you have 300MB+ just the
string, and in order to create the string, we need to parse the whole value
and store it which uses up a lot more memory and it is understandable how it
is causing out of memory exceptions.

If you are creating your reader from the XmlReader.Create method (which it
looks like you are) then you could then read the Base64 value using the
method:

ReadElementContentAsBase64

Thanks.

"B" <[e-mail] in message
news:EB**********************************@microsof t.com...

Thanks for responding.

The code below is using the ReadElementContentAsString method and converts
it to a byte.

---- CODE ----
Dim strBinaryContent As String =
inDocumentReader.ReadElementContentAsString()
DocumentBinaryContent = Convert.FromBase64String(strBinaryContent)
---- CODE -----

The XML data is just an image placed in the file as binary.

Using the ReadElementContentAsBase64 rendered the same outcome. Either
way,
the Page File jumps to around 1.6GB of usage.

"Zafar Abbas" wrote:

>Thanks for reporting the issue.
Could you share the XML data and code to reproduce this issue?

-Zafar Abbas

"B" <[e-mail] in message
news:92**********************************@microso ft.com...

>I am using the XMLReader to read an XML document with a large image
embedded
inside of it. By large, I mean +150MB. When I use the
ReadElementContentAsBase64 method to read the element content, the page
file
usage peaks and eventually the system throws a
System.OutOfMemoryException.
Does anybody have a suggestion on how to remedy this type of problem?

Thanks,
B

[B]

I've modified the code to reflect your recommendations. Here's what i have

Dim b(_fileLength.length) as byte
xmlReader.ReadElementContentAsBase64(b, 0, b.length)

Unfortunately, the ReadElementContentAs method is still drawing a lot of
memory.

"Zafar Abbas" wrote:

You are reading the data as a string which is definitely the cause of this.
We need to allocate that string in UTF-16, so you have 300MB+ just the
string, and in order to create the string, we need to parse the whole value
and store it which uses up a lot more memory and it is understandable how it
is causing out of memory exceptions.

If you are creating your reader from the XmlReader.Create method (which it
looks like you are) then you could then read the Base64 value using the
method:

ReadElementContentAsBase64

Thanks.

"B" <[e-mail] in message
news:EB**********************************@microsof t.com...

Thanks for responding.

The code below is using the ReadElementContentAsString method and converts
it to a byte.

---- CODE ----
Dim strBinaryContent As String =
inDocumentReader.ReadElementContentAsString()
DocumentBinaryContent = Convert.FromBase64String(strBinaryContent)
---- CODE -----

The XML data is just an image placed in the file as binary.

Using the ReadElementContentAsBase64 rendered the same outcome. Either
way,
the Page File jumps to around 1.6GB of usage.

"Zafar Abbas" wrote:

Thanks for reporting the issue.
Could you share the XML data and code to reproduce this issue?

-Zafar Abbas

"B" <[e-mail] in message
news:92**********************************@microsof t.com...
I am using the XMLReader to read an XML document with a large image
embedded
inside of it. By large, I mean +150MB. When I use the
ReadElementContentAsBase64 method to read the element content, the page
file
usage peaks and eventually the system throws a
System.OutOfMemoryException.
Does anybody have a suggestion on how to remedy this type of problem?

Thanks,
B

[B]

Same result with code below

Dim b(CInt(inFile.Length)) As Byte 'file length equivalent to 150mb file
Dim cnt As Integer

Do
cnt = inDocumentReader.ReadElementContentAsBase64(b, 0, 4096)
Loop Until cnt = 0
"B" wrote:

I've modified the code to reflect your recommendations. Here's what i have

Dim b(_fileLength.length) as byte
xmlReader.ReadElementContentAsBase64(b, 0, b.length)

Unfortunately, the ReadElementContentAs method is still drawing a lot of
memory.

"Zafar Abbas" wrote:

You are reading the data as a string which is definitely the cause of this.
We need to allocate that string in UTF-16, so you have 300MB+ just the
string, and in order to create the string, we need to parse the whole value
and store it which uses up a lot more memory and it is understandable how it
is causing out of memory exceptions.

If you are creating your reader from the XmlReader.Create method (which it
looks like you are) then you could then read the Base64 value using the
method:

ReadElementContentAsBase64

Thanks.

"B" <[e-mail] in message
news:EB**********************************@microsof t.com...

Thanks for responding.
>
The code below is using the ReadElementContentAsString method and converts
it to a byte.
>
---- CODE ----
Dim strBinaryContent As String =
inDocumentReader.ReadElementContentAsString()
DocumentBinaryContent = Convert.FromBase64String(strBinaryContent)
---- CODE -----
>
The XML data is just an image placed in the file as binary.
>
Using the ReadElementContentAsBase64 rendered the same outcome. Either
way,
the Page File jumps to around 1.6GB of usage.
>
>
>
"Zafar Abbas" wrote:
>
>Thanks for reporting the issue.
>Could you share the XML data and code to reproduce this issue?
>>
>-Zafar Abbas
>>
>"B" <[e-mail] in message
>news:92**********************************@microso ft.com...
>I am using the XMLReader to read an XML document with a large image
>embedded
inside of it. By large, I mean +150MB. When I use the
ReadElementContentAsBase64 method to read the element content, the page
file
usage peaks and eventually the system throws a
System.OutOfMemoryException.
Does anybody have a suggestion on how to remedy this type of problem?
>
Thanks,
B
>>
>>
>>

[Zafar Abbas]

Have you tried changing the size of the buffer you read using the Read
method? If you read small enough data each time, your memory consumption
should stay low. Below you seem to be reading 4k of data every time, which
does not sound like a lot. Could you attack some profiling tools to your
application and see where most of the memory allocations are coming from?

Also it would be great if you could provide the complete comppilable code
and XML to work with so we could look at it over here.

Thanks
..
"B" <[e-mail] in message
news:42**********************************@microsof t.com...

Same result with code below

Dim b(CInt(inFile.Length)) As Byte 'file length equivalent to 150mb file
Dim cnt As Integer

Do
cnt = inDocumentReader.ReadElementContentAsBase64(b, 0, 4096)
Loop Until cnt = 0
"B" wrote:

>I've modified the code to reflect your recommendations. Here's what i
have

Dim b(_fileLength.length) as byte
xmlReader.ReadElementContentAsBase64(b, 0, b.length)

Unfortunately, the ReadElementContentAs method is still drawing a lot of
memory.

"Zafar Abbas" wrote:

You are reading the data as a string which is definitely the cause of
this.
We need to allocate that string in UTF-16, so you have 300MB+ just the
string, and in order to create the string, we need to parse the whole
value
and store it which uses up a lot more memory and it is understandable
how it
is causing out of memory exceptions.

If you are creating your reader from the XmlReader.Create method (which
it
looks like you are) then you could then read the Base64 value using the
method:

ReadElementContentAsBase64

Thanks.

"B" <[e-mail] in message
news:EB**********************************@microsof t.com...

Thanks for responding.

The code below is using the ReadElementContentAsString method and
converts
it to a byte.

---- CODE ----
Dim strBinaryContent As String =
inDocumentReader.ReadElementContentAsString()
DocumentBinaryContent = Convert.FromBase64String(strBinaryContent)
---- CODE -----

The XML data is just an image placed in the file as binary.

Using the ReadElementContentAsBase64 rendered the same outcome.
Either
way,
the Page File jumps to around 1.6GB of usage.

"Zafar Abbas" wrote:

Thanks for reporting the issue.
Could you share the XML data and code to reproduce this issue?

-Zafar Abbas

"B" <[e-mail] in message
news:92**********************************@microso ft.com...
I am using the XMLReader to read an XML document with a large image
embedded
inside of it. By large, I mean +150MB. When I use the
ReadElementContentAsBase64 method to read the element content, the
page
file
usage peaks and eventually the system throws a
System.OutOfMemoryException.
Does anybody have a suggestion on how to remedy this type of
problem?

Thanks,
B

[B]

Thanks for helping.

Below is the method giving me issues. I can't upload an xml document,
however, you can create one with an embedded document within binarycontent
element

Private Sub ReadDocumentSubTree(ByVal inDocumentReader As XmlReader, ByVal
inFile As FileInfo)

Dim DocumentBinaryContent(CInt(inFile.Length)) As Byte

Dim objBLL As New BusinessLogic.Upload

'loop through document
While inDocumentReader.Read
If inDocumentReader.IsStartElement Then

Case "BINARYCONTENT"
Dim strBinaryContent As String =
inDocumentReader.ReadString()

*********** LINE GIVING PROBLEM ***********
' I've left the code to read as string for clarity...however, i have tried
reading a 'small amount at a time and still had the same result.
DocumentBinaryContent =
Convert.FromBase64String(strBinaryContent)
*********** LINE GIVING PROBLEM ***********

End Select
End If
End While

'clean up
If Not inDocumentReader Is Nothing Then
inDocumentReader.Close()
inDocumentReader = Nothing
End If

DocumentBinaryContent = Nothing
objData = Nothing
End Sub

"Zafar Abbas" wrote:

Have you tried changing the size of the buffer you read using the Read
method? If you read small enough data each time, your memory consumption
should stay low. Below you seem to be reading 4k of data every time, which
does not sound like a lot. Could you attack some profiling tools to your
application and see where most of the memory allocations are coming from?

Also it would be great if you could provide the complete comppilable code
and XML to work with so we could look at it over here.

Thanks
..
"B" <[e-mail] in message
news:42**********************************@microsof t.com...

Same result with code below

Dim b(CInt(inFile.Length)) As Byte 'file length equivalent to 150mb file
Dim cnt As Integer

Do
cnt = inDocumentReader.ReadElementContentAsBase64(b, 0, 4096)
Loop Until cnt = 0
"B" wrote:

I've modified the code to reflect your recommendations. Here's what i
have

Dim b(_fileLength.length) as byte
xmlReader.ReadElementContentAsBase64(b, 0, b.length)

Unfortunately, the ReadElementContentAs method is still drawing a lot of
memory.

"Zafar Abbas" wrote:

You are reading the data as a string which is definitely the cause of
this.
We need to allocate that string in UTF-16, so you have 300MB+ just the
string, and in order to create the string, we need to parse the whole
value
and store it which uses up a lot more memory and it is understandable
how it
is causing out of memory exceptions.

If you are creating your reader from the XmlReader.Create method (which
it
looks like you are) then you could then read the Base64 value using the
method:

ReadElementContentAsBase64

Thanks.

"B" <[e-mail] in message
news:EB**********************************@microsof t.com...

Thanks for responding.
>
The code below is using the ReadElementContentAsString method and
converts
it to a byte.
>
---- CODE ----
Dim strBinaryContent As String =
inDocumentReader.ReadElementContentAsString()
DocumentBinaryContent = Convert.FromBase64String(strBinaryContent)
---- CODE -----
>
The XML data is just an image placed in the file as binary.
>
Using the ReadElementContentAsBase64 rendered the same outcome.
Either
way,
the Page File jumps to around 1.6GB of usage.
>
>
>
"Zafar Abbas" wrote:
>
>Thanks for reporting the issue.
>Could you share the XML data and code to reproduce this issue?
>>
>-Zafar Abbas
>>
>"B" <[e-mail] in message
>news:92**********************************@microso ft.com...
>I am using the XMLReader to read an XML document with a large image
>embedded
inside of it. By large, I mean +150MB. When I use the
ReadElementContentAsBase64 method to read the element content, the
page
file
usage peaks and eventually the system throws a
System.OutOfMemoryException.
Does anybody have a suggestion on how to remedy this type of
problem?
>
Thanks,
B
>>
>>
>>

[Helena Kotas [MSFT]]

Hello,

Whenever you try to get the 150MB image that is encoded in base64 as a
string, it will allocate huge amount of memory.

Here is why:
150MB encoded in base64 is 200MB;
String are stored UTF-16 -400 MB
In order to read such huge string, the reader reads it in chunks to find out
how big it is and then allocates it as one big piece -2x 400 MB = 800 MB

And this is just to read is as a string, it is not converted yet.

Obviously you want to prevent such big allocations, and the way to do it is
to avoid reading it as a string. That's where the ReadContentAsBase64 ir
(ReadElementContentAsBase64) comes in. It will read the data in small (4kB)
chunks, base64-decode them and copy them into the buffer you provided. So the
only big allocation here should be your buffer.

' This is your big buffer
Dim DocumentBinaryContent(CInt(inFile.Length)) As Byte
Dim actualLength As Int

'loop through document
While inDocumentReader.Read
If inDocumentReader.IsStartElement Then

Case "BINARYCONTENT"
actualLength =
inDocumentReader.ReadElementContentAsBase(inFile, 0, inFile.Length)

End Select
End If
End While
Hope this helps,
-Helena Kotas
"B" wrote:

Thanks for helping.

Below is the method giving me issues. I can't upload an xml document,
however, you can create one with an embedded document within binarycontent
element

Private Sub ReadDocumentSubTree(ByVal inDocumentReader As XmlReader, ByVal
inFile As FileInfo)

Dim DocumentBinaryContent(CInt(inFile.Length)) As Byte

Dim objBLL As New BusinessLogic.Upload

'loop through document
While inDocumentReader.Read
If inDocumentReader.IsStartElement Then

Case "BINARYCONTENT"
Dim strBinaryContent As String =
inDocumentReader.ReadString()

*********** LINE GIVING PROBLEM ***********
' I've left the code to read as string for clarity...however, i have tried
reading a 'small amount at a time and still had the same result.
DocumentBinaryContent =
Convert.FromBase64String(strBinaryContent)
*********** LINE GIVING PROBLEM ***********

End Select
End If
End While

'clean up
If Not inDocumentReader Is Nothing Then
inDocumentReader.Close()
inDocumentReader = Nothing
End If

DocumentBinaryContent = Nothing
objData = Nothing
End Sub

"Zafar Abbas" wrote:

Have you tried changing the size of the buffer you read using the Read
method? If you read small enough data each time, your memory consumption
should stay low. Below you seem to be reading 4k of data every time, which
does not sound like a lot. Could you attack some profiling tools to your
application and see where most of the memory allocations are coming from?

Also it would be great if you could provide the complete comppilable code
and XML to work with so we could look at it over here.

Thanks
..
"B" <[e-mail] in message
news:42**********************************@microsof t.com...

Same result with code below
>
Dim b(CInt(inFile.Length)) As Byte 'file length equivalent to 150mb file
Dim cnt As Integer
>
Do
cnt = inDocumentReader.ReadElementContentAsBase64(b, 0, 4096)
Loop Until cnt = 0
>
>
"B" wrote:
>
>I've modified the code to reflect your recommendations. Here's what i
>have
>>
>Dim b(_fileLength.length) as byte
>xmlReader.ReadElementContentAsBase64(b, 0, b.length)
>>
>Unfortunately, the ReadElementContentAs method is still drawing a lot of
>memory.
>>
>"Zafar Abbas" wrote:
>>
You are reading the data as a string which is definitely the cause of
this.
We need to allocate that string in UTF-16, so you have 300MB+ just the
string, and in order to create the string, we need to parse the whole
value
and store it which uses up a lot more memory and it is understandable
how it
is causing out of memory exceptions.
>
If you are creating your reader from the XmlReader.Create method (which
it
looks like you are) then you could then read the Base64 value using the
method:
>
ReadElementContentAsBase64
>
>
>
Thanks.
>
"B" <[e-mail] in message
news:EB**********************************@microsof t.com...
>
Thanks for responding.
>
The code below is using the ReadElementContentAsString method and
converts
it to a byte.
>
---- CODE ----
Dim strBinaryContent As String =
inDocumentReader.ReadElementContentAsString()
DocumentBinaryContent = Convert.FromBase64String(strBinaryContent)
---- CODE -----
>
The XML data is just an image placed in the file as binary.
>
Using the ReadElementContentAsBase64 rendered the same outcome.
Either
way,
the Page File jumps to around 1.6GB of usage.
>
>
>
"Zafar Abbas" wrote:
>
>Thanks for reporting the issue.
>Could you share the XML data and code to reproduce this issue?
>>
>-Zafar Abbas
>>
>"B" <[e-mail] in message
>news:92**********************************@microso ft.com...
>I am using the XMLReader to read an XML document with a large image
>embedded
inside of it. By large, I mean +150MB. When I use the
ReadElementContentAsBase64 method to read the element content, the
page
file
usage peaks and eventually the system throws a
System.OutOfMemoryException.
Does anybody have a suggestion on how to remedy this type of
problem?
>
Thanks,
B
>>
>>
>>
>
>
>

[B]

Thank you both for you help. I've changed my code to use
ReadElementContentAsBase64. I no longer get the OOM exception, but have
found that the database times out. So it looks like I will need to do some
tuning. Thanks again for the good info.

"Helena Kotas [MSFT]" wrote:

Hello,

Whenever you try to get the 150MB image that is encoded in base64 as a
string, it will allocate huge amount of memory.

Here is why:
150MB encoded in base64 is 200MB;
String are stored UTF-16 -400 MB
In order to read such huge string, the reader reads it in chunks to find out
how big it is and then allocates it as one big piece -2x 400 MB = 800 MB

And this is just to read is as a string, it is not converted yet.

Obviously you want to prevent such big allocations, and the way to do it is
to avoid reading it as a string. That's where the ReadContentAsBase64 ir
(ReadElementContentAsBase64) comes in. It will read the data in small (4kB)
chunks, base64-decode them and copy them into the buffer you provided. So the
only big allocation here should be your buffer.

' This is your big buffer
Dim DocumentBinaryContent(CInt(inFile.Length)) As Byte
Dim actualLength As Int

'loop through document
While inDocumentReader.Read
If inDocumentReader.IsStartElement Then

Case "BINARYCONTENT"
actualLength =
inDocumentReader.ReadElementContentAsBase(inFile, 0, inFile.Length)

End Select
End If
End While
Hope this helps,
-Helena Kotas
"B" wrote:

Thanks for helping.

Below is the method giving me issues. I can't upload an xml document,
however, you can create one with an embedded document within binarycontent
element

Private Sub ReadDocumentSubTree(ByVal inDocumentReader As XmlReader, ByVal
inFile As FileInfo)

Dim DocumentBinaryContent(CInt(inFile.Length)) As Byte

Dim objBLL As New BusinessLogic.Upload

'loop through document
While inDocumentReader.Read
If inDocumentReader.IsStartElement Then

Case "BINARYCONTENT"
Dim strBinaryContent As String =
inDocumentReader.ReadString()

*********** LINE GIVING PROBLEM ***********
' I've left the code to read as string for clarity...however, i have tried
reading a 'small amount at a time and still had the same result.
DocumentBinaryContent =
Convert.FromBase64String(strBinaryContent)
*********** LINE GIVING PROBLEM ***********

End Select
End If
End While

'clean up
If Not inDocumentReader Is Nothing Then
inDocumentReader.Close()
inDocumentReader = Nothing
End If

DocumentBinaryContent = Nothing
objData = Nothing
End Sub

"Zafar Abbas" wrote:

Have you tried changing the size of the buffer you read using the Read
method? If you read small enough data each time, your memory consumption
should stay low. Below you seem to be reading 4k of data every time, which
does not sound like a lot. Could you attack some profiling tools to your
application and see where most of the memory allocations are coming from?
>
Also it would be great if you could provide the complete comppilable code
and XML to work with so we could look at it over here.
>
Thanks
..
"B" <[e-mail] in message
news:42**********************************@microsof t.com...
Same result with code below

Dim b(CInt(inFile.Length)) As Byte 'file length equivalent to 150mb file
Dim cnt As Integer

Do
cnt = inDocumentReader.ReadElementContentAsBase64(b, 0, 4096)
Loop Until cnt = 0


"B" wrote:

I've modified the code to reflect your recommendations. Here's what i
have
>
Dim b(_fileLength.length) as byte
xmlReader.ReadElementContentAsBase64(b, 0, b.length)
>
Unfortunately, the ReadElementContentAs method is still drawing a lot of
memory.
>
"Zafar Abbas" wrote:
>
You are reading the data as a string which is definitely the cause of
this.
We need to allocate that string in UTF-16, so you have 300MB+ just the
string, and in order to create the string, we need to parse the whole
value
and store it which uses up a lot more memory and it is understandable
how it
is causing out of memory exceptions.

If you are creating your reader from the XmlReader.Create method (which
it
looks like you are) then you could then read the Base64 value using the
method:

ReadElementContentAsBase64



Thanks.

"B" <[e-mail] in message
news:EB**********************************@microsof t.com...

Thanks for responding.
>
The code below is using the ReadElementContentAsString method and
converts
it to a byte.
>
---- CODE ----
Dim strBinaryContent As String =
inDocumentReader.ReadElementContentAsString()
DocumentBinaryContent = Convert.FromBase64String(strBinaryContent)
---- CODE -----
>
The XML data is just an image placed in the file as binary.
>
Using the ReadElementContentAsBase64 rendered the same outcome.
Either
way,
the Page File jumps to around 1.6GB of usage.
>
>
>
"Zafar Abbas" wrote:
>
>Thanks for reporting the issue.
>Could you share the XML data and code to reproduce this issue?
>>
>-Zafar Abbas
>>
>"B" <[e-mail] in message
>news:92**********************************@microso ft.com...
>I am using the XMLReader to read an XML document with a large image
>embedded
inside of it. By large, I mean +150MB. When I use the
ReadElementContentAsBase64 method to read the element content, the
page
file
usage peaks and eventually the system throws a
System.OutOfMemoryException.
Does anybody have a suggestion on how to remedy this type of
problem?
>
Thanks,
B
>>
>>
>>



>
>
>