StringBuilder and internal memory question

[DV]

I have a StringBuilder that has a string with 12,000,000 characters.
When I do a ToString(), I expect to have ~25,000,000 bytes worth of
memory, yet, I end up with ~43,000,000 bytes. That's almost double
the size. The string returned from ToString() is actually of size
StringBuilder.C apacity, NOT StringBuilder.L ength. It may have a
end-of-string character at StringBuilder.L ength, but its actual memory
size is StringBuilder.C apacity.

Does this sound right?

[DV]

Thanks all... I think i got it...
Interestingly enough..

If I do...
StringBuilder sb = new StringBuilder(1 0000000);
for (int i=0; i<100; i++)
sb.Append("Hell o World");
string a1 = sb.ToString();

and..
StringBuilder sb = new StringBuilder() ;
for (int i=0; i<100; i++)
sb.Append("Hell o World");
string a2 = sb.ToString();

in this case, a1 actually takes LESS physical memory then a2. a1 gets
trimmed while a2 returns the internal string.
i guess the lesson is to put a capacity whenever possible. but that
also has drawbacks..

[Jon Skeet [C# MVP]]

Willy Denoyette [MVP] <wi************ *@telenet.be> wrote:

Hit send too fast.
The String reference returned from SB.ToString() is just the same reference
to the existing String object, trimming would involve the creation of a new
string, this would increase the memory footprint and would take a
performance hit.

Exactly - so the OP's supposition that "The string returned from
ToString() is actually of size StringBuilder.C apacity, NOT
StringBuilder.L ength" sounds right to me.

--
Jon Skeet - <sk***@pobox.co m>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

[Willy Denoyette [MVP]]

"Jon Skeet [C# MVP]" <sk***@pobox.co m> wrote in message
news:MP******** *************** *@msnews.micros oft.com...
| Willy Denoyette [MVP] <wi************ *@telenet.be> wrote:
| > Hit send too fast.
| > The String reference returned from SB.ToString() is just the same
reference
| > to the existing String object, trimming would involve the creation of a
new
| > string, this would increase the memory footprint and would take a
| > performance hit.
|
| Exactly - so the OP's supposition that "The string returned from
| ToString() is actually of size StringBuilder.C apacity, NOT
| StringBuilder.L ength" sounds right to me.
|

Not really, what delimits a string is it's Length, the fact that the string
buffer (char[]) is larger is due to the way SB's handle the underlying
'String', they can add char's to the buffer without the need to create a new
String object (until the buffer is filled completely), which is the reason
why SBs exist. What is returned by SB.ToString() is a String reference and
what counts is the Length property value, what's stored beyond is not part
of the String, there isn't a managed way to get at the size of the buffer,
nor can you get at the contents of the store beyond the last char position +
1 (the 0x0000 terminator) in the buffer.
But this was not really my point, the OP assumes that the capacity is 43MB
for a string of 12000000 which is definitely not true.

Willy.

[Jon Skeet [C# MVP]]

Willy Denoyette [MVP] wrote:

| Exactly - so the OP's supposition that "The string returned from
| ToString() is actually of size StringBuilder.C apacity, NOT
| StringBuilder.L ength" sounds right to me.

Not really, what delimits a string is it's Length, the fact that the string
buffer (char[]) is larger is due to the way SB's handle the underlying
'String', they can add char's to the buffer without the need to create a new
String object (until the buffer is filled completely), which is the reason
why SBs exist. What is returned by SB.ToString() is a String reference and
what counts is the Length property value, what's stored beyond is not part
of the String, there isn't a managed way to get at the size of the buffer,
nor can you get at the contents of the store beyond the last char position +
1 (the 0x0000 terminator) in the buffer.
No - but given the rest of the OP's question, I thought it reasonable
to assume that the "size" he's talking about is the size in memory, not
the "logical" length of the string. Given that reading, isn't the OP's
supposition valid? The "buffer size" (in characters) of the returned
string is the same as (or maybe one or two characters more or less -
not sure) the capacity of the StringBuilder it was returned from.

Even though you can't get at the rest of the buffer, I'd say it still
"counts" in terms of being valuable information. For instance, if you
were creating a lot of long-lived strings from StringBuilders, it may
be worth creating a copy of the string which effectively trims the
extra buffer, allowing the oversized string to then be garbage
collected. (I've done this many times in Java, particularly when
reading lines from a file. The initial buffer is 80 characters, and if
you're reading a dictionary a line at a time and the average line
length is only 5 or 6 characters, the wastage can be very, very
significant.)
But this was not really my point, the OP assumes that the capacity is 43MB
for a string of 12000000 which is definitely not true.

That much is certainly true - *if* by "string of 12000000" you mean a
"string of internal buffer size 12000000 characters".

Jon

[Willy Denoyette [MVP]]

"DV" <da*****@gmail. com> wrote in message
news:11******** **************@ u72g2000cwu.goo glegroups.com.. .
| Thanks all... I think i got it...
| Interestingly enough..
|
| If I do...
| StringBuilder sb = new StringBuilder(1 0000000);
| for (int i=0; i<100; i++)
| sb.Append("Hell o World");
| string a1 = sb.ToString();
|
| and..
| StringBuilder sb = new StringBuilder() ;
| for (int i=0; i<100; i++)
| sb.Append("Hell o World");
| string a2 = sb.ToString();
|
| in this case, a1 actually takes LESS physical memory then a2. a1 gets
| trimmed while a2 returns the internal string.
| i guess the lesson is to put a capacity whenever possible. but that
| also has drawbacks..
|

Right, what you see here is the result of an optimization.
Let me try to explain what's happening ....

First you need to know how a SB and a String looks like on the managed heap,
this is how a StringBuilder object looks:

<standard object header> // to IntPtr sized values, not relevant here
IntPtr m_currentThread ;
int m_maxCapacity;
string m_StringValue;

while a String looks like:
<object header>
int m_arrayLength;
int m_stringLength;
char m_firstChar;

In the first case, you create a SB with capacity 10000000, that means that
the size of the underlying String object is larger than 85Kb, so, the String
will end on the Large Object Heap (LOH).
Then you start filling the string buffer, the result at the end of the loop
looks like:

Your StrinBuilder sb on the Gen0 heap

m_currentThread = xxxx // not important here
m_maxCapacity = 2147483647 // 2GB
m_StringValue = 03271000 // reference - points to a string on the
LOH (value as a sample)

m_arrayLength = 10000001 // Buffer space (in no. of char)
m_stringLength = 1100 // actual string Length
m_firstChar = 'H' // First char in buffer (start of buffer
.... // following chars
.... = 'd' // last char of string (buffer position 1100)
.... = 0x0000 // last char in buffer (buffer position 1101)

Now when you execute ... sb.ToString();
the CLR rightfully decides that this String object doesn't belong to the LOH
(is < 85Kb), so he creates a new String on the Gen0 heap and returns it's
reference in a1, the new string object looks now like:

m_arrayLength = 1101 // Buffer space (in no. of char)
m_stringLength = 1100 // actual string Length
m_firstChar = 'H' // First char in buffer (start of buffer
.... // following chars
.... = 'd' // last char of string (buffer position 1100)
.... = 0x0000

Notice the new m_arrayLength of 1100 ...
Note that the m_arrayLength = 10000001 has never been committed, only
reserved that is why you don't see this allocated in physical memory.


What's happening in the second case is:

A SB is created on the Gen0 heap and looks like:

m_currentThread = xxxx // not important here
m_maxCapacity = 2147483647 // 2GB
m_StringValue = 01274e34 // reference - points to a string on the Gen0
heap(value as a sample)

and the underlying String object at the end of the loop:

m_arrayLength = 2049 // Buffer space (in no. of char)
m_stringLength = 1100 // actual string Length
m_firstChar = 'H' // First char in buffer (start of buffer
.... // following chars
.... = 'd' // last char of string (buffer position 1100)
.... = 0x0000

Notice the m_arrayLength ...

But before you get this final string, a number of temporary strings need to
be build. Remember that the SB starts with an m_arrayLength = 17 (16 + 1 for
the 0x0000 string termination char).
That means that after the loop you have effectively created 8 intermediate
string objects (16, 32, 64, ...2048).
That means that you have wasted some memory, but also some CPU cycles, more,
you also put some additional stress on the GC which will have to clean-up
the intermediate objects.
Conclusion: you should try to pre-allocate SB's whenever possible. Note that
this is especially important for server applications and for client
(WinForms) applications that need to run in Terminal Server environments.
Hope this clears things up a bit :-)

Willy.

[Willy Denoyette [MVP]]

Jon, you should read his original question, that starts with ...
I have a StringBuilder that has a string with 12,000,000 characters.


"Jon Skeet [C# MVP]" <sk***@pobox.co m> wrote in message
news:11******** **************@ j33g2000cwa.goo glegroups.com.. .
| Willy Denoyette [MVP] wrote:
| > | Exactly - so the OP's supposition that "The string returned from
| > | ToString() is actually of size StringBuilder.C apacity, NOT
| > | StringBuilder.L ength" sounds right to me.
| >
| > Not really, what delimits a string is it's Length, the fact that the
string
| > buffer (char[]) is larger is due to the way SB's handle the underlying
| > 'String', they can add char's to the buffer without the need to create a
new
| > String object (until the buffer is filled completely), which is the
reason
| > why SBs exist. What is returned by SB.ToString() is a String reference
and
| > what counts is the Length property value, what's stored beyond is not
part
| > of the String, there isn't a managed way to get at the size of the
buffer,
| > nor can you get at the contents of the store beyond the last char
position +
| > 1 (the 0x0000 terminator) in the buffer.
|
| No - but given the rest of the OP's question, I thought it reasonable
| to assume that the "size" he's talking about is the size in memory, not
| the "logical" length of the string. Given that reading, isn't the OP's
| supposition valid? The "buffer size" (in characters) of the returned
| string is the same as (or maybe one or two characters more or less -
| not sure) the capacity of the StringBuilder it was returned from.
|
| Even though you can't get at the rest of the buffer, I'd say it still
| "counts" in terms of being valuable information. For instance, if you
| were creating a lot of long-lived strings from StringBuilders, it may
| be worth creating a copy of the string which effectively trims the
| extra buffer, allowing the oversized string to then be garbage
| collected. (I've done this many times in Java, particularly when
| reading lines from a file. The initial buffer is 80 characters, and if
| you're reading a dictionary a line at a time and the average line
| length is only 5 or 6 characters, the wastage can be very, very
| significant.)
|
| > But this was not really my point, the OP assumes that the capacity is
43MB
| > for a string of 12000000 which is definitely not true.
|
| That much is certainly true - *if* by "string of 12000000" you mean a
| "string of internal buffer size 12000000 characters".
Jon,
The OP asked....
I have a StringBuilder that has a string with 12,000,000 characters.

And I replied with .. you will end with a SB buffer (a String) with a
Capacity of 16777216 and a
Length of 12000000.

You see, the capacity of a SB holding a string with Length 12000000 is
always 16777216 or 33554432 bytes unless you pre-allocate the SB.
The SB does not consume 43MB in this case. Actually the OP doesn't know the
size of the SB at all, he is simply watching the memory counters in Taskman
I'm sure, these don't reveal the size of whatever object in memory. And
that's my point. Sorry if I'm confusing you ;-)
Willy.