How to break up binaryWrite file into multiple chunks

[Katie]

Hi,

I am doing a binaryWrite to allow users to download files. The problem
occurs if the file is too big. Some of the files i have are close to
100 megs. I read on msdn that if the data is greater than 4MB it is
advisable to break it up into multiple chunks

http://msdn.microsoft.com/library/de...4b95130b4a.asp

How would that be done?
Thanks for your time and help :)

ps below is the sub i use for streaming

Private Sub streamDocs(path, filename, originalFileName, contentType)
Response.AddHeader
"content-disposition","attachment;filename="&originalFileNa me
Response.ContentType = contentType
Dim BinaryStream, Fil, fs
Set BinaryStream = CreateObject("ADODB.Stream")
set fs = Server.CreateObject("Scripting.FileSystemObject")
Set Fil = fs.GetFile(path & "\" &filename) 'Open file
BinaryStream.Type = 1
BinaryStream.Open
BinaryStream.LoadFromFile Fil.path
Response.BinaryWrite BinaryStream.Read
BinaryStream.Cancel
BinaryStream.Close
set BinaryStream = nothing
End sub

[Justin Piper]

On Wed, 13 Sep 2006 09:03:42 -0500, Katie <Dn*********@gmail.comwrote:

Hi,

I am doing a binaryWrite to allow users to download files. The problem
occurs if the file is too big. Some of the files i have are close to
100 megs. I read on msdn that if the data is greater than 4MB it is
advisable to break it up into multiple chunks

The optional first argument to the Read() method will allows you to
specify the maximum number of bytes read. So where you have this:

Response.BinaryWrite BinaryStream.Read

You could change it to this to write data in thousand-byte chunks:

Dim i
For i = 0 To BinaryStream.Size
Response.BinaryWrite BinaryStream.Read(1000)
Next

You can find documentation for all of the Stream object's properties and
methods on Microsoft's website.

Stream Object Properties, Methods, and Events
http://windowssdk.msdn.microsoft.com.../ms677486.aspx

--
Justin Piper
Bizco Technologies
http://www.bizco.com/

[Justin Piper]

On Wed, 13 Sep 2006 09:22:36 -0500, Justin Piper <jp****@bizco.comwrote:

For i = 0 To BinaryStream.Size

Whoops, that should of course read, "For i = 0 To BinaryStream.Size Step
1000".

--
Justin Piper
Bizco Technologies
http://www.bizco.com/

[Evertjan.]

Justin Piper wrote on 13 sep 2006 in
microsoft.public.inetserver.asp.general:

On Wed, 13 Sep 2006 09:22:36 -0500, Justin Piper <jp****@bizco.com>
wrote:

> For i = 0 To BinaryStream.Size

Whoops, that should of course read, "For i = 0 To BinaryStream.Size
Step 1000".

or:

For i = 0 To BinaryStream.Size/1000

--
Evertjan.
The Netherlands.
(Please change the x'es to dots in my emailaddress)

[Justin Piper]

On Wed, 13 Sep 2006 09:43:34 -0500, Evertjan.
<ex**************@interxnl.netwrote:

Justin Piper wrote on 13 sep 2006 in
microsoft.public.inetserver.asp.general:

>On Wed, 13 Sep 2006 09:22:36 -0500, Justin Piper <jp****@bizco.com>
wrote:

>> For i = 0 To BinaryStream.Size

Whoops, that should of course read, "For i = 0 To BinaryStream.Size
Step 1000".

or:

For i = 0 To BinaryStream.Size/1000

Well, as long as we're splitting hairs, using "Step" is about 30% faster.
:)

E:\jpiper\Documents\Scripts\VBScript\prof

cscript step.vbs

Step 2.122266s
Div 3.476563s

step.vbs:
Option Explicit

Function Prof(f)
Dim start, finish, i, total
total = 0
For i = 0 To 9
start = Timer()
f
finish = Timer()
total = total + finish - start
Next
Prof = total/10
End Function

Sub ProfStep()
Dim i: For i = 0 To 1000000000 Step 100 : Next
End Sub
WScript.Echo "Step" & vbTab & Prof(GetRef("ProfStep")) & "s"

Sub ProfDiv()
Dim j: j = 1000000000 / 100
Dim i: For i = 0 To j : Next
End Sub
WScript.Echo "Div" & vbTab & Prof(GetRef("ProfDiv")) & "s"

--
Justin Piper
Bizco Technologies
http://www.bizco.com/

[Evertjan.]

Justin Piper wrote on 13 sep 2006 in
microsoft.public.inetserver.asp.general:

>For i = 0 To BinaryStream.Size/1000

Well, as long as we're splitting hairs, using "Step" is about 30%
faster.
:)

Oh, if you wish, try:

temp = int(BinaryStream.Size/1000)+1
For i = 0 To temp

However I do not believe you are right, Justin, with the 30%,
as BinaryStream.Read(1000) and the response.write
will take much more cpu time,
than the repetitive for-next division by 1000 overhead.

I guess it will be muh ess than 1%.
--
Evertjan.
The Netherlands.
(Please change the x'es to dots in my emailaddress)

[Justin Piper]

On Wed, 13 Sep 2006 11:40:05 -0500, Evertjan.
<ex**************@interxnl.netwrote:

Justin Piper wrote on 13 sep 2006 in
microsoft.public.inetserver.asp.general:

>>For i = 0 To BinaryStream.Size/1000

Well, as long as we're splitting hairs, using "Step" is about 30%
faster.
:)

Oh, if you wish, try:

temp = int(BinaryStream.Size/1000)+1
For i = 0 To temp

Int() doesn't change the type of the variable, but yes, once you eliminate
the floating point arithmetic they are equivalent.

However I do not believe you are right, Justin, with the 30%,
as BinaryStream.Read(1000) and the response.write
will take much more cpu time,
than the repetitive for-next division by 1000 overhead.

I guess it will be muh ess than 1%.

Oh, naturally this will all be lost in the noise. But as far as ".Size
Step 1000" itself, it is indeed significantly faster than ".Size/1000".
And I /did/ warn you I was splitting hairs. :)

--
Justin Piper
Bizco Technologies
http://www.bizco.com/

[Katie]

Hi,

Thanks for your help.
It works :)
Just wondering is there a particular value for the data size that is
optimal or can i use any e.g. 1000kb

Thanks
:)

Justin Piper wrote:

On Wed, 13 Sep 2006 11:40:05 -0500, Evertjan.
<ex**************@interxnl.netwrote:

Justin Piper wrote on 13 sep 2006 in
microsoft.public.inetserver.asp.general:

>For i = 0 To BinaryStream.Size/1000
Well, as long as we're splitting hairs, using "Step" is about 30%
faster.
:)

Oh, if you wish, try:

temp = int(BinaryStream.Size/1000)+1
For i = 0 To temp

Int() doesn't change the type of the variable, but yes, once you eliminate
the floating point arithmetic they are equivalent.

However I do not believe you are right, Justin, with the 30%,
as BinaryStream.Read(1000) and the response.write
will take much more cpu time,
than the repetitive for-next division by 1000 overhead.

I guess it will be muh ess than 1%.

Oh, naturally this will all be lost in the noise. But as far as ".Size
Step 1000" itself, it is indeed significantly faster than ".Size/1000".
And I /did/ warn you I was splitting hairs. :)

--
Justin Piper
Bizco Technologies
http://www.bizco.com/

[Justin Piper]

On Wed, 13 Sep 2006 13:35:45 -0500, Katie <Dn*********@gmail.comwrote:

Just wondering is there a particular value for the data size that is
optimal or can i use any e.g. 1000kb

Well, a 1000KB block size would mean that everyone downloading a file
would be using as much memory, so only do that if you won't have to
support more than a few connections at a time. If you want to find the
sweet spot you might try Microsoft's Web Application Stress Tool[1], but
if it were me, I'd just arbitrarily pick a nice round number and call it
good.

[1] Download details: Web Application Stress Tool
http://www.microsoft.com/downloads/d...displaylang=en

--
Justin Piper
Bizco Technologies
http://www.bizco.com/

[Anthony Jones]

"Katie" <Dn*********@gmail.comwrote in message
news:11*********************@b28g2000cwb.googlegro ups.com...

Hi,

Thanks for your help.
It works :)
Just wondering is there a particular value for the data size that is
optimal or can i use any e.g. 1000kb

Katie,

My apologies I did have your earlier post on this subject marked for a
response but I never did get round it.

Unfortunately the code posted so far contains a small bug that will result
in a truncation of the file to the nearest mutliple of the chunk size.

Const mclChunkSize = &H200000

Dim i

Response.Buffer = False

For i = 0 To BinaryStream.Size Step mclChunkSize
Response.BinaryWrite BinaryStream.Read(mclChunkSize)
Next

If (BinaryStream.Size Mod mclChunkSize) 0 Then
Response.BinaryWrite BinaryStream.Read(BinaryStream.Size Mod
mclChunkSize)
End If
Choice of chunk size will affect the download performance. You shoud choose
a good size like the 1Mb you proposed. Using a very small chunk size such
as 1000 is inefficient (even a single TCP/IP packet can carry a larger
payload) and each Response.BinaryWrite will block until all acknowledgements
for data sent has been received from the client. By using a larger chunk
size you let the TCP/IP stack do it's work far more effeciently.

Justin has pointed out there is a trade off between chunksize and server
memory use. However, it's worth bearing in mind that by default only 10
requests will be processed per CPU anyway (although this may be increased it
shouldn't really exceed 25) hence a chunksize of 2MB only represents 20MB -
50MB per CPU and that assumes all worker threads are busy responding to
download requests.

Anthony.

Thanks
:)

Justin Piper wrote:

On Wed, 13 Sep 2006 11:40:05 -0500, Evertjan.
<ex**************@interxnl.netwrote:

Justin Piper wrote on 13 sep 2006 in
microsoft.public.inetserver.asp.general:
>
>>For i = 0 To BinaryStream.Size/1000
>>>
>>
>Well, as long as we're splitting hairs, using "Step" is about 30%
>faster.
>:)
>>
>
Oh, if you wish, try:
>
temp = int(BinaryStream.Size/1000)+1
For i = 0 To temp

Int() doesn't change the type of the variable, but yes, once you

eliminate

the floating point arithmetic they are equivalent.

However I do not believe you are right, Justin, with the 30%,
as BinaryStream.Read(1000) and the response.write
will take much more cpu time,
than the repetitive for-next division by 1000 overhead.
>
I guess it will be muh ess than 1%.

Oh, naturally this will all be lost in the noise. But as far as ".Size
Step 1000" itself, it is indeed significantly faster than ".Size/1000".
And I /did/ warn you I was splitting hairs. :)

--
Justin Piper
Bizco Technologies
http://www.bizco.com/

[Justin Piper]

On Thu, 14 Sep 2006 03:32:02 -0500, Anthony Jones <An*@yadayadayada.com>
wrote:

>
"Katie" <Dn*********@gmail.comwrote in message
news:11*********************@b28g2000cwb.googlegro ups.com...

>Just wondering is there a particular value for the data size that is
optimal or can i use any e.g. 1000kb

Unfortunately the code posted so far contains a small bug that will
result in a truncation of the file to the nearest mutliple of the
chunk size.

Ah, well, there is actually a bug, but that isn't it. This works fine,
so long as test.txt is not an even multiple of the block size:

<% Option Explicit
Dim stream, i

Response.ContentType = "text/plain"
Set stream = CreateObject("ADODB.Stream")
With stream
.Type = 1
.Open
.LoadFromFile Server.MapPath("test.txt")
End With

For i = 0 To stream.Size Step 1000
Response.BinaryWrite stream.Read(1000)
Next
%>

When its size is an even multiple, we actually go through the loop once
more than necessary, and the last call to Read() returns Null. You had
the right idea, though. All that's needed is to change "For i = 0" to
"For i = 1" and then write any remaining bytes.

<% Option Explicit
Dim stream, i

Const BlockSize = 1000

Response.ContentType = "text/plain"
Set stream = CreateObject("ADODB.Stream")
With stream
.Type = 1
.Open
.LoadFromFile Server.MapPath("test.txt")
End With

For i = 1 To stream.Size Step BlockSize
Response.BinaryWrite stream.Read(BlockSize)
Next
If stream.Size Mod BlockSize 0 Then Response.Write stream.Read
%>

Justin has pointed out there is a trade off between chunksize and
server memory use. However, it's worth bearing in mind that by
default only 10 requests will be processed per CPU anyway (although
this may be increased it shouldn't really exceed 25) hence a
chunksize of 2MB only represents 20MB - 50MB per CPU and that assumes
all worker threads are busy responding to download requests.

That's a good point. I was thinking of how only a few hundred downloads
would be all that's needed to cause paging, but hadn't considered the
any of the _other_ logistical problems you'd encounter in such a
scenerio.

--
Justin Piper
Bizco Technologies
http://www.bizco.com/

[Justin Piper]

On Thu, 14 Sep 2006 09:43:53 -0500, Justin Piper <jp****@bizco.comwrote:

If stream.Size Mod BlockSize 0 Then Response.Write stream.Read

Agh. This has not been my week. That should be:

If Not stream.EOS Then Response.BinaryWrite stream.Read

It only appeared to work since Response.Write will dutifly write nothing
if you tell it to. :P

--
Justin Piper
Bizco Technologies
http://www.bizco.com/

[Katie]

Thanks for your help :)

Justin Piper wrote:

On Thu, 14 Sep 2006 09:43:53 -0500, Justin Piper <jp****@bizco.comwrote:

If stream.Size Mod BlockSize 0 Then Response.Write stream.Read

Agh. This has not been my week. That should be:

If Not stream.EOS Then Response.BinaryWrite stream.Read

It only appeared to work since Response.Write will dutifly write nothing
if you tell it to. :P

--
Justin Piper
Bizco Technologies
http://www.bizco.com/