Uploading big files

[ThunderMusic]

Hi,
I'm currently building a little web site for me and my friends where we
would like to be able to post big files (10Mb to 250Mb approx.) Right now
the ASP.Net UploadFile control can't do it as we are limited 1) by the
MaxRequestLengt h (4Mb) and 2) with big files (such as 250Mb) the server
crashes because it only uses memory to cache the upload...

Is there an easy way to develop an HTTP Module to handle the upload (links,
tutorials, ...)? I've seen components to do this on the net, but they are
too expensive for what we need to do and the budget we have (near 0.00$)...
I know we could use FTP (there are plenty of free server softwares out
there), but we could not use all the other features an asp.net has to offer
such as the ease of use and data binding and everything... right now I
would like to be able to put these files into a database, which is possible
with the FileUpload control, but because of the "big files" requirement, I
just can't use it...

Anybody have ideas?

Thanks

ThunderMusic

[Gabriele G. Ponti]

The MaxRequestLengt h limit can be changed in the web.config file.

<configuratio n>
<system.web>
<httpRuntime maxRequestLengt h="40960" />
</system.web>
</configuration>

Please see this document:
http://msdn.microsoft.com/library/de...imeSection.asp

"ThunderMus ic" <No************ *************@N oSpAm.comwrote in message
news:%2******** *******@TK2MSFT NGP06.phx.gbl.. .

Hi,
I'm currently building a little web site for me and my friends where we
would like to be able to post big files (10Mb to 250Mb approx.) Right now
the ASP.Net UploadFile control can't do it as we are limited 1) by the
MaxRequestLengt h (4Mb) and 2) with big files (such as 250Mb) the server
crashes because it only uses memory to cache the upload...

Is there an easy way to develop an HTTP Module to handle the upload
(links, tutorials, ...)? I've seen components to do this on the net, but
they are too expensive for what we need to do and the budget we have (near
0.00$)... I know we could use FTP (there are plenty of free server
softwares out there), but we could not use all the other features an
asp.net has to offer such as the ease of use and data binding and
everything... right now I would like to be able to put these files into a
database, which is possible with the FileUpload control, but because of
the "big files" requirement, I just can't use it...

Anybody have ideas?

Thanks

ThunderMusic

[ThunderMusic]

I know this part... We changed it, but there are 2 problems with this 1) it
opens the server for Denial Of Service attacks and 2)even if I put it to 2Gb
and upload a file of 200Mb, the server crashes because the FileUpload
control caches the request entirely in memory...

Thanks

ThunderMusic
"Gabriele G. Ponti" <ggponti.at.hot mail.comwrote in message
news:O2******** ******@TK2MSFTN GP05.phx.gbl...

The MaxRequestLengt h limit can be changed in the web.config file.

<configuratio n>
<system.web>
<httpRuntime maxRequestLengt h="40960" />
</system.web>
</configuration>

Please see this document:
http://msdn.microsoft.com/library/de...imeSection.asp

"ThunderMus ic" <No************ *************@N oSpAm.comwrote in message
news:%2******** *******@TK2MSFT NGP06.phx.gbl.. .

>Hi,
I'm currently building a little web site for me and my friends where we
would like to be able to post big files (10Mb to 250Mb approx.) Right now
the ASP.Net UploadFile control can't do it as we are limited 1) by the
MaxRequestLeng th (4Mb) and 2) with big files (such as 250Mb) the server
crashes because it only uses memory to cache the upload...

Is there an easy way to develop an HTTP Module to handle the upload
(links, tutorials, ...)? I've seen components to do this on the net, but
they are too expensive for what we need to do and the budget we have
(near 0.00$)... I know we could use FTP (there are plenty of free server
softwares out there), but we could not use all the other features an
asp.net has to offer such as the ease of use and data binding and
everything.. . right now I would like to be able to put these files into
a database, which is possible with the FileUpload control, but because of
the "big files" requirement, I just can't use it...

Anybody have ideas?

Thanks

ThunderMusic

[Brian P. Hammer]

I know you said your budget was $0 but I use a component called aspupload
that works great. I think the single server license is $150.00.
"ThunderMus ic" <No************ *************@N oSpAm.comwrote in message
news:%2******** *******@TK2MSFT NGP06.phx.gbl.. .

Hi,
I'm currently building a little web site for me and my friends where we
would like to be able to post big files (10Mb to 250Mb approx.) Right now
the ASP.Net UploadFile control can't do it as we are limited 1) by the
MaxRequestLengt h (4Mb) and 2) with big files (such as 250Mb) the server
crashes because it only uses memory to cache the upload...

Is there an easy way to develop an HTTP Module to handle the upload
(links, tutorials, ...)? I've seen components to do this on the net, but
they are too expensive for what we need to do and the budget we have (near
0.00$)... I know we could use FTP (there are plenty of free server
softwares out there), but we could not use all the other features an
asp.net has to offer such as the ease of use and data binding and
everything... right now I would like to be able to put these files into a
database, which is possible with the FileUpload control, but because of
the "big files" requirement, I just can't use it...

Anybody have ideas?

Thanks

ThunderMusic

[ThunderMusic]

it could be interesting... I'll look for it on google... (or maybe you have
to link to the home page og the publisher?)

I'm currently trying a free solution that is under the LGPL called :
NeatUpload... I found it a little before noon (Eastern time USA) I'll see if
it does the job... if it does, I'll use it, if it doesn't, I'll look for
your aspupload... ;)

thanks a lot...

ThunderMusic
"Brian P. Hammer" <Br*********@co mmunity.nospamw rote in message
news:Om******** ******@TK2MSFTN GP03.phx.gbl...

>I know you said your budget was $0 but I use a component called aspupload
that works great. I think the single server license is $150.00.
"ThunderMus ic" <No************ *************@N oSpAm.comwrote in message
news:%2******** *******@TK2MSFT NGP06.phx.gbl.. .

>Hi,
I'm currently building a little web site for me and my friends where we
would like to be able to post big files (10Mb to 250Mb approx.) Right now
the ASP.Net UploadFile control can't do it as we are limited 1) by the
MaxRequestLeng th (4Mb) and 2) with big files (such as 250Mb) the server
crashes because it only uses memory to cache the upload...

Is there an easy way to develop an HTTP Module to handle the upload
(links, tutorials, ...)? I've seen components to do this on the net, but
they are too expensive for what we need to do and the budget we have
(near 0.00$)... I know we could use FTP (there are plenty of free server
softwares out there), but we could not use all the other features an
asp.net has to offer such as the ease of use and data binding and
everything.. . right now I would like to be able to put these files into
a database, which is possible with the FileUpload control, but because of
the "big files" requirement, I just can't use it...

Anybody have ideas?

Thanks

ThunderMusic

[Brian P. Hammer]

http://www.aspupload.com/ is the site. I think their main site is at
http://www.persits.com/

HTH,
Brian

"ThunderMus ic" <No************ *************@N oSpAm.comwrote in message
news:%2******** *********@TK2MS FTNGP02.phx.gbl ...

it could be interesting... I'll look for it on google... (or maybe you
have to link to the home page og the publisher?)

I'm currently trying a free solution that is under the LGPL called :
NeatUpload... I found it a little before noon (Eastern time USA) I'll see
if it does the job... if it does, I'll use it, if it doesn't, I'll look
for your aspupload... ;)

thanks a lot...

ThunderMusic
"Brian P. Hammer" <Br*********@co mmunity.nospamw rote in message
news:Om******** ******@TK2MSFTN GP03.phx.gbl...

>>I know you said your budget was $0 but I use a component called aspupload
that works great. I think the single server license is $150.00.
"ThunderMusi c" <No************ *************@N oSpAm.comwrote in message
news:%2******* ********@TK2MSF TNGP06.phx.gbl. ..

>>Hi,
I'm currently building a little web site for me and my friends where we
would like to be able to post big files (10Mb to 250Mb approx.) Right
now the ASP.Net UploadFile control can't do it as we are limited 1) by
the MaxRequestLengt h (4Mb) and 2) with big files (such as 250Mb) the
server crashes because it only uses memory to cache the upload...

Is there an easy way to develop an HTTP Module to handle the upload
(links, tutorials, ...)? I've seen components to do this on the net, but
they are too expensive for what we need to do and the budget we have
(near 0.00$)... I know we could use FTP (there are plenty of free server
softwares out there), but we could not use all the other features an
asp.net has to offer such as the ease of use and data binding and
everything. .. right now I would like to be able to put these files into
a database, which is possible with the FileUpload control, but because
of the "big files" requirement, I just can't use it...

Anybody have ideas?

Thanks

ThunderMusi c

[mcrose]

I was working on a component to allow a "multi-post upload"; I had to
set it down for the moment in order to take care of more urgent work,
but I intend to get back on it next week.

It is a fairly simple server side script but it does require a client
side component that I currently have coded as a VB Application and
needs to be recoded as a downloadable .net component (it would be even
if this was a java applet, for cross browser compatability)

It was working, and the basic design is in place, but it could use some
code review and exception handling. It is pretty simple and should
allow for arbitrary small posts, and abitrarly large uploads, limitted
only by disk availability.

If anyone uses this, or improves it, please let me know so I can
benefit from the code review.

Basic concept:
1. client creates a connection and gets a hash from server
2. client reads file and post "postSize" bytes - server side opens file
associated with hash, seek() to offset sent by client and writes the
data sent. -repeat
3. client sends a completed signal.

There could be some optimization if you kept the file open between
posts, but that brings up the

The files are:
gigaUploadClien t.vb - a class that implements the client
UploadServlet.a spx
UploadServlet.a spx.vb - implements the server side storage
uploadTester.vb - The code behind a form demonstrating how to use the
client.

uploadTester.vb :
Public Class uploadTester

Private _uploader As gigaUploadClien t

Private Sub btnSend_Click(B yVal sender As System.Object, ByVal e As
System.EventArg s) Handles btnSend.Click
btnSend.Enabled = False
Dim postSize As Integer
If Not Integer.TryPars e(txtPostSizeKB .Text, postSize) Then
MessageBox.Show ("<" + txtPostSizeKB.T ext + "is not a
valid integer!")
Return
End If
_uploader = New gigaUploadClien t(txtFile.Text, txtUrl.Text,
postSize)
Dim upldThrd As New Threading.Threa d(AddressOf
_uploader.SendF ile)
upldThrd.Start( )
End Sub

Private Sub btnUpdate_Click (ByVal sender As System.Object, ByVal e
As System.EventArg s) Handles btnUpdate.Click
lblProgress.Tex t = _uploader.GetPr ogress + " : " +
Math.Round(_upl oader.GetProgre ssFraction * 100).ToString + "%"
btnSend.Enabled = Not _uploader.IsRun ning()
End Sub
End Class

*************** *************** *************** *************** *************** *************** **
UploadServlet.a spx:
<%@ Page Language="VB" AutoEventWireup ="false"
CodeFile="Uploa dServlet.aspx.v b" Inherits="_Uplo adServlet" %>

*************** *************** *************** *************** *************** *************** **
Imports System
Imports System.IO

Partial Class _UploadServlet
Inherits System.Web.UI.P age
Private savePath As String
Private Const readBlockSize As Integer = 10240

Public Sub New()
Dim settings As Configuration.A ppSettingsReade r = New
Configuration.A ppSettingsReade r()
Try
savePath = settings.GetVal ue("savePath", GetType(String) )
Catch
savePath = "c:\temp\upload er"
End Try
Try
If Not Directory.Exist s(savePath) Then
Directory.Creat eDirectory(save Path)
End If
Catch ex As Exception

End Try
End Sub

Protected Sub Page_Load(ByVal sender As Object, ByVal e As
System.EventArg s) Handles Me.Load
ProcessUploadRe quest()
End Sub

Private Sub restartUpload(B yVal fpath As String)
'/* Find the seek position */
Dim seekPos As Long = 0
Try
seekPos = Long.Parse(Requ est("offset"))
Catch nex As Exception
Response.Write( "-1")
Return
End Try

Dim fout As FileStream
If seekPos 0 Then
'* Open a RandomAccessFil e so that we can seek on it.
fout = File.Create(fpa th, FileMode.OpenOr Create,
FileOptions.Ran domAccess)
'this seek call takes significant time when the file is
big, so only do
'it if we get a seekposition which suggest a resume upload
fout.Seek(seekP os, IO.SeekOrigin.B egin)
'truncate the file to the restart position
fout.Close()
Response.Output .Write(seekPos. ToString)
Return
Else
Response.Write( "-1")
Return
End If
End Sub

Private Sub SpoolFile(ByVal fpath As String)

Dim instr As Stream = Request.InputSt ream

If (Not Directory.Exist s(Path.GetDirec toryName(fpath) )) Then
Try
Directory.Creat eDirectory(Path .GetDirectoryNa me(fpath))
Catch ex As Exception
End Try
End If

Dim fout As FileStream
If (File.Exists(fp ath)) Then
fout = File.OpenWrite( fpath)
fout.Seek(0, SeekOrigin.End)
Else
fout = File.Create(fpa th, FileMode.Create New)
End If
Dim b(readBlockSize ) As Byte
Dim i As Integer

Do
i = Request.InputSt ream.Read(b, 0, readBlockSize)
fout.Write(b, 0, i)
Diagnostics.Deb ug.Print(Text.E ncoding.ASCII.G etString(b, 0,
i))
Loop While (i = readBlockSize)
'Response.Write (fout.Length)
fout.Close()

End Sub

Public Sub ProcessUploadRe quest()
Try
Response.Conten tType = "text/html;charset=UT F-8"

Dim hash As String = Request("hash")
Dim cmd As String = Request("cmd")
Dim fname As String = Request("fname" )
If "".Equals(fname ) Or fname Is Nothing Then fname =
"downloadfile.d at"

Diagnostics.Deb ug.Print("cmd ::" + cmd)
Diagnostics.Deb ug.Print("fname ::" + fname)

If hash Is Nothing Then hash = "0" +
Guid.NewGuid(). ToString + "1"
If ("").Equals(has h) Then hash = "0" +
Guid.NewGuid(). ToString + "1"
Dim fpath As String = Path.Combine(Pa th.Combine(save Path,
hash), fname)

Diagnostics.Deb ug.Print("hash ::" + hash)

Select Case cmd

Case "getconnect ion"
Response.Output .Write(hash)
Response.Output .Close()
Return

Case "getstatus"
'this is and extra step for the current
configuration
'but it is useful if we want to implement a resume
upload.
'this is not currently implemented.
Diagnostics.Deb ug.Print("writi ng file to " + fpath)

If (File.Exists(fp ath)) Then
'* The file exists on the server. A previous
upload has failed.
'* report back how many bytes have been written
Response.Output .Write(New
FileInfo(fpath) .Length())
Else
'* report back that zero bytes have been
written
Response.Output .Write("0")
Response.Output .Close()
End If

Case "completeupload "
' this is where we could do any clean up or file
moving that we want

Response.Output .Write("Success . Bytes written: " +
FileLen(fpath). ToString)
Return

Case "restart"
restartUpload(f path)
Response.Output .Close()
Return
Case "spoolfile"
'this is the case of spooling the file
If (hash Is Nothing) Then
Response.Output .Write("null hash, null cmd is
not valid")
End If
SpoolFile(fpath )
Response.Output .Close()
Return
End Select

Catch e As Exception
Response.Output .Write("failed" )
Response.Output .Write(e.Messag e)
Diagnostics.Deb ug.Print(e.Mess age)
End Try
End Sub
End Class
*************** *************** *************** *************** *************** *************** *****

Imports System
Imports System.IO
Imports System.Net

Public Class gigaUploadClien t
Private _cli As WebClient
Private _hash As String
Private _fileSize As Long
Private _reader As IO.Stream
Private _uploadUrl As String
Private _sourceFile As String
Private _postSize As Long
Private _curPos As Long = 0
Private _lockObj As New Object
Private _isRunning As Boolean = False

Public Sub New(ByVal sourceFile As String, ByVal uploadUrl As
String, ByVal postSize As Integer)
_uploadUrl = uploadUrl
_sourceFile = sourceFile
_postSize = postSize
End Sub
Public Sub SendFile()
_isRunning = True
_cli = New WebClient()

_cli.QueryStrin g.Add("cmd", "getconnection" )
'cli.QueryStrin g.Add("hash", "")
_hash = _cli.DownloadSt ring(_uploadUrl )

'begin hack
'trim off any whitespace. think this was because I used a
WriteLine() on the servlet. duh!
_hash.TrimEnd(C onstants.vbCrLf .ToCharArray)
If _hash.EndsWith( Constants.vbCrL f) Then
_hash = _hash.Substring (0, _hash.Length - 2)
End If
'end hack

Dim fileOffset As Long = getStatus()
If fileOffset 0 Then
restartTransfer (fileOffset)
End If
spoolFile()
_isRunning = False
End Sub

Private Function getStatus() As Long
_cli.QueryStrin g.Clear()
_cli.QueryStrin g.Add("cmd", "getstatus" )
_cli.QueryStrin g.Add("hash", _hash)
_cli.QueryStrin g.Add("fname", Path.GetFileNam e(_sourceFile))

Dim fileOffsetStr As String = _cli.DownloadSt ring(_uploadUrl )
Dim fileOffset As Long
If Long.TryParse(f ileOffsetStr, fileOffset) Then
Return fileOffset
Else
Return 0
End If
End Function

Private Sub restartTransfer (ByVal offset As Long)
_cli.QueryStrin g.Clear()
_cli.QueryStrin g.Add("cmd", "restart")
_cli.QueryStrin g.Add("hash", _hash)
_cli.QueryStrin g.Add("fname", Path.GetFileNam e(_sourceFile))

Dim fileOffsetStr As String = _cli.DownloadSt ring(_uploadUrl )
SyncLock _lockObj
_curPos = offset
End SyncLock
End Sub

Private Sub confirmFileFini shed()
_cli.QueryStrin g.Clear()
_cli.QueryStrin g.Add("cmd", "completeupload ")
_cli.QueryStrin g.Add("fname", Path.GetFileNam e(_sourceFile))
_cli.QueryStrin g.Add("hash", _hash)
Dim resp As String = _cli.DownloadSt ring(_uploadUrl )
End Sub

Private Sub spoolFile()
Dim out As IO.Stream
_fileSize = FileLen(_source File)

Const readBufferSize As Integer = 4096
Dim b(readBufferSiz e) As Byte
Dim postSize As Integer = _postSize * 1024
Dim i As Integer
Dim bytesSpooled As Long = 0

_reader = File.OpenRead(_ sourceFile)
If _curPos 0 Then
_reader.Seek(_c urPos, SeekOrigin.Begi n)
End If

Do
_cli.QueryStrin g.Clear()
_cli.QueryStrin g.Add("cmd", "spoolfile" )
_cli.QueryStrin g.Add("hash", _hash)
_cli.QueryStrin g.Add("fname",
Path.GetFileNam e(_sourceFile))
_cli.Headers.Ad d("Content-Type",
"applicatio n/octet-stream")
'don't write offset unless we need to resume an upload
'cli.QueryStrin g.Add("offset", curPos.ToString )
out = _cli.OpenWrite( _uploadUrl)
Dim bytesSpooledThi sPost As Integer = 0
Do
i = _reader.Read(b, 0, readBufferSize)
out.Write(b, 0, i)
bytesSpooledThi sPost += i
bytesSpooled += i
SyncLock _lockObj
_curPos = bytesSpooled
End SyncLock
Loop While (i = readBufferSize And bytesSpooledThi sPost <
postSize)
out.Close()
Loop While (i = readBufferSize) ' i will be less then
readBufferSize when we reach the end of the file
End Sub

Public Function GetProgressFrac tion() As Double
SyncLock _lockObj
Return _curPos / _fileSize
End SyncLock
End Function

Public Function GetProgress() As String
SyncLock _lockObj
Return Math.Round(_cur Pos / 1024).ToString + "kb of " +
Math.Round(_fil eSize / 1024).ToString + "kb transfered"
End SyncLock
End Function

Public Function IsRunning() As Boolean
Return _isRunning
End Function
End Class

[mcrose]

Ok, piss-poor editing on that last post.
If anyone wants the files, let me know, and I can email them.

[Brian P. Hammer]

Please pass them along... bhammer at leas dot com

Thanks,
Brian

"mcrose" <ma*********@gm ail.comwrote in message
news:11******** *************@d 34g2000cwd.goog legroups.com...

Ok, piss-poor editing on that last post.
If anyone wants the files, let me know, and I can email them.