Uploading big files

[ThunderMusic]

Hi,
I'm currently building a little web site for me and my friends where we
would like to be able to post big files (10Mb to 250Mb approx.) Right now
the ASP.Net UploadFile control can't do it as we are limited 1) by the
MaxRequestLength (4Mb) and 2) with big files (such as 250Mb) the server
crashes because it only uses memory to cache the upload...

Is there an easy way to develop an HTTP Module to handle the upload (links,
tutorials, ...)? I've seen components to do this on the net, but they are
too expensive for what we need to do and the budget we have (near 0.00$)...
I know we could use FTP (there are plenty of free server softwares out
there), but we could not use all the other features an asp.net has to offer
such as the ease of use and data binding and everything... right now I
would like to be able to put these files into a database, which is possible
with the FileUpload control, but because of the "big files" requirement, I
just can't use it...

Anybody have ideas?

Thanks

ThunderMusic

[Gabriele G. Ponti]

The MaxRequestLength limit can be changed in the web.config file.

<configuration>
<system.web>
<httpRuntime maxRequestLength="40960" />
</system.web>
</configuration>

Please see this document:
http://msdn.microsoft.com/library/de...imeSection.asp

"ThunderMusic" <No*************************@NoSpAm.comwrote in message
news:%2***************@TK2MSFTNGP06.phx.gbl...

Hi,
I'm currently building a little web site for me and my friends where we
would like to be able to post big files (10Mb to 250Mb approx.) Right now
the ASP.Net UploadFile control can't do it as we are limited 1) by the
MaxRequestLength (4Mb) and 2) with big files (such as 250Mb) the server
crashes because it only uses memory to cache the upload...

Is there an easy way to develop an HTTP Module to handle the upload
(links, tutorials, ...)? I've seen components to do this on the net, but
they are too expensive for what we need to do and the budget we have (near
0.00$)... I know we could use FTP (there are plenty of free server
softwares out there), but we could not use all the other features an
asp.net has to offer such as the ease of use and data binding and
everything... right now I would like to be able to put these files into a
database, which is possible with the FileUpload control, but because of
the "big files" requirement, I just can't use it...

Anybody have ideas?

Thanks

ThunderMusic

[ThunderMusic]

I know this part... We changed it, but there are 2 problems with this 1) it
opens the server for Denial Of Service attacks and 2)even if I put it to 2Gb
and upload a file of 200Mb, the server crashes because the FileUpload
control caches the request entirely in memory...

Thanks

ThunderMusic
"Gabriele G. Ponti" <ggponti.at.hotmail.comwrote in message
news:O2**************@TK2MSFTNGP05.phx.gbl...

The MaxRequestLength limit can be changed in the web.config file.

<configuration>
<system.web>
<httpRuntime maxRequestLength="40960" />
</system.web>
</configuration>

Please see this document:
http://msdn.microsoft.com/library/de...imeSection.asp

"ThunderMusic" <No*************************@NoSpAm.comwrote in message
news:%2***************@TK2MSFTNGP06.phx.gbl...

>Hi,
I'm currently building a little web site for me and my friends where we
would like to be able to post big files (10Mb to 250Mb approx.) Right now
the ASP.Net UploadFile control can't do it as we are limited 1) by the
MaxRequestLength (4Mb) and 2) with big files (such as 250Mb) the server
crashes because it only uses memory to cache the upload...

Is there an easy way to develop an HTTP Module to handle the upload
(links, tutorials, ...)? I've seen components to do this on the net, but
they are too expensive for what we need to do and the budget we have
(near 0.00$)... I know we could use FTP (there are plenty of free server
softwares out there), but we could not use all the other features an
asp.net has to offer such as the ease of use and data binding and
everything... right now I would like to be able to put these files into
a database, which is possible with the FileUpload control, but because of
the "big files" requirement, I just can't use it...

Anybody have ideas?

Thanks

ThunderMusic

[Brian P. Hammer]

I know you said your budget was $0 but I use a component called aspupload
that works great. I think the single server license is $150.00.
"ThunderMusic" <No*************************@NoSpAm.comwrote in message
news:%2***************@TK2MSFTNGP06.phx.gbl...

Hi,
I'm currently building a little web site for me and my friends where we
would like to be able to post big files (10Mb to 250Mb approx.) Right now
the ASP.Net UploadFile control can't do it as we are limited 1) by the
MaxRequestLength (4Mb) and 2) with big files (such as 250Mb) the server
crashes because it only uses memory to cache the upload...

Is there an easy way to develop an HTTP Module to handle the upload
(links, tutorials, ...)? I've seen components to do this on the net, but
they are too expensive for what we need to do and the budget we have (near
0.00$)... I know we could use FTP (there are plenty of free server
softwares out there), but we could not use all the other features an
asp.net has to offer such as the ease of use and data binding and
everything... right now I would like to be able to put these files into a
database, which is possible with the FileUpload control, but because of
the "big files" requirement, I just can't use it...

Anybody have ideas?

Thanks

ThunderMusic

[ThunderMusic]

it could be interesting... I'll look for it on google... (or maybe you have
to link to the home page og the publisher?)

I'm currently trying a free solution that is under the LGPL called :
NeatUpload... I found it a little before noon (Eastern time USA) I'll see if
it does the job... if it does, I'll use it, if it doesn't, I'll look for
your aspupload... ;)

thanks a lot...

ThunderMusic
"Brian P. Hammer" <Br*********@community.nospamwrote in message
news:Om**************@TK2MSFTNGP03.phx.gbl...

>I know you said your budget was $0 but I use a component called aspupload
that works great. I think the single server license is $150.00.
"ThunderMusic" <No*************************@NoSpAm.comwrote in message
news:%2***************@TK2MSFTNGP06.phx.gbl...

>Hi,
I'm currently building a little web site for me and my friends where we
would like to be able to post big files (10Mb to 250Mb approx.) Right now
the ASP.Net UploadFile control can't do it as we are limited 1) by the
MaxRequestLength (4Mb) and 2) with big files (such as 250Mb) the server
crashes because it only uses memory to cache the upload...

Is there an easy way to develop an HTTP Module to handle the upload
(links, tutorials, ...)? I've seen components to do this on the net, but
they are too expensive for what we need to do and the budget we have
(near 0.00$)... I know we could use FTP (there are plenty of free server
softwares out there), but we could not use all the other features an
asp.net has to offer such as the ease of use and data binding and
everything... right now I would like to be able to put these files into
a database, which is possible with the FileUpload control, but because of
the "big files" requirement, I just can't use it...

Anybody have ideas?

Thanks

ThunderMusic

[Brian P. Hammer]

http://www.aspupload.com/ is the site. I think their main site is at
http://www.persits.com/

HTH,
Brian

"ThunderMusic" <No*************************@NoSpAm.comwrote in message
news:%2*****************@TK2MSFTNGP02.phx.gbl...

it could be interesting... I'll look for it on google... (or maybe you
have to link to the home page og the publisher?)

I'm currently trying a free solution that is under the LGPL called :
NeatUpload... I found it a little before noon (Eastern time USA) I'll see
if it does the job... if it does, I'll use it, if it doesn't, I'll look
for your aspupload... ;)

thanks a lot...

ThunderMusic
"Brian P. Hammer" <Br*********@community.nospamwrote in message
news:Om**************@TK2MSFTNGP03.phx.gbl...

>>I know you said your budget was $0 but I use a component called aspupload
that works great. I think the single server license is $150.00.
"ThunderMusic" <No*************************@NoSpAm.comwrote in message
news:%2***************@TK2MSFTNGP06.phx.gbl...

>>Hi,
I'm currently building a little web site for me and my friends where we
would like to be able to post big files (10Mb to 250Mb approx.) Right
now the ASP.Net UploadFile control can't do it as we are limited 1) by
the MaxRequestLength (4Mb) and 2) with big files (such as 250Mb) the
server crashes because it only uses memory to cache the upload...

Is there an easy way to develop an HTTP Module to handle the upload
(links, tutorials, ...)? I've seen components to do this on the net, but
they are too expensive for what we need to do and the budget we have
(near 0.00$)... I know we could use FTP (there are plenty of free server
softwares out there), but we could not use all the other features an
asp.net has to offer such as the ease of use and data binding and
everything... right now I would like to be able to put these files into
a database, which is possible with the FileUpload control, but because
of the "big files" requirement, I just can't use it...

Anybody have ideas?

Thanks

ThunderMusic

[mcrose]

I was working on a component to allow a "multi-post upload"; I had to
set it down for the moment in order to take care of more urgent work,
but I intend to get back on it next week.

It is a fairly simple server side script but it does require a client
side component that I currently have coded as a VB Application and
needs to be recoded as a downloadable .net component (it would be even
if this was a java applet, for cross browser compatability)

It was working, and the basic design is in place, but it could use some
code review and exception handling. It is pretty simple and should
allow for arbitrary small posts, and abitrarly large uploads, limitted
only by disk availability.

If anyone uses this, or improves it, please let me know so I can
benefit from the code review.

Basic concept:
1. client creates a connection and gets a hash from server
2. client reads file and post "postSize" bytes - server side opens file
associated with hash, seek() to offset sent by client and writes the
data sent. -repeat
3. client sends a completed signal.

There could be some optimization if you kept the file open between
posts, but that brings up the

The files are:
gigaUploadClient.vb - a class that implements the client
UploadServlet.aspx
UploadServlet.aspx.vb - implements the server side storage
uploadTester.vb - The code behind a form demonstrating how to use the
client.

uploadTester.vb:
Public Class uploadTester

Private _uploader As gigaUploadClient

Private Sub btnSend_Click(ByVal sender As System.Object, ByVal e As
System.EventArgs) Handles btnSend.Click
btnSend.Enabled = False
Dim postSize As Integer
If Not Integer.TryParse(txtPostSizeKB.Text, postSize) Then
MessageBox.Show("<" + txtPostSizeKB.Text + "is not a
valid integer!")
Return
End If
_uploader = New gigaUploadClient(txtFile.Text, txtUrl.Text,
postSize)
Dim upldThrd As New Threading.Thread(AddressOf
_uploader.SendFile)
upldThrd.Start()
End Sub

Private Sub btnUpdate_Click(ByVal sender As System.Object, ByVal e
As System.EventArgs) Handles btnUpdate.Click
lblProgress.Text = _uploader.GetProgress + " : " +
Math.Round(_uploader.GetProgressFraction * 100).ToString + "%"
btnSend.Enabled = Not _uploader.IsRunning()
End Sub
End Class

************************************************** ******************************************
UploadServlet.aspx:
<%@ Page Language="VB" AutoEventWireup="false"
CodeFile="UploadServlet.aspx.vb" Inherits="_UploadServlet" %>

************************************************** ******************************************
Imports System
Imports System.IO

Partial Class _UploadServlet
Inherits System.Web.UI.Page
Private savePath As String
Private Const readBlockSize As Integer = 10240

Public Sub New()
Dim settings As Configuration.AppSettingsReader = New
Configuration.AppSettingsReader()
Try
savePath = settings.GetValue("savePath", GetType(String))
Catch
savePath = "c:\temp\uploader"
End Try
Try
If Not Directory.Exists(savePath) Then
Directory.CreateDirectory(savePath)
End If
Catch ex As Exception

End Try
End Sub

Protected Sub Page_Load(ByVal sender As Object, ByVal e As
System.EventArgs) Handles Me.Load
ProcessUploadRequest()
End Sub

Private Sub restartUpload(ByVal fpath As String)
'/* Find the seek position */
Dim seekPos As Long = 0
Try
seekPos = Long.Parse(Request("offset"))
Catch nex As Exception
Response.Write("-1")
Return
End Try

Dim fout As FileStream
If seekPos 0 Then
'* Open a RandomAccessFile so that we can seek on it.
fout = File.Create(fpath, FileMode.OpenOrCreate,
FileOptions.RandomAccess)
'this seek call takes significant time when the file is
big, so only do
'it if we get a seekposition which suggest a resume upload
fout.Seek(seekPos, IO.SeekOrigin.Begin)
'truncate the file to the restart position
fout.Close()
Response.Output.Write(seekPos.ToString)
Return
Else
Response.Write("-1")
Return
End If
End Sub

Private Sub SpoolFile(ByVal fpath As String)

Dim instr As Stream = Request.InputStream

If (Not Directory.Exists(Path.GetDirectoryName(fpath))) Then
Try
Directory.CreateDirectory(Path.GetDirectoryName(fp ath))
Catch ex As Exception
End Try
End If

Dim fout As FileStream
If (File.Exists(fpath)) Then
fout = File.OpenWrite(fpath)
fout.Seek(0, SeekOrigin.End)
Else
fout = File.Create(fpath, FileMode.CreateNew)
End If
Dim b(readBlockSize) As Byte
Dim i As Integer

Do
i = Request.InputStream.Read(b, 0, readBlockSize)
fout.Write(b, 0, i)
Diagnostics.Debug.Print(Text.Encoding.ASCII.GetStr ing(b, 0,
i))
Loop While (i = readBlockSize)
'Response.Write(fout.Length)
fout.Close()

End Sub

Public Sub ProcessUploadRequest()
Try
Response.ContentType = "text/html;charset=UTF-8"

Dim hash As String = Request("hash")
Dim cmd As String = Request("cmd")
Dim fname As String = Request("fname")
If "".Equals(fname) Or fname Is Nothing Then fname =
"downloadfile.dat"

Diagnostics.Debug.Print("cmd ::" + cmd)
Diagnostics.Debug.Print("fname ::" + fname)

If hash Is Nothing Then hash = "0" +
Guid.NewGuid().ToString + "1"
If ("").Equals(hash) Then hash = "0" +
Guid.NewGuid().ToString + "1"
Dim fpath As String = Path.Combine(Path.Combine(savePath,
hash), fname)

Diagnostics.Debug.Print("hash ::" + hash)

Select Case cmd

Case "getconnection"
Response.Output.Write(hash)
Response.Output.Close()
Return

Case "getstatus"
'this is and extra step for the current
configuration
'but it is useful if we want to implement a resume
upload.
'this is not currently implemented.
Diagnostics.Debug.Print("writing file to " + fpath)

If (File.Exists(fpath)) Then
'* The file exists on the server. A previous
upload has failed.
'* report back how many bytes have been written
Response.Output.Write(New
FileInfo(fpath).Length())
Else
'* report back that zero bytes have been
written
Response.Output.Write("0")
Response.Output.Close()
End If

Case "completeupload"
' this is where we could do any clean up or file
moving that we want

Response.Output.Write("Success. Bytes written: " +
FileLen(fpath).ToString)
Return

Case "restart"
restartUpload(fpath)
Response.Output.Close()
Return
Case "spoolfile"
'this is the case of spooling the file
If (hash Is Nothing) Then
Response.Output.Write("null hash, null cmd is
not valid")
End If
SpoolFile(fpath)
Response.Output.Close()
Return
End Select

Catch e As Exception
Response.Output.Write("failed")
Response.Output.Write(e.Message)
Diagnostics.Debug.Print(e.Message)
End Try
End Sub
End Class
************************************************** *********************************************

Imports System
Imports System.IO
Imports System.Net

Public Class gigaUploadClient
Private _cli As WebClient
Private _hash As String
Private _fileSize As Long
Private _reader As IO.Stream
Private _uploadUrl As String
Private _sourceFile As String
Private _postSize As Long
Private _curPos As Long = 0
Private _lockObj As New Object
Private _isRunning As Boolean = False

Public Sub New(ByVal sourceFile As String, ByVal uploadUrl As
String, ByVal postSize As Integer)
_uploadUrl = uploadUrl
_sourceFile = sourceFile
_postSize = postSize
End Sub
Public Sub SendFile()
_isRunning = True
_cli = New WebClient()

_cli.QueryString.Add("cmd", "getconnection")
'cli.QueryString.Add("hash", "")
_hash = _cli.DownloadString(_uploadUrl)

'begin hack
'trim off any whitespace. think this was because I used a
WriteLine() on the servlet. duh!
_hash.TrimEnd(Constants.vbCrLf.ToCharArray)
If _hash.EndsWith(Constants.vbCrLf) Then
_hash = _hash.Substring(0, _hash.Length - 2)
End If
'end hack

Dim fileOffset As Long = getStatus()
If fileOffset 0 Then
restartTransfer(fileOffset)
End If
spoolFile()
_isRunning = False
End Sub

Private Function getStatus() As Long
_cli.QueryString.Clear()
_cli.QueryString.Add("cmd", "getstatus")
_cli.QueryString.Add("hash", _hash)
_cli.QueryString.Add("fname", Path.GetFileName(_sourceFile))

Dim fileOffsetStr As String = _cli.DownloadString(_uploadUrl)
Dim fileOffset As Long
If Long.TryParse(fileOffsetStr, fileOffset) Then
Return fileOffset
Else
Return 0
End If
End Function

Private Sub restartTransfer(ByVal offset As Long)
_cli.QueryString.Clear()
_cli.QueryString.Add("cmd", "restart")
_cli.QueryString.Add("hash", _hash)
_cli.QueryString.Add("fname", Path.GetFileName(_sourceFile))

Dim fileOffsetStr As String = _cli.DownloadString(_uploadUrl)
SyncLock _lockObj
_curPos = offset
End SyncLock
End Sub

Private Sub confirmFileFinished()
_cli.QueryString.Clear()
_cli.QueryString.Add("cmd", "completeupload")
_cli.QueryString.Add("fname", Path.GetFileName(_sourceFile))
_cli.QueryString.Add("hash", _hash)
Dim resp As String = _cli.DownloadString(_uploadUrl)
End Sub

Private Sub spoolFile()
Dim out As IO.Stream
_fileSize = FileLen(_sourceFile)

Const readBufferSize As Integer = 4096
Dim b(readBufferSize) As Byte
Dim postSize As Integer = _postSize * 1024
Dim i As Integer
Dim bytesSpooled As Long = 0

_reader = File.OpenRead(_sourceFile)
If _curPos 0 Then
_reader.Seek(_curPos, SeekOrigin.Begin)
End If

Do
_cli.QueryString.Clear()
_cli.QueryString.Add("cmd", "spoolfile")
_cli.QueryString.Add("hash", _hash)
_cli.QueryString.Add("fname",
Path.GetFileName(_sourceFile))
_cli.Headers.Add("Content-Type",
"application/octet-stream")
'don't write offset unless we need to resume an upload
'cli.QueryString.Add("offset", curPos.ToString)
out = _cli.OpenWrite(_uploadUrl)
Dim bytesSpooledThisPost As Integer = 0
Do
i = _reader.Read(b, 0, readBufferSize)
out.Write(b, 0, i)
bytesSpooledThisPost += i
bytesSpooled += i
SyncLock _lockObj
_curPos = bytesSpooled
End SyncLock
Loop While (i = readBufferSize And bytesSpooledThisPost <
postSize)
out.Close()
Loop While (i = readBufferSize) ' i will be less then
readBufferSize when we reach the end of the file
End Sub

Public Function GetProgressFraction() As Double
SyncLock _lockObj
Return _curPos / _fileSize
End SyncLock
End Function

Public Function GetProgress() As String
SyncLock _lockObj
Return Math.Round(_curPos / 1024).ToString + "kb of " +
Math.Round(_fileSize / 1024).ToString + "kb transfered"
End SyncLock
End Function

Public Function IsRunning() As Boolean
Return _isRunning
End Function
End Class

[mcrose]

Ok, piss-poor editing on that last post.
If anyone wants the files, let me know, and I can email them.

[Brian P. Hammer]

Please pass them along... bhammer at leas dot com

Thanks,
Brian

"mcrose" <ma*********@gmail.comwrote in message
news:11*********************@d34g2000cwd.googlegro ups.com...

Ok, piss-poor editing on that last post.
If anyone wants the files, let me know, and I can email them.