473,287 Members | 1,659 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,287 software developers and data experts.

How to design a file system to store uploaded files via ASP.NET

I'd like to set up a file system for the ASP.NET 2.0 application
to store user-uploaded files, since the members are more than
100,000 people, the basic requirements are as below:

(1) The file system is separate with front-end web site
(2) Need to re-size user-uploaded image file to same size
(3) Need to rename file name to avoid duplicate name
(4) How to design directory and file structure to increase disk I/O
performance
(5) How to scale the file system by adding more HD or file server
(6) What kind of file server? Window 2003 or Linux

Any suggestion? thanks in advance.
Feb 12 '06 #1
4 2515
RedHair,

(1) I would recommend that you don't permit direct access to the filesystem.
Perhaps consider using a database (e.g. SQL Server) to store the files
instead. If you wanted direct file system access, you would need to
configure special permissions to allow the web app security the correct
context for writing to the web server disks. This can pose a big security
risk, no matter which OS is used.

Using SQL Server datatype "image" (sometimes referred to as BLOB, from
Binary Large OBject), you can have up to a 2Gb file in each record, so
there's plenty of storage potential.

(2 / 3 / 4) Using a DB would also remove problems 2, 3 and 4. For (3) you
would use a unique key (primary key) for each file record created in the
storage table. This means you can have duplicate file names if necessary
without conflict. You would need to consider adding a supplemental column
to the table to hold some other value that would allow users to distinguish
the files if same name used. This could be the UserID/name of the person
uploading, or the date-time stamp of the upload, or use both for example.

For point 2, do you mean a file that is a photo, or any type of file such as
a document/spreadsheet?

(5) If you use a hot-swap capable RAID based server or NAS, you should be
able to add more / larger disks and grow your RAID partition into the new
space.

(1 / 6) Since you're posting in the dotnet forums, we can probably assume
you're at least considering to use .NET to create this app, so you'll
probably be wanting a Windows host server. Windows Server 2003 Web Edition
is perfect for this as it is much cheaper than Standard (or Enterprise)
versions and you don't need to buy separate licences. Also, this can host
an MSDE version of SQL server if required to keep infrastructure costs down
to bare minimum, though with your stated user audience, you should probably
consider a separate dedicated DB server. (E.g. SQL Server with CPU
licence(s) for Internet use).

Hope that helps.

Al
"RedHair" <re*****@ms40.url.com.tw> wrote in message
news:uo*************@TK2MSFTNGP12.phx.gbl...
I'd like to set up a file system for the ASP.NET 2.0 application
to store user-uploaded files, since the members are more than
100,000 people, the basic requirements are as below:

(1) The file system is separate with front-end web site
(2) Need to re-size user-uploaded image file to same size
(3) Need to rename file name to avoid duplicate name
(4) How to design directory and file structure to increase disk I/O
performance
(5) How to scale the file system by adding more HD or file server
(6) What kind of file server? Window 2003 or Linux

Any suggestion? thanks in advance.

Feb 12 '06 #2
Thanks for your detailed reply.
Use db instead of file system is the original idea, but many articles
mentioned
that read/write binary from db costs the performance.
Besides, it is possible to put db generated image file into the cache either
in server or client side?

"Alec MacLean" <al**********@NO19SPAM60.copeohs.com> 撰寫於郵件新聞:e9**************@TK2MSFTNGP11.phx.g bl...
RedHair,

(1) I would recommend that you don't permit direct access to the
filesystem. Perhaps consider using a database (e.g. SQL Server) to store
the files instead. If you wanted direct file system access, you would
need to configure special permissions to allow the web app security the
correct context for writing to the web server disks. This can pose a big
security risk, no matter which OS is used.

Using SQL Server datatype "image" (sometimes referred to as BLOB, from
Binary Large OBject), you can have up to a 2Gb file in each record, so
there's plenty of storage potential.

(2 / 3 / 4) Using a DB would also remove problems 2, 3 and 4. For (3) you
would use a unique key (primary key) for each file record created in the
storage table. This means you can have duplicate file names if necessary
without conflict. You would need to consider adding a supplemental column
to the table to hold some other value that would allow users to
distinguish the files if same name used. This could be the UserID/name of
the person uploading, or the date-time stamp of the upload, or use both
for example.

For point 2, do you mean a file that is a photo, or any type of file such
as a document/spreadsheet?

(5) If you use a hot-swap capable RAID based server or NAS, you should be
able to add more / larger disks and grow your RAID partition into the new
space.

(1 / 6) Since you're posting in the dotnet forums, we can probably assume
you're at least considering to use .NET to create this app, so you'll
probably be wanting a Windows host server. Windows Server 2003 Web
Edition is perfect for this as it is much cheaper than Standard (or
Enterprise) versions and you don't need to buy separate licences. Also,
this can host an MSDE version of SQL server if required to keep
infrastructure costs down to bare minimum, though with your stated user
audience, you should probably consider a separate dedicated DB server.
(E.g. SQL Server with CPU licence(s) for Internet use).

Hope that helps.

Al
"RedHair" <re*****@ms40.url.com.tw> wrote in message
news:uo*************@TK2MSFTNGP12.phx.gbl...
I'd like to set up a file system for the ASP.NET 2.0 application
to store user-uploaded files, since the members are more than
100,000 people, the basic requirements are as below:

(1) The file system is separate with front-end web site
(2) Need to re-size user-uploaded image file to same size
(3) Need to rename file name to avoid duplicate name
(4) How to design directory and file structure to increase disk I/O
performance
(5) How to scale the file system by adding more HD or file server
(6) What kind of file server? Window 2003 or Linux

Any suggestion? thanks in advance.


Feb 13 '06 #3
I've not read that deeply into the performance issues myself, not having had
to produce an app of this type (though one is planned). I'll need to do
some further reading myself, it seems!

I have been working on an app that uploads photo images (HR system, staff
photo) with no discernable speed issues, but this is a Win-forms app used on
local LAN, so direct comparison is probably not a fair one.

It seems logical that there would be a performance hit while the binary file
is "translated" into the stream required to up/down-load, though I don't
know how much impact that has. Probably varies a lot with hardware being
used as well the number of user hits per second/minute. You'd need to weigh
the pro's and cons of the available approaches (between File I/O and DB I/O)
to judge which will best suit your current project's needs.

I've not used caching myself as of this time, but I wouldn't see a technical
barrier with caching at either end. I'm more than happy to be corrected /
illuminated on this though!

Al
"RedHair" <re*****@ms40.url.com.tw> wrote in message
news:%2****************@TK2MSFTNGP14.phx.gbl...
Thanks for your detailed reply.
Use db instead of file system is the original idea, but many articles
mentioned
that read/write binary from db costs the performance.
Besides, it is possible to put db generated image file into the cache
either
in server or client side?

Feb 13 '06 #4
Thanks!
"Alec MacLean" <al**********@NO19SPAM60.copeohs.com> 撰寫於郵件新聞:em*************@TK2MSFTNGP12.phx.gb l...
I've not read that deeply into the performance issues myself, not having
had to produce an app of this type (though one is planned). I'll need to
do some further reading myself, it seems!

I have been working on an app that uploads photo images (HR system, staff
photo) with no discernable speed issues, but this is a Win-forms app used
on local LAN, so direct comparison is probably not a fair one.

It seems logical that there would be a performance hit while the binary
file is "translated" into the stream required to up/down-load, though I
don't know how much impact that has. Probably varies a lot with hardware
being used as well the number of user hits per second/minute. You'd need
to weigh the pro's and cons of the available approaches (between File I/O
and DB I/O) to judge which will best suit your current project's needs.

I've not used caching myself as of this time, but I wouldn't see a
technical barrier with caching at either end. I'm more than happy to be
corrected / illuminated on this though!

Al
"RedHair" <re*****@ms40.url.com.tw> wrote in message
news:%2****************@TK2MSFTNGP14.phx.gbl...
Thanks for your detailed reply.
Use db instead of file system is the original idea, but many articles
mentioned
that read/write binary from db costs the performance.
Besides, it is possible to put db generated image file into the cache
either
in server or client side?


Feb 21 '06 #5

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

7
by: pescott | last post by:
I am still struggling to get some files uploaded to a database as BLOB data. I have 5 includes which allows the user to upload 5 files, numbered accordingly. However, there are script errors. Duhh,...
9
by: | last post by:
hi.. i don't know if this is the right group for this.. i have a small application which involves a windows service, and web services which bascally retrieves files, process them and zips them...
13
by: Sky Sigal | last post by:
I have created an IHttpHandler that waits for uploads as attachments for a webmail interface, and saves it to a directory that is defined in config.xml. My question is the following: assuming...
9
by: Arsen V. | last post by:
Hello, What is the suggested way to store uploaded files? 1) IMAGE type data in an SQL Database table 2) As a file in the NTFS file system Thanks, Arsen
1
by: laredotornado | last post by:
Hi, I'm using PHP 4.4.4 on Apache 2 on Fedora Core 5. PHP was installed using Apache's apxs and the php library was installed to /usr/local/php. However, when I set my "error_reporting"...
12
by: SAL | last post by:
Hello, Is it possible to read a CSV from the Client, and bind my Datagrid to the data in the CSV file without uploading the file to the Server first? I have tried and in Debug mode on my...
3
by: Stephan | last post by:
Hi all, I am new to access and I face the following "issue": I would like to create a database, to which users can upload files (=pdf, doc, xls...). The files shall be stored locally on a...
5
by: djhexx | last post by:
Hello. I have an ASP.NET application (C#) that I allow users to upload files. Files get stored in a SQL2005 database. The file data is stored in a varbinary(max) column. When the user uploads...
1
by: Nosferatum | last post by:
Before I start and try to solve my problem, I wonder if anyone can help me with my challenge: I want to make a system to upload files (mostly word and pdf-files), store them in the file system and...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, youll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.