How to design a file system to store uploaded files via ASP.NET

[RedHair]

I'd like to set up a file system for the ASP.NET 2.0 application
to store user-uploaded files, since the members are more than
100,000 people, the basic requirements are as below:

(1) The file system is separate with front-end web site
(2) Need to re-size user-uploaded image file to same size
(3) Need to rename file name to avoid duplicate name
(4) How to design directory and file structure to increase disk I/O
performance
(5) How to scale the file system by adding more HD or file server
(6) What kind of file server? Window 2003 or Linux

Any suggestion? thanks in advance.

[Alec MacLean]

RedHair,

(1) I would recommend that you don't permit direct access to the filesystem.
Perhaps consider using a database (e.g. SQL Server) to store the files
instead. If you wanted direct file system access, you would need to
configure special permissions to allow the web app security the correct
context for writing to the web server disks. This can pose a big security
risk, no matter which OS is used.

Using SQL Server datatype "image" (sometimes referred to as BLOB, from
Binary Large OBject), you can have up to a 2Gb file in each record, so
there's plenty of storage potential.

(2 / 3 / 4) Using a DB would also remove problems 2, 3 and 4. For (3) you
would use a unique key (primary key) for each file record created in the
storage table. This means you can have duplicate file names if necessary
without conflict. You would need to consider adding a supplemental column
to the table to hold some other value that would allow users to distinguish
the files if same name used. This could be the UserID/name of the person
uploading, or the date-time stamp of the upload, or use both for example.

For point 2, do you mean a file that is a photo, or any type of file such as
a document/spreadsheet?

(5) If you use a hot-swap capable RAID based server or NAS, you should be
able to add more / larger disks and grow your RAID partition into the new
space.

(1 / 6) Since you're posting in the dotnet forums, we can probably assume
you're at least considering to use .NET to create this app, so you'll
probably be wanting a Windows host server. Windows Server 2003 Web Edition
is perfect for this as it is much cheaper than Standard (or Enterprise)
versions and you don't need to buy separate licences. Also, this can host
an MSDE version of SQL server if required to keep infrastructure costs down
to bare minimum, though with your stated user audience, you should probably
consider a separate dedicated DB server. (E.g. SQL Server with CPU
licence(s) for Internet use).

Hope that helps.

Al
"RedHair" <re*****@ms40.url.com.tw> wrote in message
news:uo*************@TK2MSFTNGP12.phx.gbl...

I'd like to set up a file system for the ASP.NET 2.0 application
to store user-uploaded files, since the members are more than
100,000 people, the basic requirements are as below:

(1) The file system is separate with front-end web site
(2) Need to re-size user-uploaded image file to same size
(3) Need to rename file name to avoid duplicate name
(4) How to design directory and file structure to increase disk I/O
performance
(5) How to scale the file system by adding more HD or file server
(6) What kind of file server? Window 2003 or Linux

Any suggestion? thanks in advance.

[RedHair]

Thanks for your detailed reply.
Use db instead of file system is the original idea, but many articles
mentioned
that read/write binary from db costs the performance.
Besides, it is possible to put db generated image file into the cache either
in server or client side?

"Alec MacLean" <al**********@NO19SPAM60.copeohs.com> ¼¶¼g©ó¶l¥ó·s»D:e9**************@TK2MSFTNGP11.phx.g bl...

RedHair,

(1) I would recommend that you don't permit direct access to the
filesystem. Perhaps consider using a database (e.g. SQL Server) to store
the files instead. If you wanted direct file system access, you would
need to configure special permissions to allow the web app security the
correct context for writing to the web server disks. This can pose a big
security risk, no matter which OS is used.

Using SQL Server datatype "image" (sometimes referred to as BLOB, from
Binary Large OBject), you can have up to a 2Gb file in each record, so
there's plenty of storage potential.

(2 / 3 / 4) Using a DB would also remove problems 2, 3 and 4. For (3) you
would use a unique key (primary key) for each file record created in the
storage table. This means you can have duplicate file names if necessary
without conflict. You would need to consider adding a supplemental column
to the table to hold some other value that would allow users to
distinguish the files if same name used. This could be the UserID/name of
the person uploading, or the date-time stamp of the upload, or use both
for example.

For point 2, do you mean a file that is a photo, or any type of file such
as a document/spreadsheet?

(5) If you use a hot-swap capable RAID based server or NAS, you should be
able to add more / larger disks and grow your RAID partition into the new
space.

(1 / 6) Since you're posting in the dotnet forums, we can probably assume
you're at least considering to use .NET to create this app, so you'll
probably be wanting a Windows host server. Windows Server 2003 Web
Edition is perfect for this as it is much cheaper than Standard (or
Enterprise) versions and you don't need to buy separate licences. Also,
this can host an MSDE version of SQL server if required to keep
infrastructure costs down to bare minimum, though with your stated user
audience, you should probably consider a separate dedicated DB server.
(E.g. SQL Server with CPU licence(s) for Internet use).

Hope that helps.

Al
"RedHair" <re*****@ms40.url.com.tw> wrote in message
news:uo*************@TK2MSFTNGP12.phx.gbl...

I'd like to set up a file system for the ASP.NET 2.0 application
to store user-uploaded files, since the members are more than
100,000 people, the basic requirements are as below:

(1) The file system is separate with front-end web site
(2) Need to re-size user-uploaded image file to same size
(3) Need to rename file name to avoid duplicate name
(4) How to design directory and file structure to increase disk I/O
performance
(5) How to scale the file system by adding more HD or file server
(6) What kind of file server? Window 2003 or Linux

Any suggestion? thanks in advance.

[Alec MacLean]

I've not read that deeply into the performance issues myself, not having had
to produce an app of this type (though one is planned). I'll need to do
some further reading myself, it seems!

I have been working on an app that uploads photo images (HR system, staff
photo) with no discernable speed issues, but this is a Win-forms app used on
local LAN, so direct comparison is probably not a fair one.

It seems logical that there would be a performance hit while the binary file
is "translated" into the stream required to up/down-load, though I don't
know how much impact that has. Probably varies a lot with hardware being
used as well the number of user hits per second/minute. You'd need to weigh
the pro's and cons of the available approaches (between File I/O and DB I/O)
to judge which will best suit your current project's needs.

I've not used caching myself as of this time, but I wouldn't see a technical
barrier with caching at either end. I'm more than happy to be corrected /
illuminated on this though!

Al
"RedHair" <re*****@ms40.url.com.tw> wrote in message
news:%2****************@TK2MSFTNGP14.phx.gbl...

Thanks for your detailed reply.
Use db instead of file system is the original idea, but many articles
mentioned
that read/write binary from db costs the performance.
Besides, it is possible to put db generated image file into the cache
either
in server or client side?

[RedHair]

Thanks!
"Alec MacLean" <al**********@NO19SPAM60.copeohs.com> ¼¶¼g©ó¶l¥ó·s»D:em*************@TK2MSFTNGP12.phx.gb l...

I've not read that deeply into the performance issues myself, not having
had to produce an app of this type (though one is planned). I'll need to
do some further reading myself, it seems!

I have been working on an app that uploads photo images (HR system, staff
photo) with no discernable speed issues, but this is a Win-forms app used
on local LAN, so direct comparison is probably not a fair one.

It seems logical that there would be a performance hit while the binary
file is "translated" into the stream required to up/down-load, though I
don't know how much impact that has. Probably varies a lot with hardware
being used as well the number of user hits per second/minute. You'd need
to weigh the pro's and cons of the available approaches (between File I/O
and DB I/O) to judge which will best suit your current project's needs.

I've not used caching myself as of this time, but I wouldn't see a
technical barrier with caching at either end. I'm more than happy to be
corrected / illuminated on this though!

Al
"RedHair" <re*****@ms40.url.com.tw> wrote in message
news:%2****************@TK2MSFTNGP14.phx.gbl...

Thanks for your detailed reply.
Use db instead of file system is the original idea, but many articles
mentioned
that read/write binary from db costs the performance.
Besides, it is possible to put db generated image file into the cache
either
in server or client side?