In a web farm environment, how to store the user uploading files for future
access?
Store them in db, local file system or centralized file server?
If trying to storing the file in local file system or centralized file
server, how the web server
receives the uploading file and sync to each web server's local file system?
or the centralized
file server?
Should I create only one folder or lots of subfolders to organize the file
structure for better access
from file system?
10  1555 
Depends on what you want to do with the files and what they represent. Since
you haven't said, ASP.NET uploads files into server memory, buffers it and
then writes it to disk in the simplest case. Where you choose to store it
depends on how frequently you want to access it and the responsiveness you
want from the application, among other things.
--
--
Regards,
Alvin Bruney [MVP ASP.NET]
[Shameless Author plug]
The O.W.C. Black Book, 2nd Edition
Exclusively on www.lulu.com/owc $19.99
"Redhairs" <re*****@u.s.awrote in message
news:#G**************@TK2MSFTNGP06.phx.gbl...
In a web farm environment, how to store the user uploading files for
future access?
Store them in db, local file system or centralized file server?
If trying to storing the file in local file system or centralized file
server, how the web server
receives the uploading file and sync to each web server's local file
system? or the centralized
file server?
Should I create only one folder or lots of subfolders to organize the file
structure for better access
from file system?
Thanks for your reply.
Just like simple photo album web app for user to upload image files and
review them later.
"Alvin Bruney [ASP.NET MVP]" <www.lulu.com/owcwrote in message
news:78**********************************@microsof t.com...
Depends on what you want to do with the files and what they represent.
Since you haven't said, ASP.NET uploads files into server memory, buffers
it and then writes it to disk in the simplest case. Where you choose to
store it depends on how frequently you want to access it and the
responsiveness you want from the application, among other things.
--
--
Regards,
Alvin Bruney [MVP ASP.NET]
[Shameless Author plug]
The O.W.C. Black Book, 2nd Edition
Exclusively on www.lulu.com/owc $19.99
"Redhairs" <re*****@u.s.awrote in message
news:#G**************@TK2MSFTNGP06.phx.gbl...
>In a web farm environment, how to store the user uploading files for
future access?
Store them in db, local file system or centralized file server?
If trying to storing the file in local file system or centralized file
server, how the web server
receives the uploading file and sync to each web server's local file
system? or the centralized
file server?
Should I create only one folder or lots of subfolders to organize the
file structure for better access
from file system?
In solutions that I've designed that have this as a requirement, I've
typically used Network Attached Storage.
In most cases, all the servers in the web farm are on the same gigabit
network segment, and the NAS (which is usually multi-homed) sits on that
same segment.
In most cases, storing files in the database will lead to long term
problems. As amazing as current databases are, they're still not file
systems...
--
Chris Mullins
"Redhairs" <re*****@u.s.awrote in message
news:%2****************@TK2MSFTNGP06.phx.gbl...
In a web farm environment, how to store the user uploading files for
future access?
Store them in db, local file system or centralized file server?
If trying to storing the file in local file system or centralized file
server, how the web server
receives the uploading file and sync to each web server's local file
system? or the centralized
file server?
Should I create only one folder or lots of subfolders to organize the file
structure for better access
from file system?
Chris,
What type of long term problems?
Can you share please I store images in a database.
Thank You,
LVP
"Chris Mullins [MVP - C#]" <cm******@yahoo.comwrote in message
news:%2****************@TK2MSFTNGP03.phx.gbl...
In solutions that I've designed that have this as a requirement, I've
typically used Network Attached Storage.
In most cases, all the servers in the web farm are on the same gigabit
network segment, and the NAS (which is usually multi-homed) sits on that
same segment.
In most cases, storing files in the database will lead to long term
problems. As amazing as current databases are, they're still not file
systems...
--
Chris Mullins
"Redhairs" <re*****@u.s.awrote in message
news:%2****************@TK2MSFTNGP06.phx.gbl...
>In a web farm environment, how to store the user uploading files for
future access?
Store them in db, local file system or centralized file server?
If trying to storing the file in local file system or centralized file
server, how the web server
receives the uploading file and sync to each web server's local file
system? or the centralized
file server?
Should I create only one folder or lots of subfolders to organize the
file structure for better access
from file system?
Any tips on file structure design based on performance and maintenance
concerns?
e.g. one folder for all image files? or many subfolders?
"Chris Mullins [MVP - C#]" <cm******@yahoo.comwrote in message
news:%2****************@TK2MSFTNGP03.phx.gbl...
In solutions that I've designed that have this as a requirement, I've
typically used Network Attached Storage.
In most cases, all the servers in the web farm are on the same gigabit
network segment, and the NAS (which is usually multi-homed) sits on that
same segment.
In most cases, storing files in the database will lead to long term
problems. As amazing as current databases are, they're still not file
systems...
--
Chris Mullins
"Redhairs" <re*****@u.s.awrote in message
news:%2****************@TK2MSFTNGP06.phx.gbl...
>In a web farm environment, how to store the user uploading files for
future access?
Store them in db, local file system or centralized file server?
If trying to storing the file in local file system or centralized file
server, how the web server
receives the uploading file and sync to each web server's local file
system? or the centralized
file server?
Should I create only one folder or lots of subfolders to organize the
file structure for better access
from file system?
The biggest problems I've seen with storing files in a database is that it
grows without bound, and current databases don't seem to do very well with
really large data sets. In the large range (1TB -1PB) the file systems (in
my experience) deals with large sets more effectivly. The database is great
for storing "Which computers have that image on it?", but for the actual
image.... not so hot.
If you are adding files quickly enough, you'll need to be adding disk space
into your database every hours / days / weeks. Even with current large
drives, this is both expensive and time consuming. Using a DB, there's
really no good way to scatter the data across a large number of computers -
you could use FileGroup in SQL2K5 (Enterprise only), or something similar,
but these are not very good for dynamically adding data.
... if you're not adding drives, you're archiving files. This means a job
that runs, and moves images from the DB to a disk. You then need to design
the system to first check the DB, then check the disk. This makes things 2x
as complicated as just skipping the DB step and storing files on disk.
My preference at this point is:
- Store files on a NAS, or a set of NAS devices.
- Store the index (file names, metadata, location[s]) in the database.
I would love to hear of an out-of-the-box approach people have used for
this - especially in Oracle land (I'm familar with the SQL Server
landscape).
--
Chris Mullins
"LVP" <lv**********@hotmail.comwrote in message
news:eJ**************@TK2MSFTNGP06.phx.gbl...
Chris,
What type of long term problems?
Can you share please I store images in a database.
Thank You,
LVP
"Chris Mullins [MVP - C#]" <cm******@yahoo.comwrote in message
news:%2****************@TK2MSFTNGP03.phx.gbl...
>In solutions that I've designed that have this as a requirement, I've
typically used Network Attached Storage.
In most cases, all the servers in the web farm are on the same gigabit
network segment, and the NAS (which is usually multi-homed) sits on that
same segment.
In most cases, storing files in the database will lead to long term
problems. As amazing as current databases are, they're still not file
systems...
--
Chris Mullins
"Redhairs" <re*****@u.s.awrote in message
news:%2****************@TK2MSFTNGP06.phx.gbl...
>>In a web farm environment, how to store the user uploading files for
future access?
Store them in db, local file system or centralized file server?
If trying to storing the file in local file system or centralized file
server, how the web server
receives the uploading file and sync to each web server's local file
system? or the centralized
file server?
Should I create only one folder or lots of subfolders to organize the
file structure for better access
from file system?
This seems to be somewhat one sided argument, storing in the DB works well in
some situations, it can work well if the file sizes aren't too large. If you
are storing several TB of images then you probably don't want to put them in
the database, but if the sizing is smaller it should not be a problem.
The argument you make about disk space really isn't a database issue it is a
disk issue and you'll run out of disk space just as quickly in the file
system as in the DB.
The advantages to storing in the database are a central location and
syncronized backups between the images and the database. The primary problem
is really if you have a lot of clients(in this case web servers in the farm)
hitting the database they will be chewing up bandwidth into your db server to
get the images. Another disadvantage is as noted in parent post the file
system generally is faster.
However I have used SQL Server before for storing images for web use, in
these cases though I was limiting the actual file size of the image to 55KB
or less and made use of some caching on the webserver, I never saw a
performance problem. Also because I controlled the point of entry for the
images I would scale the images down with GDI+ server side before saving to
the db.
"Chris Mullins [MVP - C#]" wrote:
The biggest problems I've seen with storing files in a database is that it
grows without bound, and current databases don't seem to do very well with
really large data sets. In the large range (1TB -1PB) the file systems (in
my experience) deals with large sets more effectivly. The database is great
for storing "Which computers have that image on it?", but for the actual
image.... not so hot.
If you are adding files quickly enough, you'll need to be adding disk space
into your database every hours / days / weeks. Even with current large
drives, this is both expensive and time consuming. Using a DB, there's
really no good way to scatter the data across a large number of computers -
you could use FileGroup in SQL2K5 (Enterprise only), or something similar,
but these are not very good for dynamically adding data.
... if you're not adding drives, you're archiving files. This means a job
that runs, and moves images from the DB to a disk. You then need to design
the system to first check the DB, then check the disk. This makes things 2x
as complicated as just skipping the DB step and storing files on disk.
My preference at this point is:
- Store files on a NAS, or a set of NAS devices.
- Store the index (file names, metadata, location[s]) in the database.
I would love to hear of an out-of-the-box approach people have used for
this - especially in Oracle land (I'm familar with the SQL Server
landscape).
--
Chris Mullins
"LVP" <lv**********@hotmail.comwrote in message
news:eJ**************@TK2MSFTNGP06.phx.gbl...
Chris,
What type of long term problems?
Can you share please I store images in a database.
Thank You,
LVP
"Chris Mullins [MVP - C#]" <cm******@yahoo.comwrote in message
news:%2****************@TK2MSFTNGP03.phx.gbl...
In solutions that I've designed that have this as a requirement, I've
typically used Network Attached Storage.
In most cases, all the servers in the web farm are on the same gigabit
network segment, and the NAS (which is usually multi-homed) sits on that
same segment.
In most cases, storing files in the database will lead to long term
problems. As amazing as current databases are, they're still not file
systems...
--
Chris Mullins
"Redhairs" <re*****@u.s.awrote in message
news:%2****************@TK2MSFTNGP06.phx.gbl...
In a web farm environment, how to store the user uploading files for
future access?
Store them in db, local file system or centralized file server?
If trying to storing the file in local file system or centralized file
server, how the web server
receives the uploading file and sync to each web server's local file
system? or the centralized
file server?
Should I create only one folder or lots of subfolders to organize the
file structure for better access
from file system?
On Jan 4, 8:45*pm, Jeff <J...@discussions.microsoft.comwrote:
This seems to be somewhat one sided argument, storing in the DB works wellin
some situations, it can work well if the file sizes aren't too large. *If you
are storing several TB of images then you probably don't want to put them in
the database, but if the sizing is smaller it should not be a problem. *
The argument you make about disk space really isn't a database issue it isa
disk issue and you'll run out of disk space just as quickly in the file
system as in the DB. *
The advantages to storing in the database are a central location and
syncronized backups between the images and the database. *The primary problem
is really if you have a lot of clients(in this case web servers in the farm)
hitting the database they will be chewing up bandwidth into your db serverto
get the images. *Another disadvantage is as noted in parent post the file
system generally is faster.
However I have used SQL Server before for storing images for web use, in
these cases though I was limiting the actual file size of the image to 55KB
or less and made use of some caching on the webserver, I never saw a
performance problem. *Also because I controlled the point of entry for the
images I would scale the images down with GDI+ server side before saving to
the db.
"Chris Mullins [MVP - C#]" wrote:
The biggest problems I've seen with storing files in a database is that it
grows without bound, and current databases don't seem to do very well with
really large data sets. In the large range (1TB -1PB) the file systems(in
my experience) deals with large sets more effectivly. The database is great
for storing "Which computers have that image on it?", but for the actual
image.... not so hot.
If you are adding files quickly enough, you'll need to be adding disk space
into your database every hours / days / weeks. Even with current large
drives, this is both expensive and time consuming. Using a DB, there's
really no good way to scatter the data across a large number of computers -
you could use FileGroup in SQL2K5 (Enterprise only), or something similar,
but these are not very good for dynamically adding data.
... if you're not adding drives, you're archiving files. This means a job
that runs, and moves images from the DB to a disk. You then need to design
the system to first check the DB, then check the disk. This makes things2x
as complicated as just skipping the DB step and storing files on disk.
My preference at this point is:
- Store files on a NAS, or a set of NAS devices.
- Store the index (file names, metadata, location[s]) in the database.
I would love to hear of an out-of-the-box approach people have used for
this - especially in Oracle land (I'm familar with the SQL Server
landscape).
--
Chris Mullins
"LVP" <lvp_agent...@hotmail.comwrote in message
news:eJ**************@TK2MSFTNGP06.phx.gbl...
Chris,
What type of long term problems?
Can you share please I store images in a database.
Thank You,
LVP
"Chris Mullins [MVP - C#]" <cmull...@yahoo.comwrote in message
>news:%2****************@TK2MSFTNGP03.phx.gbl...
>In solutions that I've designed that have this as a requirement, I've
>typically used Network Attached Storage.
>In most cases, all the servers in the web farm are on the same gigabit
>network segment, and the NAS (which is usually multi-homed) sits on that
>same segment.
>In most cases, storing files in the database will lead to long term
>problems. As amazing as current databases are, they're still not file
>systems...
>--
>Chris Mullins
>"Redhairs" <redh...@u.s.awrote in message
>>news:%2****************@TK2MSFTNGP06.phx.gbl.. .
>>In a web farm environment, how to store the user uploading files for
>>future access?
>>Store them in db, local file system or centralized file server?
>>If trying to storing the file in local file system or centralized file
>>server, how the web server
>>receives the uploading file and sync to each web server's local file
>>system? or the centralized
>>file server?
>>Should I create only one folder or lots of subfolders to organize the
>>file structure for better access
>>from file system?- Hide quoted text -
- Show quoted text -
To BLOB or Not To BLOB: Large Object Storage in a Database or a
Filesystem? http://research.microsoft.com/resear...MSR-TR-2006-45
"Alexey Smirnov" <al************@gmail.comwrote
To BLOB or Not To BLOB: Large Object Storage in
a Database or a Filesystem?
http://research.microsoft.com/resear...MSR-TR-2006-45
That's really a great paper.
I quite enjoyed reading it - having real metrics applied to this stuff is
exactly what I like to see!
--
Chris Mullins
You may want to consider a tree like hierarchy where the top folder
represents the logged on user account with a sub folder that has the image
files. It's easier to trouble shoot or retrieve images that way since you
index on the logged on user and then read all the files in that folder. When
you get the file upload from the user, simply find the logged on user
account and store the file inside the directory. If this is anonymous, that
approach won't work.
--
--
Regards,
Alvin Bruney [MVP ASP.NET]
[Shameless Author plug]
The O.W.C. Black Book, 2nd Edition
Exclusively on www.lulu.com/owc $19.99
"Redhairs" <re*****@u.s.awrote in message
news:eg**************@TK2MSFTNGP04.phx.gbl...
Thanks for your reply.
Just like simple photo album web app for user to upload image files and
review them later.
"Alvin Bruney [ASP.NET MVP]" <www.lulu.com/owcwrote in message
news:78**********************************@microsof t.com...
>Depends on what you want to do with the files and what they represent.
Since you haven't said, ASP.NET uploads files into server memory, buffers
it and then writes it to disk in the simplest case. Where you choose to
store it depends on how frequently you want to access it and the
responsiveness you want from the application, among other things.
--
--
Regards,
Alvin Bruney [MVP ASP.NET]
[Shameless Author plug]
The O.W.C. Black Book, 2nd Edition
Exclusively on www.lulu.com/owc $19.99
"Redhairs" <re*****@u.s.awrote in message
news:#G**************@TK2MSFTNGP06.phx.gbl...
>>In a web farm environment, how to store the user uploading files for
future access?
Store them in db, local file system or centralized file server?
If trying to storing the file in local file system or centralized file
server, how the web server
receives the uploading file and sync to each web server's local file
system? or the centralized
file server?
Should I create only one folder or lots of subfolders to organize the
file structure for better access
from file system?
This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics
by: Karen Grube |
last post by:
Hi!
Each week, we receive a two-page PDF file from UPS along with a separate
flat file (a CSV) The PDF file contains the overview of our weekly invoice
and the CSV contains the details of each...
|
by: Fred |
last post by:
Hi,
I want to be able to read and write to the custom properties of any Office
type file. In Windows explorer I can do this manually by right clicking on a
file selecting 'properties' from the...
|
by: Matt Jensen |
last post by:
Howdy
I've got a rather strange issue occuring.
I used forms based .NET authentication, although I'm also setting some
session variables when people login.
However, I've found when people use...
|
by: Dieter |
last post by:
Hi.
In the snippet of code below, I'm trying to understand why when the
struct dirent ** namelist
is declared with "file" scope, I don't have a problem freeing the
allocated memory. But...
|
by: sunny |
last post by:
Hi
C Ref Man says
" The typedef specifier does not reserve storage and is called a
storage class specifier only for syntactic convenience "
does this mean storage specifier for "user...
|
by: Richard Lewis Haggard |
last post by:
I have an application that can't use the registry to save various user
options and application settings. My first thought was to simply use an
application configuration file but this approach seems...
|
by: Bonzol |
last post by:
Vb.net 2003, Web application
Hey all!
Just a quick question.
I have a site which allows users the ability to upload files by the
HTML brows button control.
My question is if a user submits...
|
by: =?Utf-8?B?c3Vpcw==?= |
last post by:
Hi every body,
i have a problem with how to read .MSG file (Outlook) using vb.net ?
using this articles the author has already explain how to read .MSG file
using stuctured storage (vc++)
...
|
by: smileprince00 |
last post by:
hi friends, i am new learner of this sql server 2008.my question is if my primary data file i.e .mdf is restricted to certain amount of memory like 100mb..i have some more secondary data files.i.e...
|
by: Charles Arthur |
last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
|
by: Sonnysonu |
last post by:
This is the data of csv file
1 2 3
1 2 3
1 2 3
1 2 3
2 3
2 3
3
the lengths should be different i have to store the data by column-wise with in the specific length.
suppose the i have to...
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
|
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers,...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
|
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new...
| |
