473,322 Members | 1,352 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,322 software developers and data experts.

hundred thousands BLOBs in database

Hi

We intend create system to store documents in company.
Size of documents: 100k-2M, hundred thousands or milions files.
We are considering the possibility to store documents in DB2 or in
filesystem.

Can anybody share your experience?

What is performance solution with files in db? Have you any problems with
that?

I am looking for articles, links, hints, advantages and disadvantages for
this solutions.

Thanks in advance
Yaro
Jan 26 '06 #1
8 1547
Yaro wrote:
Hi

We intend create system to store documents in company.
Size of documents: 100k-2M, hundred thousands or milions files.
We are considering the possibility to store documents in DB2 or in
filesystem.

Can anybody share your experience?

What is performance solution with files in db? Have you any problems with
that?

I am looking for articles, links, hints, advantages and disadvantages for
this solutions.

Thanks in advance
Yaro


Yaro,

A few initial thoughts based on my (much smaller volume) experiences, if you
are going to use LOB columns -

1) Make sure you specify COMPACT on the column definition
2) Define separate DMS tablespaces for data, index and LOB objects
3) How are you going to back it all up ?
4) Are you intending to log the LOBs ? If so how are you going to handle
the log files, and be careful of individual log file sizes.
5) Perhaps consider disabling file system caching
6) Check that you set maximum file size and disk quotas appropriately (or
make them unlimited if possible)

HTH

Phil
Jan 26 '06 #2
Yaro wrote:
Hi

We intend create system to store documents in company.
Size of documents: 100k-2M, hundred thousands or milions files.
We are considering the possibility to store documents in DB2 or in
filesystem.

Can anybody share your experience?

What is performance solution with files in db? Have you any problems with
that?

I am looking for articles, links, hints, advantages and disadvantages for
this solutions.

Since storing and presumably managing these documents is the core
function of this system I'm wondering whether this discussion should be
more appropriately held around DB2 Content Manager instead of
re-inventing the wheel on the DBMS level.

Cheers
Serge
--
Serge Rielau
DB2 Solutions Development
DB2 UDB for Linux, Unix, Windows
IBM Toronto Lab
Jan 26 '06 #3
Thank you for the reply
A few initial thoughts based on my (much smaller volume) experiences, if
you are going to use LOB columns -
1) Make sure you specify COMPACT on the column definition
2) Define separate DMS tablespaces for data, index and LOB objects
3) How are you going to back it all up ?
4) Are you intending to log the LOBs ? If so how are you going to handle
the log files, and be careful of individual log file sizes.
It is one of my biggest concern.
Can I switch logging LOBs off? Maybe for table or tablespace? I didn't see
option like that.
5) Perhaps consider disabling file system caching
6) Check that you set maximum file size and disk quotas appropriately (or
make them unlimited if possible)
HTH
Phil


Thanks
Yaro


Jan 26 '06 #4
Since storing and presumably managing these documents is the core
function of this system I'm wondering whether this discussion should be
more appropriately held around DB2 Content Manager instead of re-inventing
the wheel on the DBMS level.


We are not familiar with this product.
Thanks for hint

Yaro

Jan 26 '06 #5
Yaro wrote:
Can I switch logging LOBs off? Maybe for table or tablespace? I didn't see
option like that.


Yes, you can. Use the option NOT LOGGED (see the CREATE TABLE statement
documentation) for that.

--
Knut Stolze
DB2 Information Integration Development
IBM Germany
Jan 26 '06 #6
Yaro wrote:
Since storing and presumably managing these documents is the core
function of this system I'm wondering whether this discussion should be
more appropriately held around DB2 Content Manager instead of re-inventing
the wheel on the DBMS level.

We are not familiar with this product.
Thanks for hint

Here is a link:
http://www-306.ibm.com/software/data/cm/
Maybe worthwhile to give IBM a shout, maybe get a briefing....

Cheers
Serge
--
Serge Rielau
DB2 Solutions Development
DB2 UDB for Linux, Unix, Windows
IBM Toronto Lab
Jan 26 '06 #7
Yaro schrieb:
Hi

We intend create system to store documents in company.
Size of documents: 100k-2M, hundred thousands or milions files.
We are considering the possibility to store documents in DB2 or in
filesystem.

Can anybody share your experience?

What is performance solution with files in db? Have you any problems with
that?

I am looking for articles, links, hints, advantages and disadvantages for
this solutions.

Thanks in advance
Yaro


I created a DMS system and I decided for myself, to store the documents
NOT within the database. After 6 years of usage I've about 450000
documents with about 800000 renderings (a document can be available in
different renderings ... formats). These documents need about 80GB
(or compressed 50 GB) for themselves.

The database itselfs (project oriented document management system)
needs about 1.2 GB.

When goig this way, one has to consider how to store the files within
the filesystem. Especially, when a Windows system is the server.

Unix/Linux systems have no performance problems with such large number
of files, but a Windows server can step into big trouble here.

Having the files within the filesystem also means, that you have to
have a transfer path from the storage server to the client, looking
fo a special file.
I do not assume, that you want to have these files in mounted
filesystem, visible for all users !?
One way could be FTP (remember XP-SP2 firewall !) ...

Having the file in filesystems, make them very easily to backup :-)

Marten

Jan 26 '06 #8
Thank you Marten

Your post is very helpful

Yaro
Jan 27 '06 #9

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: Jesse Wolgamott | last post by:
I've seen plenty of posts regarding the estimation of table size, usually in the processing of planning for server storage needs. Well, I've got a different problem. I need to know how much data...
7
by: Howard Lowndes | last post by:
My situation is that I am interacting PHP 4.1.2 to PostgreSQL 7.2.2 I have no difficulty inserting and managing BLOBs into the Large Object system table, and I have a user table called images...
7
by: Nilabhra Banerjee | last post by:
Hi, I am still not sure whether the BLOBS are actually stored in the database or they have the pointer to the database for that file in the filesystem. If I remove the files (sources) for BLOBS...
0
by: Bing | last post by:
Hi there, I am using the DB2 universal JDBC driver type 4 to insert BLOBs into a DB2 database. The Method I used for supplying the BLOB data value is setBinaryStream(). Everything works fine as...
0
by: .Net Newbie | last post by:
I am currently working with the C# version of the PortalSDK in a text editor and have run across a bug that I need to resolve in order for my app to function properly. Basically, I'm using the...
0
by: .Net Newbie | last post by:
Basically, I'm using the Documents section of the PortalCSSDK to load Blobs into the Portal Database on SQL Server. When I upload small binary documents everything works fine. However, when I try...
2
by: Jerry LeVan | last post by:
Hi, I am just getting into large objects and bytea "stuff". I created a small db called pictures and loaded some large objects and then tried to do a restore. Here is how I got the dump. ...
5
by: bhodgins | last post by:
Hi, I am new on here, and had a newbie question that I am stumped with. I am not new to access, but am new to VB. I am trying to export BLOBs from a field called photo to external jpeg files. I...
1
by: LongJohnBaldry | last post by:
My database is bloated, and it's not the beer. The users wanted pictures on their forms, so I set up some OLE object fields and plonked them on the forms. Lo and behold the users went mad pasting jpg...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.