473,386 Members | 1,654 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,386 software developers and data experts.

Binary data in PostgreSQL

Hi all,

AFAIK it is possible for columns to be very large, up to about 2 GB. Are
there any hints or experiences about storing binary data (jpg-images,
pdf-documents) in PostgrreSQL with or without the complicated lo-stuff?

Of course it's in many cases a good approach to store those files simply
in the file system but there's always a risk of running out of sync
(filesystem and tables), e.g. by deleting files and not deleting the
table rows with the filenames.

Any ideas and comments welcome.

---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings

Nov 22 '05 #1
4 4001
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Well, you have the option to use the bytea types. It's nice to have the stuff
in the database. I wrote an application (quite a while back) where I had to
store documents to the database. At that time I switched to mysql, not
because it's better, but because it handles binary data quite easily. Since
this is a couple of years ago it's quite likely that PostgreSQL byte handling
improved a lot (back then PG was still at a pre 7 version).
Meanwhile I'm developing an application that stores huge amounts of documents.
Since this time I need the features of a real database, like triggers etc.
I'm using PG now. The amount of data I'm storing now is huge, therefor I
decided to store the files in a directory tree. I'm storing about 40000
documents around 100k each a week, thats about 12GB a month. The file storage
directory is handled by the server side of the application and is invisible
to normal users. The main reasons for using the filesystem instead of the
database were:

1) a lot of documents
2) due to 1) I'm able to spread the documents over several
harddrives/filesystems - so this solves the storage space problem (unless you
want to use LVM or similar)
3) much easier handling on the application side. I can use rsync, ftp, scp or
whatever else to transfer files. Also most programing languages are quite
good at handling normal files.
4) You never run into a file problem regarding file type or size, at least not
up to the limit of the operating system you're on.
5) You can use conventional backup methods without interrupting the database
server. It's also much faster to restore the whole system on a different
machine - dump/restore the database and copy your directory tree.

You have some drawbacks though:
1) your application has to make sure that the referencing records and the file
storage is in sync. If there's a bug in the application you can totally
scramble the storage.
2) If your server has a filesystem problem you could lose files due to fsck,
which again would "unsync" the data from the files.

Basically it all depends on the structure of your application. If you can
guarantee that only the application has access to the file storage you can
control the synchronization problem. If not I'd probably go the extra mile to
store the stuff as blobs.
There is another option you might want to consider. If you're storing
relatively small files, like the images for a website, you could even go with
a normal "text" field and base64 encode the data before storing (as well as
decoding it after retrieval).

My $0.02

On Saturday 21 February 2004 10:53 am, Holger Marzen wrote:
Hi all,

AFAIK it is possible for columns to be very large, up to about 2 GB. Are
there any hints or experiences about storing binary data (jpg-images,
pdf-documents) in PostgrreSQL with or without the complicated lo-stuff?

Of course it's in many cases a good approach to store those files simply
in the file system but there's always a risk of running out of sync
(filesystem and tables), e.g. by deleting files and not deleting the
table rows with the filenames.

Any ideas and comments welcome.

---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings


- --
UC

- --
Open Source Solutions 4U, LLC 2570 Fleetwood Drive
Phone: +1 650 872 2425 San Bruno, CA 94066
Cell: +1 650 302 2405 United States
Fax: +1 650 872 2417
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQFAN8g4jqGXBvRToM4RAreNAJwMwT1Twtg9c35rp1Sgag hKU7XDiQCg0bQM
Z7cbgYmGYtjkCFnJfSL3tm8=
=3yhL
-----END PGP SIGNATURE-----
---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faqs/FAQ.html

Nov 22 '05 #2
It's not the most effective use of space, and I'm sure not the best way
to do it, but I store such data as base64 encoded text. Works very
well for my needs, in that regard.

Gavin

Holger Marzen wrote:
Hi all,

AFAIK it is possible for columns to be very large, up to about 2 GB. Are
there any hints or experiences about storing binary data (jpg-images,
pdf-documents) in PostgrreSQL with or without the complicated lo-stuff?

Of course it's in many cases a good approach to store those files simply
in the file system but there's always a risk of running out of sync
(filesystem and tables), e.g. by deleting files and not deleting the
table rows with the filenames.

Any ideas and comments welcome.

---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings

---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings

Nov 22 '05 #3
Hello,

I would use large objects. Easy to access, easy to use.

J
Gavin M. Roy wrote:
It's not the most effective use of space, and I'm sure not the best way
to do it, but I store such data as base64 encoded text. Works very
well for my needs, in that regard.

Gavin

Holger Marzen wrote:
Hi all,

AFAIK it is possible for columns to be very large, up to about 2 GB. Are
there any hints or experiences about storing binary data (jpg-images,
pdf-documents) in PostgrreSQL with or without the complicated lo-stuff?

Of course it's in many cases a good approach to store those files simply
in the file system but there's always a risk of running out of sync
(filesystem and tables), e.g. by deleting files and not deleting the
table rows with the filenames.

Any ideas and comments welcome.

---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings

---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings


---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ma*******@postgresql.org

Nov 22 '05 #4
bytea's suck for large data. I think my processes ate up 5x the size of
the large objects just trying to select them from the database. Anything
over 10 Meg it usually isn't useful for.

Jon

On Sat, 21 Feb 2004, Joshua Drake wrote:
Hello,

I would use large objects. Easy to access, easy to use.

J
Gavin M. Roy wrote:
It's not the most effective use of space, and I'm sure not the best way
to do it, but I store such data as base64 encoded text. Works very
well for my needs, in that regard.

Gavin

Holger Marzen wrote:
Hi all,

AFAIK it is possible for columns to be very large, up to about 2 GB. Are
there any hints or experiences about storing binary data (jpg-images,
pdf-documents) in PostgrreSQL with or without the complicated lo-stuff?

Of course it's in many cases a good approach to store those files simply
in the file system but there's always a risk of running out of sync
(filesystem and tables), e.g. by deleting files and not deleting the
table rows with the filenames.

Any ideas and comments welcome.

---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings

---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings


---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ma*******@postgresql.org

Nov 22 '05 #5

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: bissatch | last post by:
Hi, I am trying to write script that is run when a form is submitted. The form contains an image input field and when submitted, the image is uploaded, resized and added as binary information to...
2
by: Gabriele Bartolini | last post by:
Hi guys, just a quick and probably stupid question. When I make a cluster based on an index on a table, how can I remove it later? Should I 'create' a new one by using the primary key index? ...
0
by: David Garamond | last post by:
I'm making a "relocatable" Postgres binary distribution for my clients. Everything goes into postgresql-7.4.1/ directory, including libraries and binaries. This will be installed by a...
2
by: Ron Snyder | last post by:
I'm attempting to use spamassassin 3.0 (beta) with an SQL backend, and have identified one performance gain so far that makes PostgreSQL a good (IMO) candidate for the backend. I need some advice...
1
by: Jerry LeVan | last post by:
I am getting ready to try to add image viewing into my postgresql browser... I am probably going to start with storing a thumbnail and a URL for the actual image in the table. Is there any...
1
by: Igor Shevchenko | last post by:
Hi, I use binary mode for sending params and receiving data using libpq with protocol v3; PostgreSQL version is 7.4.3. Here, varchar-s are returned in binary mode; they're generated from...
1
by: David Garamond | last post by:
When I distribute a binary-only distribution of Postgre to a client, can I exclude some parts of Postgres (to make it smaller), e.g. documentation (all of doc/), some PL's, or even psql, initdb,...
2
by: Bastian Voigt | last post by:
Hallo List! I just found this old posting on google. Now my question is how can I read an integer value from the PGresult using the binary format? Can someone plz gimme a code example? (please...
4
by: Dmitry Teslenko | last post by:
Hello! I have some postgresql database that stores binary data (windows/linux executables). Field type I use for this is bytea. Also I have php script that allows users download these binaries via...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: aa123db | last post by:
Variable and constants Use var or let for variables and const fror constants. Var foo ='bar'; Let foo ='bar';const baz ='bar'; Functions function $name$ ($parameters$) { } ...
0
by: ryjfgjl | last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
0
by: ryjfgjl | last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.