473,320 Members | 1,831 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

lo_import for bytea columns

Is there an equivalent function for bytea columns that works like
lo_import?

Alternatively, is there a way to copy from a large object to a bytea
column from SQL?

Or maybe someone has another way of attacking this problem:

I've got some Perl code that does this:

undef $/;
$data = <FHFOR89MBFILE>;
$sth = $dbh->prepare("insert into data (bigbyteacolumn) values (?)");
$sth->bind_param(1, $data, DBI::SQL_BINARY);
$sth->execute;

Which has worked fine for a while, with file sizes around 10MB.

However, now I have someone who wants to use this for a file that's 89MB,
and it's taking up about 500M of memory before crashing. I'm trying to
find a less-memory-consuming way of handling this, even if just for a
temporary hack for this one file. I think what's happening is that Perl
is reading in the 89M, and then I'm guessing that either Perl or the
driver is converting that into a fully-escaped string for transfer, and
this is where the problem is occuring.

Any ideas?

Thanks,

Jonathan Bartlett
---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match

Nov 12 '05 #1
1 4064
Jonathan Bartlett wrote:
However, now I have someone who wants to use this for a file that's 89MB,
and it's taking up about 500M of memory before crashing. I'm trying to
find a less-memory-consuming way of handling this, even if just for a
temporary hack for this one file. I think what's happening is that Perl
is reading in the 89M, and then I'm guessing that either Perl or the
driver is converting that into a fully-escaped string for transfer, and
this is where the problem is occuring.

Any ideas?


If you can use 7.4, then see:
http://www.postgresql.org/docs/curre...ibpq-exec.html

Specifically look at the PQexecParams() function. You could write a
simple command line program that uses PQexecParams() to insert your
large files. Or talk the Perl Postgres DBI driver guys into supporting
it maybe.

Joe
---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend

Nov 12 '05 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

48
by: Edwin Quijada | last post by:
Hi !! Everybody I am developing app using Delphi and I have a question: I have to save pictures into my database. Each picture has 20 o 30k aprox. What is the way more optimus? That 's table will...
2
by: Howard Lowndes | last post by:
I'm working psql v7.2.2 to postmaster v7.2.2 and want to use the \lo_import function. The psql manual says that the syntax is \lo_import '<filename>' 'comment' This loads the blob OK and...
0
by: Jonathan Bartlett | last post by:
Is there an equivalent function for bytea columns that works like lo_import? Alternatively, is there a way to copy from a large object to a bytea column from SQL? Or maybe someone has another...
2
by: Alvar Freude | last post by:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, I want to change a column from text to bytea; since it seems that alter table can't change the column type, i have to add a temporary column...
0
by: Alvar Freude | last post by:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, the following I posted already on pgsql-bugs -- perhaps someone has a good workaround or fix or can say me that I'm wrong? There seems to...
2
by: Carlos | last post by:
Do I need to use the -b option in pg_dump to dump bytea fields? For a while now I have been routinely using the -b option in pg_dump to back up, restore, and copy my databases because I thought...
7
by: Dennis Gearon | last post by:
when bytea, text, and varchar(no limit entered) columns are used, do they ALWAYS use an extra table/file? Or do they only do it after a certain size of input? Also, if I wanted to put a *.pdf...
0
by: Peter Wang | last post by:
I have a function and am using cursor. All the columns I select in the cursor clause are BYTEA datatype. I need to compare the after-fetch-value for two BYTEA columns which is temp2 and temp3...
4
by: Jerry LeVan | last post by:
Hi, I am adding image and large object support in my Cocoa postgresql browser. Are there going to be any enhanced bytea support functions coming along? It seems sorta silly to have to write...
3
nehashri
by: nehashri | last post by:
I am trying to insert a large object into my dbs using: INSERT INTO table VALUES (lo_import ( ' z/image/image.jpg ' )); but i get the message saying that I need superuser privilidges to do it...
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.