473,320 Members | 2,193 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

bulk loading with DBI

Hi,

I'm just wondering if anyone has experience with bulk loading data into an
Oracle database with DBI.

My options are to use DBI or Oracle's sqlloader. I'd rather do it using DBI,
but I'm not sure what kind of perfomance hit I'll incur. I need to load 2
tables on a daily basis (~50MB).

Any comments?

- Greg.

Jul 19 '05 #1
1 7985
SQLLoader will generally be faster.

Pretty much trial an error really

For bulk inserts you should use DBI with placeholders.

Bit of pseudo code, might help...

my $sth = $dbh->prepare (SQL insert :param1, :param2 etc)
my $count = 0;

loop
$sth->bind_param(":param1", $param1);
$sth->bind_param(":param2", $param2);
...
...
eval { $sth->execute() };
if ($@) {
print "Error.......";
};
...
...
if ($count>50) { # Change this value depending on threshold,
rollback segment size etc
$self->doCommit($dbh);
$count=0;
}
$count++;

end loop

$dbh->commit

HTH
AdrianK

"Greg H" <gr********@telus.net> wrote in message news:<eV******************@news2.telusplanet.net>. ..
Hi,

I'm just wondering if anyone has experience with bulk loading data into an
Oracle database with DBI.

My options are to use DBI or Oracle's sqlloader. I'd rather do it using DBI,
but I'm not sure what kind of perfomance hit I'll incur. I need to load 2
tables on a daily basis (~50MB).

Any comments?

- Greg.

Jul 19 '05 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
by: cmars | last post by:
Hi, I want to insert massive amounts of data in near real-time into a MySQL database. The data is in a structured binary format, and I have code that parses the data into logical data structures. ...
2
by: Chris | last post by:
Any help would be appreciated. I am running a script that does the following in succession. 1-Drop existing database and create new database 2-Defines tables, stored procedures and functions...
2
by: David Sharp | last post by:
I've been doing some experiments with speeding up copying tables of approximately 1 million rows between databases using BCP and BULK INSERT. I noticed that the total time for removing the...
2
by: Mike Husler | last post by:
We have created CSV files on HPUX 11.0 and transferred them via ASCII ftp to our SQL Server machine file store to load large amounts for data using the BULK INSERT command. This is the command: ...
3
by: moonriver | last post by:
Right now I develop an application to retrieve over 30,000 records from a binary file and then load them into a SQL Server DB. So far I load those records one by one, but the performance is very...
2
by: qwerty123 | last post by:
Hi folks: We have a system requirement where a large amount of data needs to be transferred between two systems. The main system is a DB2 8.x running on a pSeries (AIX) and the archive database...
2
by: its me Rock | last post by:
Hi All, How to load bulk of records from one file to another file.Is there any memory allocation techiques available.Need suggestion in that.... Eg: just reading the data from the...
5
by: yeoj13 | last post by:
Hello, I have a db2load script I'm using to populate a large table. Ideally, my target table is required to have "Not Null" constraints on a number of different columns. I've noticed a ...
0
by: NickW | last post by:
I have written a stored procedure that is used to load data into a SQL Server 2000 and 2005 databases from some 40ish different text files. One of the text files is loading all of the rows apart from...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.