473,756 Members | 7,611 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

db design and performance for bulk insert

26 New Member
i have a problem with large data import to a db in sql server.. Actually i have an application that collects data from an environment and dispatches this data to different csv files for sql server to bulk insert.. Each csv file corresponds a table in db.. After generating csv files on the application server(as a result csv files are remote files for the sql server), the dump process takes so much time that sometimes causes transaction log to fill up.. After reading articles, for the name of performance i changed recovery mode of db to simple and locate the log file of db to some other disk drive.. i also copy csv files to sql server manually but the result is almost same.. What can i do to increase the performance of bulk import?

i have a stored procedure that takes the path of remote file and the table name for bulk insert. i am not actually sure what is going under but for another db with recovery mode set to full import operation is sometimes 10 times faster than my db.. i am stuck in solving and understanding the mechanisms.. Changing recovery mode or using the existing db's create script to regenerate the db to bulk insert did not seem to have the same performance with the other db..

To sum up all i need to do is to import very large data to a db from csv files to corresponding tables.. this db can also be dropped and created to before bulk insert to increase performance and i do not need any backup/restore utility on this db..
Dec 16 '09 #1
3 3032
Delerna
1,134 Recognized Expert Top Contributor
Have you tried this using DTS instead?
How large is very large?
Dec 21 '09 #2
nbiswas
149 New Member
I don't know what version of SQL SERVER you are using.

If it is 2000 go ahead with DTS.

If higher version(2005+), I would strongly like to propose SSIS solution

Choose your database - > Right click on that -> Tasks -> Import data.

The Import / Export wizard will appear. Click on next.

From Choose a DataSource Screen, Choose the type of datasource you want to ( in this case it will be Microsoft excel), then specify the file path and click next.

Next choose the destination datasouce(i.e. the database name ) followed by the table names and then click finish.

Hope this makes sense
Dec 21 '09 #3
akdemirc
26 New Member
i am using 2005 server, a program actually generates these large csv files (about at size of gb) and this import process should also be done through the application by means of stored procs. as the files are generated at the computer the application runs, i shared the folder for sql server machine to have direct access for bulk insert.. the program calls a procedure from the server by passing the name of the file to be imported and the name of the table to import into.. i read some articles on bulk insert and performance optimizations and tried a few of them such as TABLOCK which really seemed to improve the performance.. Is SSIS a faster way that can be called from inside an application (actually c++ application) or is there some other parameters that i can use for further improvements?

thanks
Dec 21 '09 #4

Sign in to post your reply or Sign up for a free account.

Similar topics

7
12129
by: iqbal | last post by:
Hi all, We have an application through which we are bulk inserting rows into a view. The definition of the view is such that it selects columns from a table on a remote server. I have added the servers using sp_addlinkedserver on both database servers. When I call the Commit API of oledb I get the following error: Error state: 1, Severity: 19, Server: TST-PROC22, Line#: 1, msg:
6
12357
by: pk | last post by:
Sorry for the piece-by-piece nature of this post, I moved it from a dormant group to this one and it was 3 separate posts in the other group. Anyway... I'm trying to bulk insert a text file of 10 columns into a table with 12. How can I specify which columns to insert to? I think format files are what I'm supposed to use, but I can't figure them out. I've also tried using a view, as was suggested on one of the many websites I've...
35
2834
by: sacha.prins | last post by:
Hi, I read a lot about DB2 INSERT performance here. I have a nice story as well. The thing is, I work on 2 installations of DB2 (on completely different locations) which run on different (but same generation) hardware. Benchmarking the disk throughput and CPU basically amounts to the same figures (+/- 10%).
46
13148
by: dunleav1 | last post by:
I have a process that does inserts that runs 50% slower that Oracle and Mssql. Queries normally take 50% slower than normal. DB2, Oracle, Mssql all are configured on same os, same disk array (raid 1/0 - 15k disks) but DB2 I/O seems to be significantly slower. Tablespaces are SMS with automatic prefetch set. My thought was to create the log buffer array as large as possible so I don't have to switch to disk much. I'm running circular...
2
5563
by: Arun Srinivasan | last post by:
I need to speed up the inserts through informatica, and I came across insert buf option with db2 packages for import statements and how we can use that option to do the same in java programs. Are there any packages to be bound for informatica powercenter, so that I can rebind them with this option? My informatica person says that Informatica does not bind any packages to the database. Is he missing something. Any one who has worked in...
3
2972
by: Tawfiq | last post by:
Hi, We have shifted one of our data processing to a new instance of SQL 2005. For this a new DB was created in SQL2005 and it is not an upgrade of SQL200 DB. The data processing application is a VB6 application that runs a batch process to insert about 4.5 million records everyday in this SQL 2005 database. I am using ADO
3
3643
by: oravm | last post by:
Hi, I re-write a query and used bulk collect to improve the performance of the batch process. The query below has NO compile error but when execute query there is error 'ORA-01403: no data found.' CREATE OR REPLACE PROCEDURE PROCESS_ANGKASA(REF_NO varchar2)is v_cntr_code varchar2(16); v_receipt_code varchar2(3); start_time number; end_time number;
10
2866
by: nflacco | last post by:
I'm tinkering around with a data collection system, and have come up with a very hackish way to store my data- for reference, I'm anticipating collecting at least 100 million different dataId whatevers per year, possibly much more. ---366 data tables ( one for each day of the year ), each row being assigned a unique DataId ( unique across all 366 tables too ) ---100 data_map tables, table 0 having all DataIds ending in 00, table 99...
4
6578
by: shreyask | last post by:
I have been working on doing bulk updates and inserts through sqlapi++ using a bulk udpate extention library, developed, a long time ago, internally. The database is oracle. The bulk inserts/updates are done in batches of 50K. I am facing a very peculiar performance issue with regard to these bulk operations. The initial batch of bulk insert (50K records) is finished within 3 seconds; while the next similar batch for bulk insert or update...
0
9384
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
9212
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
9973
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
9790
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
8645
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
6473
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5247
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
3742
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
3276
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.