Right now I develop an application to retrieve over 30,000 records from a binary file and then load them into a SQL Server DB. So far I load those records one by one, but the performance is very poor: it takes more than 6 hours to finish the loading.
So could I make use of the bulk-insert mechanism of SQL Server to reduce the loading time in C# codes? or other performance improvement solutions? 3 33229
moonriver wrote: Right now I develop an application to retrieve over 30,000 records from a binary file and then load them into a SQL Server DB. So far I load those records one by one, but the performance is very poor: it takes more than 6 hours to finish the loading.
So could I make use of the bulk-insert mechanism of SQL Server to reduce the loading time in C# codes? or other performance improvement solutions?
Hi moonriver,
as long as the ADO.NET doesn't provide an equivalent to IRowsetFastLoad
(OLEDB) you'll have to use BULK INSERT, BCP or DTS.
This article might help you in your decision which tool to use. http://msdn.microsoft.com/library/en...t_bcp_67oh.asp http://longhorn.msdn.microsoft.com/l...sqlclient.aspx
shows that bulk-insert-functionality might be build-in in .NET 2.0.
[BULK INSERT]
Reference of the BULK INSERT-Command: http://msdn.microsoft.com/library/?u...ba-bz_4fec.asp
This should be easy to use in C#, something like:
SqlCommand cm = new SqlCommand();
cmd.CommandText = "BULK INSERT myData\nFROM '"
+ fileName + "'\nWITH (BATCHSIZE="
+ sqlCount.ToStri ng()
+ ", FIELDTERMINATOR =';')";
//Console.WriteLi ne(cmd.CommandT ext);
cmd.ExecuteNonQ uery();
[bcp]
bcp can be used with System.Diagnost ics.Process.Sta rt
Reference: http://msdn.microsoft.com/library/?u...p_bcp_61et.asp
[DTS]
AFAIK there is no DTS object library (class) for .NET so one solution is
to use the DTS objects through the COM/interop.
An Example is here: http://sqldev.net/dts/ExecutePackage.htm
Another option will be to create a job with the DTS package as Step 1.
Then, you could use sp_start_job through a stored procedure that the
application executed through ADO.NET's command object.
HTH!
Cheers
Arne Janning
You can also make a stored procedure which has the bulk insert statement and
execute it through c#
Regards,
Sarfraz
"moonriver" <xi*******@yaho o.com> wrote in message
news:81******** *************** ***********@mic rosoft.com... Right now I develop an application to retrieve over 30,000 records from a
binary file and then load them into a SQL Server DB. So far I load those
records one by one, but the performance is very poor: it takes more than 6
hours to finish the loading. So could I make use of the bulk-insert mechanism of SQL Server to reduce
the loading time in C# codes? or other performance improvement solutions?
Bulk insert's poor, it requires admin privileges and is slower than bcp
bcp is your best bet, to run it in fast mode make sure there are no indexes on the tabl
(if your table needs indexes, drop them, bcp in, then recreate the indexes... yes this I
the fastest way of doing it This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: Chris |
last post by:
Any help would be appreciated.
I am running a script that does the following in succession.
1-Drop existing database and create new database
2-Defines tables, stored procedures and functions in the database
3-Imports data using bulk insert
4-Analyzes data using stored procedures
I would like to improve the performance of the analysis in step 4 by
|
by: php newbie |
last post by:
Hello,
I am trying to load a simple tab-delimited data file to SQL Server. I
created a format file to go with it, since the data file differs from
the destination table in number of columns.
When I execute the query, I get an error saying that only sysadmin or
bulkadmin roles are allowed to use the BULK INSERT statement. So, I
proceeded with the Enterprise Manager to grant myself those roles.
However, I could not find sysadmin or...
|
by: me |
last post by:
I'm also having problems getting the bulk insert to work. I don't know
anything about it except what I've gleened from BOL but I'm not seeming to
get anywhere...Hopefully there is some little (or big) problem with my code
that someone can point out that may save me some time.
TIA
CBL
|
by: iqbal |
last post by:
Hi all,
We have an application through which we are bulk inserting rows into a
view. The definition of the view is such that it selects columns from
a table on a remote server. I have added the servers using
sp_addlinkedserver on both database servers.
When I call the Commit API of oledb I get the following error:
Error state: 1, Severity: 19, Server: TST-PROC22, Line#: 1, msg:
|
by: pk |
last post by:
Sorry for the piece-by-piece nature of this post, I moved it from a
dormant group to this one and it was 3 separate posts in the other
group. Anyway...
I'm trying to bulk insert a text file of 10 columns into a table with
12. How can I specify which columns to insert to? I think format
files are what I'm supposed to use, but I can't figure them out. I've
also tried using a view, as was suggested on one of the many websites
I've...
| |
by: Philip Boonzaaier |
last post by:
I want to be able to generate SQL statements that will go through a list of
data, effectively row by row, enquire on the database if this exists in the
selected table- If it exists, then the colums must be UPDATED, if not, they
must be INSERTED.
Logically then, I would like to SELECT * FROM <TABLE>
WHERE ....<Values entered here>, and then IF FOUND
UPDATE <TABLE> SET .... <Values entered here> ELSE
INSERT INTO <TABLE> VALUES <Values...
|
by: Ted |
last post by:
I have BULK INSERT T-SQL statements that work for all of my basic data
tables except for one.
Here is the problem statement (with obvious replacement of irrelevant
path info):
BULK INSERT igbconts_tmp FROM 'C:\\my_code_path\\IGBCONTS.txt'
WITH (KEEPNULLS,
FORMATFILE = 'C:\\my_data_path\\contacts.fmt');
|
by: ozkhillscovington |
last post by:
We have sp's in place that do BULK INSERTS from txt files into the tables. This works fine, however they have asked us to add a field that identifies accounting ctr. The only thing that identifies accounting ctr is the last three letters of the text file being used for the BULK INSERT.
How would you suggest that I do this? Is there a way to add a default value to the .fmt or schema files for the text fields, or in the BULK INSERT...
|
by: nano |
last post by:
Does sql server have a way to handle errors in a sproc which would allow
one to insert rows, ignoring rows which would create a duplicate key
violation? I know if one loops one can handle the error on a row by row
basis. But is there a way to skip the loop and do it as a bulk insert?
It's easy to do in Access, but I'm curious to know if SQL Server proper
can handle like this. I am guessing that a looping operation would be
slower to...
|
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it.
Here is my compilation command:
g++-12 -std=c++20 -Wnarrowing bit_field.cpp
Here is the code in...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
| |
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
|
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |