473,396 Members | 1,838 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,396 software developers and data experts.

Performant Flat File SQL Insert question

Hi All,
I am receiving a flat file data feed from another system. This is
comming in every 10 minutes and is going to need to be a high
permormance system. The file will contain all the data every time,
which might require a truncate and re-fill of the data (I haven't
figured out if they'll let me update yet, but because of performance
concerns inserts are cheaper then updates) I figure to use
SystemFileWatch to catch the file comming in. Convert it to XML, which
required splitting the flat file up, and re-writting it. Then using
OPENXML to insert it into a database.
Let me know what you guys think. I'm pretty concerned about this
system. The web page that displays the data is going to have high
traffic.
Thanks for any ideas and thoughts

Apr 12 '06 #1
2 2350
For the best performance you'll want to use SqlBulkCopy, have a look at:
http://msdn2.microsoft.com/en-US/lib...py(VS.80).aspx

About the OpenXML idea, if you files are large you may run into issues using
this method.

Sayed Ibrahim Hashimi
www.sedodream.com

"Red2" wrote:
Hi All,
I am receiving a flat file data feed from another system. This is
comming in every 10 minutes and is going to need to be a high
permormance system. The file will contain all the data every time,
which might require a truncate and re-fill of the data (I haven't
figured out if they'll let me update yet, but because of performance
concerns inserts are cheaper then updates) I figure to use
SystemFileWatch to catch the file comming in. Convert it to XML, which
required splitting the flat file up, and re-writting it. Then using
OPENXML to insert it into a database.
Let me know what you guys think. I'm pretty concerned about this
system. The web page that displays the data is going to have high
traffic.
Thanks for any ideas and thoughts

Apr 14 '06 #2
used Bulk Copy, it worked great, thanks. I had to use Typed Data Sets
to assist in gettting everything lined up. SQLBulkCopy doesn't have
really good error handling. But it's fast and easy. Thanks

Apr 17 '06 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: BT3 | last post by:
I am trying to replace a single record in a flat file. The file is relatively small and no need for database. I open the file, and save the ftell() value in a variable. I read a record using...
1
by: .d.hos | last post by:
ok, fairly new to python, relatively familiar w/ ms-sql. here's my issue: my .py script parses the contents of a (tab delim.) flat file, then attempts to insert the information into the db. I've...
13
by: raykyoto | last post by:
Hi all, I'm sure this is a popular question that comes up every few months here. Indeed, I've looked at some of the past postings, but I would like to ask things differently. Basically, I'm...
22
by: Daniel Billingsley | last post by:
Ok, I wanted to ask this separate from nospam's ridiculous thread in hopes it could get some honest attention. VB6 had a some simple and fast mechanisms for retrieving values from basic text...
2
by: Bill Nguyen | last post by:
I need to parse a text file with fixed columns into a table. I have the file spec (column positions) but don't know how to use it. Thanks Bill
3
by: blackd77 | last post by:
What I would LIKE to do is noted in the subject line. What I'm finding is that "edit SQL" appears to only be an option if I am creating a table. If I select "append to" the option to edit SQL...
9
by: FFMG | last post by:
In my site I have a config table, (MySQL), with about 30 entries; the data is loaded on every single page load. This is not the only call to the db, (we do a total of about 8 calls to the db). As...
15
by: lxyone | last post by:
Using a flat file containing table names, fields, values whats the best way of creating html pages? I want control over the html pages ie 1. layout 2. what data to show 3. what controls to...
3
by: Tim | last post by:
Hi Folks, I'm used to a UNLOAD command that allows me to dump to a named flat file the results of any SELECT statement. Hence one can build a single SQL file which contains multiple SQL...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: ryjfgjl | last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
0
BarryA
by: BarryA | last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
0
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
0
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.