473,654 Members | 3,072 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Need advice on a Data Import Routine

Hello,

I have a system that I am writing to automaticly import Credit Transaction
data into a SQL Server 2000 Database. I am using a VB.Net application to
detect when the file arives and prep it for parsing. The file is
aproximately 10Mb of relatively complex hierarchal data that is defined by 2
character tokens at the begining of each data line. The structure breaks out
into aproximately 6 parent-child related tables with numerous lookup tables
on each primary table.

The previous aproach loaded each line of raw data into an Import table and
used a Stored Procedure to break down the information into the tables. The
Stored Procedure is large and very complex. It also has the problem that the
Parsing takes so long I need to implement a very long timeout on my
connection or I get errors. It is also difficult to track progress through
the Import for the user to see.

The options I see at this point are;

1. Use the current approach of loading the raw data into an Import table and
Parsing with a Stored Procedure.

2. Write the Parsing Routine in VB.NET and use ADO.NET to load the data into
the apropriate tables.

3. Use a mixed aproach of breaking the massive Stored Procedure into smaller
procedures and managing the steps with VB.Net.

I am looking for any tips and advice that the guru's out there are willing
to give. Especially a comparision of the effiency of doing ADO.NET bulk
inserts vs. Stored Procedure parsing.

Thanks for any help given.
Doug

PS - Sorry for the cross posting in sqlserver.progr amming but I am hoping to
get a balanced set of views on this issue.
Jul 21 '05 #1
0 1564

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

10
3091
by: Jeff Wagner | last post by:
I am in the process of learning Python (obsessively so). I've been through a few tutorials and read a Python book that was lent to me. I am now trying to put what I've learned to use by rewriting that Numerology program I wrote years ago in VB. There are times I am totally stuck (for instance, I just had an idea to put the numerical values of the alphabet and months of the year in a dictionary located in a function. Then, I can import the...
5
1720
by: yvan | last post by:
Approximately once a month, a client of ours sends us a bunch of comma-delimited text files which I have to clean up and then import into their MS SQL database. All last week, I was using a Cold Fusion script to upload the cleaned up files and then import the records they contained into the database, though obviously, the process took friggin' forever, and could have been done 500x quicker had I done it directly on the server. My SQL...
1
1970
by: LenS | last post by:
If this is the wrong place to post this, please advise better place. Otherwise, I have created the following python program and it works. Running on XP. I think I am now at that stage of learning python where I'm not quit a newbie and I am not really knowlegable. I know just enough to be dangerous and can really screw things up. Any suggestion on improving would be greatly appreciated. However my real question is I would like to run...
30
4145
by: nephish | last post by:
Hey there, i have tried about every graphing package for python i can get to work on my system. gnuplot, pychart, biggles, gdchart, etc.. (cant get matplot to work) so far, they all are working ok. I need to make an x y chart for some data that comes in from sensors at different times durring the day. i need it to show the value in the y and the time in the x . no problem so far. But what i cannot get to happen is to scale x (time of the...
9
4030
by: Edward S | last post by:
I budget for a Project in an Excel sheet as illustrated below. The months below are usually a 2 year period i.e. 24 months, though it could be over 24 months depending upon a Project. I then need to input this in an Access database, where I do a comparison with the Actual cost. The table “TblBudget” in Access is made of 4 fields, namely: (1) CostElement (2) CostCenter (3) Month (4) Amount$. At the moment this method is very cumbersome....
0
271
by: Doug R | last post by:
Hello, I have a system that I am writing to automaticly import Credit Transaction data into a SQL Server 2000 Database. I am using a VB.Net application to detect when the file arives and prep it for parsing. The file is aproximately 10Mb of relatively complex hierarchal data that is defined by 2 character tokens at the begining of each data line. The structure breaks out into aproximately 6 parent-child related tables with numerous...
9
1374
by: Peted | last post by:
Hi Looking for advice on best way to implement this idea I have a form with 10 text boxes, and one button. when user clicks button i want to transmit info in text boxes to a device, but i only want to transmit data from textboxes that user has changed data in since last button click How can i check if the text box contents have been changed by user, so
6
1405
by: Steven W. Orr | last post by:
Given the following code: (I hope it's as simple as possible) :-) #! /usr/bin/python import new class BASE: def __init__( self ): print 'Hello from BASE init' def m1( self ): print 'M1 Base: Self = ', self def m1replace( self ):
0
8814
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
8706
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
8475
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
8591
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
7304
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
5621
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
4149
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
4293
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
1915
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.