473,725 Members | 2,232 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Need advice on a Data Import Routine

Hello,

I have a system that I am writing to automaticly import Credit Transaction
data into a SQL Server 2000 Database. I am using a VB.Net application to
detect when the file arives and prep it for parsing. The file is
aproximately 10Mb of relatively complex hierarchal data that is defined by 2
character tokens at the begining of each data line. The structure breaks out
into aproximately 6 parent-child related tables with numerous lookup tables
on each primary table.

The previous aproach loaded each line of raw data into an Import table and
used a Stored Procedure to break down the information into the tables. The
Stored Procedure is large and very complex. It also has the problem that the
Parsing takes so long I need to implement a very long timeout on my
connection or I get errors. It is also difficult to track progress through
the Import for the user to see.

The options I see at this point are;

1. Use the current approach of loading the raw data into an Import table and
Parsing with a Stored Procedure.

2. Write the Parsing Routine in VB.NET and use ADO.NET to load the data into
the apropriate tables.

3. Use a mixed aproach of breaking the massive Stored Procedure into smaller
procedures and managing the steps with VB.Net.

I am looking for any tips and advice that the guru's out there are willing
to give. Especially a comparision of the effiency of doing ADO.NET bulk
inserts vs. Stored Procedure parsing.

Thanks for any help given.
Doug

PS - Sorry for the cross posting in sqlserver.progr amming but I am hoping to
get a balanced set of views on this issue.
Jul 21 '05 #1
0 1571

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

10
3098
by: Jeff Wagner | last post by:
I am in the process of learning Python (obsessively so). I've been through a few tutorials and read a Python book that was lent to me. I am now trying to put what I've learned to use by rewriting that Numerology program I wrote years ago in VB. There are times I am totally stuck (for instance, I just had an idea to put the numerical values of the alphabet and months of the year in a dictionary located in a function. Then, I can import the...
5
1726
by: yvan | last post by:
Approximately once a month, a client of ours sends us a bunch of comma-delimited text files which I have to clean up and then import into their MS SQL database. All last week, I was using a Cold Fusion script to upload the cleaned up files and then import the records they contained into the database, though obviously, the process took friggin' forever, and could have been done 500x quicker had I done it directly on the server. My SQL...
1
1974
by: LenS | last post by:
If this is the wrong place to post this, please advise better place. Otherwise, I have created the following python program and it works. Running on XP. I think I am now at that stage of learning python where I'm not quit a newbie and I am not really knowlegable. I know just enough to be dangerous and can really screw things up. Any suggestion on improving would be greatly appreciated. However my real question is I would like to run...
30
4164
by: nephish | last post by:
Hey there, i have tried about every graphing package for python i can get to work on my system. gnuplot, pychart, biggles, gdchart, etc.. (cant get matplot to work) so far, they all are working ok. I need to make an x y chart for some data that comes in from sensors at different times durring the day. i need it to show the value in the y and the time in the x . no problem so far. But what i cannot get to happen is to scale x (time of the...
9
4035
by: Edward S | last post by:
I budget for a Project in an Excel sheet as illustrated below. The months below are usually a 2 year period i.e. 24 months, though it could be over 24 months depending upon a Project. I then need to input this in an Access database, where I do a comparison with the Actual cost. The table “TblBudget” in Access is made of 4 fields, namely: (1) CostElement (2) CostCenter (3) Month (4) Amount$. At the moment this method is very cumbersome....
0
271
by: Doug R | last post by:
Hello, I have a system that I am writing to automaticly import Credit Transaction data into a SQL Server 2000 Database. I am using a VB.Net application to detect when the file arives and prep it for parsing. The file is aproximately 10Mb of relatively complex hierarchal data that is defined by 2 character tokens at the begining of each data line. The structure breaks out into aproximately 6 parent-child related tables with numerous...
9
1377
by: Peted | last post by:
Hi Looking for advice on best way to implement this idea I have a form with 10 text boxes, and one button. when user clicks button i want to transmit info in text boxes to a device, but i only want to transmit data from textboxes that user has changed data in since last button click How can i check if the text box contents have been changed by user, so
6
1408
by: Steven W. Orr | last post by:
Given the following code: (I hope it's as simple as possible) :-) #! /usr/bin/python import new class BASE: def __init__( self ): print 'Hello from BASE init' def m1( self ): print 'M1 Base: Self = ', self def m1replace( self ):
0
8888
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
8752
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
9401
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
9113
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
8097
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
6702
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
6011
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
4519
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
2
2635
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.