473,599 Members | 3,227 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Bulk Insert Issue...

1 New Member
Hi Guys and Girls,

Tough one for me this, Im trying to bulk insert quite a big file, iv taken a small snippet of the file so you can slowly understand what im trying to get at! Please see below

24/06/2010 19:18:23 /IServerManager. GetPublicKey (timings: authenticate=0 authorise=0 execute=0 logError=0 teardown=0 total=0)
24/06/2010 19:18:23 /IServerManager. Login (timings: authenticate=0 authorise=0 execute=4524029 logError=0 teardown=468003 total=4992032)
24/06/2010 19:20:31 /IServerManager. GetPublicKey (timings: authenticate=0 authorise=0 execute=0 logError=0 teardown=0 total=0)

I have created a new database within SQL Server with the column names as follows

CREATE TABLE [dbo].[MedWayFiles](
[RowID] [int] IDENTITY(1,1) NOT NULL,
[Date and Time] [datetime] NULL,
[Commands] [nvarchar](50) NULL,
[Timings] [nvarchar](50) NULL,
[Authenticate] [nvarchar](50) NULL,
[Authorise] [nvarchar](50) NULL,
[Execute] [nvarchar](50) NULL,
[LogError] [nvarchar](50) NULL,
[TearDown] [nvarchar](50) NULL,
[Total] [nvarchar](50) NULL,
[SLA Breach] [nvarchar](50) NULL,
CONSTRAINT [PK_MedWayFiles] PRIMARY KEY CLUSTERED

As you can see the column names reference words that are within the Text file! Can some one please help me identifying the keywords within the text-file and assigning them to the
Relevant column! i.e bulkinsert! As I have 47 Text-Files and they all hold about 300+ lines of information and doing them one by one, well we all know will take many years!!!

Thanks in Advance!
Aug 10 '10 #1
1 1474
Jerry Winston
145 Recognized Expert New Member
This is doable. Just use BULK INSERT with ROWTERMINATOR like '\n' to access log entries one row at a time. Parse the log entry using CHARINDEX, SUBSTRING, and CAST inserting the derived values into your table [medWayFiles].

Once you perfect parsing the log strings, you can write a script to create the other 300 scripts. If your file names are numbered ie 'log0001.txt',' log0002.txt' the potential to create a loop to generate the BULK INSERT functions are obvious. If they is no pattern to the file names, use
Expand|Select|Wrap|Line Numbers
  1. dir /b *.[myLogFileExtension] > myLogFileList.txt
from the windows command shell to create a list of file names.

This code creates a list of BULK INSERT statements you can copy from the file list created using the windows command line.
Expand|Select|Wrap|Line Numbers
  1. CREATE TABLE #myLogFiles(LogEntryFile VARCHAR(MAX))
  2.  
  3. BULK INSERT #myLogFiles 'C:\data\myLogFileList.txt
  4. WITH
  5. (ROWTERMINATOR = '\n')
  6.  
  7. SELECT 'BULK INSERT [#LogEntry] ''' +
  8. LogEntryFile +''' WITH (ROWTERMINATOR = ''\n'')'
  9. FROM #myLogFiles
  10.  
From SSMS copy the code from the grid view below this code into a new query window:
Expand|Select|Wrap|Line Numbers
  1. CREATE TABLE #LogEntry (Entry VARCHAR(MAX))
Insert your parsing code below your BULK INSERT statements in the new window:
Expand|Select|Wrap|Line Numbers
  1. INSERT INTO [MedWayFiles]
  2. SELECT
  3. SUBSTRING
  4. (Entry,
  5. CHARINDEX('attribute1',Entry),
  6. CHARINDEX('attribute2',Entry)-CHARINDEX('attribute1',Entry)
  7. ),
  8. ...
  9. FROM
  10. #LogEntry
It's not elegant, but it'll get the job done. An elegant solution IMO would do this all in one punch making use of sp_executesql, CURSORs, and CTE's.
Aug 10 '10 #2

Sign in to post your reply or Sign up for a free account.

Similar topics

3
12984
by: Drew | last post by:
I am trying to use Bulk Insert for a user that is not sysadmin. I have already set up the user as a member of "bulkadmin". When I run the following script: DECLARE @SQL VARCHAR(1000) CREATE TABLE amdbo.#temp ( (10) NULL , (10) NULL , (10) NULL )
2
16495
by: php newbie | last post by:
Hello, I am trying to load a simple tab-delimited data file to SQL Server. I created a format file to go with it, since the data file differs from the destination table in number of columns. When I execute the query, I get an error saying that only sysadmin or bulkadmin roles are allowed to use the BULK INSERT statement. So, I proceeded with the Enterprise Manager to grant myself those roles. However, I could not find sysadmin or...
7
12111
by: iqbal | last post by:
Hi all, We have an application through which we are bulk inserting rows into a view. The definition of the view is such that it selects columns from a table on a remote server. I have added the servers using sp_addlinkedserver on both database servers. When I call the Commit API of oledb I get the following error: Error state: 1, Severity: 19, Server: TST-PROC22, Line#: 1, msg:
6
12347
by: pk | last post by:
Sorry for the piece-by-piece nature of this post, I moved it from a dormant group to this one and it was 3 separate posts in the other group. Anyway... I'm trying to bulk insert a text file of 10 columns into a table with 12. How can I specify which columns to insert to? I think format files are what I'm supposed to use, but I can't figure them out. I've also tried using a view, as was suggested on one of the many websites I've...
9
4225
by: adi | last post by:
Hi all, Hope there is a quick fix for this: I am inserting data from one table to another on the same DB. The insert is pretty simple as in: insert into datatable(field1, field2, field3) select a1, a2, a3 from temptable...
16
17000
by: Philip Boonzaaier | last post by:
I want to be able to generate SQL statements that will go through a list of data, effectively row by row, enquire on the database if this exists in the selected table- If it exists, then the colums must be UPDATED, if not, they must be INSERTED. Logically then, I would like to SELECT * FROM <TABLE> WHERE ....<Values entered here>, and then IF FOUND UPDATE <TABLE> SET .... <Values entered here> ELSE INSERT INTO <TABLE> VALUES <Values...
3
3451
by: Tim Satterwhite | last post by:
Hi All, I think this is a thorny problem, and I'm hoping you can help. I've not found this exact issue described anywhere yet. I have a stored procedure that calls BULK INSERT on a set of text files. These files are FTP'd from a legacy system (a mainframe running MVS). Sometimes, the process steps on iteslf, whereby the bulk insert is attempted on a file whose FTP is still in progress; it's not fully written to disk on the SQL box...
0
4167
by: Peter Nofelt | last post by:
Hi all, ISSUE: ==================== In SQL 2005 (sp2) I get the following error when preforming a bulk insert with an associated xml format file: "Could not bulk insert. Unknown version of format file" Question:
0
2620
by: Tawfiq | last post by:
Hi, I have got the following situation please give me some ideas how to solve/work around it. Current situation: Everyday day about 10 million records are processed and bulk inserted in individual tables. Around 20K records are inserted at a time. At the end of the day this table is clustered indexed. The field used for indexing is not a primary key. There is no primary key in this table. After indexing read access is given and
2
5885
by: zswanson | last post by:
Hi everyone, I receive a variety of text based files that seperates each row by a single delimiter, usually '~'. I use BULK INSERT to transfer the file from a text file to a temporary table. It works fine for most of the files (which use '~' as the delimiter for the rows), but for some reason I can't get it to work for a particular file. The file looks like it uses a new line character as the row delimiter, so I put in '\n' as the row...
0
7916
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
8401
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
8404
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
8268
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
1
5866
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
3900
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
3944
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
2418
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
0
1252
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.