Hi all,
I have to parse two text files on a weekly basis. Each range from 300kb to
1mb in total. Each text file has 5 columns (name,id, dollar,
startdate,enddate), everytime,
a) I need to parse each row, extract each column
2) check if the data already exisinst in the db between startdate and end
date
3) if not, then insert them into the the database, else, modify the record
with the new data.
As you can imagine, with size like 1mb, doing this row by row is not fast
nor efficient as it hits the db so many times (around 8000 -10,000 rows)
I'm wondering what would be a faster and efficent way to do this? I'm
thinking of a solution, but would love to get some input. Read into a
dataTable first and then modify the rowState? SqlBulkInsert? Is there a way
to parse large txt file into memory FAST?
thanks. Any suggestion is grealty appreciated. 5 3732
I would use the SQL Bulk load tool (assuming you use SQL server).
This is fast and does the parsing for you. Then you have everything in
the db and just need to run a query on top of it.
Remy
Hello Remy,
I'm not familiar with the sql bulk load tool. How do I invoke it within my
asp.net C# source?
I know ADO .NET 2.0 offers sql bulk insert, but i guess that's different.
"Remy" <rb********@hotmail.com> wrote in message
news:11*********************@q12g2000cwa.googlegro ups.com... I would use the SQL Bulk load tool (assuming you use SQL server). This is fast and does the parsing for you. Then you have everything in the db and just need to run a query on top of it.
Remy
just did a quick search on sql bulk load, i have sql server 2000, but i
don't have access to its exe files. the sql server is hosted somwhere else.
(i'm using hosting solution)
"Remy" <rb********@hotmail.com> wrote in message
news:11*********************@q12g2000cwa.googlegro ups.com... I would use the SQL Bulk load tool (assuming you use SQL server). This is fast and does the parsing for you. Then you have everything in the db and just need to run a query on top of it.
Remy
Yeah, then you are most likely out of luck in terms of the SQL bulk
load tool.
What helps a little bit is if you combine a bunch of INSERT statements
into one call.
I think you can combine by separating the statements with a ;
Hi liming,
I agree with Remy on this.
Since you have limited access to the database server, bulk insert is not
possible here.
Regarding Remy's suggestion on submitting multiple sql statements in one
call, can you test and verify the speed improvement? Thanks.
Regards,
Walter Wang
Microsoft Online Community Support
==================================================
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights. This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics
by: Peter A. Schott |
last post by:
I've got a file that seems to come across more like a dictionary from what I can
tell. Something like the following format:
###,1,val_1,2,val_2,3,val_3,5,val_5,10,val_10...
|
by: Ram Laxman |
last post by:
Hi all,
I have a text file which have data in CSV format.
"empno","phonenumber","wardnumber"
12345,2234353,1000202
12326,2243653,1000098
Iam a beginner of C/C++ programming.
I don't know how to...
|
by: nate |
last post by:
Hello,
Does anyone know where I can find an ASP server side script written in
JavaScript to parse text fields from a form method='POST' using
enctype='multipart/form-data'? I'd also like it to...
|
by: Brian |
last post by:
Hi all,
I have an xml document that can contain 248 nodes with
each node containing different fields. There is the
possibility that there will be more nodes. Each node can
have up to 30...
|
by: Nat |
last post by:
Hi
<..I'm useing c# asp.net, and it's webapplications..>
I have a content.xml file with some content id's, and a text
(fx.: <content Id = bla bla bla bla></content id>
The text in...
|
by: Sam |
last post by:
Hi Everybody,
I'm using C# and System.XML and can parse XML files like this:
<LogIn>Yes</LogIn>
<Name>DB_MAGUser</Name>
but I've been asked to parse and work with a XML file that looks...
|
by: akira |
last post by:
Hello,
I need to parse this XML file or XML stream . Is it possible to parse
it with DOM reading each DataContainer one by one.
<DataContainer>
<name> </name>
<birthDate> </birthDate>...
|
by: DH |
last post by:
Hi,
I'm trying to strip the html and other useless junk from a html page..
Id like to create something like an automated text editor, where it
takes the keywords from a txt file and removes them...
|
by: markbfernandez |
last post by:
I have an "Update Customer" form that doesn't work as .xhtml. Here's
where it stops working:
<input type="text" name="ud_first" id="ud_first"
value="<? echo "$first"?>" />
-------^
When the...
|
by: Charles Arthur |
last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
|
by: aa123db |
last post by:
Variable and constants
Use var or let for variables and const fror constants.
Var foo ='bar';
Let foo ='bar';const baz ='bar';
Functions
function $name$ ($parameters$) {
}
...
|
by: ryjfgjl |
last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
|
by: emmanuelkatto |
last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud.
Please let me know.
Thanks!
Emmanuel
|
by: nemocccc |
last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
|
by: Hystou |
last post by:
There are some requirements for setting up RAID:
1. The motherboard and BIOS support RAID configuration.
2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
|
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers,...
| |