473,549 Members | 3,105 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Parsing log file using C#

I need code that can parse data from log file.
The log is pipe delimited and in following format:
time=2006-11-03 13:13:56| fw=199.23.48.12 0| user=tjolen| realm=EMEA| roles=Network Connect| src=24.141.197. 139| dstname=| duration=| msg=AUT22670: Login succeeded for tjolen/EMEA.

time=2006-11-03 13:14:07| fw=199.23.48.12 0| user=tjolen| realm=EMEA| roles=Network Connect| src=63.131.197. 139| dstname=TUN-VPN| duration=| msg=JAV20021: Connected to EPN port 443

time=2006-11-03 13:16:10| fw=199.23.38.12 0| user=tjolen| realm=EMEA| roles=Network Connect| src=34.131.127. 139| dstname=| duration=| msg=AUT22673: Logout from 24.131.127.139

Purpose is to read from text file and store in sql server database.
Please help!
Apr 8 '08 #1
7 6352
There are several good open-source implementations of CSVReader that you can
use for this. Just google "C#" CsvReader.
-- Peter
Site: http://www.eggheadcafe.com
UnBlog: http://petesbloggerama.blogspot.com
Short Urls & more: http://ittyurl.net
"Prem Parekh" wrote:
I need code that can parse data from log file.
The log is pipe delimited and in following format:
time=2006-11-03 13:13:56| fw=199.23.48.12 0| user=tjolen| realm=EMEA| roles=Network Connect| src=24.141.197. 139| dstname=| duration=| msg=AUT22670: Login succeeded for tjolen/EMEA.

time=2006-11-03 13:14:07| fw=199.23.48.12 0| user=tjolen| realm=EMEA| roles=Network Connect| src=63.131.197. 139| dstname=TUN-VPN| duration=| msg=JAV20021: Connected to EPN port 443

time=2006-11-03 13:16:10| fw=199.23.38.12 0| user=tjolen| realm=EMEA| roles=Network Connect| src=34.131.127. 139| dstname=| duration=| msg=AUT22673: Logout from 24.131.127.139

Purpose is to read from text file and store in sql server database.
Please help!
Apr 8 '08 #2
On Tue, 08 Apr 2008 12:10:52 -0700, Prem Parekh wrote:
>I need code that can parse data from log file.
The log is pipe delimited and in following format:
time=2006-11-03 13:13:56| fw=199.23.48.12 0| user=tjolen| realm=EMEA| roles=Network Connect| src=24.141.197. 139| dstname=| duration=| msg=AUT22670: Login succeeded for tjolen/EMEA.

time=2006-11-03 13:14:07| fw=199.23.48.12 0| user=tjolen| realm=EMEA| roles=Network Connect| src=63.131.197. 139| dstname=TUN-VPN| duration=| msg=JAV20021: Connected to EPN port 443

time=2006-11-03 13:16:10| fw=199.23.38.12 0| user=tjolen| realm=EMEA| roles=Network Connect| src=34.131.127. 139| dstname=| duration=| msg=AUT22673: Logout from 24.131.127.139

Purpose is to read from text file and store in sql server database.
Please help!
Have a look at String.Split()

rossum

Apr 8 '08 #3
On Apr 8, 3:10*pm, Prem Parekh wrote:
I need code that can parse data from log file.
The log is pipe delimited and in following format:
time=2006-11-03 13:13:56| fw=199.23.48.12 0| user=tjolen| realm=EMEA| roles=Network Connect| src=24.141.197. 139| dstname=| duration=| msg=AUT22670: Login succeeded for tjolen/EMEA. * * *

time=2006-11-03 13:14:07| fw=199.23.48.12 0| user=tjolen| realm=EMEA| roles=Network Connect| src=63.131.197. 139| dstname=TUN-VPN| duration=| msg=JAV20021: Connected to EPN port 443 * * *

time=2006-11-03 13:16:10| fw=199.23.38.12 0| user=tjolen| realm=EMEA| roles=Network Connect| src=34.131.127. 139| dstname=| duration=| msg=AUT22673: Logout from 24.131.127.139 *

Purpose is to read from text file and store in sql server database.
Please help!
Hi,

You can split the line using String.Split( new char[] {'|'}); and then
later split each piece by the '=' then you have a pair of key,value.
Apr 8 '08 #4
Hi,
try this....

string str = System.IO.File. ReadAllText("Pa thName");

now str hold the content of the file... you can get substring and split the values based on your criteria and use it...
Apr 9 '08 #5
Vasanth,

Thanks for your reply. I've done similar to what you suggested. Now how I can I put this parsed text into a table by removing column names (time, fw,etc) that are repeating in everyline and write it to Sql Server.
Should I use Dataset?
Please advise.

My code:
using (StreamReader sr = new StreamReader("c :\\textfiles\\m odified.txt"))
{
String str;
// Read and display lines from the file until the end of
// the file is reached.
while ((str = sr.ReadLine()) != null)
{
string[] arInfo = new string[4];

// define which character is seperating fields
char[] splitter = { '|' };

arInfo = str.Split(split ter);

for (int x = 0; x < arInfo.Length; x++)
{
Console.Write(a rInfo[x] + "<br>");
}

}
}
Apr 9 '08 #6
It depends on the length of the file.

The fastest way to get it into a SQL Server is using SqlBulkCopy -
WriteToServer.
If it is reasonable to hold it in memory, then DataTable is a
quick'n'dirty answer. If the file is very large, you will want to
stream, by impersonating an IDataReader. You could steal
SimpleDataReade r from here:
http://groups.google.com/group/micro...c7a20056ffe8e1

and the rest of your code replaces XmlDataReader (from the same post);
in particular, your DoRead override would read the next line [return
false if EOF], call SetValues(arInf o) and return true.

Marc
Apr 9 '08 #7
For info - I've just (for fun) written a LINQ-based implementation of
this that makes it easy to chain an object stream to an IDataReader.
Let me know if you'd find it useful,
Apr 9 '08 #8

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
3641
by: Willem Ligtenberg | last post by:
I decided to use SAX to parse my xml file. But the parser crashes on: File "/usr/lib/python2.3/site-packages/_xmlplus/sax/handler.py", line 38, in fatalError raise exception xml.sax._exceptions.SAXParseException: NCBI_Entrezgene.dtd:8:0: error in processing external entity reference This is caused by: <!DOCTYPE Entrezgene-Set PUBLIC...
2
3927
by: Cigdem | last post by:
Hello, I am trying to parse the XML files that the user selects(XML files are on anoher OS400 system called "wkdis3"). But i am permenantly getting that error: Directory0: \\wkdis3\ROOT\home Canonicalpath-Directory4: \\wkdis3\ROOT\home\bwe\ You selected the file named AAA.XML getXmlAlgorithmDocument(): IOException Not logged in
3
3486
by: Pir8 | last post by:
I have a complex xml file, which contains stories within a magazine. The structure of the xml file is as follows: <?xml version="1.0" encoding="ISO-8859-1" ?> <magazine> <story> <story_id>112233</story_id> <pub_name>Puleen's Publication</pub_name> <pub_code>PP</pub_code> <edition_date>20031201</edition_date>
4
6827
by: Neil.Smith | last post by:
I can't seem to find any references to this, but here goes: In there anyway to parse an html/aspx file within an asp.net application to gather a collection of controls in the file. For instance what I'm trying to do is upload a html file onto the web server, convert it to aspx file and then parse it for input tags/controls, which in turn...
3
4366
by: toton | last post by:
Hi, I have some ascii files, which are having some formatted text. I want to read some section only from the total file. For that what I am doing is indexing the sections (denoted by .START in the file) with the location. And for a particular section I parse only that section. The file is something like, .... DATAS
9
1973
by: Paulers | last post by:
Hello, I have a log file that contains many multi-line messages. What is the best approach to take for extracting data out of each message and populating object properties to be stored in an ArrayList? I have tried looping through the logfile using regex, if statements and flags to find the start and end of each message but I do not see a...
3
3298
by: Anup Daware | last post by:
Hi Group, I am facing a strange problem here: I am trying to read xml response from a servlet using XmlTextWriter. I am able to read the read half of the xml and suddenly an exception: “Unexpected end of file while parsing Name has occurred” isbeing thrown. Following is the part o xml I am trying to read: <CHECK_ITEM_OUT>
13
4475
by: Chris Carlen | last post by:
Hi: Having completed enough serial driver code for a TMS320F2812 microcontroller to talk to a terminal, I am now trying different approaches to command interpretation. I have a very simple command set consisting of several single letter commands which take no arguments. A few additional single letter commands take arguments:
13
2792
by: charliefortune | last post by:
I am fetching some product feeds with PHP like this $merch = substr($key,1); $feed = file_get_contents($_POST); $fp = fopen("./feeds/feed".$merch.".txt","w+"); fwrite ($fp,$feed); fclose ($fp); and then parsing them with PHP's native parsing functions. This is succesful for most of the feeds, but a couple of them claim to be
2
3598
by: Felipe De Bene | last post by:
I'm having problems parsing an HTML file with the following syntax : <TABLE cellspacing=0 cellpadding=0 ALIGN=CENTER BORDER=1 width='100%'> <TH BGCOLOR='#c0c0c0' Width='3%'>User ID</TH> <TH Width='10%' BGCOLOR='#c0c0c0'>Name</TH><TH width='7%' BGCOLOR='#c0c0c0'>Date</TH> and so on.... whenever I feed the parser with such file I get the...
0
7718
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
0
7956
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
1
7470
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For...
0
6041
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then...
0
5088
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert...
0
3498
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in...
1
1936
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
1
1058
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
0
763
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.