473,326 Members | 2,113 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,326 software developers and data experts.

Separating fields and values from log files

Hi everybody!

I have the following files: (MSC.080806.00, MSC.080806.01, MSC.080806.02...) each one has the same pattern, with two different START and STOP times and values:

...
START:2008/08/06 08:30:00 WED; STOP: 2008/08/06 09:00:00 WED;
...
TRK
KEY (COMMON_LANGUAGE_NAME)
INFO (OM2TRKINFO)
INCATOT PRERTEAB INFAIL NATTMPT NOVFLATB GLARE
OUTFAIL DEFLDCA DREU PREU TRU SBU
MBU OUTMTCHF CONNECT TANDEM AOF ANF
TOTU NANS ANSU ANSWER ACCCONG NOANSWER
...
75 RJOTLP1_TLMARC
2W 709 709
2272 0 0 3588 0 0
0 0 0 0 3672 0
0 0 3588 1200 0 0
3672 3369 2810 0 0 0

76 SPOBV2_FLO
2W 711 649
2272 0 0 3493 0 0
2 0 0 0 3707 0
0 0 3491 1216 0 0
3707 3334 3196 0 0 0
...
START:2008/08/06 09:00:00 WED; STOP: 2008/08/06 09:30:00 WED;
...
TRK
KEY (COMMON_LANGUAGE_NAME)
INFO (OM2TRKINFO)
INCATOT PRERTEAB INFAIL NATTMPT NOVFLATB GLARE
OUTFAIL DEFLDCA DREU PREU TRU SBU
MBU OUTMTCHF CONNECT TANDEM AOF ANF
TOTU NANS ANSU ANSWER ACCCONG NOANSWER
...
75 RJOTLP1_TLMARC
2W 709 709
3101 0 0 5095 0 0
4 0 0 0 5172 0
0 0 5091 1582 0 0
5172 4775 4025 0 0 0

76 SPOBV2_FLO
2W 711 649
3018 0 0 4960 0 0
8 0 0 0 5193 0
0 0 4952 1534 0 0
5193 4624 4402 0 0 0
...

Using another very valuable hint from you, I could separate the lines with START and STOP patterns to record them into MySQL after some cuts:

edu@ubuntu:~/routes# awk '/START/{ print; }' MSC.080806.20
START:2008/08/06 08:30:00 WED; STOP: 2008/08/06 09:00:00 WED;
START:2008/08/06 09:00:00 WED; STOP: 2008/08/06 09:30:00 WED;

And then, I separated the lines to save as columns names for my MySQL fields :

edu@ubuntu:~/routes# awk '/OM2TRKINFO/{ getline; print; }' MSC.080806.20
INCATOT PRERTEAB INFAIL NATTMPT NOVFLATB GLARE
INCATOT PRERTEAB INFAIL NATTMPT NOVFLATB GLARE

But, I have more 3 lines to do that:
edu@ubuntu:~/routes# awk '/OM2TRKINFO/{ getline; getline; print; }' MSC.080806.20
OUTFAIL DEFLDCA DREU PREU TRU SBU
OUTFAIL DEFLDCA DREU PREU TRU SBU

edu@ubuntu:~/routes# awk '/OM2TRKINFO/{ getline; getline; getline; print; }' MSC.080806.20
MBU OUTMTCHF CONNECT TANDEM AOF ANF
MBU OUTMTCHF CONNECT TANDEM AOF ANF

edu@ubuntu:~/routes# awk '/OM2TRKINFO/{ getline; getline; getline; getline; print; }' MSC.080806.20
TOTU NANS ANSU ANSWER ACCCONG NOANSWER
TOTU NANS ANSU ANSWER ACCCONG NOANSWER

I don´t think this getline; ...; getline way is the better way!!! It doesn’t look elegant or professional.

But the worst problem is how to get the fields values:

The file has:
75 RJOTLP1_TLMARC
2W 709 709
3101 0 0 5095 0 0
4 0 0 0 5172 0
0 0 5091 1582 0 0
5172 4775 4025 0 0 0

As an example, I need: values(FIELDS)

RJOTLP1_TLMARC (TRK)

3101 (INCATOT) 0 (PRERTEAB) 0 (INFAIL) 5095 (NATTMPT) 0 (NOVFLATB) 0 (GLARE)
And so on:
4 0 0 0 5172 0
0 0 5091 1582 0 0
5172 4775 4025 0 0 0

can anyone please help me with that? Sorry about the long post.

Best regards,

Eduardo
Aug 7 '08 #1
2 2475
I have never done anything like that before but if I were you I would look into a scripting language like Python. I know you could do it with c but as I don't know if you have any programming experience python might be easier. I will look into it a little more when I get home and see if I can give any specific suggestions.

A couple quick questions though:

First, are you greating a new MySQL database or are you using an existing one? If you are using an existing one, are you making new tables or using existing tables?

Second, is this a one time exercise or do you need a method to repeat this process in the future?

Edward
Aug 8 '08 #2
Hi Edward,

In fact, I noticed I might use Perl. I am studying Perl to implement that (I have many years of programming experience in VB and just some months ago I installed an Ubuntu LAMP.) I think Perl fits well for that, but it seems that Perl has some problem running Awk embebed... So it means some more code lines.

1. It´ll be a new table as that or something like that: CREATE TABLE routes(DATE, TIME, TRK, INCATOT, PRERTEAB, INFAIL, NATTMPT, NOVFLATB, GLARE, OUTFAIL, DEFLDCA, DREU, PREU, TRU, SBU, MBU, OUTMTCHF, CONNECT, TANDEM, AOF, ANF, TOTU, NANS, ANSU, ANSWER, ACCCONG, NOANSWER) and then INSERT INTO routes VALUES (mm/dd/yyyy, 08:30, RJOTLP1_TLMARC, 2272, 0, 0, 3588, 0, 0, 0, 0, 0, 0, 3672, 0, 0, 0, 3588, 1200, 0, 0, 3672, 3369, 2810, 0, 0, 0)

2. I´ll run that at crontab every half hour 24 x 7! These files have around 1.5Mb. Nowadays, it is done by excell VB and recorded on the net a sheet with data and charts. Now, I have to upgrade that to LAMP! I always used Windows, so now I am a kind of happy and exciting about the company agreed that we have to kill all legacies and put everything on the www, so I got this Ubuntu LAMP to do that.

Thank you so much!

Eduardo
Aug 11 '08 #3

Sign in to post your reply or Sign up for a free account.

Similar topics

1
by: steven virnig | last post by:
I know there has to be a way to do this, but I've gone brain dead. The scenario..a varchar field in a table contains a date range (i.e. June 1, 2004 - June 15, 2004 or September 1, 2004 -...
9
by: Anon Email | last post by:
Hi people, I'm learning about header files in C++. The following is code from Bartosz Milewski: // Code const int maxStack = 16; class IStack
13
by: Michelle | last post by:
Hi all... I could use a little TLC here for understanding and not for solving a specific problem... sorry if I've got the wrong group, but I'm using VB so I figured this was the most appropriate...
0
by: Thag | last post by:
Hi! I am currently trying to get an overview about the possibility of separating graphical data from its animations. More specifically speaking, the graphical data is made up from 3D-CAD...
2
by: Justin Fancy | last post by:
Hi everyone, I need some help. I'm placing text files into a created database using vb.Net. The problem is that, i need two seperate sql statements to add both files because they are in...
14
by: aaragon | last post by:
Hi everyone, I've been writing some code and so far I have all the code written in the .h files in order to avoid the linker errors. I'm using templates. I wanted to move the implementations to...
1
by: Darren | last post by:
Hi there. I am a hobbyist with no formal background in computer programming. Although years ago I’ve dabbled with a bit of the Microsoft QBasic and very simple MS-DOS batch files, javascript is...
3
by: gjok | last post by:
Hi. I just cant seem to get my head around n tier application architechture where you have separate UI and data layers. I just dont get it, and cant see the point I am finding myself spending more...
2
by: steph | last post by:
I have a table with 250 fields. Of course you are wondering why 250 fields... what could I possibly be storing in so many fields? I am using this table as a general import table for files that...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.