Need to know the best method to perform sort merge of log file con

[mesterak]

I'm trying to write a text log file processor but am having significant
performance issues.

* On average, there are about 100-200 files to process each file being about
1MB in size.
* Typically there are ~600k to 1m lines to process total. Every line in
each log file typically contains a date, a time, follwed by a textual message
* Unfortunately, not all log files have the same format/layout; e.g. some
may have only have month/day while others have year/moth/day; some may have
time with milliseconds (12:34:56.768) while other only have standard time
format (12:34:56)
* Log file contents from all files need to be sort merged to support either
viewing or post merge parsing/searches/whatever.

I have tried loading all of the file contents into memory, reformatting on
the fly to get dates, times, etc. in the same formart, and performing a sort
merge by comparing date and time stamps per line; which took forever (20-30
minutes on average.) I also tried using MS Log Parser which did the job
fairly but totally consumes CPU utilization (still took awhile as well.)

Surely there has to be a better approach without requiring a ton of memory
and thrashing disk I/O to read, reformat, and sort merge text log files. Any
suggestions and/or code examples?

Thanks,

Matt

[Cor Ligthert [MVP]]

Mesterak,

I answer in the way of a question. Why do you think that people has
investigated databases?

First it where random databases and because of the maintenance problems with
those they came what is now relational databases.

Have a look for SQL Express

Your alternative are doing it yourself and create binary files, however be
aware of what I wrote about maintenance.

I hope this helps,

Cor

[mesterak]

I don't require retention of the data thus, what maintenance is required?
None. Surely, there has to be some algorithm and/or best practice method out
there for doing this sort of thing without requiring a database.

Honestly, you have replied to three of my posts supplying equally vague
answers. I do appreciate your replies but am not finding the responses
beneficial in reaching solutions. You mention doing it myself and create
binary files. Ok, so how would this be done with some details and through
code examples?

"Cor Ligthert [MVP]" wrote:

Mesterak,

I answer in the way of a question. Why do you think that people has
investigated databases?

First it where random databases and because of the maintenance problems with
those they came what is now relational databases.

Have a look for SQL Express

Your alternative are doing it yourself and create binary files, however be
aware of what I wrote about maintenance.

I hope this helps,

Cor

[Cor Ligthert [MVP]]

Mesterak,

Honestly, you have replied to three of my posts supplying equally vague
answers. I do appreciate your replies but am not finding the responses
beneficial in reaching solutions. You mention doing it myself and create
binary files. Ok, so how would this be done with some details and through
code examples?

I was busy searching for the linefeed character for you. However if this is
your idea, feel free.

But honetsly, don't ask something to a newsgroup, if you are sure you know
the answers better. Most people don't want to be in a newsgroup with text
answering direct to the given solution, which is so bad, that it can be
thought that it was there idea afterwards.

Succes

Cor

[mesterak]

There was absolutely no need to take it personal. I was thanking you for
responding but was requesting more detail beyond you "hinting" as to a
solution path. You are the MVP with the answers, yes? If I knew the
answers, I wouldn't post questions. So again, I thank-you for responding and
only ask for more precision in your answers...only you know what you are
thinking, so you must please elaborate when responding or "break-it-down" for
us newbies...

Kindly,

Matt

"Cor Ligthert [MVP]" wrote:

Mesterak,

Honestly, you have replied to three of my posts supplying equally vague
answers. I do appreciate your replies but am not finding the responses
beneficial in reaching solutions. You mention doing it myself and create
binary files. Ok, so how would this be done with some details and through
code examples?

I was busy searching for the linefeed character for you. However if this is
your idea, feel free.

But honetsly, don't ask something to a newsgroup, if you are sure you know
the answers better. Most people don't want to be in a newsgroup with text
answering direct to the given solution, which is so bad, that it can be
thought that it was there idea afterwards.

Succes

Cor

[TT (Tom Tempelaere)]

Hi Matt,

"mesterak" wrote:

There was absolutely no need to take it personal. I was thanking you for
responding but was requesting more detail beyond you "hinting" as to a
solution path. You are the MVP with the answers, yes? If I knew the
answers, I wouldn't post questions. So again, I thank-you for responding and
only ask for more precision in your answers...only you know what you are
thinking, so you must please elaborate when responding or "break-it-down" for
us newbies...

That's not what Cor meant. He means that there are better tools for doing
what you do without having to code your own engine. Try to reuse as much as
possible. Joining is a *natural* operation in databases. You write a simple
query, 3-4 lines long and get the results in a table, while you would code an
entire engine only to find the same results.

Anyway, using a database may not be an option for you. In that case you
could use the ODBC driver for text files and join (merge) the logs that way.
It supports several data/time formats and other. You will have to write a
short descriptor file for the log file formats, what this does is describe
the log file as a table to the ODBC driver. Search the net or respective
newsgroups for more info. Then you write a SQL query that gets the data how
you want it.

If that isn't an option, then is not much more to say. Write it by hand.

Kind regards,
--
Tom Tempelaere.

[mesterak]

Thanks, I will check it out ;)

-Matt

"TT (Tom Tempelaere)" wrote:

Hi Matt,

"mesterak" wrote:

There was absolutely no need to take it personal. I was thanking you for
responding but was requesting more detail beyond you "hinting" as to a
solution path. You are the MVP with the answers, yes? If I knew the
answers, I wouldn't post questions. So again, I thank-you for responding and
only ask for more precision in your answers...only you know what you are
thinking, so you must please elaborate when responding or "break-it-down" for
us newbies...

That's not what Cor meant. He means that there are better tools for doing
what you do without having to code your own engine. Try to reuse as much as
possible. Joining is a *natural* operation in databases. You write a simple
query, 3-4 lines long and get the results in a table, while you would code an
entire engine only to find the same results.

Anyway, using a database may not be an option for you. In that case you
could use the ODBC driver for text files and join (merge) the logs that way.
It supports several data/time formats and other. You will have to write a
short descriptor file for the log file formats, what this does is describe
the log file as a table to the ODBC driver. Search the net or respective
newsgroups for more info. Then you write a SQL query that gets the data how
you want it.

If that isn't an option, then is not much more to say. Write it by hand.

Kind regards,
--
Tom Tempelaere.

[Cor Ligthert [MVP]]

Tom,

Beside this is about my other vague answers where Matt is talking about the
code from Jon (which Mat uses now) beside the buffer part completely based
on the same as my first answer to him.

Only Jon did write it in code for him.

Cor

[mesterak]

Whatever...

"Cor Ligthert [MVP]" wrote:

Tom,

Beside this is about my other vague answers where Matt is talking about the
code from Jon (which Mat uses now) beside the buffer part completely based
on the same as my first answer to him.

Only Jon did write it in code for him.

Cor