Log file data mining

[David R]

I am writing an application in .net 1.1 using c#.
I need to read and store data from a log file.
The log file has 3 types of information:
-the date
-the time
-the event
All 3 are on seperate lines.
I can open the file and read each line.

Any suggestions on the collection I should use to store this information?
I will need to report information from this data.
i.e. the of occurances of a specific event
i.e. times between certian events

[Frinavale]

Consider using a DataBase, XML file, or CSV file that you can easily parse or select from.

[David R]

So storing the log file to an xml file would be better than an arraylist?
Do you have an example?

[Curtis Rutland]

When you say store, do you mean write to disk, or store in your program in memory....there's a big difference.

Frinny's suggestions are aimed at persistent storage.

[David R]

Sorry, I mean store in my program. I do not need to store to a db.

I'm looking for the best way to read the log file and then do some calculations.
ie. number of occurances, times between certain events, etc

[Curtis Rutland]

Write a simple class with those properties. Then store your data in a List<YourClass>.

[David R]

can u give an example?

[Curtis Rutland]

Sure. Your class would look something like this:

public class LogEntry
{
    public DateTime DateAndTime;
    public string Event;
    LogEntry(DateTime dAndT, string e)
    {
        DateAndTime = dAndT;
        Event = e;
    }
}

And your collection would look something like this:

System.Collections.ArrayList list = new System.Collections.ArrayList();

Add items to your collection using list.Add.

[Frinavale]

@insertAlias
Or instead of using an ArrayList you can use a List(Of LogEntry)....this lets you be more specific about the types of objects in your list.

[Curtis Rutland]

I had thought of that, but I'm pretty sure that doesn't exist in 1.1 :(

In C# it's List<LogEntry>

[Frinavale]

@insertAlias
1.1 was ages ago!
I can't even remember that far back!

You're right the List(T) Class doesn't exist in the 1.1 Framework: just 2.0 and forward (Generics are so helpful!)

[David R]

Thank You.
One last question -
Like I said, I'm using 1.1 for a stand alone windows app. No db.
Just reading/writing to a file.

Are there any benefits to upgrading to .Net 2 or even 3?

[Frinavale]

@David R
Simply put: Yes.

There are many new features that will make your life (as a developer) much easier. Some new controls have been added as well that can also make the end user experience more enjoyable as well. Check them out for yourself :)

However, if you're just reading and writing files...then there's no need to upgrade....unless of course you're working with XML documents.

In general I would strongly recommend upgrading to 3.5 because of the benefits it offers (LINQ is probably one of the most powerful)

[Curtis Rutland]

I would say yes as well. Especially if you are doing database stuff...

And if you go to .NET 3, you might as well go to .NET 3.5. It has all kinds of great tools. Plus, Visual Studio 2008 is just soooo much better than VS.NET

[David R]

getting back to my collection question -
so I will have 2 variable from the LogEntry class; a DateTime and Event string.
How do I add them both to a Arraylist?

[Curtis Rutland]

You don't. You use that class as a variable.

ArrayList list = new ArrayList();
LogEntry le = new LogEntry(DateTime.Parse("12/1/2008","Something");
list.Add(le);

Except that you need to do something like that in a loop, reading your lines in, and then creating a new LogEntry, then adding that LogEntry into the ArrayList.

You can use DateTime.Parse (which should work with a string like this: 03/01/2009 05:42:00 see this link for more details) or you can create a new DateTime(year, month, day, hour, minute, second.

[David R]

An arraylist would be best if I need to do some searching and some calcualtions?

[Curtis Rutland]

Hard to say. An array list is a useful way to store a non-fixed-length array of objects in memory. If you were using Framework 2.0 or later I'd tell you to use a System.Collections.Generic.List or even a Dictionary, but that's a moot point. A dictionary would make searching easier.

If you know the length of the array, you could just use LogEvent[numOfEvents] for less memory overhead.

Without knowing what you need to do to these objects, yes, I'd say use an arraylist.

But at a certain point, you're just going to have to try it out...I can't necessarily tell you what's best for your project until you have tried it and determined what the difficulties and problems are. See what you can do with this, and if you have any other problems, I'll be glad to help.

Make sure you make a new thread for new problems, if they don't directly relate to this thread topic.

[David R]

If i upgrade to Framework 3.5 and use the List class...
How would i add a "Date", "Time" and "Event" to the list?
How would I access the data in a list to count occurrances and perform calculations?

[Curtis Rutland]

Well, for one thing, I'd suggest holding both the date and time in one field, a DateTime field.

As for using a List, you would do things the same way. Here's a few examples:

//using statement
using System.Collections.Generic;
 
//creating the list
List<LogEntry> list = new List<LogEntry>();
 
//adding an element:
string date = "10/1/2008";
string time = "13:15:05";
string event = "something";
    //you would have gotten these values from your log file
LogEvent le = new LogEvent(DateTime.Parse(date + " " + time,event);
list.Add(le);
 
//retrieving an entry and accessing it's data:
LogEvent le2 = list[0]; //use whatever number it would be in the array, probably from a loop variable
string event2 = le2.Event;
DateTime dt = event2.DateAndTime;

Hope that clears things up. The only difference is if you use an arraylist, you will have to cast the value like this:

//assuming list is an arraylist:
LogEvent le2 = (LogEvent)list[0];