Hi
I'm trying to build a log viewer, its going to be part of a
contentmanageme nt system and needs some extra features that commersial
logviewers don´t have.
The log files (IIS log) are 200-500mb large and the problem is that it
taket to long to parse them.
I use textreader.read line and split on spases, and then I use that
infrormation.
When looping through the file (without actually doing anything) it
takes about 2-3 seconds and thats the time that a commercial log parser
takes to parse the log.
The commercial log parser also stores the data in a database, I tried
that just for fun, I used a access database and the procedure took
about 10 minutes, not to good.
My question is Can I speed up this?
I would like to have a dataset containing a number of tables,
containing Unique Useragents, ip adresses, visited pages (parsed from
querystring), visitors per hour and so on.
And I would like to do this in the time the commercial application did
it, lets be generous under 10 seconds.
Can it even be done using c#?
How would you guyes solve it?
Thanks
/Jimmy
3  8331 
"Jmc"
I'm trying to build a log viewer,
Can it even be done using c#?
Use the free Microsoft Log Parser 2.2
If Log Parser does not do what you need
then you can add your own "actions" called
"COM plug-ins or input processors" to make
Log Parser 2.2 do it!
And yes Log Parser can work with C# too!
Log parser is a powerful, versatile tool
that provides universal query access to
text-based data such as log files, XML
files and CSV files, as well as key data
sources on the Windows operating system
such as the Event Log, the Registry, the
file system, and Active Directory.
Download details: Log Parser 2.2 http://www.microsoft.com/downloads/d...displaylang=en
Windows Live Search http://www.live.com/?q=Log+Parser
Read the Log Parser Book!
Windows Live Search http://www.live.com/?q=Log+Parser+Book
Use the Log Parser forum
Windows Live Search http://www.live.com/?q=Log+Parser+forum
The many ways of Log Parser 2.2
will most likely do what you need! :)
Hi
You are right I could use Log Parser, I actually tried that as a first
resort, but to get all the information that I wanted I needed it make 5
(or so) queries so that didn't make things go very fast.
When the logfile went over 1,3GB (for one day) it just wasn't funny
anymore =).
I solved it by opening the file my self and used Hashtables to collect
the data that intereseted me.
The result was that my application parsed 1,3 gb logfile in about 40
seconds (about 10-15 of them was to pretty up User agents).
I'm "happy" and my customer is satified, so I'll leave it like it is
right now =).
For those who are thinking of writing something like this my recent
experience says: Log parser is the easiest and most nice way to read
log files, but it seems (at least when using .net) its faster to loop
through the file and collect the data.
Thanks for your help.
/Jimmy
Flowering Weeds wrote:
"Jmc"
I'm trying to build a log viewer,
Can it even be done using c#?
Use the free Microsoft Log Parser 2.2
If Log Parser does not do what you need
then you can add your own "actions" called
"COM plug-ins or input processors" to make
Log Parser 2.2 do it!
And yes Log Parser can work with C# too!
Log parser is a powerful, versatile tool
that provides universal query access to
text-based data such as log files, XML
files and CSV files, as well as key data
sources on the Windows operating system
such as the Event Log, the Registry, the
file system, and Active Directory.
Download details: Log Parser 2.2
http://www.microsoft.com/downloads/d...displaylang=en
Windows Live Search
http://www.live.com/?q=Log+Parser
Read the Log Parser Book!
Windows Live Search
http://www.live.com/?q=Log+Parser+Book
Use the Log Parser forum
Windows Live Search
http://www.live.com/?q=Log+Parser+forum
The many ways of Log Parser 2.2
will most likely do what you need! :)
"Jmc"
You are right I could use Log Parser,
Glad you got it running but Log Parser
should be able to do it faster (using any
programming language)! :)
So perhaps (when you get the time) you
could go over to the LogParser forum and
see if they can help you improve your Log
Parser queries (you know the customer will
want much more "log" information next time)! :) This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics | |
by: Steven Burn |
last post by:
The application;
Service on my webserver that allows a user to upload their HOSTS file for functions to verify the contents are still valid.
Uses;
1. XMLHTTP (MSXML2)
2. FileSystemObject
3. CrazyBeavers Upload control (couldn't get the Dundas one to work)
|
by: sahukar praveen |
last post by:
Hello,
I have a question.
I try to print a ascii file in reverse order( bottom-top). Here is the logic.
1. Go to the botton of the file fseek(). move one character back to avoid the EOF.
2. From here read a character, print it, move the file pointer (FILE*) to 2 steps back (using fseek(fp, -2, SEEK_CUR)) to read the previous character.
This seems to be ok if the file has a single line (i.e. no new line character). The above logic...
|
by: Michael Mair |
last post by:
Cheerio,
I would appreciate opinions on the following:
Given the task to read a _complete_ text file into a string:
What is the "best" way to do it?
Handling the buffer is not the problem -- the character
input is a different matter, at least if I want to remain within
the bounds of the standard library.
|
by: Rajorshi Biswas |
last post by:
Hi folks,
Suppose I have a large (1 GB) text file which I want to read in
reverse. The number of characters I want to read at a time is
insignificant. I'm confused as to how best to do it. Upon browsing
through this group and other sources on the web, it seems that there
are many ways to do it.
Some suggest that simply fseek'ing to 8K bytes before the end of file,
and going backwards is the way. In this case, am I guaranteed best
results...
|
by: rnorthedge |
last post by:
I am working on a code library which needs to read in the data from
large binary files. The files hold int, double and string data. This
is the code for reading in the strings:
protected internal override string ReadString()
{
stringLength = fileStream.ReadByte();
moInput.Read(byteArrayBuffer, 0, stringLength);
return asciiEncoding.GetString(byteArrayBuffer, 0, stringLength );
}
| | |
by: Hutty |
last post by:
I have a program that open text files and compares them, however, when
reading files larger than 500kb the programs seems to bomb. I get
re-directed to "page not found". Any idea how to get around this issue?
Ultimate goal is to read text files as big as 50mb or more. I'm using the
"input type=file " to upload file, and then using Streamreader to read the
text.
Thanks
--
Hutty
|
by: Brad |
last post by:
I'm working on a web app which will display LARGE tiff image files (e.g
files 10-20+ mb). Files are hidden from users direct access. For other,
smaller image files I have used FileStream to read in a file in a single
Read and so my quesitons are:
(1) What is a practical file size limit for reading using FileStream.Read
(reading the file in a single read)...especially on a web server where I
don't think I'd want to tax memory...
|
by: blazedaces |
last post by:
Ok, so you know my problem, java is running out of memory reading with SAX, the event-based xml parser intended more-so than DOM for extremely large files. I'll try to explain what I've been doing and why I have to do it. Hopefully someone has a suggestion...
Alright, so I'm using a gps-simulation program that outputs gps data, like longitude, lattitude, altitude, etc. (hundreds of terms, these are just the well known ones). In the newer...
|
by: akalmand |
last post by:
Hi there,
I am writing a code to read some data from the text files. The number of text files is not fixed and could be more that 15. the length of each file is large... close to 100,000 on an average. some of them are extra large. The data that I have to read will always be at the bottom and will be in the last 5 -20 line in the files depending upon their size. small files will have 5 line and large files will have 20 lines to read.
Can...
|
by: byte8bits |
last post by:
How does C++ safely open and read very large files? For example, say I
have 1GB of physical memory and I open a 4GB file and attempt to read
it like so:
#include <iostream>
#include <fstream>
#include <string>
using namespace std;
int main () {
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
| | |
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it.
Here is my compilation command:
g++-12 -std=c++20 -Wnarrowing bit_field.cpp
Here is the code in...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
|
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
| | |
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
|
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |
