473,471 Members | 1,744 Online
Bytes | Software Development & Data Engineering Community
Create Post

Home Posts Topics Members FAQ

Scrolling Large Text File Without Hogging Memory

Tom
I don't want to re-invent the wheel and am looking for a simple
implementation of a text viewer or RichTextBox in read only mode that
allows rapid file positioning within large data files without the time
consuming and memory hogging associated with loading the entire file.

Using the thumb to get close and then paging and scrolling to get
exact placement within the file. Perhaps with only a few pages of data
loaded into memory.

I can't seem to snap out of my unmanaged way of thinking and how I'd
tackle this problem with pointers in C++. After too many years of
structured programming and with much effort I am beginning to gain
some working knowledge of these amazingly flexible objects. Can
someone please point me in a smart and C# style direction for solving
this not so uncommon and rather entry level task?

Thanks.

-- Tom
Dec 11 '07 #1
3 4499
Tom,

Well, how would you do it in C++? There isn't a facility out-of-the-box
to do this in .NET, but if you provide some insight as to how you would do
it in C++, I'm sure a number of people would be able to chime in to show how
you would do the equivalent in .NET.
--
- Nicholas Paldino [.NET/C# MVP]
- mv*@spam.guard.caspershouse.com

"Tom" <Th********@earthlink.netwrote in message
news:jf********************************@4ax.com...
>I don't want to re-invent the wheel and am looking for a simple
implementation of a text viewer or RichTextBox in read only mode that
allows rapid file positioning within large data files without the time
consuming and memory hogging associated with loading the entire file.

Using the thumb to get close and then paging and scrolling to get
exact placement within the file. Perhaps with only a few pages of data
loaded into memory.

I can't seem to snap out of my unmanaged way of thinking and how I'd
tackle this problem with pointers in C++. After too many years of
structured programming and with much effort I am beginning to gain
some working knowledge of these amazingly flexible objects. Can
someone please point me in a smart and C# style direction for solving
this not so uncommon and rather entry level task?

Thanks.

-- Tom

Dec 11 '07 #2
On Tue, 11 Dec 2007 22:08:06 -0800, Tom <Th********@earthlink.netwrote:
[...]
#1) One important observation is that StreamReader correctly imports
ASCII text data files into a RichTextBox because it defaults to UTF-8.
Why is this confusing? My texts state that .Net uses Unicode(UTF-16)?
..NET uses UTF-16 internally, but it supports a wide range of text formats
for reading and writing. Even in situations where the default encoding
used to read a file doesn't work, you can specify an encoding yourself and
..NET will convert.
[...]
#2) StreamReader calls File.OpenText that has attribute CanSeek =
false. Thus the only other C# option I have found so far is to use
FileStream. Unfortunately FileStream supports the reading of bytes and
not lines. There was some happy dancing here when I figured out the
byte_array -string -RichTexBox conversion and input.
I don't think your effort with FileStream was lost, since you learned more
about using encodings to convert bytes to or from strings.

But you can in fact seek with a StreamReader. You just need to use the
BaseStream for seeking, calling DiscardBufferedData() on the StreamReader
after you seek to synchronize the StreamReader with the underlying stream.
[...]
#3) Using the thumb to "Position" the pointer within the file,
clearing the RichTextBox and performing another RichTextBox input is
level one performance. Being able to smoothly scroll in both
directions using a buffer would kick it up several notches.

[...]
Wiki references the following >>

http://www.winterdom.com/dev/dotnet/index.html

The above has a FileMap and Patch that looks promising ... I hope to
explore them soon and hopefully understand them. Yikes! Most likely
over my head.
Well, as Nicholas asked, how would you do it in C++? Assuming you're
familiar with the native Win32 memory mapping API, then assuming the
memory mapping libraries you found work in a similar fashion, I'd guess
they would be able to handle the buffering useful for better performance..
It kind of depends how elaborate the implementations are, especially with
respect to going backwards (would they pre-load data from the file that
precedes data you're reading?) I didn't actually look at the libraries,
so I obviously don't know.

Of course, you don't have to use memory mapping techniques. You could
buffer the file yourself, keeping decent-size blocks in a linked list or
even just referenced in a small array (one small enough that shifting the
array elements is a trivial, fast operation).
>
#4) A crude scrolling/thumb implementation simple scales the percent
thumb travel to that of the pointer being Positioned to that same
percentage of the file.Length. Some clever scroll and cursor event
trapping within the RichTextBox might can effectively reload the
RichTextBox ... but I suspect less than smooth operation. Even so,
viewing portions of a huge file quickly and without hogging resources
is a good thing. :)
I'm not really sure what you're asking here. The default minimum and
maximum range for a ScrollBar is suited to percentages, but you can set
the range to whatever you want, up to the range of the Int32 type. Is
there something you're looking to do other than just setting the Maximum
property to the file length?

Finally, I'll suggest that whatever you do, trying to do this with an
actual RichTextBox is always going to be on the slow side (because you
need to reset the text of the RichTextBox whenever you scroll), and
possibly even prone to UI oddities depending on how you do it (having your
own scrollbar rather than using the textbox's solves some issues, but may
introduce new ones). You may find that in the long run it makes more
sense to implement your own textbox control that supports the
virtualization you're looking for.

Or, for that matter, maybe someone already has. Have you looked to see if
a class like this already exists?

Pete
Dec 12 '07 #3
Tom
Thanks Nicholas & Pete !!

Nicholas >>

You got me thinking and digging much harder to be able to respond
after my first posting. You'd probably be amazed at how many times I
rewrote that reply as I continuously found more useful tips in the
doco. At some point I needed to reply out of courtesy and let you know
progress was being made and your little push helped!!

Pete >>

Calling the BaseStream is so logical! I just don't think at that level
yet. Obviously there is an internal pointer that allows StreamReader
to function. I dug a little using Reflector (a new tool for me!) ...
but I never connected the dots.

Your comments on using a simplified class and handling the buffering
myself has me thinking outside my little box. I've searched a few
hours for enhancements on this project and I focused more on the
enhanced DataGridView offerings, tutorials and examples on binding,
and the WPF overviews than buffered text viewers. I need to spend some
additional time searching more thoroughly in that narrowed topic. Wow
there is a lot to absorb!

A one line comment I received the other day about CodeProject.com sure
has opened my eyes to the vastness of public domain class
enhancements. Once I began trying to get fancy with OwnerDraw in
ListView several realizations hit me. I'm still trying to comprehend a
bigger picture and the guidance I receive in here is absolutely the
MOST helpful.

Your comment on the ScrollBar range being Int32 leads to much cleaner
handling than working with percentages. The smallest thumb increment
on the largest files might span several pages and I'll have to work
out those issues with scroll and cursor events.

----------------------------------------------------------------

The good news is the help I receive in here also carries over to
improved usage of the online doco. The bad news is I still have a ton
of stupid to overcome. Hopefully a little less every day. ;)

Both you guys are batting 1,000 in my ledger. Thanks again.

-- Tom

Dec 12 '07 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
by: Jason Murry | last post by:
I have a camera system (Axis) which stores JPG via FTP 1-10fps. There is also a motion jpg live stream. I am trying to store these images either in JPG or in video format so they can be...
8
by: Rune Johansen | last post by:
Hi, I'm sorry if these questions are trivial, but I've searched the net and haven't had any luck finding the information I need. I need to perform some regular expression search and replace on...
11
by: CSN | last post by:
Is it possible to iterate over an array in plpgsql? Something like: function insert_stuff (rel_ids int) .... foreach rel_ids as id insert into table (rel_id, val) values (id, 5);
1
by: lwickland | last post by:
Summary: System.Net.ScatterGatherBuffers.MemoryChuck allocates inordinately large bytes when sending large post data. The following application consumes inordinate quantities of memory. My code...
1
by: Oberfuhrer | last post by:
Hello all VB.net friends ! i have done most of my programming in assembly, at least so far. Recently i decided to learn a high level language for windows programming. I didnt take long to...
8
by: knoxautoguy | last post by:
This problem has consumed a lot of my time, and I'm aftraid someone will tell me that my whole approach to this objective is wrong; however, I would like to know if there is a way to do this. I...
2
by: Sin Jeong-hun | last post by:
In short, is there any way to know the very code that is currently hogging the CPU? I've written an application which continously GETting web pages using HttpWebRequest's asynchronous methods...
7
by: robert.waters | last post by:
I have an Access database frontend linked via ODBC to a large (gigabytes) mysql database. I need to view a large amount of data in a a textbox (variable up to 300K), but I receive a 'there isnt...
20
by: Simon Strobl | last post by:
Hello, I tried to load a 6.8G large dictionary on a server that has 128G of memory. I got a memory error. I used Python 2.5.2. How can I load my data? SImon
0
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
0
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...
0
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing,...
1
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new...
0
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The...
0
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
0
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated ...
0
muto222
php
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.