472,989 Members | 2,975 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 472,989 software developers and data experts.

different string representation (buffer gap)

Hi all --

I'm contemplating the idea of writing a simple emacs-like editor in
python (for fun and the experience of doing so). In reading through
Craig Finseth's "The Craft of Text Editing":

http://www.finseth.com/~fin/craft/

, I've come across the "buffer gap" representation for the text data
of the buffer. Very briefly, it keeps the unallocated memory of the
character array at the editing point, so that as long as there is
memory available, an insert/delete is very low (constant) cost. Of
course, moving the editing point means copying some character data so
that the gap moves with you, but...

Anyway, I'm wondering what straightforward ways to leverage /
implement this representation in Python. Ideally, it would be great
if one could use a BufferGap class in all the places you'ld use a
python string transparently, to use standard regular expressions, for
example. Glancing quickly at the regexmodule.c, and its use of
PyString_Whatever, I'm not certain this is easy to do efficiently
(must one copy the buffer's contents into a Python-native string
before one can use something like a regular expression match on the
buffer's contents?).

Anyone have ideas / suggestions on how one would represent an editing
buffer in a way that would remain most (transparently) compatible with
the Python standard library string operations (and yet remain
efficient for editing)? If one is embedding the interpreter in the
editor (or writing the editor in pure python), and using python for
editor extensibility, it seems desireable to keep complexity down for
extension writers , and to allow them to think of the buffer as a
string.

Thanks...
Jul 18 '05 #1
6 2162
[snip]

My advice: find some pre-existing editor component and adapt it to suit
your needs. Writing an editor component, whether it be for a GUI or
console, is a pain in the ass.

If you are looking for a GUI editor component, stick with wxPython and
the wxStyledTextCtrl, it is a binding to the fabulous scintilla editing
component: http://scintilla.org

I have no advice for a console editor, but I would suggest you take a
look at curses or ncurses, whichever one is the open source version.

Once you have that thing embedded in your application, the rest should
basically write itself. How do I know? Because I wrote an editor with
it myself: http://pype.sourceforge.net
- Josiah
Jul 18 '05 #2
ma**************@yahoo.com (manders2k) writes:
Anyway, I'm wondering what straightforward ways to leverage /
implement this representation in Python. Ideally, it would be great
if one could use a BufferGap class in all the places you'ld use a
python string transparently, to use standard regular expressions, for
example. Glancing quickly at the regexmodule.c, and its use of
PyString_Whatever, I'm not certain this is easy to do efficiently
(must one copy the buffer's contents into a Python-native string
before one can use something like a regular expression match on the
buffer's contents?).


You can do some of that with the array module, but Python's regexp
library doesn't give any way to search backwards for a regexp, so that's
another problem you'll face trying to write an editor in Python.
Jul 18 '05 #3
Josiah Carlson <jc******@nospam.uci.edu> wrote in message news:<bv**********@news.service.uci.edu>...
My advice: find some pre-existing editor component and adapt it to suit
your needs. Writing an editor component, whether it be for a GUI or
console, is a pain in the ass.


:-) It might be a pain in the ass, but it sounds like the most
edifying (and probably the most fun) part of the process to me.

Reading up on what's involved, getting a basic editor put together
sounds actually quite easy. A few tens of hours of work, maybe.
Making the editor feature-rich, extending it in a substantial way,
well doesn't sound hard just work-intensive.

Thanks for the pointers though...
Jul 18 '05 #4
Paul Rubin <http://ph****@NOSPAM.invalid> wrote in message news:<7x************@ruckus.brouhaha.com>...
You can do some of that with the array module, but Python's regexp
library doesn't give any way to search backwards for a regexp, so that's
another problem you'll face trying to write an editor in Python.


Yeah, I have a feeling that it might be easier to code up the buffer
in C/C++, and embed it in the interpreter. I'm not sure how much of a
performance bottleneck having this very low-level component written in
python will be on modern machines; probably not such a big deal.
Writing a buffer class and fiddling with pointers and whatnot actually
sounds easier to do in C++ than in emulating this style of thing in
Python (then again, I'm a heck of a lot more comfortable with C++ than
Python at this point, so that might not speak to the difficulty of the
task).

What I guess I wish were the case is that I could implement the
"string interface" on my BufferGap, so that everywhere that Python (at
the C API level) expects a string, a BufferGap could be used instead.
That way, all the libraries that inspect and operate on strings would
work transparently, without having to be recoded (copy / paste, end up
with a lot of mostly identical, redundant code) to operate on this
other string representation. Maybe this just isn't possible with the
current C-Python implementation. I suspect it would be with many
possible C++-Python implementations, but we don't have one of those
lying around so...

I'm pretty sure that for modest size buffers (even a megabyte of
text), copying the contents of the buffer into a python-string
representation before operating on it with python-libraries would be
transparently fast. It just seems...wasteful, and potentially very
bad news if someone ever tried to do a regex search on a buffer that
occupied more than half of the physical memory of the machine or
somesuch.
Jul 18 '05 #5
manders2k:
I'm not sure how much of a
performance bottleneck having this very low-level component written in
python will be on modern machines; probably not such a big deal.
The performance bottleneck in split buffers is often the cost of copying
array ranges. I once wrote a patch for Python's array class to provide
copying within an array but the patch contents didn't make it to SourceForge
and I haven't had time to follow it up.

http://mail.python.org/pipermail/pat...il/012043.html
Writing a buffer class and fiddling with pointers and whatnot actually
sounds easier to do in C++ than in emulating this style of thing in
Python (then again, I'm a heck of a lot more comfortable with C++ than
Python at this point, so that might not speak to the difficulty of the
task).
Split buffers don't need to use pointers. I have written several split
buffer implementations including

* the implementation in Scintilla (scintilla/src/CellBuffer.[h,cxx])
http://cvs.sourceforge.net/viewcvs.p...lla/scintilla/

* a templated C++ implementation
http://mailman.lyra.org/pipermail/sc...ch/000903.html

* a generic implementation that is part of my SinkWorld project written in a
subset of C++ that can be automatically translated into Java or C#
http://cvs.sourceforge.net/viewcvs.p...lla/sinkworld/

Also in SinkWorld is a split buffer based data structure for partitioning
a document into segments such as lines called lv which is in lv.h. While the
line starts could be stored in a standard split buffer, inserting text would
then lead to adding to all following line start positions. To fix this,
there is also a 'step', with all positions after the step position adding
the step value to their values. The step is moved to the position where text
is being inserted or deleted but due to locality of modification, the move
is mostly short.
What I guess I wish were the case is that I could implement the
"string interface" on my BufferGap, so that everywhere that Python (at
the C API level) expects a string, a BufferGap could be used instead.
IIRC, at one stage there was explicit support in Python (perhaps in the
buffer class) for multiple segment buffers but it was never used so has
probably rotted.
That way, all the libraries that inspect and operate on strings would
work transparently, without having to be recoded (copy / paste, end up
with a lot of mostly identical, redundant code) to operate on this
other string representation.


I'd like to see this implemented and have been meaning to look into it
myself.

Neil
Jul 18 '05 #6

"manders2k" <ma**************@yahoo.com> wrote in message

[snip]
What I guess I wish were the case is that I could implement the
"string interface" on my BufferGap, so that everywhere that Python (at
the C API level) expects a string, a BufferGap could be used instead.
That way, all the libraries that inspect and operate on strings would
work transparently, without having to be recoded (copy / paste, end up
with a lot of mostly identical, redundant code) to operate on this
other string representation. Maybe this just isn't possible with the
current C-Python implementation.


Have you looked at the mmap standard module?
http://www.python.org/doc/current/lib/module-mmap.html
"Memory-mapped file objects behave like both strings and like file objects.
Unlike normal string objects, however, these are mutable. You can use mmap
objects in most places where strings are expected; for example, you can use
the re module to search through a memory-mapped file"

Michael
Jul 18 '05 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

6
by: Marco Herrn | last post by:
Hi, I need to serialize an object into a string representation to store it into a database. So the SOAPFormatter seems to be the right formatter for this purpose. Now I have the problem that...
7
by: Eric | last post by:
Hi All, I need to XOR two same-length Strings against each other. I'm assuming that, in order to do so, I'll need to convert each String to a BitArray. Thus, my question is this: is there an...
2
by: Alpha | last post by:
Hi, I'm able to make connection to a server using socket connection. However, when I send a command string the server just ignores it. All command string needs to start with "0xF9" at Byte 0. ...
33
by: Jordan Tiona | last post by:
How can I make one of these? I'm trying to get my program to store a string into a variable, but it only stores one line. -- "No eye has seen, no ear has heard, no mind can conceive what God...
14
by: ern | last post by:
Does a function exist to convert a 128-bit hex number to a string?
232
by: robert maas, see http://tinyurl.com/uh3t | last post by:
I'm working on examples of programming in several languages, all (except PHP) running under CGI so that I can show both the source files and the actually running of the examples online. The first...
6
by: Javier | last post by:
Hello people, I'm recoding a library that made a few months ago, and now that I'm reading what I wrote I have some questions. My program reads black and white images from a bitmap (BMP 24bpp...
14
by: Aman JIANG | last post by:
hi i need a fast way to do lots of conversion that between string and numerical value(integer, float, double...), and boost::lexical_cast is useless, because it runs for a long time, (about 60...
17
by: let_the_best_man_win | last post by:
How do I print a pointer address into a string buffer and then read it back on a 64 bit machine ? Compulsion is to use string (char * buffer) only. printing with %d does not capture the full...
0
by: Aliciasmith | last post by:
In an age dominated by smartphones, having a mobile app for your business is no longer an option; it's a necessity. Whether you're a startup or an established enterprise, finding the right mobile app...
0
tracyyun
by: tracyyun | last post by:
Hello everyone, I have a question and would like some advice on network connectivity. I have one computer connected to my router via WiFi, but I have two other computers that I want to be able to...
2
by: giovanniandrean | last post by:
The energy model is structured as follows and uses excel sheets to give input data: 1-Utility.py contains all the functions needed to calculate the variables and other minor things (mentions...
3
NeoPa
by: NeoPa | last post by:
Introduction For this article I'll be using a very simple database which has Form (clsForm) & Report (clsReport) classes that simply handle making the calling Form invisible until the Form, or all...
1
by: Teri B | last post by:
Hi, I have created a sub-form Roles. In my course form the user selects the roles assigned to the course. 0ne-to-many. One course many roles. Then I created a report based on the Course form and...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 1 Nov 2023 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM) Please note that the UK and Europe revert to winter time on...
3
by: nia12 | last post by:
Hi there, I am very new to Access so apologies if any of this is obvious/not clear. I am creating a data collection tool for health care employees to complete. It consists of a number of...
0
isladogs
by: isladogs | last post by:
The next online meeting of the Access Europe User Group will be on Wednesday 6 Dec 2023 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, Mike...
4
by: GKJR | last post by:
Does anyone have a recommendation to build a standalone application to replace an Access database? I have my bookkeeping software I developed in Access that I would like to make available to other...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.