473,320 Members | 2,164 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

different string representation (buffer gap)

Hi all --

I'm contemplating the idea of writing a simple emacs-like editor in
python (for fun and the experience of doing so). In reading through
Craig Finseth's "The Craft of Text Editing":

http://www.finseth.com/~fin/craft/

, I've come across the "buffer gap" representation for the text data
of the buffer. Very briefly, it keeps the unallocated memory of the
character array at the editing point, so that as long as there is
memory available, an insert/delete is very low (constant) cost. Of
course, moving the editing point means copying some character data so
that the gap moves with you, but...

Anyway, I'm wondering what straightforward ways to leverage /
implement this representation in Python. Ideally, it would be great
if one could use a BufferGap class in all the places you'ld use a
python string transparently, to use standard regular expressions, for
example. Glancing quickly at the regexmodule.c, and its use of
PyString_Whatever, I'm not certain this is easy to do efficiently
(must one copy the buffer's contents into a Python-native string
before one can use something like a regular expression match on the
buffer's contents?).

Anyone have ideas / suggestions on how one would represent an editing
buffer in a way that would remain most (transparently) compatible with
the Python standard library string operations (and yet remain
efficient for editing)? If one is embedding the interpreter in the
editor (or writing the editor in pure python), and using python for
editor extensibility, it seems desireable to keep complexity down for
extension writers , and to allow them to think of the buffer as a
string.

Thanks...
Jul 18 '05 #1
6 2194
[snip]

My advice: find some pre-existing editor component and adapt it to suit
your needs. Writing an editor component, whether it be for a GUI or
console, is a pain in the ass.

If you are looking for a GUI editor component, stick with wxPython and
the wxStyledTextCtrl, it is a binding to the fabulous scintilla editing
component: http://scintilla.org

I have no advice for a console editor, but I would suggest you take a
look at curses or ncurses, whichever one is the open source version.

Once you have that thing embedded in your application, the rest should
basically write itself. How do I know? Because I wrote an editor with
it myself: http://pype.sourceforge.net
- Josiah
Jul 18 '05 #2
ma**************@yahoo.com (manders2k) writes:
Anyway, I'm wondering what straightforward ways to leverage /
implement this representation in Python. Ideally, it would be great
if one could use a BufferGap class in all the places you'ld use a
python string transparently, to use standard regular expressions, for
example. Glancing quickly at the regexmodule.c, and its use of
PyString_Whatever, I'm not certain this is easy to do efficiently
(must one copy the buffer's contents into a Python-native string
before one can use something like a regular expression match on the
buffer's contents?).


You can do some of that with the array module, but Python's regexp
library doesn't give any way to search backwards for a regexp, so that's
another problem you'll face trying to write an editor in Python.
Jul 18 '05 #3
Josiah Carlson <jc******@nospam.uci.edu> wrote in message news:<bv**********@news.service.uci.edu>...
My advice: find some pre-existing editor component and adapt it to suit
your needs. Writing an editor component, whether it be for a GUI or
console, is a pain in the ass.


:-) It might be a pain in the ass, but it sounds like the most
edifying (and probably the most fun) part of the process to me.

Reading up on what's involved, getting a basic editor put together
sounds actually quite easy. A few tens of hours of work, maybe.
Making the editor feature-rich, extending it in a substantial way,
well doesn't sound hard just work-intensive.

Thanks for the pointers though...
Jul 18 '05 #4
Paul Rubin <http://ph****@NOSPAM.invalid> wrote in message news:<7x************@ruckus.brouhaha.com>...
You can do some of that with the array module, but Python's regexp
library doesn't give any way to search backwards for a regexp, so that's
another problem you'll face trying to write an editor in Python.


Yeah, I have a feeling that it might be easier to code up the buffer
in C/C++, and embed it in the interpreter. I'm not sure how much of a
performance bottleneck having this very low-level component written in
python will be on modern machines; probably not such a big deal.
Writing a buffer class and fiddling with pointers and whatnot actually
sounds easier to do in C++ than in emulating this style of thing in
Python (then again, I'm a heck of a lot more comfortable with C++ than
Python at this point, so that might not speak to the difficulty of the
task).

What I guess I wish were the case is that I could implement the
"string interface" on my BufferGap, so that everywhere that Python (at
the C API level) expects a string, a BufferGap could be used instead.
That way, all the libraries that inspect and operate on strings would
work transparently, without having to be recoded (copy / paste, end up
with a lot of mostly identical, redundant code) to operate on this
other string representation. Maybe this just isn't possible with the
current C-Python implementation. I suspect it would be with many
possible C++-Python implementations, but we don't have one of those
lying around so...

I'm pretty sure that for modest size buffers (even a megabyte of
text), copying the contents of the buffer into a python-string
representation before operating on it with python-libraries would be
transparently fast. It just seems...wasteful, and potentially very
bad news if someone ever tried to do a regex search on a buffer that
occupied more than half of the physical memory of the machine or
somesuch.
Jul 18 '05 #5
manders2k:
I'm not sure how much of a
performance bottleneck having this very low-level component written in
python will be on modern machines; probably not such a big deal.
The performance bottleneck in split buffers is often the cost of copying
array ranges. I once wrote a patch for Python's array class to provide
copying within an array but the patch contents didn't make it to SourceForge
and I haven't had time to follow it up.

http://mail.python.org/pipermail/pat...il/012043.html
Writing a buffer class and fiddling with pointers and whatnot actually
sounds easier to do in C++ than in emulating this style of thing in
Python (then again, I'm a heck of a lot more comfortable with C++ than
Python at this point, so that might not speak to the difficulty of the
task).
Split buffers don't need to use pointers. I have written several split
buffer implementations including

* the implementation in Scintilla (scintilla/src/CellBuffer.[h,cxx])
http://cvs.sourceforge.net/viewcvs.p...lla/scintilla/

* a templated C++ implementation
http://mailman.lyra.org/pipermail/sc...ch/000903.html

* a generic implementation that is part of my SinkWorld project written in a
subset of C++ that can be automatically translated into Java or C#
http://cvs.sourceforge.net/viewcvs.p...lla/sinkworld/

Also in SinkWorld is a split buffer based data structure for partitioning
a document into segments such as lines called lv which is in lv.h. While the
line starts could be stored in a standard split buffer, inserting text would
then lead to adding to all following line start positions. To fix this,
there is also a 'step', with all positions after the step position adding
the step value to their values. The step is moved to the position where text
is being inserted or deleted but due to locality of modification, the move
is mostly short.
What I guess I wish were the case is that I could implement the
"string interface" on my BufferGap, so that everywhere that Python (at
the C API level) expects a string, a BufferGap could be used instead.
IIRC, at one stage there was explicit support in Python (perhaps in the
buffer class) for multiple segment buffers but it was never used so has
probably rotted.
That way, all the libraries that inspect and operate on strings would
work transparently, without having to be recoded (copy / paste, end up
with a lot of mostly identical, redundant code) to operate on this
other string representation.


I'd like to see this implemented and have been meaning to look into it
myself.

Neil
Jul 18 '05 #6

"manders2k" <ma**************@yahoo.com> wrote in message

[snip]
What I guess I wish were the case is that I could implement the
"string interface" on my BufferGap, so that everywhere that Python (at
the C API level) expects a string, a BufferGap could be used instead.
That way, all the libraries that inspect and operate on strings would
work transparently, without having to be recoded (copy / paste, end up
with a lot of mostly identical, redundant code) to operate on this
other string representation. Maybe this just isn't possible with the
current C-Python implementation.


Have you looked at the mmap standard module?
http://www.python.org/doc/current/lib/module-mmap.html
"Memory-mapped file objects behave like both strings and like file objects.
Unlike normal string objects, however, these are mutable. You can use mmap
objects in most places where strings are expected; for example, you can use
the re module to search through a memory-mapped file"

Michael
Jul 18 '05 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

6
by: Marco Herrn | last post by:
Hi, I need to serialize an object into a string representation to store it into a database. So the SOAPFormatter seems to be the right formatter for this purpose. Now I have the problem that...
7
by: Eric | last post by:
Hi All, I need to XOR two same-length Strings against each other. I'm assuming that, in order to do so, I'll need to convert each String to a BitArray. Thus, my question is this: is there an...
2
by: Alpha | last post by:
Hi, I'm able to make connection to a server using socket connection. However, when I send a command string the server just ignores it. All command string needs to start with "0xF9" at Byte 0. ...
33
by: Jordan Tiona | last post by:
How can I make one of these? I'm trying to get my program to store a string into a variable, but it only stores one line. -- "No eye has seen, no ear has heard, no mind can conceive what God...
14
by: ern | last post by:
Does a function exist to convert a 128-bit hex number to a string?
232
by: robert maas, see http://tinyurl.com/uh3t | last post by:
I'm working on examples of programming in several languages, all (except PHP) running under CGI so that I can show both the source files and the actually running of the examples online. The first...
6
by: Javier | last post by:
Hello people, I'm recoding a library that made a few months ago, and now that I'm reading what I wrote I have some questions. My program reads black and white images from a bitmap (BMP 24bpp...
14
by: Aman JIANG | last post by:
hi i need a fast way to do lots of conversion that between string and numerical value(integer, float, double...), and boost::lexical_cast is useless, because it runs for a long time, (about 60...
17
by: let_the_best_man_win | last post by:
How do I print a pointer address into a string buffer and then read it back on a 64 bit machine ? Compulsion is to use string (char * buffer) only. printing with %d does not capture the full...
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.