Pre-PEP Proposal: Codetags

[Micah Elliott]

I also have this living as a wiki <http://tracos.org/codetag/wiki/Pep>
if people would like to add comments there. I might try to capture there
feedback from this group anyway. First try at a PEP -- thanks for any
feedback!
Please add **NOTE:** comments to the bottom of this wiki document
using `WikiRestructur edText`:trac:.

=============== =============== =========
Codetag PEP (*or* Tao of Codetagging)
=============== =============== =========

PEP: XXX
Title: Codetags
Version: $Revision$
Last-Modified: $Date$
Author: Micah Elliott <mde at tracos.org>
Status: Active
Type: Informational
Content-Type: text/x-rst
Created: 27-Jun-2005
Post-History: XXX
Abstract
========

This informational PEP aims to provide guidelines for consistent use
of Codetags, which would enable the construction of standard utilities
to take advantage of the Codetag information, as well as making Python
code more uniform across projects. Codetag is a also a very
lightweight programming micro-paradigm and becomes useful for project
managment, documentation, change tracking, and project health
monitoring. This is submitted as a PEP because I feel its ideas are
Pythonic, although the concepts are not unique to Python programming.
Herein are the definition of a Codetag, a philosophy rant, a
motivation for standardized conventions, a specification, a toolset
description, and possible objections to the Codetag project/paradigm.
What's a Codetag?
=============== ==

Programmers widely use ad-hoc code comment markup conventions to serve
as reminders of sections of code that need closer inspection or
review. Examples of markup include ``FIXME``, ``TODO``, ``XXX``,
``BUG``, but there many more in wide use in existing software. Such
markup will be henceforth referred to as a *Codetag*. These Codetags
may show up in application code, unit tests, scripts, general
documentation, or wherever suitable.
Philosophy
==========

NOTE: **I'm not certain Philosophy_ belongs in the PEP, but it
somewhat explains the usefulness of Codetags** <mde>

If you subscribe to most of these values, then Codetags will likely be
useful for you.

1. As much information as possible should be contained **inside the
source code** (application code or unit tests). This along with use
of Codetags impedes duplication. Most documentation can be
generated from that source code Eg, by using help2man, man2html,
docutils, epydoc/pydoc, ctdocgen, etc.

2. Information should be almost **never duplicated** -- it should be
recorded in a single original format and all other locations should
be automatically generated from the original, or simply be
referenced. This is the *SPOT* rule.

3. Documentation that gets into customers' hands should be
**auto-generated** from single sources into whatever output formats.
People want documentation in many forms. It is thus important to
have a documentation system that can generate all of these.

4. Whatever information is subject to (and suited for) user
feedback/input should be contained in a **wiki** (or maybe usenet or
maillists). Eg, FAQ, RFC, PEP.

5. There should not be a dedicated, disjoint **documentation team**
for any non-huge project. The developers writing the code know the
code best, and should be the ones to describe it.

6. **Plain text** (with non-invasive markup) is the best form of writing
anything. All other formats are to be generated from the plain
text.

7. **Revision control** should be used for almost everything. And
modifications should be checkin'd at least daily.
Motivation
==========

**Various productivity tools can be built around Codetags.**
See `Toolset Possibilities`_ .

**Encourages consistency.**
Historically, a subset of these Codetags has been used informally in
the majority of codes in existence, whether Python or some other
language. Tags have been used in an inconsistent manner with
different spellings, semantics, format, and placement. Eg, some
programmers might include datestamps and/or user identifiers, limit to
a single line or not, spell the Codetag differently than others, etc.

**Encourages adherence to SPOT/DRY principle.**
Eg, generating a roadmap dynamically from Codetags instead of keeping
TODOs in sync with separate roadmap document.

**Easy to remember.**
All Codetags must be concise, intuitive, and semantically
non-overlapping with others. Format is also simple.

**Use not required/imposed.**
If you don't use Codetags already, there's no obligation to start, and
no risk of affecting code (but see Objections_). A small subset can be
adopted and the Tools_ will still be useful (a few are already
implicitly adopted anyway). Also very easy to identify and remove if a
Codetag is no longer deemed useful. Then it is effectively *completed*
and recorded by revision control simply by checkin'ing.

**Gives a global view of code.**
Use tools to generate documentation and reports.

**A logical location for capturing CRCs/Stories/Requirements.**
The XP community often does not electronically capture Stories, but
Codetags seem like a good place to locate them.

**Extremely lightweight process.**
Creating tickets in a tracking system for every thought degrades
development velocity. Even if a ticketing system is employed, Codetags
are useful for simply containing links to those tickets.
Examples
========

This shows a simple Codetag as commonly found in sources everywhere
(with the addition of a trailing ``<>``).

::

def foo():
# FIXME: Seems like this loop should be finite. <>
while True: ...

This contrived example demonstrates more common use of Codetagging.
It uses some of the available fields to specify the owners (a pair of
developers with initials *mde* and *cle*), the Work Week of expected
completion (*w14*), and the priority of the item (*p2*).

::

def foo():
# FIXME: Seems like this loop should be finite. <mde,cle w14 p2>
while True:
Specification
=============

This describes the format: parsing layout, mnemonic names, fields,
and semantics.

General Layout
--------------

Each Codetag should be inside a comment, and can be any number of lines.
It should match the indentation of surrounding code. The end of the
Codetag is marked by a ``<...>``, which must not be split onto
multiple lines.

There are multiple fields per Codetag, all of which are optional.

To be succinct, a Codetag is a mnemonic, a colon, a commentary, an
opening broket, a list of optional fields, and a closing broket. Ie,::

# MNEMONIC: Some (maybe multi-line) commentary. <field field ...>

... FIXME: Add completion vs target date?? <mde>
Mnemonic Semantics
------------------

The Codetags of interest (``recommended mnemonic (synonym list)``, *canon*,
semantics, and **NOTEs**) are as follows.

Some of these are temporary (eg, ``FIXME``) while others are
persistent (eg, ``REQ``). Synonyms should probably be deprecated in
the interest of minimalism and consistency. I chose a mnemonic over a
synonym for three criteria: descriptive, short, common usage trends.

``TODO (TBD, MLSTN, DONE)``
*To Do*, An informal task/feature that is pending completion.
Relevant to roadmap.

NOTE: **DONE would really be a completed TODO item, but these
should probably be done through the revision control system.** <mde>

``FIXME (XXX, DEBUG, BROKEN, RFCTR, OOPS, SMELL)``
*Fix Me*, Problematic or ugly code. Needs refactoring or cleanup.

NOTE: **Choosing between FIXME and XXX is difficult. AFAICT XXX is
more common, but so much less descriptive. Furthermore, XXX is a
useful placeholder in a piece of code having a value that is
unknown. Sun says that XXX and FIXME are slightly different,
giving XXX higher severity.** <mde>

``REQ (STORY)``
*Requirement*, Satisfaction of a specific, formal requirement.

``RFE (FEETCH, NYI, FR, FTRQ, FTR)``
*Request For Enhancement*, A roadmap item not yet implemented.

``IDEA``
*Idea*, Possible ``RFE`` candidate, but less formal than ``RFE``.

``??? (QUEST, WTF, TBD, QSTN)``
*Question*, Misunderstood detail. Product of coincidental programming.

``HACK (WKRD)``
*Hack*, Temporary code to force inflexible functionality, or
simply a test change, or workaround a known problem.

``PORT``
*Portability*, Workaround specific to OS, Python version, etc.

``BUG (BUGFIX)``
*Bug*, Reported defect tracked in bug database.

``NOTE (HELP)``
*Note*, Implementation detail that stands out as non-intuitive. Or
a code reviewer found something that needs discussion or further
investigation.

NOTE: **Maybe a useful metric where a high count of NOTEs indicates
a problem.** <mde>

``FAQ``
*Frequently Asked Question*, Interesting area that requires
external explanation.

NOTE: **This is probably more appropriately documented in a wiki
where users can more easily contribute.** <mde>

``GLOSS``
*Glossary*, Item definition for project glossary.

``STAT``
*Status*, File-level statistical indicator of work needing done on this
file.

``RVDBY``
*Reviewed By*, File-level indicator of programmer(s) who performed
recent code review.

``SEE (REF)``
*Reference*, Pointer to other code, web link, etc.

NOTE: **File-level Codetags might be better suited as properties in the
revision control system.** <mde>

Fields
------

All fields are optional. It should be possible for groups to
define/add their own, but the proposed standard fields are as follows:

``wNN``
Workweek target completion (estimation). Origination and
completion are freebees with revision control.

``pN``
Priority level.

``xxx[,yyy]...``
List of initials of owners of completion responsibility. There
should be no digits for initials.
Toolset Possibilities
=============== ======

Currently, programmers (and sometimes analysts) typically use *grep*
to generate a list of items corresponding to a single Codetag. However,
various hypothetical productivity tools could take advantage of a
consistent Codetag format. Some example tools follow.

NOTE: Codetag tools are mostly unimplemented (but I'm getting started!) <mde>

Document Generator
Possible docs: glossary, roadmap, manpages

Codetag History
Track (with revision control system interface) when a BUGtag (or any codetag)
originated/resolved in a code section

Code Statistics
A project Health-O-Meter

Codetag Lint
Notify of invalid use of Codetags, and aid in porting to Codetag

Story Manager/Browser
An electronic means to replace XP notecards. In MVC terms, the
Codetag is the Model, and the Story Manager could be a graphical
Viewer/Controller to do visual rearrangment, prioritization, and
assignment, milestone management.

Any Text Editor
Used for changing, removing, adding, rearranging Codetags.

There are some tools already in existence that take advantage of a
smaller set of pseudo-Codetags (see References_)

Objections
==========

**Objection**: Extreme Programming argues that such Codetags should not ever
exist in code since the code is the documentation.

**Defense**: Maybe put the Codetags in the unit test files instead.
Besides, it's tough to generate documentation from uncommented source
code.

----

**Objection**: Too much existing code has not followed proposed
guidelines.

**Defense**: [Simple] utilities (*ctlint*) could convert existing
codes.

----

**Objection**: Causes duplication with tracking system.

**Defense**: Not really -- If an item exists in the tracker, a simple
ticket number as the Codetag commentary is sufficent. Maybe a
duplicated title would be acceptable. Furthermore, it's too
burdensome to have a ticket filed for every item that pops into a
developer mind on-the-go.

----

**Objection**: Codetags are ugly and clutter code.

**Defense**: That is a good point. But I'd still rather have such info
in a single place (the source code) than various other documents,
likely getting duplicated or forgotten about.

----

**Objection**: Codetags (and all comments) get out of date.

**Defense**: Not so much if other sources (externally visible
documentation) depend on them being accurate.

----
References
==========

Some other tools have approached defining/exploiting Codetags.

See http://tracos.org/codetag/wiki/Links
Comments
========

Please add comments below following line. Or feel free to comment
inline in above sections with **NOTE:** Codetags. Objections_ might be
a popular area for comments. :-)

Thank you!

----

<First comment.>

----
...
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
End:
}}}

--
Micah Elliott

[Martin v. Löwis]

Micah Elliott wrote:

I also have this living as a wiki <http://tracos.org/codetag/wiki/Pep>
if people would like to add comments there. I might try to capture there
feedback from this group anyway. First try at a PEP -- thanks for any
feedback!

I think you somewhat misunderstood the purpose of the PEP process.
This is meant primarily for enhancements to Python (the language
and its library), and it is meant to save the PEP author from
implementing unagreeable functionality - if the PEP is accepted,
the PEP author is typically expected to implement the proposed
functionality (in many cases, having a draft implementation is
prerequisite to accepting it).

Both elements seem to be missing your document: it does not propose
changes to the Python language; instead, it proposes a specific
way of writing comments (ie. something that is not relevant to the
Python interpreter or libraries, only to the Python developer).
Also, there is no indication that you would like to implement
something for the PEP: what tool would you like to change in what
specific way?

If you want this as a PEP so that Python somehow "endorses" code
tags (which appear to be independent from the programming language),
I'm -1: Python has traditionally abstained from pushing unrelated
technologies. Instead, it integrates with the technologies people
find useful; in this spirit, I would rather like to see code tags
become popular on their own merits, and *then* perhaps integrating
support for it into IDLE could be considered.

Regards,
Martin

[Fuzzyman]

I'm sure Martin's comment is basically correct. *However* - you could
take your proposal forward by developing a set of tools (e.g.
documentation tools) that use your proposal. That way people have a
working implementation to use.

e.g. Tools to generate HTML TODO lists from source code

It would then be easier to integrate your ideas into existing IDEs.

Regards,

Fuzzy
http://www.voidspace.org.uk/python

[Aahz]

In article <42************ **@v.loewis.de> ,
=?ISO-8859-1?Q?=22Martin_v =2E_L=F6wis=22? = <ma****@v.loewi s.de> wrote:

Micah Elliott wrote:

I also have this living as a wiki <http://tracos.org/codetag/wiki/Pep>
if people would like to add comments there. I might try to capture there
feedback from this group anyway. First try at a PEP -- thanks for any
feedback!

I think you somewhat misunderstood the purpose of the PEP process.
This is meant primarily for enhancements to Python (the language
and its library), and it is meant to save the PEP author from
implementing unagreeable functionality - if the PEP is accepted,
the PEP author is typically expected to implement the proposed
functionalit y (in many cases, having a draft implementation is
prerequisite to accepting it).

Both elements seem to be missing your document: it does not propose
changes to the Python language; instead, it proposes a specific
way of writing comments (ie. something that is not relevant to the
Python interpreter or libraries, only to the Python developer).
Also, there is no indication that you would like to implement
something for the PEP: what tool would you like to change in what
specific way?

However, it's also true that there are plenty of informational PEPs,
most notably PEP 8 (Python style guide). OTOH, it is also generally the
case that such PEPs codify existing practice rather than attempting to
create new practices (PEP 6 being a notable counter-example).
--
Aahz (aa**@pythoncra ft.com) <*> http://www.pythoncraft.com/

The way to build large Python applications is to componentize and
loosely-couple the hell out of everything.

[A.M. Kuchling]

On Thu, 11 Aug 2005 08:47:37 +0200,
Martin v. Löwis <ma****@v.loewi s.de> wrote:

I think you somewhat misunderstood the purpose of the PEP process.
This is meant primarily for enhancements to Python (the language
and its library), ...

PEP 0 disagrees:

PEP stands for Python Enhancement Proposal. A PEP is a design
document providing information to the Python community, or describing
a new feature for Python. The PEP should provide a concise technical
specification of the feature and a rationale for the feature.

...

There are two kinds of PEPs. A Standards Track PEP describes a new
feature or implementation for Python. An Informational PEP describes
a Python design issue, or provides general guidelines or information
to the Python community, but does not propose a new feature.

Most of PEP 0 is concerned with describing the workflow for Standards Track
PEPs, of course, but I guess there's not much to say for informational PEPs.
("Publish your PEP. Eventually, freeze the PEP and stop making changes to it."
would be about the sum of it.)

I think we need to encourage writing detailed specifications of interfaces
for the community's use, such as the WSGI interface (PEP 333, IIRC), so
using the PEP repository for this purpose is a good idea. If such things
are deemed off-topic for PEPs, then I think we should have a separate set of
documents for this (perhaps the suggested PEEPS: Python Environment
Enhancement Proposals).

--amk

[Terry Reedy]

""Martin v. Löwis"" <ma****@v.loewi s.de> wrote in message
news:42******** ******@v.loewis .de...

Both elements seem to be missing your document: it does not propose
changes to the Python language; instead, it proposes a specific
way of writing comments (ie. something that is not relevant to the
Python interpreter or libraries, only to the Python developer).
It is similar in this way and complementary to
http://www.python.org/peps/pep-0287.html
:reStructuredTe xt Docstring Format

However, that was only submitted after the specification and tool
development was much more advanced, a user community existed, and proposals
in hand to integrate with existing Python development mechanisms (pydoc and
PEPs). And indeed, it was adopted as an allowed PEP format and separately
for other things such as PyDev summaries.
Python has traditionally abstained from pushing unrelated technologies.
I remember that there was also opposition to simply 'endorsing' reST.
Instead, it integrates with the technologies people
find useful; in this spirit, I would rather like to see code tags
become popular on their own merits, and *then* perhaps integrating
support for it into IDLE could be considered.

The OP might also look for codetag usage in Python source (I know of XXX)
and produce a useful collation/reporting tool if there is not one already.

Terry J. Reedy

[Bengt Richter]

On Thu, 11 Aug 2005 08:47:37 +0200, =?ISO-8859-1?Q?=22Martin_v =2E_L=F6wis=22? = <ma****@v.loewi s.de> wrote:

Micah Elliott wrote:

I also have this living as a wiki <http://tracos.org/codetag/wiki/Pep>
if people would like to add comments there. I might try to capture there
feedback from this group anyway. First try at a PEP -- thanks for any
feedback!

I think you somewhat misunderstood the purpose of the PEP process.
This is meant primarily for enhancements to Python (the language
and its library), and it is meant to save the PEP author from
implementing unagreeable functionality - if the PEP is accepted,
the PEP author is typically expected to implement the proposed
functionalit y (in many cases, having a draft implementation is
prerequisite to accepting it).

Both elements seem to be missing your document: it does not propose
changes to the Python language; instead, it proposes a specific
way of writing comments (ie. something that is not relevant to the
Python interpreter or libraries, only to the Python developer).

Erm, (cough) PEP 263 specifies comment syntax ;-)

But I agree that the OP's proposal is not a core language matter.
Perhaps this is another PEEP candidate (i.e., a "Python Evironment Enhancement Proposal"
to put in a "PEEP" application/tools/environment-oriented clone of the PEP process).

Maybe a PEEP top page in the wiki with links to specific PEEPs could be a way to
try it out.

Regards,
Bengt Richter

[mdelliot]

Wow, thanks for all the quick responses!

Martin wrote:

the PEP author is typically expected to implement the proposed
functionality (in many cases, having a draft implementation is
prerequisite to accepting it).
Fuzzyman wrote: ...you could take your proposal forward by developing a set of tools
(e.g. documentation tools) that use your proposal. That way people
have a working implementation to use.
Yes, I'm working on that and should have some tools reasonably
functional within the next few months. I didn't want to commit a lot
of time to writing tools if no one showed interested in the proposed
conventions. Since no one is flaming the idea yet, I will get a
move-on. The five example tools (that have come to mind so far) which
I mentioned in the pre-pep were:

* Document Generator (glossary, roadmap, manpages, bug lists, etc.)

* Codetag History (useful for estimating, autopsy, etc.)

* Code Statistics (a project Health-O-Meter)

* Codetag Lint (aid in fixing almost-Codetags)

* Story Manager/Browser (graphical viewer for various potential
purposes)

I'd love to hear any other ideas.

Aahz and Terry wrote: it's also true that there are plenty of informational PEPs...
...It is similar in this way and complementary to reStructuredTex t
Docstring Format...
Right, that's why I thought the PEP process was appropriate for this,
like it was for restructuredtex t. This of course is much smaller
scale. I just want the proposal to live somewhere where I can get some
feedback. I haven't seen much in the way of "+/-" yet, but that's
fine since Codetags are not implemented/in use yet.

amk wrote: If such things are deemed off-topic for PEPs, then I think we should
have a separate set of documents for this (perhaps the suggested
PEEPS: Python Environment Enhancement Proposals).
Sounds great. Make it a PEEP! I couldn't find anything official about
PEEPs yet :-) I did create a link to PEEPs from the Front Page wiki.

Terry wrote: The OP might also look for codetag usage in Python source (I know of
XXX)
Great idea. I tried this some months ago and this was one of the
things that justified the idea. For some cursory stats:

$ csrcs=$(find ~/archive/Python-2.4.1 -name *.c)
$ for tag in XXX FIXME TODO BUG HACK IDEA NOTE RFE; do echo -n "$tag: "; grep $tag $csrcs |wc -l
done XXX: 376
FIXME: 11
TODO: 12
BUG: 109
HACK: 0
IDEA: 0
NOTE: 32
RFE: 0
$
$ pysrcs=$(find /usr/lib/python2.3/ -maxdepth 2 -name '*.py')
$ for tag in XXX FIXME TODO BUG HACK IDEA NOTE RFE; do echo -n "$tag: "; grep $tag $pysrcs |wc -l
done

XXX: 457
FIXME: 10
TODO: 57
BUG: 149
HACK: 0
IDEA: 0
NOTE: 50
RFE: 0

So I guess I'll now ask for ongoing feedback on the PEEP wiki site.
I'm happy to wait for that feedback while the toolsmithing gets
underway. Maybe a PEEP process discussion is warranted for another
c.l.py thread.

--
Micah Elliott

[Martin v. Löwis]

Bengt Richter wrote:

Both elements seem to be missing your document: it does not propose
changes to the Python language; instead, it proposes a specific
way of writing comments (ie. something that is not relevant to the
Python interpreter or libraries, only to the Python developer).

Erm, (cough) PEP 263 specifies comment syntax ;-)

Indeed. However, it turns out that the syntax *is* relevant to
the Python interpreter. So despite it looking like a comment, it is
not.

I personally had wished if it would have been proper syntax (and
had proposed PEP 244 in that direction), however, the BDFL rejected
that idea an preferred a comment-like syntax.

Regards,
Martin