469,922 Members | 2,190 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,922 developers. It's quick & easy.

Tokenizer inconsistency wrt to new lines in comments

The tokenize.generate_tokens function seems to handle in a context-
sensitive manner the new line after a comment:
>>from StringIO import StringIO
from tokenize import generate_tokens

text = '''
.... # hello world
.... x = (
.... # hello world
.... )
.... '''
>>>
for t in generate_tokens(StringIO(text).readline):
.... print repr(t[1])
....
'\n'
'# hello world\n'
'x'
'='
'('
'\n'
'# hello world'
'\n'
')'
'\n'
''

Is there a reason that the newline is included in the first comment
but not in the second, or is it a bug ?

George
Apr 4 '08 #1
1 1128
On 4 Apr., 18:22, George Sakkis <george.sak...@gmail.comwrote:
The tokenize.generate_tokens function seems to handle in a context-
sensitive manner the new line after a comment:
>from StringIO import StringIO
from tokenize import generate_tokens
>text = '''

... # hello world
... x = (
... # hello world
... )
... '''
>for t in generate_tokens(StringIO(text).readline):

... print repr(t[1])
...
'\n'
'# hello world\n'
'x'
'='
'('
'\n'
'# hello world'
'\n'
')'
'\n'
''

Is there a reason that the newline is included in the first comment
but not in the second, or is it a bug ?

George
I guess it's just an artifact of handling line continuations within
expressions where a different rule is applied. For compilation
purposes both the newlines within expressions as well as the comments
are irrelevant. There are even two different token namely NEWLINE and
NL which are produced for newlines. NL and COMMENT will be ignored.
NEWLINE is relevant for the parser.

If it was a bug it has to violate a functional requirement. I can't
see which one.

Kay
Apr 4 '08 #2

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

6 posts views Thread by Maarten van Reeuwijk | last post: by
8 posts views Thread by Matthew Thorley | last post: by
4 posts views Thread by Java Guy | last post: by
17 posts views Thread by Javier Gomez | last post: by
19 posts views Thread by Alex Vinokur | last post: by
82 posts views Thread by Edward Elliott | last post: by
18 posts views Thread by Robbie Hatley | last post: by
1 post views Thread by Karl Kobata | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.