By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
440,235 Members | 1,011 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 440,235 IT Pros & Developers. It's quick & easy.

Tokenizer inconsistency wrt to new lines in comments

P: n/a
The tokenize.generate_tokens function seems to handle in a context-
sensitive manner the new line after a comment:
>>from StringIO import StringIO
from tokenize import generate_tokens

text = '''
.... # hello world
.... x = (
.... # hello world
.... )
.... '''
>>>
for t in generate_tokens(StringIO(text).readline):
.... print repr(t[1])
....
'\n'
'# hello world\n'
'x'
'='
'('
'\n'
'# hello world'
'\n'
')'
'\n'
''

Is there a reason that the newline is included in the first comment
but not in the second, or is it a bug ?

George
Apr 4 '08 #1
Share this Question
Share on Google+
1 Reply


P: n/a
On 4 Apr., 18:22, George Sakkis <george.sak...@gmail.comwrote:
The tokenize.generate_tokens function seems to handle in a context-
sensitive manner the new line after a comment:
>from StringIO import StringIO
from tokenize import generate_tokens
>text = '''

... # hello world
... x = (
... # hello world
... )
... '''
>for t in generate_tokens(StringIO(text).readline):

... print repr(t[1])
...
'\n'
'# hello world\n'
'x'
'='
'('
'\n'
'# hello world'
'\n'
')'
'\n'
''

Is there a reason that the newline is included in the first comment
but not in the second, or is it a bug ?

George
I guess it's just an artifact of handling line continuations within
expressions where a different rule is applied. For compilation
purposes both the newlines within expressions as well as the comments
are irrelevant. There are even two different token namely NEWLINE and
NL which are produced for newlines. NL and COMMENT will be ignored.
NEWLINE is relevant for the parser.

If it was a bug it has to violate a functional requirement. I can't
see which one.

Kay
Apr 4 '08 #2

This discussion thread is closed

Replies have been disabled for this discussion.