473,320 Members | 2,107 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

readline tokenizer newline sticky wicket

Given a "linemess.py" file with inconsistent line ending:

line 1 \r
\r\n
line \n

tokenized as per:

import tokenize
f=open('linemess.py','r')
tokens=tokenize.generate_tokens(f.readline)
for t in tokens:
print t

get output as follows:

(1, 'line', (1, 0), (1, 4), 'line 1\r\n')
(2, '1', (1, 5), (1, 6), 'line 1\r\n')
(4, '\r\n', (1, 6), (1, 8), 'line 1\r\n')
(1, 'line', (2, 0), (2, 4), 'line 2\n')
(2, '2', (2, 5), (2, 6), 'line 2\n')
(4, '\n', (2, 6), (2, 7), 'line 2\n')
(0, u'', (3, 0), (3, 0), u'')

So that the Windows \r\n is tokenized as a single literal token rather
than as \n under the convention of universal newline support.

Isn't this a problem?

I think this must have been at the route of the issue I ran into when a
file of messy inconsistent line ending that nonetheless compiled and ran
without a problem was rejected by tokenizer.py as having an indent issue.

On the theory that if tokenizer needs to fail when crap is thrown at it,
it should do so more gracefully - is this bug reportable?

Art


Feb 7 '06 #1
0 1348

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

12
by: Mike Maxwell | last post by:
When I invoke readline() in a for loop, why does it return a series of one-char strings, rather than the full line? >>> for sL in sys.stdin.readline(): print sL .... abc a b c
5
by: Knackeback | last post by:
task: - read/parse CSV file code snippet: string key,line; typedef tokenizer<char_separator<char> > tokenizer; tokenizer tok(string(""), sep); while ( getline(f, line) ){ ++lineNo;...
4
by: Java Guy | last post by:
This must be a classical topic -- C++ stgring tokenizer. I just switched from C to C++ ( in Unix ). It turns out that there is no existing C++ string tokenizer. Searching on the Web, I found...
4
by: alberto | last post by:
In this code for a console application: int n; n = Console.Read(); Console.WriteLine(n); Console.ReadLine();
6
by: Christian Convey | last post by:
Hello, I've got a program that (ideally) perpetually monitors sys.stdin for lines of text. As soon as a line comes in, my program takes some action. The problem is, it seems like a very large...
2
by: Jan | last post by:
Hello there, I've got a strange problem using the readline method in vb2005 for reading the serial port. If I use the readchar method I can receive one character from serialport. This is the...
1
by: mike.biang | last post by:
I am using the textstream object to read a CSV file and parse through the fields. I use the readline() method to read in each line of the file, and parse the contents into the values of the...
1
by: Horta | last post by:
Hi folks, Sometimes, when I do an os.popen*(), the process executed by the command hangs, and the script stops forever on the readline()/ readlines() calls. I found that I can use select, but I'm...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.