Given a "linemess.p y" file with inconsistent line ending:
line 1 \r
\r\n
line \n
tokenized as per:
import tokenize
f=open('linemes s.py','r')
tokens=tokenize .generate_token s(f.readline)
for t in tokens:
print t
get output as follows:
(1, 'line', (1, 0), (1, 4), 'line 1\r\n')
(2, '1', (1, 5), (1, 6), 'line 1\r\n')
(4, '\r\n', (1, 6), (1, 8), 'line 1\r\n')
(1, 'line', (2, 0), (2, 4), 'line 2\n')
(2, '2', (2, 5), (2, 6), 'line 2\n')
(4, '\n', (2, 6), (2, 7), 'line 2\n')
(0, u'', (3, 0), (3, 0), u'')
So that the Windows \r\n is tokenized as a single literal token rather
than as \n under the convention of universal newline support.
Isn't this a problem?
I think this must have been at the route of the issue I ran into when a
file of messy inconsistent line ending that nonetheless compiled and ran
without a problem was rejected by tokenizer.py as having an indent issue.
On the theory that if tokenizer needs to fail when crap is thrown at it,
it should do so more gracefully - is this bug reportable?
Art
0  1372  This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics | |
by: Mike Maxwell |
last post by:
When I invoke readline() in a for loop, why does it return a series of
one-char strings, rather than the full line?
>>> for sL in sys.stdin.readline(): print sL
....
abc
a
b
c
|
by: Knackeback |
last post by:
task:
- read/parse CSV file
code snippet:
string key,line;
typedef tokenizer<char_separator<char> > tokenizer;
tokenizer tok(string(""), sep);
while ( getline(f, line) ){
++lineNo;
tok.assign(line, sep);
|
by: Java Guy |
last post by:
This must be a classical topic -- C++ stgring tokenizer.
I just switched from C to C++ ( in Unix ). It turns out that there is no
existing C++ string tokenizer. Searching on the Web, I found several and
tried one or two of them. Not very satisfied.
Any suggestions? Thx!
|
by: alberto |
last post by:
In this code for a console application:
int n;
n = Console.Read();
Console.WriteLine(n);
Console.ReadLine();
|
by: Christian Convey |
last post by:
Hello,
I've got a program that (ideally) perpetually monitors sys.stdin for
lines of text. As soon as a line comes in, my program takes some
action.
The problem is, it seems like a very large amount of data must
accumulate on sys.stdin before even my first invocation of readline()
returns. This delay prevents my program from being responsive in the
way it must be.
| | |
by: Jan |
last post by:
Hello there,
I've got a strange problem using the readline method in vb2005 for
reading the serial port.
If I use the readchar method I can receive one character from
serialport. This is the code I use:
Function ReceiveSerialData() As String
' Receive strings from a serial port.
Dim returnStr As String = ""
|
by: mike.biang |
last post by:
I am using the textstream object to read a CSV file and parse through
the fields. I use the readline() method to read in each line of the
file, and parse the contents into the values of the different fields.
With files generated on a windows system, this works fine.
However, when working with CSV files genereated through Excel on a Mac,
the entire contents of the file is read when I make the first ReadLine
call. I imagine that this...
|
by: Horta |
last post by:
Hi folks,
Sometimes, when I do an os.popen*(), the process executed by the
command hangs, and the script stops forever on the readline()/
readlines() calls. I found that I can use select, but I'm thinking...
if, after a sellect() call returns, the stdout (for example) has more
than one line? or if it has just some characters, no newline, and the
process just hangs?
I just want a readline(timeout) and readlines(timeout) like
functions....
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look !
Part I. Meaning of...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
| | |
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: adsilva |
last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
|
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| |
