473,326 Members | 2,813 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,326 software developers and data experts.

RE: ka-ping yee tokenizer.py

Hi Fredrik,

This is exactly what I need. Thank you.
I would like to do one additional function. I am not using the tokenizer to
parse python code. It happens to work very well for my application.
However, I would like either or both of the following variance:
1) I would like to add 2 other characters as comment designation
2) write a module that can readline, modify the line as required, and
finally, this module can be used as the argument for the tokenizer.

Def modifyLine( fileHandle ):
# readline and modify this string if required
....

For token in tokenize.generate_tokens( modifyLine( myFileHandle ) ):
Print token

Anxiously looking forward to your thoughts.
karl

-----Original Message-----
From: py*************************************@python.org
[mailto:py*************************************@pyt hon.org] On Behalf Of
Fredrik Lundh
Sent: Monday, September 15, 2008 2:04 PM
To: py*********@python.org
Subject: Re: ka-ping yee tokenizer.py

Karl Kobata wrote:
I have enjoyed using ka-ping yee's tokenizer.py. I would like to
replace the readline parameter input with my own and pass a list of
strings to the tokenizer. I understand it must be a callable object and
iteratable but it is obvious with errors I am getting, that this is not
the only functions required.
not sure I can decipher your detailed requirements, but to use Python's
standard "tokenize" module (written by ping) on a list, you can simple
do as follows:

import tokenize

program = [ ... program given as list ... ]

for token in tokenize.generate_tokens(iter(program).next):
print token

another approach is to turn the list back into a string, and wrap that
in a StringIO object:

import tokenize
import StringIO

program = [ ... program given as list ... ]

program_buffer = StringIO.StringIO("".join(program))

for token in tokenize.generate_tokens(program_buffer.readline):
print token

</F>

--
http://mail.python.org/mailman/listinfo/python-list

Sep 16 '08 #1
1 1803
On Sep 16, 2:48*pm, "Karl Kobata" <karl.kob...@syncira.comwrote:
Hi Fredrik,

This is exactly what I need. *Thank you.
I would like to do one additional function. *I am not using the tokenizer to
parse python code. *It happens to work very well for my application.
However, I would like either or both of the following variance:
1) I would like to add 2 other characters as comment designation
2) write a module that can readline, modify the line as required, and
finally, this module can be used as the argument for the tokenizer.

Def modifyLine( fileHandle ):
* # readline and modify this string if required
...

For token in tokenize.generate_tokens( modifyLine( myFileHandle ) ):
* * * * Print token

Anxiously looking forward to your thoughts.
karl

-----Original Message-----
From: python-list-bounces+kkobata=syncira....@python.org

[mailto:python-list-bounces+kkobata=syncira....@python.org] On Behalf Of
Fredrik Lundh
Sent: Monday, September 15, 2008 2:04 PM
To: python-l...@python.org
Subject: Re: ka-ping yee tokenizer.py

Karl Kobata wrote:
I have enjoyed using ka-ping yee's tokenizer.py. *I would like to
replace the readline parameter input with my own and pass a list of
strings to the tokenizer. *I understand it must be a callable object and
iteratable but it is obvious with errors I am getting, that this is not
the only functions required.

not sure I can decipher your detailed requirements, but to use Python's
standard "tokenize" module (written by ping) on a list, you can simple
do as follows:

* * *import tokenize

* * *program = [ ... program given as list ... ]

* * *for token in tokenize.generate_tokens(iter(program).next):
* * * * *print token

another approach is to turn the list back into a string, and wrap that
in a StringIO object:

* * *import tokenize
* * *import StringIO

* * *program = [ ... program given as list ... ]

* * *program_buffer = StringIO.StringIO("".join(program))

* * *for token in tokenize.generate_tokens(program_buffer.readline):
* * * * *print token

</F>

--http://mail.python.org/mailman/listinfo/python-list

This is an interesting construction:
>>a= [ 'a', 'b', 'c' ]
def moditer( mod, nextfun ):
.... while 1:
.... yield mod( nextfun( ) )
....
>>list( moditer( ord, iter( a ).next ) )
[97, 98, 99]

Here's my point:
>>a= [ 'print a', 'print b', 'print c' ]
tokenize.generate_tokens( iter( a ).next )
<generator object at 0x009FF440>
>>tokenize.generate_tokens( moditer( lambda s: s+ '#', iter( a ).next ).next )
It adds a '#' to the end of every line, then tokenizes.
Sep 17 '08 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

7
by: bobsled | last post by:
For class A to reuse functionality of a class B, A could either contains B or inherits from B. When should pick which? In a book the author says that "Don't inherit from a concrete class." Does...
8
by: Workgroups | last post by:
I've got a page where the nature of the beast is such that the user clicks a submit button to ransomize some data in somewhat rapid succession (once per second, give or take). The page generates a...
5
by: gimme_this_gimme_that | last post by:
I'd like to create my own version of google suggest GS. If you haven't seen GS check it out at : http://www.google.com/webhp?complete=1&hl=en I've reviewed several AJAX write-ups on the...
4
by: Ken Allen | last post by:
I have been using Visual Studio 2005 for some time now, and generally it works fine. Recently I have been adding some threading logic to one of my applications to permit the user interface to be...
7
by: James Johnson | last post by:
Are there structs in JavaScript? If not, what's the closest thing, or do I just use parallel arrays? I'm populating a JavaScript array from ColdFusion query, but I don't think I can do this: ...
12
by: rwise5 | last post by:
Hello, I was wondering if someone could help me with the following? Write a function that shifts the stored value of five character variables in a circular fashion. Your function should work in...
45
nemisis
by: nemisis | last post by:
Hi Everyone I have am doing an object oriented C++ program and I have no idea as to how start it................. I am not that good at coding so i tried and made an overview of what...
6
by: happyhondje | last post by:
Hello everyone, I've got a little issue, both programming and performance-wise. I have a set, containing objects that refer to other sets. For example, in a simple notation: (<a, b, c>, <d, e>)...
12
alpnz
by: alpnz | last post by:
Hi all ... checking in after a long hiatus, major DB problem in the outer galaxy ... nothing to strenuous though ... bring me up to speed, whos in charge here now? :-) OOOO got a couple of curly...
11
by: VK | last post by:
In the continuation of the discussion at "Making Site Opaque -- This Strategy Feasible?" and my comment at http://groups.google.com/group/comp.lang.javascript/msg/b515a4408680e8e2 I have...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.