C# - Parsing lines of C# source code with Regular expressions

[ahropak]

Hi,

I have a question regarding a regular expression within Regex.Split() method which will help me to break each line of code into tokens.

I'm trying to parse some lines of C# source code and split them into tokens.
My logic is very simple: construct a regular expression with 'operators and punctuators' which will serve as delimiters and split a string into array of tokens including those delimiters.

For example:

I have the following line of C# code

public static void DoStuff ( string[] strArray, int someArgument )
 

What regular expression should I use to get an array of tokens like this:

public
static
void
DoStuff
(
string
[
]
strArray
,
int
someArgument
)
 

And is it possible to include in that regular expression such delimiters as one line and multiline comments?
For example:

public /*should be private*/  static void DoStuff( string[] strArray ) //int someArgument )
 

after the split will be:

public
/*should be private*/
static 
void 
DoStuff
( 
string
[
] 
strArray 
) 
//int someArgument )
 

And how to handle binary delimiters such as <<, >>, ==, !=, >=, or <= ?


Many thanks in advance.