how do i tokenize a string on the base of special characters such as ' and . ect?

[Ahsan123pk]

I have used StringTokenizer class to make tokens of a string but my program tokenizes a string only on the basis of spaces but it does not tokenizes on the basis of special characters such as comma(,),full stop(.),single quotes etc.

Consider a sentence or string:
"I don't know who is he."

After compiling it the output of my program is:
I: valid token
don't: valid token
know: valid token
who: valid token
is: valid token
he!:valid token


but my requirement is:
the short form "don't" should be replaced by "do not" and the out should be:

do: valid token
not: valid token

and

he: valid token
!: valid token

I want your help.Can you suggest me what to do?
I'll be thankful to you

[rotaryfreak]

@Ahsan123pk
take a look at the java api for the StringTokenizer class, there you will find that you can specifically change the default delimiter to whatever you want to with the method
public StringTokenizer(String theString, String delimiters)

http://java.sun.com/j2se/1.4.2/docs/...Tokenizer.html

and if you're still confused on the topic, i encourage you to visit my old professor's website,

http://aimanhanna.com/concordia/comp248/index.htm

there you will find lots of examples on various java topics but the example in particular that will help you is titled "Strings2.java"(.doc)

hope this helps!

[Ahsan123pk]

Thankyou for the answer it worked.