By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
437,691 Members | 2,041 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 437,691 IT Pros & Developers. It's quick & easy.

How to tokenize input

P: 4
my input is

hi, my name is chungku. I am 24 years old.
want to toknize even the punctuation and number....

output should be in this form

hi
,
my
name
is
chungku
.
I
am
24
years
old
.

thanks for all you generous help
Feb 1 '10 #1
Share this Question
Share on Google+
3 Replies


P: 2
Please enhance your question.
1.Do you need to replace space with new line ?
2.How will you differentiate with the chars that what should be in next line or in the same line.

If case 1, then copy the text in "VI" editor, press escape and enter these line
Expand|Select|Wrap|Line Numbers
  1. %s:\s:\r:g
and press enter.

Thanks,
PerlLearner
Feb 1 '10 #2

P: 2
If you need perl code for (1) then here it is,

Expand|Select|Wrap|Line Numbers
  1. #!/usr/bin/perl
  2. use strict;
  3. my $str="hi, my name is chungku. I am 24 years old.";
  4. $str=~s/\s/\n/g;
  5. print "str is |$str|\n";
Thanks,
PerlLearner
Feb 1 '10 #3

numberwhun
Expert Mod 2.5K+
P: 3,503
I can't help but think that the different questions you are posting sound distinctly like school work. Are these questions in fact related to school work? If they are, then you should not be asking with intent to get answers. That is against the sites posting policy.

Regards,

Jeff
Feb 1 '10 #4

Post your reply

Sign in to post your reply or Sign up for a free account.