By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
455,798 Members | 1,378 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 455,798 IT Pros & Developers. It's quick & easy.

How to tokenize a collection of text file?

P: 8
I am working on Information Retrieval field.For this project I need to Tokenize a collection of documents such as text files. I have done how to tokenize a string and one text file.but in the text file i am able to tokenize on the whitespace only,not able to work on hyphen or comma etc.So,I need the java code which will actually tokenize the character while getting , or - or ' etc for a collection of text files. pls help pls....
Aug 13 '13 #1
Share this Question
Share on Google+
10 Replies


Expert 100+
P: 785
Just simply replace the whitespace " " in the code with a comma "," etc. to tokenize on other characters.
But most likely you want all of the mentioned characters together to be a token separator. Then tokenize by using regular Expression:
Expand|Select|Wrap|Line Numbers
  1. String tokens[] = textString.split("[\\s\\-,']+".
If you still have problems, then please show the code you have done so far here, so that we can improve and change it.

One tip: to get more and faster answers in general do NOT write "pls help pls". For me it makes the impression that you are in a hurry (abbreviation) and will not value any answer. It's clear that we are here to help you, but I feel pressure if you mention this self-speaking fact and emphasize it with two pleas. You are just lucky that I am in a very good mood right now, else I would not have answered you because of this sentence.
Aug 13 '13 #2

P: 8
thank you very much...I am able to tokenize the text file on getting white space and any other punctuation.Now I want to tokenize a collection of text files not only a single text file.
I am attaching my code here.one error has occured and i am unable to find why actualy it is occuring.
Attached Files
File Type: txt filetoken.txt (1.1 KB, 708 views)
Aug 13 '13 #3

Expert 100+
P: 785
Ok, here is the code from the file. I put it here directly using code tags instead of a text file, because then it's easier for others to read (and understand and providing help based on the line number). For the same reason, I cleaned up by removing commented-out code and then indented it properly.
Cleaned-up original code:
Expand|Select|Wrap|Line Numbers
  1. package stemmer; 
  2. import java.util.*;  // Provides TreeMap, Iterator, Scanner  
  3. import java.io.*;    // Provides FileReader, FileNotFoundException  
  4.  
  5. public class NewEmpty  
  6. {  
  7.    public static void main(String[ ] args)  
  8.    {  
  9.         Scanner br;  
  10.  
  11.         //**READ THE DOCUMENTS**  
  12.         for (int x=0; x<Docs.length; x++)  
  13.         {  
  14.             br = new Scanner(new FileReader(Docs[x]));                   
  15.         }
  16.  
  17.         try
  18.         {
  19.             String strLine= " ";
  20.             String filedata="";
  21.             while ( (strLine =br.readLine()) != null)
  22.             {
  23.                 filedata+=strLine+" ";
  24.             }
  25.             StringTokenizer stk=new StringTokenizer(filedata," .,-'");
  26.             while(stk.hasMoreTokens())
  27.             {
  28.                String token=stk.nextToken();
  29.                System.out.println(token);
  30.             }
  31.             br.close();
  32.         }  
  33.         catch (Exception e)
  34.         {
  35.             System.err.println("Error: " + e.getMessage());
  36.         }
  37.  
  38.         // Array of documents  
  39.         String Docs [] = {"words.txt", "words2.txt","words3.txt", "words4.txt",};  
  40.     } 
  41. }
Aug 14 '13 #4

Expert 100+
P: 785
First, I wonder how it compiled at all. In line 39, you defined the string-array of docs that you want to loop through in line 12. So it must be defined BEFORE line 12

Second, you open a text file for reading in your for-loop and assigning it to "br", but instead of parsing its content, you close the for-loop which will assign the next text file and so on, until you assign the last text file and then you parse only this last one. To fix the code, you must do all the parsing (line 17 to 36) inside the for-loop, not outside.

Third, you have a memory leak. If there occurs an exception while reading a file, you don't close the file, leaving it open forever. You must do your close-command in the "finally" part of your try-catch-command. (Unfortunately the close-command can also throw an error, so it needs a try-catch itself).

Fourth, the string array should be named "docs" instead of "Docs". Only classes should start with uppercase letters, but instances not. Every professional java programmer follows this coding style for good reasons, which I will not explain further here, because it leads too far away.

There are some other minor issues and enhancements, but they don't hinder you to get it running, so I will not mention them now.

Here is the corrected source code. (I cannot try to run it at the moment, but you should do it anyway, so tell me if it's ok now.)
Expand|Select|Wrap|Line Numbers
  1. package stemmer; 
  2. import java.util.*;  // Provides TreeMap, Iterator, Scanner  
  3. import java.io.*;    // Provides FileReader, FileNotFoundException  
  4.  
  5. public class NewEmpty  
  6. {  
  7.    public static void main(String[ ] args)  
  8.    {  
  9.         // Array of documents  
  10.         String docs [] = {"words.txt", "words2.txt","words3.txt", "words4.txt",};  
  11.  
  12.         // process all documents  
  13.         for (int x=0; x<docs.length; x++)  
  14.         {
  15.             // read document and parse it
  16.             Scanner br = new Scanner(new FileReader(docs[x]));                   
  17.             try
  18.             {
  19.                 String strLine= " ";
  20.                 String filedata="";
  21.                 while ( (strLine =br.readLine()) != null)
  22.                 {
  23.                     filedata+=strLine+" ";
  24.                 }
  25.                 StringTokenizer stk=new StringTokenizer(filedata," .,-'");
  26.                 while(stk.hasMoreTokens())
  27.                 {
  28.                    String token=stk.nextToken();
  29.                    System.out.println(token);
  30.                 }                
  31.             }  
  32.             catch (Exception e)
  33.             {
  34.                 System.err.println("Error: " + e.getMessage());
  35.             }
  36.             finally
  37.             {
  38.                 try
  39.                 {
  40.                     br.close();
  41.                 }
  42.                 catch (Exception e2)
  43.                 {
  44.                     // NOPMD 
  45.                 }
  46.             }
  47.         }
  48.     } 
  49. }
Aug 14 '13 #5

P: 1
The following code is working fine in NetBeans for the above problem :D


Expand|Select|Wrap|Line Numbers
  1. package FinalizedPrograms;
  2. import java.io.BufferedReader;
  3. import java.util.*;  // Provides TreeMap, Iterator, Scanner  
  4. import java.io.*;    // Provides FileReader, FileNotFoundException  
  5.  
  6. public class TokenizingMultipleFiles  
  7. {  
  8.    public static void main(String[ ] args)  
  9.    {  
  10.      // Scanner br;  
  11.    // Array of documents  
  12.   String Docs [] = {"temp.txt", "temp1.txt",};
  13. //**FOR LOOP TO READ THE DOCUMENTS**  
  14. for (int x=0; x<Docs.length; x++)  
  15. {  
  16.   try  
  17.       {  
  18.           File f=new File(Docs[x]);
  19.           BufferedReader br = new BufferedReader(new FileReader(f));
  20.          //br = new Scanner(new FileReader(Docs[x]));  
  21.          try{
  22. String strLine= " ";
  23. String filedata="";
  24. while ( (strLine = br.readLine()) != null)   {
  25. filedata+=strLine+" ";
  26. }
  27. StringTokenizer stk=new StringTokenizer(filedata," .,-'[]{}/|@#!$%^&*_-+=?<>:;()");
  28.    while(stk.hasMoreTokens()){
  29.        String token=stk.nextToken();
  30.        System.out.println(token);
  31.    }
  32.    br.close();
  33.    }  
  34.    catch (Exception e){
  35.      System.err.println("Error: " + e.getMessage());
  36.    }
  37.  
  38.       }  
  39.      catch (FileNotFoundException e)  
  40.      {  
  41.  System.err.println(e);  
  42.  return;  
  43.       }  
  44.      } //End of for loop *]
  45.  
  46. }  
  47.  
Aug 14 '13 #6

P: 8
Expand|Select|Wrap|Line Numbers
  1. package IR;
  2. import java.io.BufferedReader;
  3. import java.util.*;  // Provides TreeMap, Iterator, Scanner  
  4. import java.io.*;    // Provides FileReader, FileNotFoundException  
  5.  
  6. public class FilesTokenization 
  7. {  
  8.    public static void main(String[ ] args)  
  9.    {  
  10.      // Scanner br;  
  11.    // Array of documents  
  12.   String Docs [] = {"words.txt", "words2.txt","words3.txt", "words4.txt",};
  13.   //start for loop
  14.   for (int x=0; x<Docs.length; x++)  
  15. {  
  16.   try  
  17.       {  
  18.           File f=new File(Docs[x]);
  19.           BufferedReader br = new BufferedReader(new FileReader(f));
  20.          //br = new Scanner(new FileReader(Docs[x]));  
  21.  
  22. try{
  23. String strLine= " ";
  24. String filedata="";
  25. while ( (strLine = br.readLine()) != null)   
  26. {
  27. filedata+=strLine+" ";
  28. }
  29. StringTokenizer stk=new StringTokenizer(filedata," .,-';{}?()");
  30.    while(stk.hasMoreTokens())
  31.    {
  32.        String token=stk.nextToken();
  33.        System.out.println(token);
  34.    }
  35.    br.close();
  36.    }  
  37.      catch (FileNotFoundException e)  
  38.      {  
  39.  System.err.println(e);  
  40.  return;  
  41.       }
  42.       }
  43. catch (Exception e){
  44.      System.err.println("Error: " + e.getMessage());
  45.   }
  46.       }  
  47. }  
  48. }
Aug 14 '13 #7

P: 8
thank you very much chaarman for pointing out the faults.and I am able to successfully run program and it gives correct output.the code is mentioned in the above post by me.
Aug 14 '13 #8

P: 2
It is very helpful content for me.
Thanks for provide such type of content
Aug 27 '13 #9

P: 2
Hi.Presently I am doing a project on personalized web search which was related to information retrieval concepts like stemming and tokenization. Can any one help me in providing the related code for my project.
Jan 6 '15 #10

P: 2
Please mail the code for tokenization to me. My mail id: mailmevinay1994@gmail.com
Jan 6 '15 #11

Post your reply

Sign in to post your reply or Sign up for a free account.