I am working on Information Retrieval field.For this project I need to Tokenize a collection of documents such as text files. I have done how to tokenize a string and one text file.but in the text file i am able to tokenize on the whitespace only,not able to work on hyphen or comma etc.So,I need the java code which will actually tokenize the character while getting , or - or ' etc for a collection of text files. pls help pls....
10 8114
Just simply replace the whitespace " " in the code with a comma "," etc. to tokenize on other characters.
But most likely you want all of the mentioned characters together to be a token separator. Then tokenize by using regular Expression: - String tokens[] = textString.split("[\\s\\-,']+".
If you still have problems, then please show the code you have done so far here, so that we can improve and change it.
One tip: to get more and faster answers in general do NOT write "pls help pls". For me it makes the impression that you are in a hurry (abbreviation) and will not value any answer. It's clear that we are here to help you, but I feel pressure if you mention this self-speaking fact and emphasize it with two pleas. You are just lucky that I am in a very good mood right now, else I would not have answered you because of this sentence.
thank you very much...I am able to tokenize the text file on getting white space and any other punctuation.Now I want to tokenize a collection of text files not only a single text file.
I am attaching my code here.one error has occured and i am unable to find why actualy it is occuring.
Ok, here is the code from the file. I put it here directly using code tags instead of a text file, because then it's easier for others to read (and understand and providing help based on the line number). For the same reason, I cleaned up by removing commented-out code and then indented it properly.
Cleaned-up original code: - package stemmer;
-
import java.util.*; // Provides TreeMap, Iterator, Scanner
-
import java.io.*; // Provides FileReader, FileNotFoundException
-
-
public class NewEmpty
-
{
-
public static void main(String[ ] args)
-
{
-
Scanner br;
-
-
//**READ THE DOCUMENTS**
-
for (int x=0; x<Docs.length; x++)
-
{
-
br = new Scanner(new FileReader(Docs[x]));
-
}
-
-
try
-
{
-
String strLine= " ";
-
String filedata="";
-
while ( (strLine =br.readLine()) != null)
-
{
-
filedata+=strLine+" ";
-
}
-
StringTokenizer stk=new StringTokenizer(filedata," .,-'");
-
while(stk.hasMoreTokens())
-
{
-
String token=stk.nextToken();
-
System.out.println(token);
-
}
-
br.close();
-
}
-
catch (Exception e)
-
{
-
System.err.println("Error: " + e.getMessage());
-
}
-
-
// Array of documents
-
String Docs [] = {"words.txt", "words2.txt","words3.txt", "words4.txt",};
-
}
-
}
First, I wonder how it compiled at all. In line 39, you defined the string-array of docs that you want to loop through in line 12. So it must be defined BEFORE line 12
Second, you open a text file for reading in your for-loop and assigning it to "br", but instead of parsing its content, you close the for-loop which will assign the next text file and so on, until you assign the last text file and then you parse only this last one. To fix the code, you must do all the parsing (line 17 to 36) inside the for-loop, not outside.
Third, you have a memory leak. If there occurs an exception while reading a file, you don't close the file, leaving it open forever. You must do your close-command in the "finally" part of your try-catch-command. (Unfortunately the close-command can also throw an error, so it needs a try-catch itself).
Fourth, the string array should be named "docs" instead of "Docs". Only classes should start with uppercase letters, but instances not. Every professional java programmer follows this coding style for good reasons, which I will not explain further here, because it leads too far away.
There are some other minor issues and enhancements, but they don't hinder you to get it running, so I will not mention them now.
Here is the corrected source code. (I cannot try to run it at the moment, but you should do it anyway, so tell me if it's ok now.) - package stemmer;
-
import java.util.*; // Provides TreeMap, Iterator, Scanner
-
import java.io.*; // Provides FileReader, FileNotFoundException
-
-
public class NewEmpty
-
{
-
public static void main(String[ ] args)
-
{
-
// Array of documents
-
String docs [] = {"words.txt", "words2.txt","words3.txt", "words4.txt",};
-
-
// process all documents
-
for (int x=0; x<docs.length; x++)
-
{
-
// read document and parse it
-
Scanner br = new Scanner(new FileReader(docs[x]));
-
try
-
{
-
String strLine= " ";
-
String filedata="";
-
while ( (strLine =br.readLine()) != null)
-
{
-
filedata+=strLine+" ";
-
}
-
StringTokenizer stk=new StringTokenizer(filedata," .,-'");
-
while(stk.hasMoreTokens())
-
{
-
String token=stk.nextToken();
-
System.out.println(token);
-
}
-
}
-
catch (Exception e)
-
{
-
System.err.println("Error: " + e.getMessage());
-
}
-
finally
-
{
-
try
-
{
-
br.close();
-
}
-
catch (Exception e2)
-
{
-
// NOPMD
-
}
-
}
-
}
-
}
-
}
The following code is working fine in NetBeans for the above problem :D - package FinalizedPrograms;
-
import java.io.BufferedReader;
-
import java.util.*; // Provides TreeMap, Iterator, Scanner
-
import java.io.*; // Provides FileReader, FileNotFoundException
-
-
public class TokenizingMultipleFiles
-
{
-
public static void main(String[ ] args)
-
{
-
// Scanner br;
-
// Array of documents
-
String Docs [] = {"temp.txt", "temp1.txt",};
-
//**FOR LOOP TO READ THE DOCUMENTS**
-
for (int x=0; x<Docs.length; x++)
-
{
-
try
-
{
-
File f=new File(Docs[x]);
-
BufferedReader br = new BufferedReader(new FileReader(f));
-
//br = new Scanner(new FileReader(Docs[x]));
-
try{
-
String strLine= " ";
-
String filedata="";
-
while ( (strLine = br.readLine()) != null) {
-
filedata+=strLine+" ";
-
}
-
StringTokenizer stk=new StringTokenizer(filedata," .,-'[]{}/|@#!$%^&*_-+=?<>:;()");
-
while(stk.hasMoreTokens()){
-
String token=stk.nextToken();
-
System.out.println(token);
-
}
-
br.close();
-
}
-
catch (Exception e){
-
System.err.println("Error: " + e.getMessage());
-
}
-
-
}
-
catch (FileNotFoundException e)
-
{
-
System.err.println(e);
-
return;
-
}
-
} //End of for loop *]
-
-
}
-
}
-
- package IR;
-
import java.io.BufferedReader;
-
import java.util.*; // Provides TreeMap, Iterator, Scanner
-
import java.io.*; // Provides FileReader, FileNotFoundException
-
-
public class FilesTokenization
-
{
-
public static void main(String[ ] args)
-
{
-
// Scanner br;
-
// Array of documents
-
String Docs [] = {"words.txt", "words2.txt","words3.txt", "words4.txt",};
-
//start for loop
-
for (int x=0; x<Docs.length; x++)
-
{
-
try
-
{
-
File f=new File(Docs[x]);
-
BufferedReader br = new BufferedReader(new FileReader(f));
-
//br = new Scanner(new FileReader(Docs[x]));
-
-
try{
-
String strLine= " ";
-
String filedata="";
-
while ( (strLine = br.readLine()) != null)
-
{
-
filedata+=strLine+" ";
-
}
-
StringTokenizer stk=new StringTokenizer(filedata," .,-';{}?()");
-
while(stk.hasMoreTokens())
-
{
-
String token=stk.nextToken();
-
System.out.println(token);
-
}
-
br.close();
-
}
-
catch (FileNotFoundException e)
-
{
-
System.err.println(e);
-
return;
-
}
-
}
-
catch (Exception e){
-
System.err.println("Error: " + e.getMessage());
-
}
-
}
-
}
-
}
thank you very much chaarman for pointing out the faults.and I am able to successfully run program and it gives correct output.the code is mentioned in the above post by me.
It is very helpful content for me.
Thanks for provide such type of content
Hi.Presently I am doing a project on personalized web search which was related to information retrieval concepts like stemming and tokenization. Can any one help me in providing the related code for my project.
Sign in to post your reply or Sign up for a free account.
Similar topics
by: Ron |
last post by:
I have a text file with the following format (pipe delimited)
|column1=value|column2=value|column3=value|column4=value|...
|column1=value|column2=value|column3=value|column4=value|...
I have...
|
by: Job Lot |
last post by:
How can I modify values in text file? File is tab delimited as follows
Date Buy Sell
13-Jan-2005 0.9970776 0.9901224
18-Jan-2005 0.9910566 0.9841434
I want to modify Buy and Sell...
|
by: Ram Laxman |
last post by:
Hi all,
I have a text file which have data in CSV format.
"empno","phonenumber","wardnumber"
12345,2234353,1000202
12326,2243653,1000098
Iam a beginner of C/C++ programming.
I don't know how to...
|
by: Raghavendra Mahuli |
last post by:
Hello,
I have a text file in which records are stored in a particular format. For
ex:
Node1( att1, att2, node2(attx))
I need to convert it to xml. I know xsl can be used to convert *xml to*...
|
by: Ilias Lazaridis |
last post by:
within a python script, I like to create a collection which I fill with
values from an external text-file (user editable).
How is this accomplished the easiest way (if possible without the need...
|
by: Stupid48 |
last post by:
I'm trying to do a simple task but can't seem to find a solution. How
do I read lines from a text file backwards.
i.e. I want to select the last 20 lines of a text file and display them
in...
|
by: DN2UK |
last post by:
I am making a windows form in vb.net that needs to read form a text file and insert the text as objects when the form loads . The textboxes are as follow
TxtJobtitle.Text
TxtSalary.Text...
|
by: Faith0G |
last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
|
by: ryjfgjl |
last post by:
In our work, we often need to import Excel data into databases (such as MySQL, SQL Server, Oracle) for data analysis and processing. Usually, we use database tools like Navicat or the Excel import...
|
by: taylorcarr |
last post by:
A Canon printer is a smart device known for being advanced, efficient, and reliable. It is designed for home, office, and hybrid workspace use and can also be used for a variety of purposes. However,...
|
by: Charles Arthur |
last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
|
by: aa123db |
last post by:
Variable and constants
Use var or let for variables and const fror constants.
Var foo ='bar';
Let foo ='bar';const baz ='bar';
Functions
function $name$ ($parameters$) {
}
...
|
by: ryjfgjl |
last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
|
by: emmanuelkatto |
last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud.
Please let me know.
Thanks!
Emmanuel
|
by: nemocccc |
last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
|
by: Sonnysonu |
last post by:
This is the data of csv file
1 2 3
1 2 3
1 2 3
1 2 3
2 3
2 3
3
the lengths should be different i have to store the data by column-wise with in the specific length.
suppose the i have to...
| |