Negation detection in text

[maral]

Hi every one,

I'm working on negation detection for sentiment analysis in text files. Most common negation words are not, no and never. for example "good" is classified as positive but "not good" should be classified as negative.

So far I can only detect negations where the adjective is exactly right after the negation word ("not good"). But I need to consider different scopes. I need to detect the negation effect till the end of the sentence. i.e. if the system finds "not" in sentence, I want it to consider all the adjectives till the "." mark. For example: "this movie was not a very good one." Here although "good" is not right after "not" but since is before the end of the sentence, it will be classified as negative.

There are three input arguments: 1- a file containing a list of negative words like "bad" and "boring" 2- a file containing a list of positive words like "good" and "interesting" 3- a directory which includes text files which are movie reviews.

My problems is that I can not extend the scope of my negation. How can I check next 5 words after negation word or how can I find the adjectives till the end of the sentence and after the negation word.



Here is my code of what I have done so far. I also tried few "While()" and "for()" loops for my purpose but I could not manage to find the correct one.

I hope it is clear. Please don't hesitate to ask any further questions.

regards,
Maral

import java.io.*;
import java.util.*;
import java.util.regex.*; 
 
public class Find
{
    public static void main(String args[])
    {
 
        if(args.length!=3)
        {
 
            System.exit(1);
        }
        try
        {
            //read the word lists
            HashSet<String> positiveWords = new HashSet<String>();
            HashSet<String> negativeWords = new HashSet<String>();
 
            BufferedReader br = new BufferedReader(new FileReader(args[0]));
 
 
            String line;
 
            while((line=br.readLine())!=null)
            {
 
                positiveWords.add(line.toLowerCase());
 
            }
            br.close();
 
            br = new BufferedReader(new FileReader(args[1]));
            line = "";
            while((line=br.readLine())!=null)
            {
                negativeWords.add(line.toLowerCase());
            }
            br.close();
 
            System.out.println("number of positive words read: "+positiveWords.size());
            System.out.println("number of negative words read: "+negativeWords.size());
 
            //read each file from the input directory
            File folder = new File(args[2]);
            File[] listOfFiles = folder.listFiles();
            if(listOfFiles!=null && listOfFiles.length > 0)
            {
                for(int i=0; i<listOfFiles.length; i++)
                {
                    if(listOfFiles[i].isFile()){;}
                    else{continue;}
 
                    System.out.println("Dealing with "+listOfFiles[i]);
                    System.out.println(i);
                    int posCounter = 0;
                    int negCounter = 0;
 
                    HashMap<String,Integer> positives = new HashMap<String,Integer>();
                    HashMap<String,Integer> negatives = new HashMap<String,Integer>();
 
                    Scanner sc = new Scanner(new BufferedReader(new FileReader(listOfFiles[i])));
 
                    boolean isNegation = false;
                    while(sc.hasNext()) 
                    {
                        String w = (sc.next()).toLowerCase();
 
 
                        // Set flag if word is not
                        if (w.equals("not"))
                        {
                            isNegation = true;
                        }
                        else
                        {
 
                            if(positiveWords.contains(w))
                            {
                                if (isNegation)
                                {
                                    negCounter++;
                                }
                                else
                                {
                                    posCounter++;
 
                                    if(positives.containsKey(w))
                                    {
                                        int v = positives.get(w).intValue();
                                        v++;
                                        positives.put(w,new Integer(v));
                                    }
                                    else
                                    {
                                        positives.put(w,new Integer(1));
                                    }
                                }
                            }
 
                            if(negativeWords.contains(w))  
                            {
                                if (isNegation)
                                {
                                    posCounter++;
                                }
                                else
                                {
                                    negCounter++;
                                    if(negatives.containsKey(w))
                                    {
                                        int v = negatives.get(w).intValue();
                                        v++;
                                        negatives.put(w,new Integer(1));
                                    }
                                    else
                                    {
                                        negatives.put(w,new Integer(1));
                                    }
                                }
                            }
                        }
                            isNegation = false;
 
                    }
 
                    Iterator<String> it = positives.keySet().iterator();
                    while(it.hasNext())
                    {
                        String w = it.next();
 
                        System.out.println(w+": "+positives.get(w));
                    }
 
 
                    it = negatives.keySet().iterator();
                    while(it.hasNext())
                    {
                        String w = it.next();
                        System.out.println(w+": "+negatives.get(w));
                    }
                    System.out.println("number of positives: "+posCounter);
                    System.out.println("number of negatives: "+negCounter);
 
                }    
            }    
        }
        catch(Exception e)
        {
            e.printStackTrace();
        }
 
    }
}
 

[jkmyoung]

In terms of style, I would set isNegation=false at the beginning of the loop as opposed to before the loop and at the end of it.

You definitely need more documentation, particularly around your incoming arguments.

You might also want equalsIgnoreCase instead of equals.