By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
464,589 Members | 1,022 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 464,589 IT Pros & Developers. It's quick & easy.

Problem with Numpy Standard Deviation

P: 3
Can anyone please tell me where I am going wrong with this for loop which is meant to take as input a specific corpus, sample size and number of samples and then give the averages of the expected sentiment tokens, normalised lexical diversity and probability of short sentences. It is also meant to give me the standard deviation of these three statistics too. I'm a real beginner with Python so not really sure where I've gone wrong Thanks a lot in advance.

Expand|Select|Wrap|Line Numbers
  1. def test_iterate(corpus_reader, sample_size, number_of_samples):
  2. for i in xrange(number_of_samples):
  3. tokens = corpus_reader.sample_words_by_sents(sample_size)
  4. sents = corpus_reader.sample_sents(sample_size)
  5. expected_sentiment_tokens(tokens)
  6. normalised_lexical_diversity(tokens)
  7. prob_short_sents(sents)
  8. stats = expected_sentiment_tokens(tokens)
  9. stats_two = normalised_lexical_diversity(tokens)
  10. stats_three = prob_short_sents(sents)
  11. print "Average expected no of sentiment tokens: %s" % average(stats)
  12. print "Average normalised lexical diversity: %s" % average(stats_two)
  13. print "Average probability of short sentences: %s" % average(stats_three)
  14. print "Standard deviation of sentiment tokens: %s" % std(stats)
  15. print "Standard deviation of normalised lexical diversity: %s" % std(stats_two)
  16. print "Standard deviation of probability of short sentences: %s" % std(stats_three)
When I call for example
Expand|Select|Wrap|Line Numbers
  1. test_iterate(tcr, 500, 3)
, the following output is given:

Expand|Select|Wrap|Line Numbers
  1. 127.333333333 
  2. 2.08398681196 
  3. 0.506 
  4. 116.25 
  5. 2.21737363871 
  6. 0.518 
  7. 123.333333333 
  8. 1.9821801535 
  9. 0.534 
  10. Average expected no of sentiment tokens: 110.416666667 
  11. Average normalised lexical diversity: 2.89485940038 
  12. Average probability of short sentences: 0.518 
  13. Standard deviation of sentiment tokens: 0.0 
  14. Standard deviation of normalised lexical diversity: 0.0 
  15. Standard deviation of probability of short sentences: 0.0
Oct 14 '12 #1
Share this Question
Share on Google+
6 Replies

Expert 100+
P: 626
I'm a real beginner with Python so not really sure where I've gone wrong
What is wrong with what you have? What do you expect vs. what is printed? It looks like you are not saving the results from each iteration in the for() loop so are only printing the final pass but I can't tell from what you have submitted. Add some print statements similar to this
Expand|Select|Wrap|Line Numbers
  1. def test_iterate(corpus_reader, sample_size, number_of_samples):
  2.      for i in xrange(number_of_samples):
  3.          tokens = corpus_reader.sample_words_by_sents(sample_size)
  4.          sents = corpus_reader.sample_sents(sample_size)
  5.          expected_sentiment_tokens(tokens)
  6.          normalised_lexical_diversity(tokens)
  7.          prob_short_sents(sents)
  8.  
  9.          stats = expected_sentiment_tokens(tokens)
  10.          print "stats in for loop =", stats
  11.  
  12.          stats_two = normalised_lexical_diversity(tokens)
  13.          stats_three = prob_short_sents(sents)
  14.  
  15.      print "using stats =", stats, type(stats)
  16.      print "Average expected no of sentiment tokens: %s" % average(stats) 
If "stats" is anything other than a numpy array then you are probably averaging one number instead of a list of numbers.
Oct 14 '12 #2

P: 3
Thanks for the quick reply. I just tried that and this is what outputted
Expand|Select|Wrap|Line Numbers
  1. stats in for loop = 191.473684211
  2. stats in for loop = 186.277777778
  3. stats in for loop = 182.473684211
  4. stats in for loop = 182.611111111
  5. using stats = 182.611111111 <type 'float'>
  6. Average expected no of sentiment tokens: 182.611111111
Do you know how I could fix my code to make it work correctly? Just need the averages and standard deviations for each statistic.

Thanks again.
Oct 14 '12 #3

Expert 100+
P: 626
stats in for loop = 182.611111111 <----- same value
using stats = 182.611111111 <type 'float'> on both lines
You have to append each of the values in the loop to a numpy array and average the array (some definitions). Start with a simple average only and then expand the code. I am sure there are many tutuorials/examples on the web for arrays, average, and standard deviation.
Oct 15 '12 #4

P: 3
Could you please tell me what's wrong with this following code?
Expand|Select|Wrap|Line Numbers
  1. def test_iterate(corpus_reader, sample_size, number_of_samples):
  2. for i in xrange(number_of_samples):
  3. tokens = corpus_reader.sample_words_by_sents(sample_size)
  4. sents = corpus_reader.sample_sents(sample_size)
  5. print expected_sentiment_tokens(tokens)
  6. s = ([expected_sentiment_tokens(tokens)])
  7. s.append(expected_sentiment_tokens(tokens))
  8. print "Average expected no of sentiment tokens: %s" % average(s)
Expand|Select|Wrap|Line Numbers
  1. test_iterate(rcr, 500, 3)
gives output

Expand|Select|Wrap|Line Numbers
  1. 191.823529412
  2. 185.117647059
  3. 185.166666667
  4. Average expected no of sentiment tokens: 185.166666667
the average is being assigned just the last value
Oct 15 '12 #5

Rabbit
Expert Mod 10K+
P: 12,441
You're just getting the last value because that's all you have available. In line 6, you replace whatever you had with the most current value. Then in line 7, you append the most current value. The result being an average of the most current value. Then in the next iteration, you do it all over again. You will only ever have 2 of the most current values in there.
Oct 15 '12 #6

Rabbit
Expert Mod 10K+
P: 12,441
I am reverting your posts for posterity.

Please don't edit your posts and remove all traces of your question. If someone were to visit this thread to see the answer, we would like them to be able to view everything. Also, if you've solved the issue, please post the answer so that others facing the same issue may benefit from your solution.
Oct 15 '12 #7

Post your reply

Sign in to post your reply or Sign up for a free account.