446,412 Members | 1,009 Online
Need help? Post your question and get tips & solutions from a community of 446,412 IT Pros & Developers. It's quick & easy.

floating-point precision

 P: n/a Hi guys, I use LUTs to accelerate certain parts of my app. The LUTs contain float numbers, and I generated the numbers very precisely, up to 10 decimals. All seems fine, however, when I printf the numbers in the LUT, I do not get the same exact numbers. The first 6 decimals are ok, but it got beserk after that. To illustrate further: float myLUT = [0.123456789, 0.123456789, 0.123456789, ................] ; printf("myLUT: .10f", myLUT[1]); ##### output myLUT: 0.123456111 I am not quite sure why that is the case, although I believe float do not have enough precision, so the compiler (VC 8) gives the closest approximation that can be had with float. Is that so? Can somebody enlighten me on that issue? Thanks in advance, Enrique Mar 15 '07 #1
6 Replies

 P: n/a Enrique Cruiz

 P: n/a Enrique Cruiz San Diego Supercomputer Center <* "We must do something. This is something. Therefore, we must do this." -- Antony Jay and Jonathan Lynn, "Yes Minister" Mar 16 '07 #3

 P: n/a On Mar 15, 5:18 pm, Enrique Cruiz

 P: n/a Enrique Cruiz wrote: > I use LUTs to accelerate certain parts of my app. The LUTs contain float numbers, and I generated the numbers very precisely, up to 10 decimals. All seems fine, however, when I printf the numbers in the LUT, I do not get the same exact numbers. The first 6 decimals are ok, but it got beserk after that. To illustrate further: float myLUT = [0.123456789, 0.123456789, 0.123456789, ..........] ; printf("myLUT: .10f", myLUT[1]); ##### output myLUT: 0.123456111 I am not quite sure why that is the case, although I believe float do not have enough precision, so the compiler (VC 8) gives the closest approximation that can be had with float. Is that so? Can somebody enlighten me on that issue? Look in mm -- Posted via a free Usenet account from http://www.teranews.com Mar 16 '07 #5

 P: n/a William Hughes wrote: > .... snip ... > Have you checked to see if using the LUT actually does increase execution speed? (It is quite possible for the opposite to happen). Have you checked whether the putative incease in speed is significant? What is a LUT? A capitalized abbreviation for a lute, perhaps. If so, who is playing it. -- Chuck F (cbfalconer at maineline dot net) Available for consulting/temporary embedded and systems. -- Posted via a free Usenet account from http://www.teranews.com Mar 16 '07 #6

 P: n/a CBFalconer