zzh
/
gatk-3.8
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Fix corner case rounding issue in MathUtils unit test: 10^logFactorial(4)) was 23.999999... which if cast directly yielded 23 - so, do pre-rounding to ensure correct integer result if caller will cast value.

This commit is contained in:
Guillermo del Angel 2012-05-02 09:57:06 -04:00
parent 76a95fdedf
commit 429800a192
1 changed files with 2 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
public/java/src/org/broadinstitute/sting/utils/MathUtils.java
View File

@ -1463,7 +1463,8 @@ public class MathUtils {
}
public static double factorial(int x) {
return Math.pow(10, log10Factorial(x));
// avoid rounding errors caused by fact that 10^log(x) might be slightly lower than x and flooring may produce 1 less than real value
return (double)Math.round(Math.pow(10, log10Factorial(x)));
}
public static double log10Factorial(int x) {