spec: round values #636
spec: round values #636
Conversation
SHA: d4dea52 Reason: push, by JensenPaul Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
|
Hey @JensenPaul, I don't think the rounding algorithm here accomplishes its intended goal. We should round values up or down in such a way that the expected value of the result is the original number. In your steps 6/7/8, if precisionScaledValue is 12.3, then truncatedScaledValue will always be 12, while we want it to come out 13 in 30% of the cases. Seems like the way to do that is to define the fractional part of precisionScaledValue, and then compare that to a uniformly random value between 0 and 1. If the fractional part is smaller than the random value we round down, while if the fractional part is larger than the random value we round up. It also seems like the spec should mention the intended mathematical property of the algorithm, so that people can easily see the goal and verify that we're meeting it. |
|
D'oh, as @brusshamilton just pointed out to me, what I wrote is exactly the same as just changing the random value in step 7 to be between 0 and 1, rather than the current version that is between 0 and 0.5. |
Preview | Diff