• ShortFuse@lemmy.world
    cake
    link
    fedilink
    arrow-up
    3
    ·
    10 months ago

    I just recently worked on fixed point 8.8 and basically the way fractional values work, you take an integer and say that integer is then divided by another one. So you represent the number in question with two numbers not one. 0.3 can be presented in a number of ways, like 30 % 10, or 6 % 20.

    The problem is the way 0.1 is represented and 0.2 represented don’t jive when you add them, so the compiler makes a fractional representation of 0.3 based on how 0.1 and 0.2 were expressed that just comes out weird.

    That’s also why 0.3 + 0.3 is fine. When you wrote 0.3, the compiler/runtime already knew how to express 0.3 without rounding errors.