• Qfuiyh
    link
    fedilink
    arrow-up
    3
    arrow-down
    1
    ·
    1 year ago

    Title feels misleading, it gets stack overflow questions wrong 52% of the time

    However it got 77% of easy Leetcode questions correct. Also I believe that’s first try, which is not generally how chatgpt should be used.

    Also also, you should probably be using a coding specific model if you want good coding results

    • nanoUFO@sh.itjust.works
      link
      fedilink
      arrow-up
      5
      ·
      1 year ago

      Every leetcode question has been answered a billion times and you train it on those billions of answers, it should get those right.

    • OskarAxolotl@lemmy.world
      link
      fedilink
      arrow-up
      3
      ·
      1 year ago

      Probably because the model has seen thousands of possible solutions to those exact Leetcode problems. Actual questions people ask on StackOverflow tend to be much more specialized.

    • HackerJoe@sh.itjust.works
      link
      fedilink
      arrow-up
      2
      ·
      1 year ago

      But it confidently explains the wrong answers.

      I just hope politicians don’t find out how to use it. It’ll be our doom.