Hi everyone,

ever since I switched to Arch about two months ago, most applications segfault multiple times a day. There doesn’t seem to be any pattern for the crashes, sometimes it’s even happening while idling (e.g. reading a news article).

Things I’ve tried without any luck so far:

  • Running Firefox in safe-mode without any extensions
  • Switching from regular to LTS kernel
  • Disable Hardware Acceleration in Firefox
  • Change RAM speed and timings
  • Run Memtest successfully
  • Replace entire RAM with a new certified kit
  • Use only a single RAM slot
  • Apply Ryzen fixes (iommu=soft, limit c-states)
  • Use only a single CPU core (maxcpus=1)
  • Downgrade Nvidia driver to 535xx
  • Use Nouveau instead of the nvidia driver
  • Use Openbox instead of KDE
  • Disable zswap and THP

Here’s full journalctl from a day where both Spotify and Firefox crashed at the end, a few seconds after each other:

https://pastebin.com/BH0LMnD9

Some more info about my system:

  • Ryzen 5 3600X
  • MSI B450M PRO-VDH Max
  • 32GB RAM @ 3200MHz
  • Geforce RTX 2070 SUPER (using nvidia-dkms)
  • Plasma 5.27.10 on X11

I’m pretty sure that it’s not hardware related, because I’ve booted up a Debian 12 live image where everything ran for several hours without a crash. But it seems to be Arch related, as I also booted up a fresh EndeavourOS live image (so basically Arch), where applications also randomly segfaulted. Any idea why everything works fine on Debian but not on Arch? Debian uses the 6.1 kernel, which I already tried, so that’s not it.

Let me know if you need any more information that might help solve this issue. Thanks!

Edit [solved]: It looks like disabling PBO in the UEFI/BIOS did the trick. The strange thing is, after enabling it again, it’s still not crashing again. Someone suspected that the MoBo default/training settings were faulty, so I guess this was a very rare case here. That’s probably why it took so long to find a solution. Thanks everyone for helping me out!

    • NoisyFlakeOP
      5·
      6 months ago

      Hm, I’ve had this problem since my initial setup about 2-3 months ago, I think that if there’s something wrong with the software in the repos, it would’ve been fixed by now and I wouldn’t be the only one having this problem, right?

      But of course, if you want I can give the testing repos a try :)

  • gbin@lemmy.ca
    10·
    6 months ago

    The crashes are in the middle of browsers (both Firefox and chrome embedded in Spotify), if you try a simple mprime stress test (from the AUR mprime-bin) does it crash too?

    • cbarrick@lemmy.worldEnglish
      5·
      6 months ago

      Yeah, this sounds somewhat like unstable hardware.

      Definitely start with a stress test or memory test.

  • Michael Murphy (S76)@lemmy.worldEnglish
    9·
    6 months ago

    Make sure you have the latest firmware for your motherboard. This sounds like unstable voltages for memory, or an overly-aggressive PBO curve. Did you try disabling the XMP profile on the RAM, disabling PBO, and upping the voltages (within safe limits) of the SOC, DDR, and VDDP? You might find some useful info here[0] or here[1] if you intend to run your memory at 3200 MHz.

    • NoisyFlakeOP
      2·
      6 months ago

      Motherboard firmware is up-to-date, and I’ve already tried disabling XMP. I’ll give disabling PBO a try, thanks!

      I don’t necessarily have to run at 3200MHz, if it means that the system is finally stable. But since it’s already crashing at the default 2133MHz, I suppose there’s no use in playing with the voltages?

      • Michael Murphy (S76)@lemmy.worldEnglish
        2·
        6 months ago

        It’s difficult to say for sure with certainty what the issue is without trial and error. I would expect that the motherboard’s manufacturer would make sure that their board can successfully pass all tests with the standard JEDEC spec for DDR4 (2133 MHz).

        Since you say that you’ve tried different RAM kits, another alternative could be the cleanliness of power from the power supply. Perhaps there is intermittent voltage droop, and you need to experiment with the Load Line Calibration settings to adjust for vdroop between idle and load. Disabling frequency boosting and manually setting the CPU frequency could help check if it’s related to that. PBO curves might be undervolting too much while idle.

        • NoisyFlakeOP
          3·
          6 months ago

          I’m a bit speechless right now. I’ve disabled PBO and didn’t have a single crash since then, everything’s been running fine for hours. Just to make sure that this really was the issue, I’ve enabled PBO again - but still haven’t experienced any crashes in the last hours. I have no idea how simply disabling and then enabling the feature again fixed my issue, but for now it seems like all is well.

          Do you have any explanation for this weird behavior?

          Anyway, thank you very much for your suggestion, looks like this actually did the trick!

          • Michael Murphy (S76)@lemmy.worldEnglish
            3·
            6 months ago

            Sounds like voltage droop and/or a motherboard with faulty automatic “training” settings. I don’t recall if the Ryzen 3000 had custom PBO curves, but tweaking this can fix it. Upping LLC and the SOC and CPU voltage slightly alternatively could help. Though I’ve had my most stable overclock by disabling PBO entirely and using a manual CPU multiplier.

      • Tempy@lemmy.temporus.me
        1·
        5 months ago

        Try running a memtest, if it’s not voltages it could be a faulty ram stick. I’ve had it where data gets written, but what is read is garbage, corrupted some pretty important files on my system when I ran an update and it used that faulty section for it’s buffer.

  • Avid Amoeba@lemmy.ca
    7·
    6 months ago

    Could be a defective library that’s used by many apps. Glibc, etc. That said, if something like this is that broken, others should be complaining about it too.

    • gbin@lemmy.ca
      6·
      6 months ago

      One crash was in libxul and the other in libcef I doubt this is a specific lib

      • Avid Amoeba@lemmy.ca
        1·
        6 months ago

        This is a good idea, but they probably need to figure out which lib is shitting the bed first. There’s too many libs to try otherwise.

      • DefederateLemmyMl@feddit.nlEnglish
        1·
        6 months ago

        Maybe a corrupt download/copy of a library… Try a reinstall of say glibc ?

        Doesn’t explain why it also crashes in an EndeavourOS live image…

  • lemming741@lemmy.world
    6·
    6 months ago

    I had a 3700x that was doing that sort of thing. It seemed mostly random, but moving big files would crash it pretty often. It ran memtest86 for 3 days no problem. I replaced part by part, and it ended up being the CPU. I’d bought it second hand so it may have been abused.

    • NoisyFlakeOP
      1·
      6 months ago

      But if it’s a faulty CPU, wouldn’t it also crash on Debian?

      • Avid Amoeba@lemmy.ca
        6·
        6 months ago

        Wild guess, there could be differences in compilation optimization that expose this hypothetical proc defect on Arch but not on Debian. Try a day or two of mprime as some others suggested.

        • DefederateLemmyMl@feddit.nlEnglish
          3·
          6 months ago

          Try a day or two of mprime as some others suggested.

          That wouldn’t necessarily reveal a faulty CPU or firmware. I used to have a 3600x that would sometimes crash on idle at low clocks but would run cinebench or geekbench all day and all night.

      • lemming741@lemmy.world
        1·
        6 months ago

        I would think so, but it sounds similar enough with the symptoms and very similar on the model CPU so I thought I’d mention it

  • DefederateLemmyMl@feddit.nlEnglish
    4·
    6 months ago

    I’m pretty sure that it’s not hardware related

    Random segfaulting is not something that “just happens” because of an OS misconfiguration, then if the same problem happens on Arch as well as on a clean EndeavourOS live image it convinces me that it is in fact hardware related somehow. As you have already replaced the RAM, my guess is CPU or motherboard issue.

    Zen2/B450 is a widely used and well supported configuration on Linux that you normally shouldn’t have issues with, but Zen2 CPUs are rather notorious for having fragile memory controllers, and sometimes dodgy AGESA firmware releases that can cause issues on some CPUs. I used to have a 3600X myself that started crashing at idle around a particular firmware release of my motherboard, and it was fixed by a subsequent release.

    BTW the fact that it doesn’t happen on Debian doesn’t necessarily mean that Arch is the culprit. It could just be that Debian is not triggering the fault because of different, perhaps more conservative, compiler optimizations.

    As a last ditch effort, you could try resetting your entire UEFI (bios) settings to default, preferably by pulling the CMOS battery.

    BTW, is it only GUI applications that are segfaulting? Or other programs as well? Do you have an old spare GPU you can test with?

    • NoisyFlakeOP
      2·
      6 months ago

      I already did a UEFI reset, that didn’t help. As far as I can tell, it’s only GUI applications, I haven’t seen a segfault for something else so far. Unfortunately I don’t have any other GPU right now.

      It seems that a solution was found though (at least for now, it didn’t crash since a few hours) here: https://lemm.ee/comment/8161085

      • DefederateLemmyMl@feddit.nlEnglish
        3·
        6 months ago

        Glad to hear that disabling PBO helped, but it does indicate that something may not be entirely healthy with your CPU (or with the way the motherboard is driving it, that also can’t be excluded)

  • vzq@lemmy.blahaj.zone
    4·
    6 months ago

    Can you enable core dumps and get stack traces? From there you should be able to figure out which shared library is broken.

    • NoisyFlakeOP
      2·
      6 months ago

      Uhm, isn’t that what can be found at the end of the journalctl log I posted? Or are you talking about something different?

  • Ludrol@szmer.info
    2·
    6 months ago

    I would guess that this is CPU SSD issue you ran an live debian image from an usb and did not encounter any crashes.

    • NoisyFlakeOP
      2·
      6 months ago

      But I also ran a live EndeavourOS from USB and the same crashes happened.

  • Possibly linux@lemmy.zip
    English
    1·
    6 months ago

    Do you have a old computer to test the software? You could pull out the drive and put in a known good system to test. I also would use a live system for a while to see if it has problems.

    This sounds like a hardware problem. Maybe your ram controller on the CPU is faulty? Try reseating the CPU and check for bent pins.

    If its a software problem you could also go the nuclear option and start over. Its a pain but it might be worth it. I don’t know on Arch but with Debian you can reinstall all packages. Doing that should wipe out any corruption.

    • NoisyFlakeOP
      1·
      6 months ago

      Starting over won’t probably fix anything, since even the EndeavourOS live image has the segfaults. Of course I could just start over on Debian, but I really like Arch and would only switch as a last resort.

      I have another computer where I can test it, yes. It’s probably enough to run EndeavourOS live for a while, but then again, I’m 99% sure that no crashes are going to happen, otherwise the EndeavourOS forums would be flooded with this issue.

          • Possibly linux@lemmy.zip
            English
            2·
            6 months ago

            There is one line that catches my attention.

            ccp 0000:2b:00.1: ccp: unable to access the device: you might be running a broken BIOS

            This theoretically shouldn’t causes crashes but from my research it looks like AMD CCP can cause system instability in some cases. I would update your bios to the latest release and if that doesn’t fix it you should try disabling AMD CCP in bios as I doubt you need it anyway.

  • CameronDev@programming.dev
    1·
    6 months ago

    Try increasing RAM voltage? Might make it more stable under load. I had a similar issue, clean memtest, but games would randomly crash. Increasing RAM voltage fixed it.