Back on Christmas Eve of last year there were some reports that Elon Musk was in the process of shutting down Twitter’s Sacramento data center. In that article, a number of ex-Twitter employees wer…

  • Maybe
    link
    fedilink
    English
    arrow-up
    13
    arrow-down
    81
    ·
    10 months ago

    I really dislike Musk, but I find it hard to criticize this when it generally worked.

    The platform formerly known as Twitter is still running, and there’s no more $100 million/year data center.

    6-9 months would have meant $50-75 million dollars. I don’t know what the outages and re-engineering ended up costing them, but that’s a ton of money.

    • stopthatgirl7@kbin.socialOP
      link
      fedilink
      arrow-up
      69
      arrow-down
      2
      ·
      10 months ago

      Looks like you’re the type the writer talks about at the end:

      There’s something to be said for pushing back on needless rules and bureaucracy, but it helps if you actually understand stuff before doing so, rather than doing something like this that had half a dozen ways it could have ended in serious disaster and possible tragedy. The fact that it “only” resulted in Twitter falling over every few weeks for months likely means that Musk and his supporters got the very wrong lesson out of this.

      • Maybe
        link
        fedilink
        English
        arrow-up
        5
        arrow-down
        25
        ·
        10 months ago

        What risks, exactly? Twitter goes down? Proprietary Twitter data gets stolen in some server heist scenario?

        • Womble@lemmy.world
          link
          fedilink
          English
          arrow-up
          23
          ·
          10 months ago

          Millions of people’s personal data gets leaked, Musk’s cowboy “pry open the floor and electrical panels with a knife” electrocutes him, or blows the power for the room/floor/building or starts a fire.

          • Nepenthe@kbin.social
            link
            fedilink
            arrow-up
            8
            ·
            10 months ago

            Musk’s cowboy “pry open the floor and electrical panels with a knife” electrocutes him

            That one is a risk I’m willing to take. I had to stop reading the article for a moment to marvel at just how close we really were.

          • Maybe
            link
            fedilink
            English
            arrow-up
            1
            ·
            10 months ago

            Isn’t all of it encrypted though? Like I understand physical access to servers is generally bad, but you’d think once the the things are unplugged it would be difficult to access the data again without bypassing encryption. I’m clearly not a software engineer though lol

        • chaogomu@kbin.social
          link
          fedilink
          arrow-up
          17
          ·
          10 months ago

          The servers were not actually secured in the truck properly, so another scenario would have been the damage and destruction of some or all of them.

          Plus, yes, theft. And it’s not just proprietary data, it was also personal and financial data for users and advertisers.

          • seang96@spgrn.com
            link
            fedilink
            English
            arrow-up
            7
            ·
            10 months ago

            I imagine thousands of pounds of unsecured load would be potentially dangerous for the driver and all other drivers on the road too.

    • exussum@lemmy.world
      link
      fedilink
      English
      arrow-up
      41
      arrow-down
      1
      ·
      10 months ago

      The problem is risk. A lot of the bureaucracy that exists for any company is risk mitigation. The wiping of servers, or using suction cups, or any of that is a security against a large dollar amount to spend if something goes wrong. But that’s just the cost of security, it’s worthless if it isn’t tested. If a locked door isn’t rattled or deter someone, it might as well have been unlocked.

      He took a gamble and the doors were not rattled and everything worked. The thing to criticize here is really the carelessness. What if one of those servers got out and somebody stole all of that data? What if while under those floorboards he got damaged, or something related did? And it’s not just these two questions, there’s stuff in that article that probably wasn’t covered that we can question.

      There may be things that are not in the book that we can question, and that is the problem with Elon. He needs a string of bad luck to show how truly dangerous he is.

    • Everblue@lemmy.ca
      link
      fedilink
      English
      arrow-up
      36
      ·
      10 months ago

      That’s like saying why wear PPE and follow safety protocols on a construction site, it’ll save us money if we don’t do any of that. Nobody died or got hurt? Perfect.

      There’s a reason things need to be done a certain way, if something had gone wrong what would’ve been the consequences? What if all those data racks full of personal user information were just straight up stolen by the random movers they hired off the street?? What if the floor had collapsed under the weight of the servers being moved, tipped the server over and crushed someone? Just because things worked out relatively fine doesn’t mean no harm no foul.

      Musk is an idiot. Deciding to do things his way to save money and time reflects poorly on all his other companies.

      • fruitleatherpostcard
        link
        fedilink
        English
        arrow-up
        19
        ·
        10 months ago

        Are the severs running SSDs or hard disks?

        I remember the story of the guy moving a shopping cart full of his company’s HDs across the street or something and destroying them all just rattling the fucking cart across the shitty surface.

    • edric
      link
      fedilink
      English
      arrow-up
      23
      ·
      edit-2
      10 months ago

      No, they were just extremely lucky that nothing worse than twitter going down happened. There’s a reason protocols exist for data center moves. The infrastructure manager told Elon that the destination DC in Oregon had different rack and power setups and you just can’t plug and play a server you pulled out of Sacramento. Elon also went under the floorboards and disconnected the power cables and seismic detectors which could’ve caused electrocution, fire, earthquake false alarms, or compromising the detection system itself. Then they were moving equipment that weighed more than what the floor was rated for, which could’ve caused cave-ins or compromising the structural integrity of the floor. Not to mention the possible damage to the equipment by moving them the same way you’d move a couch. They also hired some random cheap moving company they found on yelp to move the servers because they charged 90% less than the existing contractor. No contracts and paid in cash.

      Tons of things could’ve gone wrong. Just because downtime was the worst that happened, doesn’t mean it’s ok to do. It is also those same data center protocols that help prevent idiots like Musk from causing catastrophic issues when they pull off stunts like this.

    • SatanicNotMessianic@lemmy.ml
      link
      fedilink
      English
      arrow-up
      16
      ·
      10 months ago

      First of all, Musk burdened twitter with a level of debt that cost (last estimate that I saw) $1B/year to service. This data center would not have been a problem if he had actually been a good businessman and, you know, didn’t massively overpay with a debt-funded takeover while waiving due diligence on a company he didn’t want in a market he completely doesn’t understand. He set fire to $44B. Twitter’s current valuation has been estimated to be as low as $4B. I personally think that’s low, but the May estimate was $15B (which didn’t include the loss of branding hit).

      So his recklessness and complete lack of understanding combined with his overconfidence and incompetence made the $100M savings into peanuts compared to what he destroyed by pulling exactly the same kind of move throughout the business.

      Now combine that with the very probable fact that this saved no where near $100M. Shitty shifting of servers breaks hardware. They weren’t prepped to receive them at the destination. They ended up with major drops in service, including Elmo having to shut twitter down for a weekend because they couldn’t handle the traffic. Now he’s whining about “scraping” and trying to squeeze blood from a stone in the face of advertisers abandoning him.

      This in no way generally worked. Things are absolutely falling apart around their ears. I’ve stopped even trying to follow twitter links because they work less than half the time since I don’t have an account.

      Elon is Dilbert’s pointy-haired boss with a lot of money and a PR firm.

      • eestileib@sh.itjust.works
        link
        fedilink
        English
        arrow-up
        4
        ·
        10 months ago

        Dilbert’s boss is lazy, Musk is full of passionate intensity.

        My first gig out of college in the Valley was working with (later for) a relatively charismatic “I know better” untreated bipolar guy. This dude actually had the chops, he was actually smarter than you. His demos and product ideas were amazing, legit visionary. Inspirational.

        But gods it was soul-destroying to try to work for the guy, he kept pulling exactly this kind of “it’s not that complicated!” stunt, changing plans on a whim, editing history to make himself consistent, hair-trigger switching between praise and abuse…

        It got a lot of good work out of me, I learned a lot, I was well-compensated, but I now that I know the signs I’m never working for a person like that again.

        (see also: the subject of another fawning Isaacson hagiography, Steve Jobs)