What’s the ‘freakiest’ event or situation you’ve experienced in production?
My first job at a university working in the data centre. Came across one of our water cooled super computers spewing water all over the cluster. It’s amazing how frequently water sneaks into data centers.
Ouch… that must’ve been expensive to fix
Having to literally build my own network to work off of because the organization’s network is locked down to un-usability for anything. I also now hate T-mobile’s CGNAT with a passion that I never thought I could feel over a networking architecture decision.
Look into VPN-based network overlays if you’re stuck behind a DNS or NAT based protective network.
It’s why I always have my hotspot on me.
I was looking at getting a wireguard up with policy based routing, but apparently they have something in there which messes that up too, leading a bunch of folks to try using openVPN in https mode. At some point you just have to stop jumping through hoops and ask why you’re giving business to an organization which simply doesn’t deliver the bare minimum for no discernable good reason.
Also the tower reception is flapping all over the place from the hotspot in my location, so I’m a very unhappy bunny.
I hear ya. I had a VeloCloud engineer told me they don’t support wildcard domain names. smacks head
Not quite freaky but we had a good laugh:
Your small shop has a client that works closely with a local government research institute, to the point of having an office suite on their campus which has its own network with all that that entails.
It is Thursday morning, and your are informed by the research institute that they have decided to give you responsibility of a bay that is supposedly shared between both teams, but really they send techs and you ask for changes and they never use it, so it just simplifies things for everyone for you to be responsible of that one specific bay. You decide to send a tech the next day to take a look at it, inventory what goes where and such.
It is the next day, so a Friday, and the tech goes out. As usual, he is stopped multiple times by people on the way to the bay, and is delayed to the point that he arrives at the bay at about 5:10 PM. No matter, he’ll do a bit of overtime and if it’s too much he’ll come back next week. The tech opens the door and finds this:
https://sh.itjust.works/pictrs/image/f70b5935-98d3-4a5a-a1dd-fd5b6b27cb38.jpeg
He sends the picture to us and we all have a good laugh. He spends bout half of the next week untangling and inventorying this mess.
Looks like 1994 in that shit!
Working at a college in Oregon, we had a significant amount of snow/rain one winter and spring and facilities reported to IT that one of the dorm’s basement was flooding. We had equipment there that was my responsibility, so… sure, send the 20-something network guy down to a flooding basement.
Luckily, the drains were working, so not much water standing, more like flowing - from the service tunnel into the ‘room’ on which the network equipment was housed. There was more than one kind of flow happening on that modular HP ProCurve 4000 switch! Soaked. Blinky lights, just humming along like normal.
RIP HP ProCurves - you were beasts.
Once one of the backbone Zyxels stopped working, so we went to the floor on which it was located. We look in the rack, as expected no blinking lights.
We didn’t have a spare and I’m into electronics, so in order to keep things going, I decided to take a look at it. Open it up… jeus fucking christ 😲… a swarm of roaches rushes in all directions! I just rushed out and told the cleaning ladies to get a vacuum cleaner in there ASAP. They did and got most of them, but eventually we did have a minor roach infestation in the office and had to disinsect a few times.
After the storm was over, I took a look at it… apparently, they laid eggs underneath the PSU and when the eggs started hatching, they short circuited it. The PSU was fried. Luckily, the output voltage of the PSU was marked on the PCB - 12V. So, I just modded an ATX to power the switch. Still works like that to this day, lol 😂.
Holy shit. What a roller coaster. Would read again! 🤓
I wish I can unsee that swarm of roaches, haunts me to this day 😂.
