We got a call from a developer stating he had lost his development machine for a new application. I asked for the name of the machine and set off to find the backup. After search returned no results I asked the developer for the system name again to confirm I had the right one. He confirmed I had that right name which left me to tell him the bad news, there was no backup of that machine. The dev team had added several new machines and never said anything. Apparently they were using them for months assuming they were backed up. Agents were of course deployed to the new systems, but this set them back quite a few days.

All of the sudden our network connections to our cloud provider disappeared. This of course caused a mad scramble to find out what happened. Someone made a change to the router config, but it was “unrelated”. Turns out it wasn’t unrelated, and combining the required change with some additional config items got everything back up and running.