Put That Server Down, Flip It, and Reverse It

Reverse Engineering a Hosting Setup

I have landed in this spot dozens of times in my career. The spot where I end up responsible for supporting or untangling something I didn’t build. Something that has very minimal or zero documentation and where I don’t have access to previous development resources. But here I am, with admin access or root credentials, being told to figure it out. I need to find a scent. I need to put my nose to the ground and see where the people that built this thing used to frequent.

Usually, I like to start with generating a list of questions for the client/party giving me access. They might not think they know anything. But if they can give you an idea of common issues, architecture choices, it’s something and it could potentially save you a whole lot of time. If the first time you ask a slew of questions and they have no answers, rephrase them and try again. People always know more than they think. You just need to ask them in a way that helps them remember.

First things first.

Generating those questions for the client. Ask yourself, and write down every question you could possibly have. Here are some examples to help you get started:

• What is the code base?
• Is the code base in source control? (Boy, this is a fun one sometimes).
  • Client: "What's source control?"
  • Me: "......."
• What kind of server?
• Can you provide any documentation or back and forth conversations regarding how things were built? Jira tasks, basic architecture diagrams, deployment steps, how-to documents, anything?
  • If no, or they provide you with nothing of value – you need to set realistic expectations. You can’t make any promises. “This is going to be a strategic mining operation; we will need to do this one chunk at a time.”
• How many people use this thing?
• Is it ok if its broken?
• Does this site/application/program ever have any issues? Regular maintenance?
  • Maybe a particular piece often needs massaged.
    • Database gets full
    • Logs get full
    • Run out of disk space
    • Slow annoying memory leaks that become problems after a few months?
  • If it goes down. How long can it comfortably be down? Might be a bad question, but it gives you an idea of how serious and careful your investigating needs to be, or just what supporting it will be like. Possible answers:
    • People will die!
    • We schedule down time often, but it needs to be available during all business hours.
    • Not a big deal. We have one client and they only use it on Mondays. And if it was broken on Monday. Just give us the heads up and it wouldn't be a big deal.
    • Literally no one uses this. We have lost all hope. Back it up, and break it.
  • Do you have multiple environments?
  • What’s your pipeline situation like?
  • Establish with owner of the thing you are working on: What is it that this thing does?
  • Can I have access to original people that created this?
    • If no, more expectations should be set. This may take a long time if there are a lot of moving parts.

Digging in and creating documentation

Start digging in and get a handle on everything. Try to answer as many of the questions you initially generated as you are diving into the System or architectural diagrams (Lucid Charts, Vizio, dry erase board). No matter how simple they seem, they are useful. Map out the pieces you know and how they connect to each other. Every time you find a new component you didn’t know about, throw it on the diagram with a placeholder so that it’s on the shelf in your brain until you discover its role (if it actually has one).

If you are stuck at a certain point and can’t figure out how something works or how it’s generated – overwhelm yourself with as much information as possible. Then let it marinate. Step away. Sleep. Overwhelm yourself again. Freak out a bit (how do I end up in these situations? ... If I figure this out completely - I am truly awesome ... Must. Keep. Hope). At a certain point you will have either reached your goal and have complete ownership, understanding, and control of the systems or maybe you will have determined there is something that cannot be figured out because the original source control was lost and can’t be reverse engineered from DLLs. Whatever the case, document all the questions you generated and their answers, your diagrams of the different moving parts, and all your notes from your struggles. Package them up and make sure they live on so that the next lucky person doesn’t has to suffer. And has a solid starting point for picking up where you left off.