For software engineers, making one small mistake while deploying code into production can lead to disastrous consequences, such as data loss or outages. That’s why many teams put careful processes in place to avoid issues before they occur.
For Canoe Senior Application Developer Casey Flynn, two key ingredients make up successful code deployment: automation and confidence. Leveraging an automated deployment pipeline in particular ensures that all necessary actions are taken without the need for human intervention.
“Automation ensures that any issues encountered during deployment are likely due to an actual flaw in the release rather than any step we forgot to take,” Flynn said.
While preparation and strong strategy are essential to efficient deployment, it’s also important to retain a good sense of humor throughout the process. This is something Enigma Software Engineer Sandra Shaefer knows well. And she has found that the “gods of deployment” are partial to songs by Queen.
“Everyone puts on their fanciest outfit and burns incense in honor of the code being deprecated while singing ‘Another One Bites the Dust’ before finishing with a rousing rendition of ‘Don’t Stop Me Now,’ ‘Under Pressure,’ or ‘We Will Rock You,’ depending on what best fits the moment,” Shaefer joked. “I’m just kidding. We cross our fingers and hope for the best.”
Built In NYC caught up with Flynn and Shaefer to garner their insightful — and humorous — perspectives on successfully deploying code into production.
Canoe’s AI-powered technology automates document and data workflows related to alternative investments, such as private equity and managed futures.
What are the key ingredients that go into a successful deployment into production?
Automation is the first ingredient. We rely heavily on an automated deployment pipeline to ensure we execute all the steps necessary each time, without human intervention. With a single click, the application is built, database migrations are executed, unit, integration and end-to-end tests are run, and the code is deployed to all of our servers.
Confidence is the second ingredient. We rely upon a stringent and well-defined development process for all changes we release. We accompany all development work by a set of requirements, test plans and an automated unit/integration test suite to confirm behavior at all levels. Before work reaches our mainline branch, our QA team and product stakeholders perform thorough testing, and it goes through a peer review process. All features slated for release are also tested in tandem within our staging environment. The long pipeline of feature confirmation ensures that we are releasing stable, quality work.
What are the most common mistakes engineers make when deploying code?
One of the most common mistakes is failing to automate everything that can be automated. We have a strict and automated process in place to reduce the possibility of human error. Releasing code to a production environment can be stressful, especially if manual processes are involved, considering one small mistake can have drastic consequences, such as an outage or loss of data. We operate at a scale that cannot afford these types of issues, so we have eliminated as many of them as possible through a deployment automation process.
Sometimes, engineers fail to release often enough. We have a complex and feature-rich application that is the by-product of years of contribution from countless developers. As we’ve grown, we’ve learned that the longer we delay the release of changes to production, the more likely we are to have issues in the end. Divergent changes, competing requirements and merge conflicts will keep a release from being on time. Our goal is to release to production every week. Over time, we’ve learned to break up features in smaller and incremental changes that slowly build up to the final feature. Feature flags enable us to turn off unfinished work in releases until it’s done.
We’ve learned that the longer we delay the release of changes to production, the more likely we are to have issues in the end.”
Does your team practice any rituals before or after deploying code?
We have a “staging testing” group chat where our QA and DevOps teams keep us posted about release testing and preparation progress. Usually, once QA has signed off on the release, our DevOps team makes a final pull request to confirm release sign-off from developers. Once the pull request is approved, DevOps will announce that they are merging the release in the chat. In a joking sense, this message is often met with reactions of catastrophic failure GIFs or other ironic humor to make light of the seriousness of shipping code to production. I’m glad that the team’s confidence in our release process allows us to do this. We rarely encounter issues following the release, which is mostly due to the automation and confidence we have in place.
Enigma’s infrastructure, APIs and tools are designed to help organizations access and integrate data needed to understand and serve small businesses.
What are the key ingredients that go into a successful deployment into production?
A deployment’s success is often related to how seamlessly changes are integrated into existing functionality. Ensuring current systems aren’t broken as they’re updated or replaced is key to the system’s smooth operation. This is why comprehensive planning is essential. My team drafts a product requirements document and a corresponding engineering design specification. Stakeholder teams review and discuss each document to refine the design and address any foreseeable issues. Then, the eng spec becomes a series of tickets in an epic on our Jira board. For large projects, we split the epic into phases and track each with a plan. This planning and documentation helps make project implementation and deployment seamless.
The other key element is thorough unit and integration testing, as well as larger scale testing. Intelligent testing takes time and careful thought, and one of the challenges of software engineering is achieving a balance between speed and circumspection. There’s usually a point in the testing journey where deploying and handling any issues that arise post-deployment saves time and resources rather than continue exhaustively testing.
By including testing in the planning phase, engineers can help catch design flaws.”
What are the most common mistakes engineers make when deploying code?
One of the most common mistakes is not including incremental phases with a large change. If you try to jump up a mountain, you will fail. You have to break the jump down into smaller steps and slowly make your way up, ensuring you’re on stable ground the whole way. Another common mistake is to not spend enough time planning testing. Leaving testing as an afterthought leads to poor coverage and increases the number of bugs. By including testing in the planning phase, engineers can help catch design flaws.
Many companies find it beneficial to have separate roles for test and QA engineers. Yet at Enigma, every engineer is responsible for testing their own code, which can often take the majority of the time allotted for a change. Because it’s such a time drain, it’s tempting to cut corners with testing. To mitigate this, our team has put ample time into making our systems as easy to test as possible.
