In the world of quality assurance, there’s no one set process: Each company — and each QA team — has its own set of requirements that ultimately depends on the product at hand.
However, there are a few noteworthy similarities across the world of QA. For example, without a constant feedback loop between the product and engineering teams, QA can breakdown.
For Jeff Rogers, head of QA at the healthcare technology company Tempus, any changes to requirements need to be circulated amongst relevant teams.
“When requirements change, the agreed-upon success criteria is evaluated by engineering, product and QA team members, and commitments are restated and/or re-forecasted,” Rogers said.
Similarly, at security company Trail of Bits, a good unit-testing feedback cycle is essential for its QA process. According to Assurance Practice Lead Stefan Edwards, this feedback cycle minimizes rework for the QA team as well as keeps the product in the planning stages rather than the reaction stages.
At Bread, a rapidly-growing e-commerce company, documentation, in addition to communication, are key during the QA process. Both allow for each team member to have access to the data in order to understand its purpose, scope and edge cases, said Christina Kung, director of engineering.
Built In NYC recently caught up with Rogers, Edwards, Kung and two other leaders to dig deeper into their respective QA processes and how it helps them create a better product for their customers.
Technology and design company Work & Co defines and launches digital experiences that transform companies all over the world. At the company, writing test cases is one of their most important QA practices. “The act of writing test cases enables our QA team to sit down and think critically about what they’re testing, how they test each feature, and the personas and scenarios required to test the functionality fully,” QA Director Neil Duggan said.
What’s the most important best practice your QA team follows, and why?
One of our most important best practices is writing test cases. While test cases are not currently seen as “cool” in many circles, they are a fundamental part of our testing process for a few reasons. The act of writing test cases enables our QA team to sit down and think critically about what they’re testing, how they test each feature, and the personas and scenarios required to test the functionality fully. Test cases are a valuable tool for regression testing at the end of a project, and they form an essential part of our handoff to a client. At Work & Co, we build our products to live on after launch, so handing off test cases that our clients can utilize for future iterations of the product is a duty of care.
Quality can be a subjective term, so the most important thing is to make your release criteria specific and measurable.”
How do you determine your release criteria?
Quality can be a subjective term, so the most important thing is to make your release criteria specific and measurable. As part of our QA process at Work & Co, we take a two-tiered approach to platform support. Tier one platforms are fully supported — these are usually the most up-to-date browsers, newest devices and operating systems, whereas tier two is typically one OS or browser version back.
This helps us when it comes to defect prioritization. We classify defects into four priority categories: blocker, critical, major and minor. At Work & Co, we commit to releasing software with zero tracked blocker and critical defects left open, and a maximum of 10 percent of the total remaining major defects open. We work closely with our clients throughout their user accepting testing (UAT) phase to ensure their satisfaction with our product, and only when both teams are happy are we ready to release the product.
How do you ensure the QA team stays up to date on shifting requirements?
Work & Co’s model is based on our entire team sitting together and collaborating closely. Our QA team is a part of the project team and sits side by side with our strategists, designers, developers and product managers. This allows the whole team to collaborate and adjust together as requirements change. Since March, our teams have been working remotely from our offices due to COVID-19, so we’ve had to embrace technologies like Zoom, Slack and Dropbox to ensure we remain just as collaborative apart as we are when we’re physically together in our offices.
From a planning perspective, the QA team is fully integrated into our project lifecycle. Most notably, we have them begin their work simultaneously with the development team. We’ve found this increases the ability to properly account for the QA process on each project, rather than reactively responding to tickets as they are assigned.
Financial technology and insurance company Narmi helps thousands of community banks and credit unions compete with megabanks by creating a better online banking experience that’s built on the foundation of collaboration and teamwork. According to Co-Founder Chris Griffin, a tight communication loop between the product and engineering teams is key for the QA and release processes. As new requirements come in from the product side, engineers complete each one and pass it back to product for QA.
What’s the most important best practice your QA team follows, and why?
Our applications are used for everyday banking, so dependability is critical. Before a change has even been merged into our codebase, it has gone through a number of checks and balances that help prevent and catch issues well before they could negatively impact production.
Every pull request is peer-reviewed, all changes are tested, and changes go through unit, integration and acceptance tests before being merged. We use feature flags to allow us to develop in smaller chunks, and team members are encouraged to test-drive their code.
After the merge, changes are deployed to staging and customer-specific UAT environments that allow us to test the change against various configurations of our platform. Once any bugs are discovered and fixed, we deploy to production, confident that the code we’ve just released is safe for our customers to use.
How do you determine your release criteria?
As a service provider for banks and credit unions, our release process is tailored to the needs of our customers. Internally, we deploy continuously to staging, which helps to identify classes of bugs quickly and encourages a more iterative QA process.
Externally, our production release process is tailored to the needs of our customers. Some financial institutions are very forward-thinking, while others prefer the software they know and love as they proceed through their digital transformation.
We’ve taken the best of both worlds to create a release process that allows us to QA and release bug-free software to financial institutions that want the latest features while allowing more comfortable customers to change only every couple of months. This keeps both sides of our customer base happy while letting us continue to develop, merge code and QA new features for everyone.
We make sure releases are smooth by keeping a tight loop of communication between product and engineering.”
How do you ensure the QA team stays up to date on shifting requirements?
Communication is key! We make sure releases are smooth by keeping a tight loop of communication between product and engineering. As new requirements come in from the product side, engineers complete each one and pass it back to product for QA. We find that having the people who create the requirements be the same ones who confirm them really tightens up our process and makes sure that what’s requested is what gets delivered.
Tempus enables physicians to deliver personalized patient care through an interactive and analytical machine learning platform that makes data accessible and useful. At Tempus, effective QA practices require all hands on deck. “Our products are built, managed, sold and supported via an equal partnership between our engineering, product management and operations team,” Jeff Rogers, head of QA, said.
What’s the most important best practice your QA team follows, and why?
Pairing continuous delivery with an extreme focus on our foundational principles: patients (our customers), security of their data and regulatory compliance. We do this by limiting the blast radius for changes in product design and optimizing our test strategies around the various failure modes. This allows us to push the envelope on speed to market and continuous improvement.
There is no standard-quality bar for release across all systems; each group assesses each change against the potential risks to our core principles.”
How do you determine your release criteria?
Tempus products are built, managed, sold and supported via an equal partnership between our engineering, product management and operations team. Each group brings its own perspective on quality and the criteria for change and release, and testing activities are aligned and negotiated as necessary. There is no standard-quality bar for release across all systems; each group assesses each change against the potential risks to our core principles.
How do you ensure the QA team stays up to date on shifting requirements?
At Tempus, QA is embedded in product scrum teams, which are tasked with consulting and coaching on adequate success criteria for all requirements, with an eye towards risk, compliance and customer impact. When requirements change, the agreed-upon success criteria is evaluated by engineering, product and QA team members, and commitments are restated and/or re-forecasted. If and when issues arise, QA is accountable for all root cause findings and remediations.
E-commerce company Bread aims to transform the world of paper credit card applications and hidden interest rates by providing leading point-of-sale financing options for merchants. Because Bread’s QA team has grown so rapidly, communication is the key for its success. “This rapid growth means documenting our features clearly so that anyone can look at our work and understand its purpose, scope and edge cases,” Director of Engineering Christina Kung said.
What’s the most important best practice your QA team follows, and why?
This past quarter, we’ve built a new QA team from scratch, onboarding a new QA lead and five new QA engineers — and we are still hiring. This rapid growth reminds us to take communication seriously. This means documenting our features clearly so that anyone can look at our work and understand its purpose, scope and edge cases. The most important practice is for our QA to ensure that everyone is filling out acceptance criteria and test cases as thoroughly as possible, and secondly keeping them up to date as the project evolves.
How do you determine your release criteria?
We have prioritized automating as much as possible. From unit tests to smoke tests, our goal is to have early and consistent feedback for any code that is committed. For example, whenever a pull request (PR) is opened our unit and integration tests are run. When the PR is merged to master our smoke tests are run, and then rerun, that’s when it goes to staging.
Of course, our automation suite is always a work in progress and it is never 100 percent. While we continue to fill in the gaps in test coverage, wherever they may be, the entire team is encouraged to do exploratory testing on features before they go to production. When our QA lead gives the blessing, we then move to production.
The most important practice is for our QA to ensure that everyone is filling out acceptance criteria and test cases as thoroughly as possible.”
How do you ensure the QA team stays up to date on shifting requirements?
This is a nod to the earlier question on best QA practices — we are diligent in keeping our acceptance criteria up to date. The ticket details are our single source of truth for engineers, product managers and QA. Whenever these details are changed we are notified, and we can also use daily standups as an opportunity to discuss those changes.
Security company Trail of Bits helps secure some of the most targeted organizations and products around the world by combining high-end security research with a real-world attacker mentality to reduce risk and secure code. According to Stefan Edwards, assurance practice lead, having a good unit-testing feedback cycle is essential. This feedback cycle minimizes rework for the QA team, as well as keeps the product in the planning stages rather than the reaction stages.
What’s the most important best practice your QA team follows, and why?
Having a good unit-testing feedback cycle to minimize rework. For example, if QA catches something, and you generate a new test case and find a new breakage, making sure it feeds back to a unit test so QA doesn’t have to test that again is huge.
Make sure your QA testing practices are as “engaged left” as possible, meaning they focus heavily on the planning or proactive stages rather than the reaction stages.”
How do you determine your release criteria?
When clients ask how to determine these sorts of things, I’d say you need really good tasking and prioritization because you need to understand how your software development life cycle (SDLC) leads to actual work being done. Once you’re as close as possible to completing that set of goals, you’re at version 1.0.
How do you ensure the QA team stays up to date on shifting requirements, and how do you plan ahead to ensure changes are handled smoothly?
Generally, when you QA and test things, it’s extremely difficult to stay up to date with those requirements. So, make sure your QA testing practices are as “engaged left” as possible, meaning they focus heavily on the planning or proactive stages rather than the reaction stages.
