Building a Quality Strategy and Executing on it
As a Engineering Manager you may often hear Scrum teams complaining that ‘Testing’ is taking too long, or too slow, QA is the bottleneck and thus slowing down Speed 2 Market. So what do you do… you get more testers to alleviate the bottleneck. This may sound like a plausible solution. However, before you decide to hire a whole bunch of QAs (or invest in Test Automation) you should probably look at a couple of areas of your Quality strategy before you do so. And if you do head down the Test automation path how to execute effectively.
In this article we will explore:
- Are you expecting too much from the QAs? Do you have the right Quality Culture?
- Have you exhausted all areas of aspects of Quality before investing in more resources or test automation?
- How to implement a Test Automation Strategy effectively with the right metrics and stakeholder buy-in
A couple assumptions in this article:
- Running Scrum, Lean or Kanban (or any combination of those methodologies)
- You Understand basics of Quality and Testing
Let’s get started with …
Quality is everyone’s Responsibility
Absolutely everyone should know by now that ‘Quality is everyone’s responsibility’. However, I have rarely seen this in action. I still see Engineers blaming QAs when there is a defect, Operations Blaming QAs when there’s a Production issue, QAs blaming BAs when acceptance criteria is incorrect, so forth and so forth. Creating a culture of Quality is hard and needs dedicated effort to up-skill everyone in Quality.
We need to move away from QA as Gatekeepers and move towards a wholistic approach to Quality embedded at every part of the Product & Sprint lifecycle. This will be key to unlocking Speed to Market and a Quality Product. Let’s look at a couple of areas of Quality you should be assessing before even looking at stuff like Test Automation:
Risk Based Testing
Are your QAs testing EVERYTHING with EVERY Story and Release? Depending on the size of the application testing an app end to end can be a significant task. You may benefit from Risk Based testing. This involves testing only those areas of the application that have significant risk based upon the code changes made. This can reduce Testing time significantly but does require the right process and tools to ensure it’s done right:
- Assessment of Business Critical Path
Ensure that the critical business features are tested always as part of each release. - Impact Analysis
Identify the right tools to help QAs identify impact areas of code changes. This can be a simple matrix of Features and dependencies between them. This will help you identify which features could impact others.
There are also Static Code analysers such as those in Visual Studio Code that can help generate a call graph at which you can identify areas of code that would be impacted.
Once you have the above you can focus the testing efforts in High impact areas of the application.
The ‘T’ skills
Hopefully the ‘T Skills’ is not new to you. If it is I will direct you towards: Why T-Shaped People. I have found the T Skills to be quite controversial, particularly with Engineers who have become so accustomed to some else testing their code. The premise is that there are experts in the team but capable of doing the skills of other functions. Engineers, however, believe that someone with a fresh pair of eyes will test better than someone who consciously knows all the impacts of their code (and probably tests the happy path). The latter may be true. But there’s also the argument that Engineer’s subconsciously hand-off quality because they know someone else will test. I have worked in organisations with and without QAs. And I can say there’s pros and cons to both. However, I won’t delve into it here. The point is Engineers help QA when it is required. Some of the most successful teams I have seen is when individuals helped other functions when needed, particularly at the end of a Sprint when it comes down to the crunch.
Pair Testing
A great way to start the process of building compassion between Testers and Engineers is Pair Testing. Just like Pair Programming — it involves 2 individuals, in this case a tester and engineer, sitting next to each other while testing a feature. A good tester will show the engineer all the edge cases or crazy tests they will be punching at the new feature.
The problem I have found with Pair testing is that Business often think it’s a waste of Engineers’ time. However, in the long run the Quality coaching will pay off by dividends by minimising the number of hand-offs between Development and QA. An evolution of this is …
Quality Assistance
If you haven’t heard of Quality Assistance already it’s about embedding Quality at every part of the lifecycle. Scaling Quality Assurance is difficult with limited resources. But through the right Quality coaching everyone can become a tester (most importantly the engineers). However, this is a significant shift and investment in time. It starts with Pair Testing, as above, but then evolves to a point where Engineers are accountable for testing their own code. The QA team takes the responsibility of coaching everyone from BAs, Engineers, and POS in looking out for quality in their respective functions.
Over time, as all the functions become skilled in Quality, the QAs can focus on testing edge cases and exploratory testing. And this is truly where I think QA can add value.
Bug Bash
Bug Bashes are a great way for the entire org (not just your team) to have a go at testing. I found it an opportune time for Engineers, BAs, Stakeholders and others to have a go at Quality. This improves the empathy and connection of quality across all the different functions of the business.
In essence a bug bash is where you allocate a set amount of time for EVERYONE to test a particular feature or product. Those who have decided to get involved input the defects into a Bug Tracker or a Spreadsheet for those who don’t have access to the bug tracker of choice.
The QA team review all the defects after the allotted time to assess whether defects are known. You can even game-ify the process by giving a prize to the person who identifies the most defects.
Automate, Automate, Automate … then do some Manual testing
Let’s say you have covered all the areas above around Quality as a Culture it’s time to move on to the fancier aspect of QA which is Test Automation.
Any repeatable test should be automated and added to the regression test suite. Not to say there is no room for Manual testing. Manual testing will always play a part in edge case testing, exploratory testing and validating customer experiences. If you are at this level of maturity, you can reap the benefits of a great Quality Assurance team as I believe this is where a Quality team can really add value.
So let’s look at a few methods of test automation below so we can get your Engineering org to a level of maturity that gives the QA team capacity to focus on the stuff that matters.
What tests do you need to Automate?
What sort of tests you need to automate will depend heavily on the application in question. If you don’t know the test pyramid check out this Article on Martin Fowler blog, or this on Re-thinking the test pyramid. Below I have identified areas you may need to consider:
- Unit tests are a must
TDD is not a must. But Unit tests most definitely are. Unit tests are the first point of call to ensure that Engineers are accountable to the Quality of their code. And ensure integrity is maintained as changes happen over time. - Integration tests over UI tests
As Engineering has become an art of connecting 2 services or systems , development has increasingly become a work of integration. Just take some of the applications you work on today and identify how many dependencies they have on external or third party systems. I bet there’s many, whether it’s SDKs, Frameworks, External Services/APIs, or Cloud hosting. - UI Tests for Complex Consumer Applications
Automate UI tests for applications with heavy UI experience. These days you can automate visual regression testing to pick up changes in design and UI. - Cross Browser/Device Testing
If the app is used across multiple devices consider automating tests across devices using tools such as BrowserStack or Saucelabs driven by Selenium tests. - Security and Performance Testing for Highly Trafficked Applications
It can be impossible to test all permutations of Performance and Security testing. Look at Static Code Analysers for your respective language (SonarQube as an example for C# projects) to help you identify security holes early. As for Performance testing, anything that builds upon the Jmeter suite is a good start.
Behaviour Driven Development
Let’s take Unit tests and Integration tests to the next level by connecting them with Acceptance criteria. One of the things I love about BDD is that you can directly connect value of your code to the business and most importantly the customer. There are Frameworks and tools are out there that accommodate for almost all the popular languages. Such tools include Cucumber, Gherkin and Specflow.
All (in combination with other tools) can scaffold out Tests for developers to fulfil. Please see an example workflow below for BDD:
Consider Page Object Pattern
If you are working on a Complex UI you should seriously consider the Page Object Pattern for structuring your Test Automation code. The Page Object Pattern enables to reuse UI components across a variety of tests. This means encapsulating the Application UI component in a class that is reused across multiple test scenarios. It also means if the UI happen to change you only have to modify one area.
https://martinfowler.com/bliki/PageObject.html
Treat your Test Automation code like you would your Application Code!
- This includes Code Reviews, Code Review Checklists, Linting, Security analysis, Continuous Delivery, Merging and Branching patterns… Everything you would do for commercial application code should be the same for your Test automation.
Choose the right Tools
Experience tells me that QAs and Engineers will push the tools and frameworks they are most familiar with. There will be times you may have to succumb to the tools purported by the team because you know going any other way will have significant impact on team health. However, where possible take an objective approach to assessing tools to build out your Test Automation Toolbox.
Because there are so many frameworks and tools in the industry at the moment I recommend an RFC (Request for Comment) and POV (Proof of Value) approach to assessing new frameworks and tools. It is a great way to get an objective view but also involving the entire team as part of the process.
Identify a person (or 2) in your team who can do an RFC (Request for Comment) to recommend a Test automation framework and language. Ask your entire team for a set of tools to be assessed so you don’t hear surprises from the team later as to why you haven’t analysed their framework of choice. The RFC should come up with a recommendation based upon a set of criteria most important to the Software Engineering Org and the Business and give some weightage to each set of criteria. Below is an example:
- Language/Framework — Programming language for the Test automation suite
- Maturity and Community support
- Ease of Learning (for those writing tests and those writing Acceptance Criteria into BDD frameworks)
- Cost
- Multi-Client (Mobile, Web, Voice?)
The RFC should be open for comment to the entire team (Engineer and QA) for no longer than 2 weeks. At the end of the RFC the nominated lead should present it back to the Leads (key decision makers) for feedback.
My Opinion on Test Automation Tools and Languages
I am a fan of utilising the same language for Development as for Test Automation. This makes for easier cross-over in skills and empathy between developers and QAs. It also allows the Engineers to take more ownership of the Test automation suite. It could change depending on circumstances such as Size of organisation, number of applications to maintain, number of languages and frameworks to support. In a larger organisation where each application is developed in a different language you may want to implement a test automation suite in a consistent language across all domains. However, smaller teams may want to align languages between the application and test suite.
And Most importantly you should set up a process
Do you have support from the RIGHT stakeholders
When I say the RIGHT stakeholders, you need to identify the people who will actually help or break your cause. All too often I see Managers get buy-in from Senior Stakeholders and Senior managers for their strategies only to fail in execution because they don’t have support from Product, and other Technology functions.
Product often tout the need for Test Automation but are the first to forego when it comes down to the crunch. Ensure they are on board with the Test Automation and Quality strategy and understand the importance. I have found getting external speakers talking about the importance of Test and Quality automation a great way to influence colleagues, particularly if they are skeptical of your motives. I will talk more about how to keep Scrum and Product teams accountable to Test Automation strategies a bit later in this article.
Identify the key metrics you will be presenting back to your Stakeholders to show the value of your Quality strategy. Here are some macro level metrics you should be tracking:
- Deployment Frequency
How many deployments per given time period ie. # Deployments per week. This should influence the right behaviours in automation. - Change Fail % -> MTTR
Personally not a huge fan of Change Fail %. However, it may be a good start if your team is not mature in Testing automation and Continuous Delivery just yet.
Eventually you want to track (Mean time to Resolution). This shows your Engineering Org’s ability to fix forward. - Team Health
Whilst undergoing any significant shift in your Strategy (Be Quality or otherwise), make sure you continue to get a pulse on the team’s health. Don’t be surprised if you take a hit but it shouldn’t be significant.
So where do I start with my Quality Strategy
You do not need to implement all the initiatives listed above, however, use your discretion to know where to start.
Firstly, Identify the key metrics you will hold yourself and your team accountable to as shown above. You may also want to consider micro metrics such as % of Regression Test Suite Automation to kick-start your Quality strategy in the right direction. You should take into account the maturity of your Quality and Software Engineering organisation before considering which metrics to track. Please see below:
Secondly, Identify the teams that are most suited to be the ‘Guinea pigs’. The criteria for a suitable team should :
- Have the capacity to trial a new Testing Framework or Process
- Own an application simple enough to trial, but complex enough to nut out the finer details.
- Have People who are accepting of change (this includes Product, Engineers, QAs and other peripheral functions). You want to create Positive deviants in the org that can showcase the possibilities of your Quality Strategy.
If you don’t have people to drive the same you may want to hire an evangelist or 2 who can push the cause. But not everyone has the headcount or budget to go around.
Allow the team to embed the process / and or framework for at least 3 sprints. First sprint for understanding the changes, second to embed, third to optimize and retrospect. Set a clear objective at the end of 3 sprints. It may be % of Test automated, or number of Stories Pair tested.
Set up a showcase for that team to present their findings, learnings and next steps. It’s an opportunity for the rest of the Engineering org to question, understand and learn from tangible examples of the Quality direction. Ensure all key stakeholders are present too!
If successful it’s now time to take it to the rest of the Engineering org through the process.
Again, Identify the key metrics you will be going after and make your leads accountable and responsible.
Track Progress
There’s no point creating a strategy if you are not going to follow through. If the Quality Strategy is important to you then make sure to track progress and milestones along the way. Here are some methods to keep yourself accountable and ensure other’s care about your Quality strategy too:
- Celebrate Milestones
I hope you are already tracking metrics like # of Releases, # Bugs in Releases, and if so celebrate every time you achieve a significant number like 20% reduction in Regression testing duration, or X number of releases per week. - Have a dashboard that tracks all the key metrics over an extend period of time that everyone reviews (particular tech leadership) on a regular basis.
- Add Quality metrics to your Sprint Reviews and Reports
Make the squad and everyone in it accountable by adding Test Automation Metrics, # Deployments and Defects to your Sprint reports. - Find every opportunity to showcase improvements
However, be transparent when you aren’t making improvements. And be clear how you will mitigate the obstacles.
To Summarise …
We have covered a broad variety of areas to improve Quality and how to implement a Technology Strategy to support the same. There are many aspects of Quality Culture you should consider before you simply scale out your QA team or invest in Test Automation. Once you have identified the Test Automation as a gap, invest in the right tools and frameworks and make sure everyone is involved in the process (because it’s a big investment!). Keep yourself accountable by measuring the right metrics and track progress over time. I truly believe Quality done well can enable improvements in Speed 2 Market and Customer experience. I hope this helps you on your Quality journey.
References and Inspiration
- Accelerate: The Science of Lean Software and Devops: Building and Scaling High Performing Technology Organizations
By Nicole Forsgren - https://medium.com/@jchyip/why-t-shaped-people-e8706198e437
- https://martinfowler.com/articles/practical-test-pyramid.html
- https://dev.to/juristr/challenging-the-testing-pyramid-4hnl
- https://tyrrrz.me/blog/unit-testing-is-overrated
