5 Minute DevOps: Solving the Uncertainty Problem
I recently gave a talk called “Agile Rehab” at QConNYC 2023 about the techniques we used to transition from delivering once per quarter to delivering multiple times per day. We did this by ditching SAFe and taking Continuous Delivery and software engineering seriously. We asked ourselves, “Why can’t we deliver today’s work to the end-user today?” and relentlessly improved our engineering practices until we could. Along the way, we found several techniques for solving that problem. One that was very useful for our improved workflow was Behavior Driven Development (BDD).
What’s BDD? BDD isn’t a technology or a coding practice, though many confuse it for both. BDD is a process where we define how something should work from the consumer's perspective in a declarative, testable way.
Early Roadblocks to CD
First, we needed to own the quality gates in the pipeline and have testing lead development rather than handing it off to a “testing team” after the fact because CD requires “Test During Development.” However, having a “testing mindset” was not a habit we had. We needed to build that muscle.
Second, we needed better requirements. You cannot write tests first for things you don’t understand. Having a product owner hand over half-baked requirements and say things like, “Oh, you can decide that during development,” followed by, “That’s not what I meant!” and endless debates about if it’s a defect or an “enhancement” isn’t useful doesn’t add business value. We had to fix these to meet our delivery goals.
I stumbled onto Liz Keogh’s BDD material while trying to come up with solutions for these. I highly recommend digging into her material. It was very helpful for developing the correct point of view; “describe the interaction with the consumer as if done by magical pixies.” In other words, if implementation details leak into the scenarios, remove them and stick to behaviors. You don’t “select a country from the dropdown.” Instead, “you choose a country.” The test still passes no matter what UX we decide on. BDD seemed like an easy win. It would help spread the right mindset for testing, make everyone a stakeholder in the outcomes, and provide a contract for “story complete” with the business. “It will do exactly this, and we all agree to that.” No more arguments about “defect” or “enhancement.” There are no enhancements in product development anyway, only “what’s next.” As my wife Dana says, “Every release is missing the next feature!”
Defining Scenarios
It took several iterations to develop a workflow that was efficient and effective at achieving clarity. When we developed this workflow, we were a team of 15, including a BA and a Product Owner. On a smaller team, fewer steps may be required. Try and see.
Drive Out Uncertainty
A user story is a problem that needs to be solved for the user, not a solution. The scenarios are specific behaviors for how the problem will be solved. Unambiguous scenarios require specific language. We had already been leveraging DDD to align teams and services, and having a word mean exactly one thing within your bounded context is a crucial DDD practice. This becomes even more apparent when you’re writing scenarios, and the business has three different meanings for a single word. We had to work through several instances of that and decided on different words with the business, so we all spoke the same language. We established a glossary to ensure clarity.
Define “Ready”
We had a set of specific acceptance criteria we tested our work decomposition process with:
If a task met that criteria, it was ready to start. The whole team was involved with this decision, and everyone was encouraged to throw up a flag if they disagreed that a task met this. We kept improving the task until it did.
Drive Out Friction
When we first tried this approach, we had the entire team meet to refine stories. That was painful and slow. Large meetings are good for pushing out information. They are less useful for creating things. To make that better and to allow the entire value stream to be more productive, we created two meetings.
In the first meeting, a group of 3 or 4 met to create rough drafts. Following the “Three Amigos” pattern, we had people with product, testing, and development knowledge get the scenarios to about 70% complete. Then, we’d schedule a meeting with the rest of the team. In that meeting, we’d review each story, review the scenarios we created, and have the team do final edits. The team would add scenarios, clarify the language, or even remove some scenarios. As the team learned more about the business domain, they became more comfortable challenging the need for some scenarios. A scenario not implemented is the best kind of development because it’s code that will never need support. Since we were using a pull system of work, rather than the misguided practice of assigning work, anyone could be developing any scenario. Therefore, everyone was personally invested in all of the scenarios being good.
After defining all of the scenarios, we’d organize them by priority and, if required, use them to split stories. Since each scenario was a unit of behavior that was independently deliverable, as long as we kept related positive and negative behavior tests in the same story, we could split them into tiny, independently deliverable units.
From Story to Task
Each tiny story is a full stack change until they are broken down into technical tasks. Because we had several small, domain-specific services, a story might impact two or more services and the UI. This presented an opportunity to deliver value sooner. If a story were too big for a single person to complete in a day or two, we would decompose that into technical tasks where each task would impact a single service. We would define any service contract changes and use evolutionary development practices so that any service could be delivered in any sequence. This allowed us to swarm the story with as many team members as possible. After all, we should always focus on the highest priority; the faster we deliver the highest priority, the faster we can learn from it.
Results
We tried this for the first time after I got frustrated with the time wasted arguing about how many points a story was. We still had “feature complete” habits at the time, and our stories were massive. So, we applied this technique. We met, defined all of the scenarios, story pointed the scenarios (they were also too big), added up the story points, and decided the “story” was 54 points or about 90% of our velocity. Then we went through technical decomposition to identify which services needed what changes, defined any service contract changes, split up the work, and at the end of the 2-week sprint, everything integrated cleanly.
When I left the team, everyone had learned how to “think” in tests, business knowledge grew across the team, and the coding style and application knowledge were leveled because everyone touched everything. The delivered quality was higher because we’d driven out uncertainty. There were no more sprints. Since the average time to deliver a story to production was 0.89 days, we had moved to a continuous flow of work and “just in time” story refining. That also meant no need for story points and endless debates with a deck of cards. The work sucked less.
Developers want to code!
Cool. I want to get paid to do my hobbies, too. Alas, getting paid requires work. It can’t all be fun. The work is not coding. The work is deciding what to code. Coding is the activity of converting that work into usable value. I enjoy coding. It’s fun watching things grow and seeing the machine dance to my commands (or not). I don’t enjoy decomposing features. I don’t know a developer who does. If I find one, I’ll make sure they are on my team. However, driving out uncertainty before development is the real work, and the outcomes are universally poor if the expectation is that developers will be handed work that’s “ready to code.” Using a disciplined engineering approach to drive out uncertainty makes this less painful. I promise that it’s less painful than the outcomes of “I just want to code.”
Free Resources
For more tips on this topic and many others for improving the flow of delivery, check out the information at DojoConsortium.org.
