5 Minute DevOps: Continuous Delivery FAQ
There are many misconceptions about CD. There are also many things about CD that are not obvious. If you are “CD curious,” perhaps this will help.
Do you think CD is unobtainable or want to start the journey? Check out MinimumCD.org.
“Why do you care if we are using CD?”
Because the work should suck less. When I had my first experience working this way and realized that all of the pain I’d suffered previously as a developer wasn’t required, I decided I needed to help others have less sucky lives.
“Is it CI/CD or CD?”
It’s just “CD.” Just as DevOps encompasses security, business, compliance, user experience, and everything else in the value stream, CD encompasses the entire development process.
“In an ideal world, you can do this, but we…”
I’ve worked to solve this problem in multiple contexts, and I have worked with others who have done this as well. I’ve yet to see or hear of a special situation where you can’t work this way, even in air-gapped environments. Jez Humble has a great talk on this: “Continuous Delivery sounds great, but it won’t work here.”
“There is no objective definition for CI”
There is an objective definition. Some may believe that pulling updates from the trunk daily but only integrating changes after several days of work to ensure the change is “feature complete” is CI. People believe all sorts of things that aren’t true. Believing really hard doesn’t make it more true. The point of CI is team communication and rapid quality feedback. The code integrates very frequently and is tested together. This allows everyone to see everyone else's changes and helps identify problems early and when they are small problems. It reduces the problems with dependency between changes, code conflicts, etc. Smaller is better.
- CI Definition on Wikipedia
- CI practice and tips on MinimumCD.org
- Martin Fowler on CI
- Trunk Based Development patterns
“We automated the build. Are we doing CD?”
Build and deploy automation are essential. Test automation is even more critical. However, it takes people, processes, and automation to execute CD. Automation is about 10% of the CD problem. Check out MinimumCD.org’s minimum list of problems to solve and see how many are behavior-dependent.
Continuous delivery is continuous product development and continuous quality feedback. The robots enable us to standardize and accelerate delivery to reduce the cost of change and improve the safety of change. This makes it possible to get faster feedback on quality and makes it more economically viable to run value experiments with hypothesis-driven development.
People frequently underestimate the definition of “small.” We aren’t talking about epics, features, or even stories. We are talking hours or minutes worth of work, not days or weeks. A CD pipeline includes the robots, but it starts with the value proposition and ends when we get feedback from production on the actual value delivered so that we can make decisions about future value propositions. This takes teamwork from everyone aligned to the product flow and unrelenting discipline for maintaining and improving quality feedback loops.
“CD is just for unicorn startups?”
CD is a quality process, not an edge case used by dotComs iterating quickly on MVPs. In fact, many startups ignore the discipline of CD early on and then need to slow down to resolve the resulting tech debt. Many will grind to a halt and fail because they didn’t include “how will we reliably deliver future changes?” in their business plans. Hopefully, they get enough funding to clear the massive tech debt before that occurs.
The reasons for delivering very frequently are:
- Ensure that we can deliver patches quickly and safely in an emergency.
- Establish an efficient and effective quality signal to reduce mean time to detect.
- Reduce the size of changes to reduce the size of the defects in each change.
- Reduce the cost of change to enable more freedom to try new ideas.
CD, when used to drive down batch size and learn from production, acts as a forcing function for stability, efficiency, effective value delivery, and improved quality of life.
“We will focus on CD after we deliver these features.”
There’s really no point in building a feature if you cannot reliably test and deliver the feature to validate the value. Also, you have no idea if your tests are valid until you deliver and find out.
- The requirements are probably wrong
- We probably misunderstood them
- They will probably change before we deliver
How much investment do you want to make in development before you find out how many of the above are true?
The correct order of operations is:
- Pipeline to production
- Production delivery of “hello world” to validate the pipeline
- Next change.
1 & 2 should occur on day 1.
“We need to hire smarter people first.”
No, CD is the tool that grows better engineers, better teams, and more effective organizations. Solving the problem of “Why can’t we deliver this change safely to production today?” in your context is an engineering problem that takes teamwork and disciplined development. If this is the primary focus, skills such as effective use of feature flagging, continuous integration, developer-driven testing, better team communication, etc., will grow rapidly. It also improves morale due to the reduced stress of repetitive toil and Fear Driven Development.
“Developers don’t want to maintain tests.”
Want to know how to spot a competent developer? They get twitchy when they don’t have good tests. Good developers live and breathe testing. For CD to function, developers need to own the implementation and maintenance of the tests that run in their pipelines. There may be, and often are, tests done outside the pipeline, but they are not used as delivery decisions. They are used to inform improvements in the tests running in the pipeline. Those tests may or may not be done by the developers. If you have developers who think testing is too hard, train them. If they say it’s not their job, then I believe there still may be a labor shortage in the food service industry. I’ve zero tolerance for incompetent peers. I’m a software engineer. I deliver working solutions. I know they work because I test them. They keep working as I continue to improve them because I test them. Only script kiddies whine about testing.
If you want to learn more, I’d read xUnit Test Patterns, Google Testing Blog, Martin Fowler’s blog posts on testing, and Working Effectively with Legacy Code.
“If the developers own the testing, what does QA do now?”
Having people who are skilled at breaking things is very important. QA should focus on helping teams design more effective and efficient test suites. They should continuously look for holes in the tests and provide feedback to the developers so the tests can be improved. They should be Developer Advocates for effective patterns and help to create platforms and examples that make testing easier. If all QA is doing is testing code for developers, they are a waste of talent. Google has some thoughts on this.
“Our users don’t want changes that often.”
I assure you that when you have an impacting incident, your users want changes that often. There’s a big difference between generally exposing new features daily and delivering changes daily to verify they do not break. Again, this is a quality process, not a feature delivery process. We need quality feedback from continuous testing in production instead of hoping our next big release doesn’t break anything. Your next big release will break something.
“We don’t believe these wild claims about CD.”
That’s pretty common. Most people don't understand how applying discipline to the development process and focusing on shrinking change-set sizes can dramatically improve business outcomes and quality of life. Those of us who live that are trying daily to show that to you. Most of us aren’t paid consultants or tool vendors. We are telling you this because we want your lives to be better too, and we want to maintain our quality of life if we join your organization. Many of us won’t join an organization where this isn’t allowed and will leave organizations where it is banned.
“How hard is it to start?”
That depends on how teams are structured and how applications are structured. I recommend learning about Conway’s Law and reading Domain-Driven Design and Team Topologies. Take a look at MinimumCD.org and begin solving those problems. It will take a few months for a team already organized to be small and product-focused, given the right support and a knowledgeable person or two. For larger, legacy systems, it will take the same broader organizational changes your competitors are working on today. The best time to plant a tree was 20 years ago. The second best time is today.
“What happens to teams that are prevented from improving CD?”
Once developers discover their lives are improving, they will not want to stop. If you are in a management role for a team focusing on CD and then decide to send the team on a death march that prevents them from sustainably improving or impose processes that prevent CD, the good ones will resist you by voting with their feet. You’ll be left with below-average developers who didn’t fully embrace improvement. Good luck with your goals.
“There’s no value in delivering partially complete features!”
This is a common misconception. There’s no value in NOT delivering. Undelivered code is a liability. If we ship it, we can begin getting feedback from it, and we can reduce the risk of the next delivery. The smaller, the better. Where’s the value?
- We know the code will not negatively impact existing code.
- We reduce the risk of subsequent deliveries.
- Because we don’t have high levels of undelivered change, we don’t need to use workarounds to resolve incidents. This means we can use our standard quality processes when they’re most important.
The cost of delivering a single line of code is the same as delivering 10k lines of code. There are no economies of scale at play here. If you can’t afford to deliver a single line of code, you should look into that. It’s a problem.
I will continue adding to this list as I think about other common questions. Feel free to check back or to comment with questions, challenges, or arguments you have where CD isn’t a valid process.
Changelog
- 2021–01–23: Initial commit
- 2021–03–28: Added “We automated the build. We are doing CD.”
- 2022–09–02: Added section on CI. Added links to MinimumCD.org project. Minor typo edits.
- 2024–02–21: There’s no value in delivering partially complete features!
