5-Minute DevOps: Review Board Dramatic Theater
Occasionally, I see well-written, well-meaning, but misguided advice that inspires me to respond with, “Oh God! No! Don’t do that!” This happened recently when I saw an article about how to do Architecture Review Boards (ARBs) correctly, and the conclusion wasn’t “Don’t use review boards.” This resulted in conversations about the value of ARBs, Change Approval Boards (CABs), and the emotional attachment many have for them.
Let’s discuss their intended purpose and goals, why they are ineffective at accomplishing them, and how we can accomplish them more effectively.
Tl;dr: Don’t fall for the theatrics of approval boards. Build it into the process. If you feel out of control, Move closer to the work.
Change Approval Boards
“A CAB not only approves changes within a company’s IT landscape but also helps with assessing, prioritizing, and mitigating risk for a company.”
Bless your heart, Atlassian. We finally got you to fix your GitFlow training. Imagine if we could fix the rest.
A CAB is a recurring meeting where changes are brought to be approved/rejected. None of the change board members are involved in making the change, and, depending on the organization's size, they may not even be aware of the reason for the change. I’ve attended many CABs over the years with scopes ranging from only a few teams to every team in the organization (yes, hundreds of people in a change meeting every morning at 8:30 am). The workflow is inevitably the same:
- The team ensures that a change is deliverable.
- The team prepares the documentation for the change's purpose, scope, etc.
- The team waits for the next change meeting.
- At the meeting, the team represents the change and assures the CAB that everything will be fine.
- If the CAB doesn’t like the answers or isn’t happy with the way the paperwork is filled out, the change is rejected and sent back for improvement.
- GOTO 1
Step 5 is interesting. I’ve seen changes rejected becasue the previous change from another team broke something, so the CAB board wanted every other team to be REALLY sure. People are emotional creatures. Rejecting changes due to minor paperwork issues isn’t hyberbole. I’ve seen it. Others have reported the same. That means the team will be presenting a large change at the next meeting. Risk and changes size are tightly coupled.
Let’s go through some of the stated goals of a CAB and apply a testing approach to the goals. Are the goals best met by a meeting?
Calculate the risks and consequences of potential changes
Based on what information? Your best-case scenario as a CAB member is that my team spends the morning developing a deliverable change and the afternoon prepping the paperwork. This way, you are presented with a small enough change that you might be able to get into some of the details during your meeting. Of course, it’s unlikely that my team is the only team needing to deploy something, so hopefully, it’ll be a long enough meeting for team in the area to finish presenting.
The reality is that the transaction cost of preparing for the CAB’s dramatic theater is too high for that, and we will avoid doing it very often. So, we’ll probably only do it at the end of the sprint, and you’ll be presented with several completed features that the business expects to receive now. There is no possible way for you to achieve this goal with a manual review process of a batch that large.
Why are we developing something without understanding the risks? This calculation should have happened before development began. If we believed a change had more than the usual risk, we should have designed the mitigation for that into the change. You cannot “inspect in” lower risk just as you cannot “inspect in” quality. Larger batches of work are more likely to cause incidents. By creating an approval meeting for risk mitigation, you’ve created a process that’s designed to increase delivery risk.
Review requests for change and evaluate their impact against available resources
How? With what data in the paperwork? Let’s assume you see a snapshot number about the system resource requirements of the change and can make a judgment call during the meeting that it’s too high; then what? Naturally, you tell the team to make changes to improve it. Again, we are trying to “inspect in” quality at the end.
Why are we not testing for this on every commit as part of the delivery pipeline? If we have known resource constraints, we create tests for those and see our trendlines against those constraints while we work to avoid the rework of completed features. Optimize for business value, not your ITSM training.
Produce comprehensive change-related documentation
What documentation? Surely not product documentation because the teams are better equipped to do that. Documentation for auditors? The best you can do is show them that a meeting occured. How does that keep us safer? There is no way for a meeting to do more than rubber stamp compliance because the CAB members lack the data on how the work was done. If they have the data, they don’t need the meeting.
This should be part of the development process and in the form of updated architectural diagrams and decisions, automated release notes, and any required product documentation. As for documentation for auditors, that should be instrumented and come from the platform logs, not meeting notes.
Ensure company-wide understanding of changes
To what end? If everyone in the company needs to know about a change, then the worst possible time is just before it’s implemented. I assure you no one who isn’t directly impacted is paying attention because they are fucused on preparing their changes for your meeting.
Instead, reduce the coupling, be transparent with roadmaps, talk about new features coming out, and celebrate after they are delivered and you begin seeing value from them.
Implement change management software and facilitate training
This is what developer platforms are for. Platform engineering is about making it difficult to make mistakes and easy to deliver change at the same time. The platform should contain the controls so that the development process becomes compliant. Approve the process, not the changes that result.
Non-destructive CABs?
I have spoken to people with a CAB that I’d consider effective. There’s a scheduled meeting, but you only attend if you do not have a CAB-certified delivery pipeline or have a recent history of breaking things. The CAB audits the pipelines for quality, security, and compliance. The meeting is the pain you want to avoid, and they make it painful on purpose so that you are incentivized to fix your problems. This is what good looks like.
Architecture Review Boards
“The Architecture Review Board (ARB) serves as a governance body ensuring IT initiatives align with Ecosystem Architecture and ultimately align with MIT IT goals, strategies, and objectives.”
— MIT
You create this review board if you have bored enterprise architects who don’t want to get their hands dirty. Effective EAs go where they are needed; they do not sit on a review board and have teams seek their blessing.
Here are some of the goals MIT lists in its ARB documents. I’ve reordered them into things the EA team should do as a team, things they should provide other teams as a service, and things they should not do.
EA Team Should…
“Establish architecture baseline and develop and maintain target architecture.”
Yes, they should set some baseline standards. However, those should not be so comprehensive that they prevent teams from applying patterns that aren’t documented, and they should review the impact of their recommendations, not fire and forget.
“Establish and promote architecture best practices.”
I’d use “good” and not “best” here, but training people to make better decisions is far more valuable than training them that only the EAs get the final word.
“Identify new technologies.”
Absolutely. Be looking ahead and bring in better ideas from outside.
EA Team Should Help With…
“Plan, design, and implement an efficient, effective, and comprehensive architecture for Infrastructure and Application.”
This should be done in collaboration with the teams doing the work. They should work with the teams as decisions are made so that they have the context of the decision and can help formulate the final decision. Shift quality left.
“Improve quality ”
They can certainly help with this by training on patterns, teaching people how to separate domains correctly, working with management on better team structures to make better architecture easier to achieve, etc. However, they can’t do this in an ARB meeting. They need to work for a living to do this.
EA Team Should Not…
“Lead architecture review process”
A competent architect knows you don’t wait for a building to be built to determine if it’s been done correctly. With software, it’s even more critical that we make quality and governance part of the workflow and not a review process because review processes increase the size of delivered work and the risk of failure.
Arguments Review Board Apologists Use
The following are direct quotes, not strawmen.
“The problems with your proposed approach begin to emerge in multi-billion dollar companies, in highly regulated industries, and when the planned changes have upstream/downstream/cross-cutting impacts.”
My recommendations come from years of working with good EAs at a Fortune 1. I wonder why processes that scale well for $600B companies would not scale to companies two orders of magnitude smaller? There must be some unique problem with being that small, which I’m unaware of. As for regulated environments, do you trust a review board to ensure you’re meeting your controls? Perhaps you’re unaware of how easy it is to sidestep those controls if the tooling doesn’t enforce them. You must also trust manual testing and only use “Oops, we got hacked!” for security. You probably need a Review Board Review Board (RBRB) to review the CAB and ARB decisions before allowing them to proceed.
Please read Investments Unlimited. It’s a fictionalized version of real things real people are doing to solve this problem in FinTech.
“Not every change needs to be reviewed. Only the big changes.”
As I said, it’s dramatic theater, not an effective quality or compliance process. If we don't use the same process to deliver all changes, then we are skipping steps that we feel are important for some changes based on how we feel about this particular change. It’s all feelings, and feelings don’t keep us safe. Let’s ignore the fact that you cannot use inspection at the end to “assure” anything and discuss other ways this is broken.
- Transaction cost: If it’s too expensive to perform one of our quality gates for a low-risk change and we have the option to skip it, then we will skip it for a marginal change because of delivery pressure.
- Incident management: If it’s too slow or cannot be performed in an emergency, we use less safe methods to resolve an incident. We should use our safest methods for every change, including hotfixes.
- Ship of Theseus: How many tiny changes equate to a big change? If we ignore good practices for the small changes, does that make us safer?
Use Engineering, Not Dramatic Theater
I remember the long and painful process of getting a security review to enable my first production delivery of a new product and asking Security when I would need the next one.
“In two years or if you make a major change.”
This is another example of dramatic theater by people who don’t understand the work they are tasked with securing. With the next commit, we could cause an unintentional security issue. Oh well. They didn’t care about security beyond it being a checkbox; why should I? The application underwent significant changes after that, but we never made a major change, so it was probably fine.
“Shift left” and “build quality in” are not buzzwords. If you care about building things correctly, delivering useful solutions, keeping them secure, and keeping them in compliance with regulations, then focus on how things are built and how to put the right guardrails in place to make it easy for every code change to meet those goals. Teach, help, and provide tooling. Don’t create meetings for “visibility” or “governance.” If you feel out of control, move closer to the work.
