If you want to learn from some of the best engineers and engineering minds in the business, register for INTERACT: the community-driven, digital conference designed by engineering leaders, for engineers leaders. This single day conference will feature 10+ speakers, interactive sessions and a community built by 100’s of engineers and engineers leaders, all for free.
In our rapidly changing technology landscape, it can be difficult to maintain a competitive advantage. Challenges present themselves daily and staying ahead of your competition can feel daunting. At the INTERACT conference for engineering leaders on September 30th, we’ll be exploring two of the most impactful ways that have emerged for companies to differentiate themselves - streamlining engineering processes and maintaining high developer velocity.
These drivers of success make companies more innovative and productive while enhancing team performance and alignment, and two of the key talks at INTERACT focus on solving these challenges:
Maria Gutierrez, VP of Engineering for Strategy and Operations at Twitter, is an expert at engineering processes. Before her time at Twitter, Maria fine tuned her abilities as the VP of Engineering at FreeAgent in addition to stints at Adobe, Intercom and Living Social.
Henrik Gütle, is the General Manager of Microsoft Azure, responsible for leading the commercial cloud business for Microsoft Canada. Leading a team at Microsoft exploring the origins of developer velocity, Henrik has a keen understanding of business impact.
Both of these talented individuals will be appearing at INTERACT on September 30th to share why engineering processes and developer velocity are critical to business success.
Let’s take a quick look at what these two will be presenting.
A sneak peak at Maria’s viewpoint on streamlining engineering processes
At its core, an organization is nothing more than a collection of moving parts. A combination of people and resources moving towards a common goal. Delivering on your objectives requires alignment at the highest levels - something that becomes increasingly difficult as companies scale.
Growth increases team sizes creating more dependencies and communication channels within an organization. Collaboration and productivity issues can quickly arise in a fast-scaling environment.
It has been observed that adding members to a team drives inefficiency with negligible benefits to team efficacy. This may sound counterintuitive but is a result of the creation of additional communication lines, which increases the chance of organizational misalignment.
The addition of communication lines brought on by organization growth also increases the risk of issues related to transparency as teams can be unintentionally left “in the dark.” This effect is compounded if decision making is done on the fly, especially if multiple people are making decisions independent of each other.
In order to maintain overall business alignment, clarity and structure, the implementation of business processes becomes necessary. By defining these processes, your organization will be able to thrive as it scales, unburdened by its own growth as if it were still a small startup.
In effect, processes allow us to codify our success, providing us with systemic and scalable ways to repeat behaviors that lead to success and avoid past mistakes.
The processes that help scale engineering organizations, the implementation of these processes, the effect of these processes, and the 3 most important processes for scaling, will be shared by Maria during her discussion with me at INTERACT.
An excerpt from Henrik’s talk
In early 2020 Microsoft conducted an exhaustive survey of over 400 of the largest companies around the world. The goal was to understand the impact of developer velocity on business performance. A follow up to this survey was then performed in May of 2021 to validate the findings of the original survey.
They found a direct correlation between high development velocity and business impact. The business impact of developer velocity was substantial. Companies with the highest velocity had 4 to 5 times higher profit margins than the companies with the lowest velocity. The companies with high velocity were also more innovative and productive.
The first survey attempted to capture the best overall picture of drivers of velocity. While the second survey sought to identify the impact of Covid on developer velocity and whether or not the shift to remote work impacted the findings of the initial survey.
Companies that were successful in both surveys shared several key similarities. Chief among them were technological updates and organizational practices such as:
Cloud Adoption: The best performing companies were companies whose leaders pushed for early cloud adoption. Companies that fell behind in this area had a difficult time catching up.
Agile: The adoption of agile practices is one of the biggest indicators of high developer velocity and is one of the first things companies should look at if they want to increase developer velocity.
Product management: Building strong product management capabilities and treating it as a discipline is highly critical. As we move towards a world with autonomous teams that own their product end to end, the importance of product management increases.
Henrik and Dev Interrupted Community Leader Conor Bronsdon will be diving into all of Microsoft’s research findings into developer velocity - and what engineers can learn from these findings - during his presentation at INTERACT.
INTERACT: September 30th, 2021
If you want to learn from some of the best engineers and engineering minds in the business, register for INTERACT: the community-driven, digital conference designed by engineering leaders, for engineers leaders. This single day conference will feature 10+ speakers, interactive sessions and a community built by 100’s of engineers and engineers leaders, all for free.
Not only will you have a chance to have access to Maria and Henrik’s research but also to other brilliant engineering leaders like:
With over 2500 members, the Dev Interrupted Discord Community is the best place for Engineering Leaders to engage in daily conversation. No sales people allowed. Join the community >>
To fight the wars of the future the US Air Force tasked a small group of software engineers with a simple job - revolutionize the way the military thinks about software development.
The group tasked with this not-so-tiny problem came to call themselves “Kessel Run” after the famed smuggling route used by Han Solo in Star Wars.
Since starting in 2017, the team at Kessel Run has expanded to include over 1,300 people across multiple locations, helping build, test, deliver, operate and maintain cloud-based infrastructure and warfighting software. These applications are used by airmen worldwide and represent the future of warfare. That’s because the wars of the future will be fought with software and system architecture as much as any other weapon.
What is Kessel Run?
“[Kessel Run] was kicked off about four years ago as a way to prove that the Department of Defense didn't have to be terrible at building and delivering software, regardless of being within the world's largest bureaucracy.” - Adam Furtado, on the Dev Interrupted podcast at 1:35
As an Airforce organization, Kessel Run delivers a wide variety of mission capabilities to warfighters around the world, utilizing industry best practices around DevOps and Agile. At the time of its inception, it represented such a radical departure from the normal way of thinking within the Department of Defense (DoD) that people joked it would have to be “smuggled” into the DoD. That’s how Kessel Run came to earn its name - a scruffy team outfitted with a mission to upend a stodgy and cumbersome bureaucracy.
A shift in thinking needed to start with culture. The team at Kessel Run decided to bring a startup like mentality to the behemoth that is the federal government with a goal of introducing modern software methodologies at scale. Pockets within the DoD were practings things like continuous delivery, but prior to Kessel Run, previous attempts to adopt modern software principles had largely failed. Warfighters weren’t getting the capabilities or tools they needed.
Problem Solvers
One of the biggest institutional problems that Kessel Run was tasked with trying to improve were the Air Force’s Air Operations Centers. Spread around the world across twenty two locations, these organizations manage all the details that involve fighting an air war. Everything from strategy, to planning, to tasking aircraft to perform certain actions, to providing real time intelligence data and feedback, are handled at Air Operations Centers.
The challenge was modernizing these centers while maintaining operational readiness and current hardware - much of which was 20 to 30 years old. All of the hardware across these locations came with its own integrated software, built from various third party sources over decades.
To tackle this challenge the team at Kessel Run applied the principles of Gall’s Law, which states that all complex systems that work evolved from simpler systems that worked.
By starting small and focusing on rapidly achievable solutions, they began to see the network effects of their actions. Small, precise fixes can have tremendous impacts on an organization and are less prone to failure than attempting systematic change overnight.
“So we knew that using Gall’s Law in history, that we needed to start small, in order to make this work. We couldn't just have a big bang approach to replace this entire system. Right? You did that by chipping away at some core parts of the system from a user functionality perspective.” - Adam Furtado, on the Dev Interrupted podcast at 13:31
Practical Success
One of the first small changes achieved by Kessel Run was with the Air Force’s air refueling program.
A remarkable acrobatic feat performed at more than 20,000 feet above ground, at speeds close to 400 miles per hour, replenishing the fuel of an aircraft is dangerous but necessary work. Everyday, fighter jets and bombers rendezvous with fuel air tankers to perform air-to-air refueling before continuing on with their mission.
Optimizing the details of such a delicate dance would be difficult, but the folks at Kessel Run believed they could do it. First, they needed software engineers. One of the problems of developing software at the federal government is a lack of engineers. Or rather, a lack of native engineers that can be found in-house. Historically speaking, the government outsources everything to contractors.
Scrounging the Air Force for active duty software engineers scattered across separate programs, Kessel Run was able to stitch together it’s own homegrown software engineering team.
With their mission in hand, they set to work building an initial application nicknamed “Jigsaw” to improve the air refueling process. By optimizing every aspect of the process from the timing, to the altitude, Jigsaw became an enormous success. Within a year of implementation the Air Force was saving $12.8 million a month on fuel.
Tiny, targeted successes like these continued. But Kessel Run was up against more than just inefficient programs.
A New Way of Thinking
Changing company culture is notoriously difficult. Changing culture inside the world’s largest bureaucracy is as hard as it gets.
The most difficult problem that Kessel Run had to tackle wasn’t the lack of software developers, or the difficulty of integrating third party software applications, or figuring out how to optimize and build combat applications, it was how to communicate with their peers in the DoD.
Part of the difficulty was due to the security implications of such work. The production environments are all on classified systems, making things like cloud implementation and tooling availability difficult.
However, navigating the business side of the DoD was always the most challenging. In the past 30 years the government has spent over a billion dollars trying to update their systems to provide the best capabilities possible to warfighters in order to prepare for a war that may never happen.
Until Kessel Run, the government didn’t have much to show for their efforts. A perception existed that new software methodologies and practices were just the next iteration of technologies that overpromised and underdelivered. It took a lot of trust to explain that doing something in a more agile way or using DevOps, would actually reduce risk and increase success for the organization.
“The problem we have is we go and talk about how deployment frequency is going to buy down risk for us. That sounds counterintuitive to everybody in the world, particularly in a military environment, where they're like, ‘What do you mean? Change is scary. I don't change stuff.’ So we're having these kind of counterintuitive conversations around why moving to this way of working is less risky and increases our chances of success.” - Adam Furtado, on the Dev Interrupted podcast at 6:39
Solving that problem came down to nothing more than old fashioned relationship building. It took years of evangelism and continued success, but eventually Kessel Run started to win the approval of the right people in the right places.
Proof is in the Pudding
From starting as an organization with only 5 software engineers, to expanding into a program that currently has over 1300 people, Kessel Run has proven itself to be an ingenious concept: bring startup culture to an old organization in need of modern ways of thinking.
Government has never been the place that attracted the top talent in technology, but with Kessel Run it’s become that. They provide access to the newest technologies, competing with some of the best companies in the industry.
They do have one ace up their sleeve when it comes to hiring: fighter jets. And those are pretty cool.
Dev Interrupted is a weekly podcast featuring a wide array of software engineering leaders and experts, exploring topics from dev team metrics to accelerating delivery.
Join the Dev Interrupted Community
With over 2500 members, the Dev Interrupted Discord Community is the best place for Engineering Leaders to engage in daily conversation. No sales people allowed. Join the community >>
I try not to get involved in arguments - but when a debate started in the Dev Interrupted Discord about if exceptional continuous improvement (CI) or continuous delivery CD) makes a group agile or not, I had to jump in. I’ve helped build many high-performing teams with agility, and I know that neither CI/CD nor Scrum makes an organization Agile.
It’s Not What You Do, It’s How You Respond
Probably my favorite way I’ve ever heard someone describe agility was that it’s about moving away from believing we can predict and plan everything to sensing reality and responding to it instead.
Consider two extremes. In the “predict and plan” view of the world, we build a feature list, come up with screens, represent that in a backlog, and dutifully execute in two-week chunks. This approach focuses on accurate plans and predictions in the early stages. Mistakes in the early stage compound rapidly later on.
Are these folks agile? What did they sense and respond to?
A different group has a sense of problems to solve and their relative importance to one another. This group decides the goal for their next investment in a sprint. Every day they look at what happened and determine if they need to change course. They frequently shift direction mid-sprint because they’ve learned something new. Are they agile? What are they sensing and responding to?
Both teams are following Scrum’s rules, but one has lost all its ability to respond to new information, and the other, while it seems chaotic, is using near real-time data to guide their next steps.
Regardless of how you do it, agility is a lot more like the second group than the first. You can tell this not by the meetings they have or titles like Scrum Master, but rather by the frequency with which they make decisions that significantly change their direction.
You Have To Have Working Feedback Loops
A way to detect weakness in a group’s pursuit of agility is to examine the strength and length of their feedback loops.
Scrum is a framework, which is a weird way of expressing that it has a few useful pieces and room to figure feedback loops for yourself. The parts it does have, though, are a series of events and structures that provide feedback loops.
A feedback loop is an activity where you can pause to look at the results at the end. A chef that tastes their own food as they prepare it creates a feedback loop. A chef that never tastes their own food or waits until the dish is finished to take a taste, does not.
If you think about the events of Scrum in terms of feedback loops, every activity has a counterpart that closes the feedback loop. Sprints have Retrospectives, Increments have Reviews, and each day has a Daily Scrum.
All too often a Sprint Review is nothing more than a team saying they did a bunch of stuff with no actual feedback happening. The same can be said of Retrospectives, where the team produces many stickies, but no change manifests. The Daily Scrum Is a daily feedback mechanism, but all too often reduced to merely a task status update.
Scrum puts the potential to leverage these feedback loops in place, but they don’t work by themselves. So you can easily wind up with a team building a two year roadmap, executing on that roadmap, but never recognizing the signals that there is actually no market for what is being built.
Sponsored Section
One way to ensure quick feedback loops internally? Cutting the review time for code in your development process. WorkerB developer automation tools by LinearB can give your team proactive notifications of changes in their PRs and update developers on long PR review times while correlating the PR to the relevant project issue.
I often see groups espousing that they are agile because of some activity they do or capability they have. In fact, that very behavior is what inspired me to write this article.
It can be tempting to look at agility as a feature list or series of checkboxes that, if you complete enough, you’ll summit the mythical agile mountain. But if those things aren’t so important, are the rules important at all?
Well, they are, but it’s just not enough.
Organizations and teams put a lot of emphasis on getting the basic structure of Scrum in place but tend to get into the rut of standing pat with those structures, and so the promise of agility remains elusive. These groups similarly work hard to cut things that they’ve been told are not agile anymore. Basically, they focus too much on the form, and not enough on the function.
You have my permission and support to keep using work-breakdown schedules as you pursue agility.
On the one hand, you need the form to be correct, or you can’t get the benefit. On the other, if you only follow the form, you stand in place.
So with my teams, I look to see if our process is “alive.” With a clear purpose for each activity, I can quickly see which of them are exhibiting signs of weakness that will disrupt feedback. Then I can intervene and bring the form back to accomplish its purpose.
For example, in a talk I give about Daily Scrums, I say, “The purpose of the daily scrum is to begin the day’s collaboration and coordination towards a sprint goal.” Most daily scrum meetings miss the mark of that purpose.
Looking at things from this angle, we can mature our forms and get the benefit back.
Change Takes Time
There is a fascinating intersection between these concepts where we recognize we want to address weaknesses in how we practice agility, but have difficulty estimating how to move forward and how long these changes will take.
One aspect to this is that we do need those forms to be in place before we can sense and respond to it. In my experience, a change in these forms takes about three months to normalize in a group.
When I introduce more advanced concepts like Test-Driven Development or Work-in-Progress Limits, I introduce those forms and also hold them in place for three months. This time frame might feel slow, but it comes down to how we as people handle change.
What we’ve done in our past is what frames what makes sense to us today. Our brains crave that feeling that the world works a specific way. When we disrupt that, people’s instincts are to reject the change. Even if it is a rational and logical change, we’ve disrupted what is normal. The three month adjustment period seems to be enough to allow change to take hold and for a new normal to be realized.
During that whole time, though, the form has to be sustained.
So as much as adherence to form can be a trap, it also is the scaffolding for change.
Learn to be Comfortable with Change
My honest advice for leaders who want to make things a bit better is to get really comfortable with microscopic change for a while.
An example might be phrasing a question differently -- a small change.
Take an honest look at how you view things working best along a spectrum of plan and predict and sense and respond. Make sure to compare how you think you do to the actions that actually take place. There is usually a disconnect there.
Next, look at an opportunity within the existing forms to strengthen a feedback loop. Using something as simple as phrasing a question differently -- a small thing -- can completely alter the course of a Review or Retrospective or Daily Scrum. A thoughtful question that encourages feedback can easily improve things..
From here, you can continue this pattern of making small adjustments to how you and others move towards sensing and responding and leveraging the forms of Scrum to become truly agile.
Join the Dev Interrupted Community
With over 2500 members, the Dev Interrupted Discord Community is the best place for Engineering Leaders to engage in daily conversation. No sales people allowed. Join the community >>
