Agility, Complexity and Empirical Method

How can complex problems be simplified? How can we prevent uncertainly from becoming a black hole that disrupts the entire process?

We aim to answer these questions to help distinguish when Agile methods can be a valuable tool and when they might not be necessary.

Origins, Context, and Challenges

Agile was born and evolved within the software industry. Initially, this industry relied on exhaustive analyses based on assumptions, driven by the belief that launching fully developed products was essential. This approach was meant to protect ideas from being copied, avoid customer rejection due to low quality, and preserve the brand’s reputation.

However, these slow processes clashed with the reality that technology and software development were accelerating the pace of the world. Needs and markets change rapidly, increasing uncertainty. Additionally, many problems businesses face are complex and often require input from diverse areas of expertise.

To maintain market position and build competitive advantages, businesses must be quick and effective in:

• Understand their users, their needs, and how those needs evolve
• Validate that their solutions address real problems
• Developing innovative and distinctive solutions while minimizing risks and losses

In other words, they must:

• Harness collective intelligence
• Streamline their processes
• Fail early and learn quickly

Agile is a philosophy—a way of working, managing, and leading. Using various methodologies and frameworks, Agile addresses these priorities. In environments where changes happen rapidly, uncertainty is high, and complexity abounds, rigid formulas and predefined steps often fail. There are no guaranteed recipes for success. Instead, Agile frameworks aim to improve communication, collaboration, and decision-making by:

• Aligning teams around shared values
• Reducing waste (both tangible and intangible) through short iterations
• Turning assumptions into hypotheses, testing those hypotheses, and continuously learning from experience

Values as a Guide

Over 20 years ago, the Agile Manifesto introduced four core values (and 12 guiding principles):

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

These values encourage reflection on what truly matters at any given moment. However, they don’t suggest that processes, tools, documentation, contracts, or plans are unnecessary. In any Agile framework, these elements exist—but only to the extent that they help achieve what’s most valuable at any given time:

• Processes that improve efficiency.
• Documentation that provides context quickly and prevents silos.
• Contracts that clarify and align expectations.
• Plans that are flexible, reviewable, and adaptable to help meet commitments.

Values shape our preferences, guiding our choices and actions. They influence how we approach our work, define goals, and set our priorities.

The people involved in a project often have diverse values. Success doesn’t hinge on individual contributions, but on how well people collaborate. A powerful way to improve teamwork and decision-making is to align the team around shared values, ensuring everyone interprets and understands them in the same way.

The Empirical Method in Scrum

Agile practices in general, and Scrum in particular, emphasize and facilitate the use of the empirical method. This approach helps us learn from experience and make decisions based on what we observe—decisions driven by the data we collect through experiments. Applying the empirical method in short cycles is essential, as the pace of change is rapid. This approach is vital for understanding our customers, learning about the market, getting to know our team, and grasping the ecosystem where we operate.

In Scrum, the empirical method is built on three fundamental pillars: transparency, inspection, and adaptation.

Transparency ensures that both the processes and the resulting work are visible to everyone involved—those doing the work and those receiving it. It’s important to ensure that information is up-to-date, provides just the right level of detail needed by its users, and is easy for them to understand.

Transparency enables effective inspection. It allows us to verify whether expectations are being met or if there are deviations, as well as identify risks and problems. Without transparency, the inspection process may fail, leading to mismanagement of expectations and wasted effort. When we need to assess progress, we must ask, “Progress relative to what?”—the budget, timeline, plan, or goals. A lack of transparency around these variables makes it hard to answer that question or to identify issues that require immediate attention.

Conducting a thorough inspection equips us to decide whether or not an adaptation is needed. In other words, it allows us to correct processes, stay the course, make adjustments, or abandon the solution we are developing for a specific problem.

To adapt effectively, people need to feel empowered, manage themselves, and make decisions. However, any changes should align with the available resources and be agreed upon by all key stakeholders. For example, we can’t decide on our own to change a solution’s release date, reduce a project’s scope without consensus, or remove promised features without properly addressing and managing those adjustments.

It’s important to stay aligned with the current context, manage expectations, and focus the team’s efforts based on how much flexibility is available. Coming up with ideas that won’t actually be considered wastes the team’s potential and can lead to frustration and demotivation. Shifting to this way of working is a process—it’s something that happens gradually as you start to see the benefits over time.

In a nutshell, Agile helps us deal with uncertainty by using an empirical approach. It gives us a way to work and organize ourselves where we’re always experimenting and learning from results. That means defining hypotheses, figuring out how to test them, measuring and analyzing the data, and then deciding whether to keep going, make changes, or scrap something altogether.

It’s also great for handling rapid change—using short cycles makes it easier to inspect and adapt. And it helps manage complexity by breaking down problems, focusing on what really matters, and staying on track.

When shouldn’t we use Agile? When things are stable, there’s little change, and you already have reliable practices in place. It’s better suited for situations with uncertainty or complexity, not for straightforward problems, small teams, or projects with fixed deadlines, designs, and scopes that can’t be adjusted.

Managing Complexity and Uncertainty with Scrum

Scrum is a lightweight framework*(¹) designed to deliver value through adaptive solutions for complex problems. This complexity often stems from rapid changes, high uncertainty, and the need to integrate diverse specialties to solve problems effectively.

Unlike detailed step-by-step methodologies, Scrum provides a set of rules to guide teams in defining their approach. Every event, artifact and role in Scrum is intended to promote the use of the empirical method. Scrum doesn’t prescribe exactly how to execute these elements, but instead offers principles to guide team interactions and decisions.

Scrum fosters success by aligning teams around core values: commitment, focus, openness, respect, and courage. These values ensure that team members row in the same direction, prioritize effectively, and communicate openly. Scrum’s events and artifacts are specifically designed to continuously apply transparency, inspection, and adaptation. These elements not only facilitate learning and data-driven decision-making but also help teams establish medium- and long-term goals.

(¹) The distinction between methodologies and frameworks is subtle yet significant:

1. Methodologies: A set of procedures designed to achieve specific objectives, requiring specialized skills.
2. Frameworks: A standardized set of concepts, practices, and criteria used as a reference for addressing similar problems.

Agile, at its core, is a philosophy—a way of working, managing, and leading. It leverages methodologies and frameworks to achieve its objectives.