THE AGILE CONSULTANT

GUIDING CLIENTS TO ENTERPRISE AGILITY

Rick Freedman

Th e Agile Consultant: Guiding Clients to Enterprise Agility

Rick Freedman Lenexa, Kansas USA

ISBN-13 (pbk): 978-1-4302-6052-3 ISBN-13 (electronic): 978-1-4302-6053-0

DOI 10.1007/978-1-4302-6053-0

Library of Congress Control Number: 2016951233

Copyright © 2016 by Rick Freedman

This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dis-similar methodology now known or hereafter developed.

Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringe-ment of the trademark.

The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights.

While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein.

Managing Director: Welmoed Spahr Acquisitions Editor: Robert HutchinsonDevelopment Editor: Matthew MoodieEditorial Board: Steve Anglin, Pramila Balan, Laura Berendson, Aaron Black,

Louise Corrigan, Jonathan Gennick, Robert Hutchinson, Celestin Suresh John, Nikhil Karkal, James Markham, Susan McDermott, Matthew Moodie, Natalie Pao, Gwenan Spearing

Coordinating Editor: Rita FernandoCopy Editor: Lori JacobsCompositor: SPi GlobalIndexer: SPi GlobalCover Designer: Isaac Ruiz Soler

Distributed to the book trade worldwide by Springer Science+Business Media New York, 233 Spring Street, 6th Floor, New York, NY 10013. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail orders-ny@springer-sbm.com , or visit www.springeronline.com . Apress Media, LLC is a California LLC and the sole member (owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation.

For information on translations, please e-mail rights@apress.com , or visit www.apress.com .

Apress and friends of ED books may be purchased in bulk for academic, corporate, or promotional use. eBook versions and licenses are also available for most titles. For more information, reference our Special Bulk Sales–eBook Licensing web page at www.apress.com/bulk-sales .

Any source code or other supplementary materials referenced by the author in this text is available to readers at www.apress.com . For detailed information about how to locate your book’s source code, go to www.apress.com/source-code/ .

Printed on acid-free paper

Apress Business: The Unbiased Source of Business Information

Apress business books provide essential information and practical advice, each written for practitioners by recognized experts. Busy managers and pro-fessionals in all areas of the business world—and at all levels of technical sophistication—look to our books for the actionable ideas and tools they need to solve problems, update and enhance their professional skills, make their work lives easier, and capitalize on opportunity.

Whatever the topic on the business spectrum—entrepreneurship, finance, sales, marketing, management, regulation, information technology, among others—Apress has been praised for providing the objective information and unbiased advice you need to excel in your daily work life. Our authors have no axes to grind; they understand they have one job only—to deliver up-to-date, accurate information simply, concisely, and with deep insight that addresses the real needs of our readers.

It is increasingly hard to find information—whether in the news media, on the Internet, and now all too often in books—that is even-handed and has your best interests at heart. We therefore hope that you enjoy this book, which has been carefully crafted to meet our standards of quality and unbiased coverage.

We are always interested in your feedback or ideas for new titles. Perhaps you’d even like to write a book yourself. Whatever the case, reach out to us at editorial@apress.com and an editor will respond swiftly. Incidentally, at the back of this book, you will find a list of useful related titles. Please visit us at www.apress.com to sign up for newsletters and discounts on future purchases.

The Apress Business Team

For Terri Doubt that the stars are fire,

Doubt that the sun doth move, Doubt truth to be a liar,

But never doubt I love . . .

ContentsAbout the Author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix

Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii

Part I: What Is Agile Consulting?. . . . . . . . . . . . . . . . . . . .1Chapter 1: The Agile Consultant. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Chapter 2: Agile Evolution: More Than Methodology . . . . . . . . . . . . . . 19

Chapter 3: The EVOLVE Framework for Agile Evolution. . . . . . . . . . . 35

Part II: The EVOLVE Agile Consulting Framework . . . .45

Chapter 4: Explore and Engage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

Chapter 5: Visualize Success . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

Chapter 6: Observe and Plan. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

Chapter 7: Lead Teams to Agility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

Chapter 8: Visible Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

Chapter 9: Evolve the Enterprise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107

Part III: Engaging at Enterprise Level . . . . . . . . . . . . . . .123

Chapter 10: Agile Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

Chapter 11 The Leadership Commitment . . . . . . . . . . . . . . . . . . . . . . 141

Chapter 12: The Agile Enterprise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155

Chapter 13: The Agile Consulting Model . . . . . . . . . . . . . . . . . . . . . . . . 167

Part IV: Running the Agile Consulting Practice . . . . . . .177

Chapter 14: The Agile Consulting Skill Set . . . . . . . . . . . . . . . . . . . . . . 179

Chapter 15: Agile Domain Expertise . . . . . . . . . . . . . . . . . . . . . . . . . . . 191

Chapter 16: Conclusion: Toward the Agile Enterprise . . . . . . . . . . . . . . 201

Appendix A: The Roots of Agile: History and Background . . . . . . . . . . 205

Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225

About the Author Rick Freedman , author of The IT Consultant (Pfeiffer, 2000), has experienced the agile transition from the inside. He has been working as an agile coach and trainer since the beginning of the agile movement. As Worldwide Project Management Director for Intel from 2001–2005, Rick evangelized, trained, and coached Intel project managers in the United States, UK, China, India, Germany, Sweden, and Australia. Since leaving Intel, Rick has trained and coached agile teams at clients such as Credit Suisse, Bank of New York, Turner Broadcasting, Motorola, U.S. Department of Homeland Security, U.S. Customs and Border Patrol, the IRS, Wells Fargo, TransCanada, and many others. He authored ESI’s original Agile Project Management course and consulted with ESI as it expanded this offering into a suite of agile courses now delivered internationally. Rick’s articles about agility for TechRepublic and DZone enable him to remain current with agile trends and engage with a worldwide audi-ence of agilists.

Acknowledgments Thanks to Jim Highsmith for introducing me to agility through our interview in 2000, to Tom Conrad for helping me understand how agility works at Pandora. Also to Ludwig and Rudolf Melik for their kind friendship. Thanks to agile geniuses Pete Behrens, Jean Tabaka, Ronica Roth, Mike Griffiths, Denise Vestin, Munir Bhimani, Scott Bird, Kay Harper, Jenny Tarwater: knowing you made me smarter. Matt Holt, my friend and my original believer: rock on, my brother. To the originators: Mike Cohn, Lyssa Adkins, Ken Schwaber, Jeff Sutherland, Dean Leffingwell, Jon Katzenbach and Doug Smith, Craig Larman, Rob Thomsett, etc.: You started a revolution that changed the world . . . no small feat.

Introduction I began learning the basics of project management as a rookie in one of the biggest information technology (IT) shops in America, Citicorp’s global data center. I was hired as a specialist in Citicorp’s new PC and Network group. My first project went way long and over budget. I was invited to a chat with a systems manager, for what I expected to be a scolding. Instead, Jerry took me to a blackboard and explained the fundamentals of managing an IT project. Jerry drew four boxes on the blackboard, and put a “D” into each one. “What do you think the D’s are for?” he asked. I just stared, stupefied. Jerry then explained a simple 4D methodology for delivering IT programs.

Discover, Design, Develop, Deploy Being a typical New York guy, Jerry didn’t explain this to me in terms of critical paths or network diagrams. “Discover what the client wants, and what you’re walking into.” Jerry advised. “Design a solution to fix the problem. Develop that thing that you designed, then Deploy it.”

I, and thousands of other project managers, took Jerry’s commonsense “4D” project philosophy and subsequently expanded it into phases, then tasks. We estimated those tasks, even though anyone who’d been in the IT field for more than a minute knew that estimates were invariably wrong. We created “phase gates” to ensure that no one illicitly progressed without permission. Those gates became codified, and eventually developed into the 17-binder propri-etary project methodologies that were the rage in the 1990s.

I went to work for one of the Big 5 consulting firms, and was trained in its particular version of the multichecklist, highly enforced methodology. After a few months, I became such a zealot of the regimented, gated approach that I joined the Project Committee.

When I got some experience in the field I quickly understood why the “pre-dict and plan” model was unworkable in a consulting context. Simply put, no client being billed by the hour will stand for dozens of hours of overhead, filling out forms, and passing through gates that don’t add business value. The client, of course, believed that project management was just the consultant planning his own project, and so was reluctant to pay for it, while we were under orders to get at least 15% of the total gig in project management fees.

Introductionxiv

Most significantly, “predict and plan” didn’t work. Upfront requirements were incomplete, as users couldn’t articulate what they needed. Estimates were “magical,” as if we could see over the horizon and know how the mix of tech-nology, personality, and culture would play out. IT projects routinely failed. As I began to understand the fallacy of the “predict and plan” model, I noticed the shoots of a “light methodologies” movement budding, with thinkers like Rob Thomsett writing books like Radical Project Management . 1 I was determined to find a method that maintained the rigor of a phase-gated approach with the low-overhead of these new “radical” project ideas.

When the overhead of the big consulting giant became too much, I moved to a local system integrator. We dealt with smaller clients and less complex IT challenges, but nonetheless struggled with consistent delivery. We spent a long time experimenting with different versions of “project toolkits” and eventually reached a light, adaptable 4D-style model that was less likely to scare away clients. With the simple application of a lean project management discipline, we grew the consulting practice significantly, and delivered con-trolled, successful results.

I left this gig to write my first book, The IT Consultant , 2 and began to travel as an advisor to other IT shops. The more small IT shops I visited, the more obvious the patterns. Either they were strangling in bureaucratic predictive project regimes that destroyed their flexibility, or they were improvising their way through projects, usually by throwing technical bodies against them, and delivering chaotic, unmanaged engagements.

In advising these small IT shops or consulting firms, I discovered a secret: insti-tute a few, simple project disciplines, apply some consulting skills, like collabo-ration and communication, and most IT shops will solve their own problems.

On a writing assignment during this time, I interviewed Jim Highsmith, soon-to-be signatory of the yet-to-be-written Agile Manifesto, and Highsmith intro-duced me to ideas that were about to revolutionize the world of software development. The concepts of high-performing teams that managed and moti-vated themselves, of enforced client participation, of lean thinking throughout the process, fell so neatly in line with the on-the-ground experience I was having in the 3 field, that I quickly became an agilist.

1 Rob Thomsett, Radical Project Management (Prentice Hall, 2002), www.amazon.com/Rob-Thomsett/e/B001KD577M/ref=dp_byline_cont_book_1 . 2 Rick Freedman, The IT Consultant: A Commonsense Framework for Managing the Client Relationship (San Francisco: Jossey-Bass/Pfeiffer, 2000), www.amazon.com/Rick-Freedman/e/B000APKF5U/ref=dp_byline_cont_book_1 . 3 www.techrepublic.com/article/consultant-employs-quotlight-methodologiesquot-for-application-development/ .

Introduction xv

The “predict and plan,” PERT (Program Evaluation and Review Technique)-method, Gantt-chart-style of project management taught us a lot, but it got strangled in its inconsistency. You can’t possible predict how projects will go, no matter how many gates you set up or papers you sign. The technology changes constantly and unpredictably. We can’t know what clients want, since they rarely know themselves. We can’t accurately estimate huge programs, but we can time-box and cost-box projects and deliver valuable, useful incre-ments quickly. We can’t manage talented knowledge workers into compliance, but we can lead them to glory.

Agile is revolutionary for those who came up in a gated, waterfall world. It’s going to become a lot more revolutionary in the next few years, as these agile ideas of collaboration, speed-to-value, adaptability, and iterative, incremental delivery evolve from a strictly IT discipline into agile marketing, and then to agile strategic planning in the executive suite.

4 www.infoq.com/articles/standish-chaos-2015 .

Successful42%

Challenged49%

Failed9%

AgileSuccessful

14%

Challenged57%

Failed29%

Waterfall

Figure I-1. Agile vs. waterfall success rates: The Standish Group CHAOS study, 2015

One thing is certain; the verdict on agile is in. Agile works, at least in some circumstances and for some organizations. The latest Standish Group CHAOS study, 4 a periodic review of project success, illustrated clearly that agile is more successful than waterfall.

Another thing is also true; the vast majority of self-proclaimed agile enter-prises are far from being agile; most are not even doing agile. The distinction is meaningful; simply performing the ceremonies and techniques of agile won’t help the marketing department prepare for agile’s iterative cadence, nor will it convince leaders to accept the ambiguity and emergent nature of agile proj-ects. Agile evolution is an agile, iterative process itself, and it usually takes a lot of iterations before the culture of agility begins to penetrate beyond the development team.

Introductionxvi

Companies gravitate to agile approaches for a few simple reasons. First, as we saw, they work; the agile community is full of success stories from enterprises that have adopted agile techniques and gained significant advantage. The ranks of developers are exerting pressure from below. Many have experienced agile, and bridle at reverting to traditional techniques. The cadence of technology, the expectations of markets, the fickleness of customers—all increase the pace of change, putting more businesses at risk of disruption. It’s no wonder executives are now looking at agility as the next holy grail of management theories. Agile consultants often see a significant drop in enthusiasm when leaders discover the depth of radical change agility requires.

My experience across the gamut, from strict Project Management Institute (PMI), CMM (Capability Maturity Model)-style predictive project management techniques to agile methods, leaves no doubt for me that agile is a revolution-ary way to think about software development. Like many revolutionary ideas, now that it’s the norm it seems obvious and inevitable. These practices and principles have the potential to profoundly enhance the competitiveness and productivity of global business; that’s how important I believe agile’s ultimate impact will be. Agile proves the business advantages of iterative, incremental, dynamic design in a collaborative, open atmosphere of continuous improve-ment. Knowledge workers, from the chief information officer (CIO) to the pair of programmers huddling over a terminal, want to move beyond fantasy strategies that will never be implemented, beyond fantasy specifications that the organization doesn’t need, won’t use, and can’t afford, beyond arbitrary estimates and schedules that guarantee “death march” projects. The move to agility was a hard-fought battle against an entrenched incumbent, and it’s triumphing only because it works.

Agile appeals to me because it’s rational, and because I evolved to it based on my own experience. But its real attraction is on the human side; it creates a team-based, collaborative, consensus culture, and it promotes the values that I admire. Honesty, openness, intrinsic motivation, robust collaboration, reflection, and improvement; these values seem so obvious from a human perspective, and yet the enterprises we build are often the exact opposite. Siloed, politically charged, hierarchical, and demotivating cultures are still the norm in many enterprises. Agile values of teamwork, incremental refinement, and change readiness are inevitable in our tumultuous business environment. Once executives, their teams, and their customers absorb these ideas, agile will change everything.

5 www.leadingagile.com/2011/01/untangling-adoption-and-transformation/ .

Introduction xvii

Who Is This Book For? Both Jim Highsmith and Mike Cottmeyer, influential agile thinkers, have recently weighed in on the distinction between “doing agile” and “being agile.” As Mike Cottmeyer phrases it,

Agile adoption is about changing the agile doing side of the equation, while transformation is about changing the agile being side of the equation. 5

Highsmith writes passionately and persuasively about the need to take agility beyond methodology for agile to achieve its potential:

Many organizations seem to be stuck at Agile 101, the rule-based approach to Agile (do this, don’t do that) that is a necessary first step towards becoming Agile, but it’s only a first step. To take advantage of the fast-paced responsiveness of a continuous delivery environment, the entire organization—from delivery teams to executive management—needs to embrace the process changes required to respond rapidly, collaborate effectively between development and operations, and embrace an adaptive, exploratory mindset. 6

My ultimate goal as an agile consultant is to help clients evolve from doing agile, with select teams following the process-based methods and techniques of scrum, for example, to being agile, through deep cultural change to the structure, norms, purpose and style of the enterprise. As there is a contrast between adopting agile and transforming to an agile culture, I see a contrast between the agile coach and the agile consultant. This isn’t a value judgment —there’s plenty of need, at the current state of agile adoption, for both.

Coaches are often consultants as well, focusing on the entire enterprise and helping organizations discover the best approach to improving agility across their firm. In this early stage of agile adoption, however, many agile coaches will have limited interaction with the executive team, primarily spending time with the product development teams and product owners, helping them under-stand and adapt to agile practices.

The agile consultant plays that coaching role but, in the ideal scenario, is also working with senior managers to create an adoption framework for agile in the enterprise. The agile coach is often having a methods-and-practices conversation, while the agile consultant should be having a return-on-invest-ment (ROI)-based, strategy-focused engagement across the entire enterprise,

6 https://assets.thoughtworks.com/articles/adaptive-leadership-accelerating-enterprise-agility-jim-highsmith-thoughtworks.pdf .

Introductionxviii

evangelizing the benefits of agility and building support at the sponsor level. Agile consultants are also coaches, working on the nitty-gritty methodologi-cal details with teams and individual contributors if that’s what the enterprise needs, or where it decides to begin. In my ideal scenario, the agile consultant is simultaneously building trust and expertise within the agile teams while advising leaders on the benefits and challenges of evolving to agility across the enterprise.

The methodology focus by many teams and firms in the beginning of their agile journey is understandable; in fact, it’s inevitable. Despite all of the evidence of agile success, clients are skeptical until they see it for themselves. It’s not rea-sonable to expect organizations to jump into a radical new set of behaviors and attitudes without some demonstration of its efficacy and fit. The agile adoption phase is a critical and necessary step on the path to enterprise agility. After all, even with all the hype around agility, in most situations the migration is still driven bottom-up, by teams of developers who have expe-rienced or heard about the speed, transparency, collaboration, and self-man-agement associated with agile methods. “Grassroots” agile sometimes works in the open, exhibiting quick wins to gain support, and sometimes it works in stealth, displaying the outward signs of predictive, traditional waterfall meth-ods. The stealth teams incrementally build the confidence of the organization by reporting and delivering faster.

Existing practices are sticky, company history is sticky, culture is sticky, and hierarchy is sticky. Stickiest of all is personality. Agile evolution requires us to help the enterprise unstick itself from its traditions. It sometimes requires us to help teams and individuals unstick their most fundamental behaviors. The effort to change norms and habits requires much more than change at the tactical level. While the use of a small “tiger team” to introduce agile into the product development function is a good practice, any improvements solely within that team will be undone by the stickiness of “the way things work here.” To make the leap from more efficient IT to a more agile enter-prise, culture, history, and management style must be disrupted. I believe this requires, in addition to methodological expertise, mature consulting skills. Communication, facilitation, negotiation, organizational diagnosis, change man-agement, strong business context skills—all are mandatory to effect evolution rather than mere adoption.

My hope is that, in reading this book, agile coaches, consultants, managers, and practitioners will gain some insight from my experiences and observa-tions. For coaches and consultants, the obvious targets of this work, I’ll point out tips, tricks, and pitfalls from my consulting experience, and describe tac-tics I’ve learned for working through challenges and barriers. For managers, I’ll define the role of leadership in agile organizations and in enterprises as they become agile. I’ll discuss in depth the evangelizing, consensus building, and reassurance that leaders are called upon to provide as organizations gain

Introduction xix

agility. For the stakeholder, such as the client or the product owner, I’ll talk about the profound adjustments in the relationship between developer and client, and the heightened level of participation and collaboration that the agile enterprise expects. And for the practitioner, the developer sitting in a cubicle writing code or designing data structures, we’ll review what it means to be self-directed , self-managed , and self-motivated , and the behaviors that these phrases imply.

The real skill of the agile consultant, or any consultant, is not in her ability to transmit the technical or domain knowledge in which she specializes; it is rather her sensitivity to the culture, history, and personality of the client, and her ability to guide this particular enterprise to the leanest, most efficient business model it can achieve. The tactical elements of agile practice are easy to teach and to learn, but, as with all profound ideas, enterprise agility is a journey. Agile consultants, whether acting as coaches or strategic advisors, are a necessary element of agile at this stage of its development. I believe we’re on the cusp of a torrent of agile evolution, up to the boardroom, that will change organizational culture forever. Like Sherpas guiding climbers, it’s our responsi-bility to thoughtfully and sensitively guide our clients to the top.