Why Agile doesn’t sell with Management ?

Agile thought movement has been around in different stages of evolution (and not to mention, different vocabulary from different preachers and schools of thought)  from 1960s – perhaps it always co-existed with its better-known but half-effective cousin, Waterfall, all those decades without ever becoming a mainstream / preferred method of solving complex problems. Declaration of Agile Manifesto in 2001 was a watershed event, and the body of literature that has grown since then is simply mindboggling. There is a huge experience base on various Agile methodologies and methods that seems to be gaining firm ground globally every single passing day.

Divide and Rule

There can’t be any doubt that the fundamental idea to approach a problem in smaller pieces is great – it is logical, it is result-oriented, it is economical and it is highly intuitive. After all, the Imperial British conquered half the world by their brilliantly cunning doctrine of ‘divide and rule’. It allows the human mind to focus on what is ‘visible’ here and now as opposed to worrying endlessly about the entire length and breadth of the problem which may never ever be crisply definable, or might require a disporportionately large amount of time (and effort) to clearly define it – by which time it is quite likely that the problem might have undergone changes due to natural ageing or changes in environment around it, or an improved clarity into the problem has given rise to another thought process about its solution, or both. Frankly, none of them is bad in the sense it is always better to know more clear definition of the problem or a more clear definition of the solution. The only problem is that it takes an inordinately large amount of time to reach there, with no mechanism to know if what is being considered will actually work as envisaged. While executing, one could encounter several more unknowns on the way that force changing the laid-out strategy. However, when you slice the problem in meaningful increments, you allow the solution to grow, one slice at a time. Of course, you still need to have the strategy to accomplish the top-level vision – but the working plans must not attempt to address that entire strategy in one single-pass. There must be a mechanism to (a) breakdown a problem into meaningful slices that (b) can be accomplished individually and incrementally to (c) make a definitive progress towards the eventual goals (d) without unduly front-loading the progress by later-day uncertainties. Agile encourages establishing a beachhead and then reinforcing it, slowly and steadily as opposed to landing in the middle of a battlefield and getting fired at from all directions. As a concept, this is splendid.

Sensible Trade-offs

It also allows a sensible trade-off between cost (resources), time and scope of work to be performed – the good old ‘triple constraints’ of any project. A small team could be highly efficient and could be easily composed of experts that don’t have to worry about a lot of non-work related problems that will eventually happen to any larger team. However, a small team can only achieve so much in a given time period. So, if you want to accomplish more work than what a bunch of experts can undertake at a time, you either must be willing to have a larger team to deliver in the same short time period, or allow the highly-efficient team to work for longer period. There is enough historical evidence to suggest that small teams are the best in solving just about any problem that requires intense human interaction and usage of individual human cognitivate skills. However, not every large task could wait for a small team to complete over prolonged time (as also attested to by Fred Brooks in his classic, The Mythical Man-month). Commercial pressures (“time to market” to meet the shopping season and to beat the competitor), rapid obsolesence of technology, fluctuating customer needs in short-term and uncertain preferences over long time, etc. are just some of the market-driven reasons why running a long development-cum-release cycle is often a bad idea. Add to that indirect factors like manpower attrition risks over long-term, uncertain economic situation in the coming quarter, etc. and you have a complete set of risks that have nothing to do with the problem at hand! However, when we work with long-cycle single-pass development, we tend to create a Frankenstien of project management because most such problems will now have to be ‘supported’ just because of choosing a particular problem-solving method! Mind you, most of these were not the problems originally stated as the technical challenges, but we piled them on because of our management decisions. Developing products incrementally allows us to filter out such indirect risks and only focus on core risks related to problem at hand.

Short feedback cycle

Agile promotes ‘doing it now’ over ‘deliberating it forever’ and rely on continuous feedback on working software as a faster and more reliable means to improve software than doing a rather late integration and a back-loaded testing cycle that is always ‘too little, too late’. Agile makes it possible for teams to develop (and deliver) software faster thereby providing early visibility to Customers by way of ‘working software’ and get an early feedback on important aspects like functionality, usability, scalability, workflow, and so on. Customers are happy to see a working software because they are mostly used to documents that don’t mean much to them. Management is happy because the customer is happy, and the teams are happy because they get an immediate feedback on their work that allows them to improve their products. There is clearly a merit in having a short feedback cycle.

Painless Change

Agile handles changes elegantly throughout the development cycle. Short of handling a mid-flight change, it allows just about any change to be introduced at any stage of development. If we believe that it is beyond the human capability to correctly specify every single requirement of a system unambiguously – not just in one go but ever – then this tweak of handling changes on the go makes very smart sense. As we have seen from RUP, the ‘phases’ of a project don’t really mimic engineering activities undertaken at a given point in the project lifecycle – it might as well be the case of shifting weights in each of the phases, but it can never be the case that we could be done with all Requirements in any phase and then move on (if that were the case, Waterfall would be little more effectives, perhaps). Of course, this strategy to handle changes flexibly means that there is never a clear and firm requirements baseline, and hence it might be near impossible to define a fixed-price work contract (or an outsourcing contract), but if the customer and the delivery organization both understand the new way of doing business, it will just come out as good, or better. The point is, a work contract exists to support the project and not to dictate how a project must handle changes. If it is the fundamental nature of the beast, we better have contracts that recognize this hard truth and allow both the parties to engage in a win-win relationship than a contractual relationship that frustrates everyone.

Team Ownership

Traditional teams have long followed the model of ‘division of labor’, or some such variant. In a small team, everyone does everything, but the larger the teams grow, the more is need for ‘vertical differentiation’ and ‘horizontal differentiation’ and it allows people to make decisions for themselves commensurate to their functional skills and their level of competency in them. Again, this is not bad, for we can’t confuse between ‘desire’ and ‘competency’. So, in theory, the traditional distribution of responsibility works well, and perhaps worked well in yesteryears (or maybe it didn’t, but that is not the point here). It certainly doesn’t work anymore. The historical advent of ‘trade unions’ in traditional industrial environment was largely motivated by the desire to participate in management process, and we see similar traits in knowledge industry as well. I think this is a positive thing, and allows ‘workers’ to become better informed, and hence better engaged in decision-making process, have better ownership and accountability of the results. Agile has done tremendous service institutionalizing this so important issue.

Agile improves team ownership of tasks and creates ‘self-organizing’ teams. A team that collectively makes its decision is more likely to own it and support its team members in realizing their individual goals that add up to the team goals. A ‘mutually exclusive and collectively exhaustive’ cross-functional team will also be highly interdependent on its members to achieve their shared goals, hence likely to have a heathy mutual respect. All these enhance a team’s sense of pride and ownership. While I strongly believe just a mere adoption of Agile practices is not likely to make teams self-organizing overnight, it still is a tremendous improvement from the traditional management structure.

So then, what is the problem ?

These are great strategies to make software development more meaningful and value-adding to customers, and less painful to developers. Still, how come most companies (read ‘managers’) have not warmed up to the idea, let alone adopted it ? In a recent blog, Allan Kelly says the adoption of Agile is probably in the range of 5%-15%. I would imagine the number more closer to lower bound than upper bound, and even then, I am little sceptic about it real implementation and actual effectiveness. How do we know that the Agile implementation in those companies is not just limited to peripheral pilot projects ? How do we know that when a 10,000 people company says they adopt Agile, they are not saying that only some projects use Agile ? Joe Little maintains list of firms practicing Scrum, clearly the favorite Agile methodology, but there are just 160+ companies in it. Definitely, many companies doing Agile are not listed here, but I also know of companies listed in that list who just pay lip service to Agile. There were also recent discussions on implementations of Agile in large companies in some Agile / Scrum lists, and even though the debate is inconclusive (which is not so relevant), the fact remains that there are real issues in adoption of Agile in larger world. My question is, why is it so?

I think there are several reasons, and in this blog, I will not include the most favorite punching bag 🙂 but rather focus internally. I believe the way Agile buffet has been laid out, it offers considerable challenges that self-limits any further growth, at least in its current avatar. For one, Agile is clearly portrayed as anti-establishment. Now, this is not necessarily bad, but an outright contempt of management, so much so that they are not even wildly considered in the category of ‘chickens’, in my view, makes ‘management’ uninterested in knowing about it any further. If the so-called command and control was one-sided (towards management) and bad, Agile is equally one-sided (towards teams) and equally bad. I am pro-empowerment, so this statement needs to be understood well before I am misunderstood. There is all merit in the team owning up its own decisions and being accountable for its decisions, but they also owe to the management to be answerable and accountable for their decisions, progress and results. In traditional projects, the management had a grip on the overall project scope and schedule, even if there was an element of uncertainty in those commitments. In Agile projects, we took away the big picture, denied the sense of control to management (again, neither good nor bad – just stating the facts as they are) and worse, we are in no better condition to give them a sense of the overall project. Yes, we have a better grip in the current iteration, but are we any better informed with similar margin of errors as before, in predicting the overall projects ? I doubt.

So, if the result of an Agile adoption is that management is not involved, is getting no better commitment from teams on the final project delivery, is kept away from periodic progress, is not allowed to be part of project meetings, and has no say on the work being undertaken or when dropped from a sprint without any prior two-way discussion, I don’t see how that same management is going to bless the adoption of Agile in his teams? If it doesn’t help him having a better confidence about his business’s collective ability to solve problems, howsomuch it might help his teams, I don’t think any human being is going to be enthusiastic about it. So, it is not surprising that the Agile adoption continues to be so low, even when the benefits clearly outweigh the costs of adopting Agile. I think the bottleneck is not management, we are the bottleneck. We have failed to include management as part of the solution team. We just conveniently decided (unilaterally, I must add) that management is the classic villian who must be kept away if the team has to achieve something worthwhile.

I think in our zeal to fix the ‘problem’, we just let the pendulum swing to the other side, and it must eventually settle down somewhere in-between to be of use because that is the real world. Perched on extremes, it has no value because of the extreme positions it must take to stay put there which are not closely related to the real world. I believe that a New Agile will evolve (in my view, it is already there – just that some people are so steadfastly holding on to the remnants of the old Agile) to become a better fit in the real world and include all parts of the workforce as respectable members of the solution team. Smart among us will freely drop the dogmatic aspects of Old Agile and quickly adopt things that work for them, whether prescribed or not.

We already see people experimenting with Lean, Kanban, Scrumban…the smart ones will not limit their potential with someone else’s prescription.

The rest will perish together.

13 thoughts on “Why Agile doesn’t sell with Management ?

  1. Pingback: OnStrategies Perspectives » The Little Method that Might

  2. TV Post author

    @Patrick: I agree that off-late, there are lot of fresh ideas in agile thinking – and that is very encouraging and highly promising for our industry (even though I see it mostly limited on LeanAgile and Kanbandev lists – others are pretty much playing the old music). In my blog too, I have also alluded to the fact that there are people among us who are learning from other disciplines and improving their toolkit -in my view, they will thrive.

  3. Patrick Wilson-Welsh

    I question your premise. In my judgement, your “New Agile” has been here for at least a couple of years. There is certainly a dogmatic old guard, but there is plenty of awesome value-oriented, business-oriented, pragmatic practice blending going on.

    In my consulting work, I work more and more with managers who think of agility as very pro-business, pro-value-flow, pro-management. And some of these managers are at the C-Level. When we go in, we go in at a high level, and we immediately start talking — not about development practices — but about value flow, ROI, codebase asset retention, muda. And yes, pretty soon we are mowing down cube walls and putting together real teams. But not before we have solid support from people who totally get it, on high. So, agile transformation, top down, bottom up, and side to side.

    The agile community was, at one time, pretty much entirely development oriented. At one old XP Universe, when Kent Beck asked how many of us in the room were developers, pretty much every hand went up. But as far as I can tell now, and with all due respect: that’s old news. And much of this may very well be flying under the radar of those who collect agile adoption and agile pilot specs for websites.

    Check out the stage design and approved track and schedule for Agile 09. There will be more CIO’s and dev managers and “establishment” types there than ever before. In fact, I know devs who complain that that is now too much the case.

  4. TV Post author

    @AkitaOnRails: Thanks for sharing great perspective. I think agile is too good a concept to be left to software teams alone! I mean think about it – of what use could be a brilliant problem-solving idea if it is selectively denied based on pay levels in an organization ? It should really be where the bulk of the problem is – and by denying such tool to management, are we implying that there is no problem with management 🙂 ?

    I find Lean as a more powerful systemic way to bring about sustainable changes with 360 degrees of involvement (thanks for no pigs, thanks for no chickens). My problem with Agile is that it refuses to adapt to the real-world that starts much before software team and ends much beyond software team. By adopting Agile locally, at best we move the constraints out of the team and – worse – move it at the entry / exit points thereby bottlenecking any possible gains we might have made during the software phase. I don’t think this ‘optimization of the parts’ will go a long way if not done in tandem with ‘optimization of the whole’, which is the Lean approach.

  5. TV Post author

    @Jason: Thanks for your insightful comments. I agree with you that management should neither be fearful of Agile adoption, nor be left out. However, I think Agile slogans need to be modified to reflect this fact of life – today it clearly sounds anti-inclusive of management, if not explicitly anti-establishment, and hence an impediment by itself for greater adoption of Agile practices.

  6. TV Post author

    @Tim: I guess that’s how a change is viewed by people most affected by it – with suspicion. Of course, in the context of this topic, the need to make ‘management’ and ‘teams’ see eye to eye and work as allies remains stronger than ever before.

  7. AkitaOnRails

    I too have this problem trying to explain why most companies don’t adopt Agile philosophy if it is so inherently “good”. And I think I know a few things. There are 3 main reasons for that, in my book:

    * First of all is Plato’s “Allegory of the cave” syndrome. Most people have been living in the cave. The shadow is the “real world”. It doesn’t matter if we try to present a more colorful world, they won’t immediately believe it. Worst, most will want to go back to the shadows they think they “understand”.

    * Second, most companies are actually doing financially well in the short term. This is the “Delta T” syndrome. Meaning most people project the current time slice into the future. If they are currently growing, they feel like they will grow continuously, ignoring chaotic dynamics. “Why change if we are doing well?”

    * Third, companies as a whole behave like complex systems. They were born out of chaotic circumstances, the founders didn’t have any boundaries or control mechanisms in the beginning and that’s the reason they were able to survive and grow. Then they want to “professionalize” destroying the very circumstances that allowed them to grow in the first place. Every single management methodology, HR policy and so on are a mean to put everything into “equilibrium”. Companies make efforts to put themselves in the mean, cutting the standard deviation at maximum. This is a complex system striving to keep themselves in a dynamic equilibrium, a zero sum effort.

    So, what they complex systems in dynamic equilibrium need is a chaotic agent, outsiders that will create disorder, taking people out of their current order and work toward a chaotic state. The trick is to put the company in a state which is neither order, nor chaos (adhoc disorder), but what is called “the Edge of Chaos” which many believe is a very maintainable state of a “Learning Organization”, where things keep improving continuously. The Agile philosophy is just the first step, but it is not the end game.

    The end game is to create an Organizational Democracy. Very few companies in the world were able to get into such an state. Look for W.L. Gore, SRC Holdings, Semco.

    Agile projects is just a temporarily good thing, but it won’t last for long if the organization as a whole don’t embrace it. Meaning, it doesn’t matter if a few project teams start to get agile if HR, the board of directors and other back-office stuff don’t walk the same path.

    This is a multi-year effort that really need to encompass the entire organization.

  8. Pingback: Why does Lean/Kanban/Agile/Scrum or “Whatever” work (versus Lean/Kanban/Agile/Scrum or “Whatever-Else”)? « Si Alhir (Sinan Si Alhir)

  9. Jason

    I agree that the pendulum with Agile is swung in the favour of developers and teams. After all the manifesto was developed BY DEVELOPERS. I don’t think management needs to fear agile adoption and I don’t think they need to be excluded from it either. A managers job isn’t to tell the team what to do, it’s to facilitate the learning, remove organization obstacles and help the team work better as a team.

    Management can absolutely have visibility into the scope and project progress, agile makes it easy to create that visibility. Sure managers shouldn’t be controlling and dictating, but their role can certainly change so they can fit into Agile. Maybe most managers evolve into ScrumMasters so they can deal with inpersonal issues and team conflicts. They are certainly better qualified to remove organizational obstacles compared to a Scrum Master who might not have the soft skills or clout to remove them.

  10. Tim Ottinger

    What’s really funny? When I was doing transitions, developers would often tell me that they thought Agile was a trick by managers. They thought it was making programmers work in a way that makes life easier for their bosses.

Leave a Reply