Agile thought movement has been around in different stages of evolution (and not to mention, different vocabulary from different preachers and schools of thought) from 1960s – perhaps it always co-existed with its better-known but half-effective cousin, Waterfall, all those decades without ever becoming a mainstream / preferred method of solving complex problems. Declaration of Agile Manifesto in 2001 was a watershed event, and the body of literature that has grown since then is simply mindboggling. There is a huge experience base on various Agile methodologies and methods that seems to be gaining firm ground globally every single passing day.
Divide and Rule
There can’t be any doubt that the fundamental idea to approach a problem in smaller pieces is great – it is logical, it is result-oriented, it is economical and it is highly intuitive. After all, the Imperial British conquered half the world by their brilliantly cunning doctrine of ‘divide and rule’. It allows the human mind to focus on what is ‘visible’ here and now as opposed to worrying endlessly about the entire length and breadth of the problem which may never ever be crisply definable, or might require a disporportionately large amount of time (and effort) to clearly define it – by which time it is quite likely that the problem might have undergone changes due to natural ageing or changes in environment around it, or an improved clarity into the problem has given rise to another thought process about its solution, or both. Frankly, none of them is bad in the sense it is always better to know more clear definition of the problem or a more clear definition of the solution. The only problem is that it takes an inordinately large amount of time to reach there, with no mechanism to know if what is being considered will actually work as envisaged. While executing, one could encounter several more unknowns on the way that force changing the laid-out strategy. However, when you slice the problem in meaningful increments, you allow the solution to grow, one slice at a time. Of course, you still need to have the strategy to accomplish the top-level vision – but the working plans must not attempt to address that entire strategy in one single-pass. There must be a mechanism to (a) breakdown a problem into meaningful slices that (b) can be accomplished individually and incrementally to (c) make a definitive progress towards the eventual goals (d) without unduly front-loading the progress by later-day uncertainties. Agile encourages establishing a beachhead and then reinforcing it, slowly and steadily as opposed to landing in the middle of a battlefield and getting fired at from all directions. As a concept, this is splendid.
It also allows a sensible trade-off between cost (resources), time and scope of work to be performed – the good old ‘triple constraints’ of any project. A small team could be highly efficient and could be easily composed of experts that don’t have to worry about a lot of non-work related problems that will eventually happen to any larger team. However, a small team can only achieve so much in a given time period. So, if you want to accomplish more work than what a bunch of experts can undertake at a time, you either must be willing to have a larger team to deliver in the same short time period, or allow the highly-efficient team to work for longer period. There is enough historical evidence to suggest that small teams are the best in solving just about any problem that requires intense human interaction and usage of individual human cognitivate skills. However, not every large task could wait for a small team to complete over prolonged time (as also attested to by Fred Brooks in his classic, The Mythical Man-month). Commercial pressures (“time to market” to meet the shopping season and to beat the competitor), rapid obsolesence of technology, fluctuating customer needs in short-term and uncertain preferences over long time, etc. are just some of the market-driven reasons why running a long development-cum-release cycle is often a bad idea. Add to that indirect factors like manpower attrition risks over long-term, uncertain economic situation in the coming quarter, etc. and you have a complete set of risks that have nothing to do with the problem at hand! However, when we work with long-cycle single-pass development, we tend to create a Frankenstien of project management because most such problems will now have to be ‘supported’ just because of choosing a particular problem-solving method! Mind you, most of these were not the problems originally stated as the technical challenges, but we piled them on because of our management decisions. Developing products incrementally allows us to filter out such indirect risks and only focus on core risks related to problem at hand.
Short feedback cycle
Agile promotes ‘doing it now’ over ‘deliberating it forever’ and rely on continuous feedback on working software as a faster and more reliable means to improve software than doing a rather late integration and a back-loaded testing cycle that is always ‘too little, too late’. Agile makes it possible for teams to develop (and deliver) software faster thereby providing early visibility to Customers by way of ‘working software’ and get an early feedback on important aspects like functionality, usability, scalability, workflow, and so on. Customers are happy to see a working software because they are mostly used to documents that don’t mean much to them. Management is happy because the customer is happy, and the teams are happy because they get an immediate feedback on their work that allows them to improve their products. There is clearly a merit in having a short feedback cycle.
Agile handles changes elegantly throughout the development cycle. Short of handling a mid-flight change, it allows just about any change to be introduced at any stage of development. If we believe that it is beyond the human capability to correctly specify every single requirement of a system unambiguously – not just in one go but ever – then this tweak of handling changes on the go makes very smart sense. As we have seen from RUP, the ‘phases’ of a project don’t really mimic engineering activities undertaken at a given point in the project lifecycle – it might as well be the case of shifting weights in each of the phases, but it can never be the case that we could be done with all Requirements in any phase and then move on (if that were the case, Waterfall would be little more effectives, perhaps). Of course, this strategy to handle changes flexibly means that there is never a clear and firm requirements baseline, and hence it might be near impossible to define a fixed-price work contract (or an outsourcing contract), but if the customer and the delivery organization both understand the new way of doing business, it will just come out as good, or better. The point is, a work contract exists to support the project and not to dictate how a project must handle changes. If it is the fundamental nature of the beast, we better have contracts that recognize this hard truth and allow both the parties to engage in a win-win relationship than a contractual relationship that frustrates everyone.
Traditional teams have long followed the model of ‘division of labor’, or some such variant. In a small team, everyone does everything, but the larger the teams grow, the more is need for ‘vertical differentiation’ and ‘horizontal differentiation’ and it allows people to make decisions for themselves commensurate to their functional skills andÂ their level of competency in them. Again, this is not bad, for we can’t confuse between ‘desire’ and ‘competency’. So, in theory, the traditional distribution of responsibility works well, and perhaps worked well in yesteryears (or maybe it didn’t, but that is not the point here). It certainly doesn’t work anymore. The historical advent of ‘trade unions’ in traditional industrial environment was largely motivated by the desire to participate in management process, and we see similar traits in knowledge industry as well. I think this is a positive thing, and allows ‘workers’ to become better informed, and hence better engaged in decision-making process, have better ownership and accountability of the results. Agile has done tremendous service institutionalizing this so important issue.
Agile improves team ownership of tasks and creates ‘self-organizing’ teams. A team that collectively makes its decision is more likely to own it and support its team members in realizing their individual goals that add up to the team goals. A ‘mutually exclusive and collectively exhaustive’ cross-functional team will also be highly interdependent on its members to achieve their shared goals, hence likely to have a heathyÂ mutual respect. All these enhance a team’s sense ofÂ pride and ownership.Â While I strongly believe just a mere adoption of Agile practices is not likely to make teams self-organizing overnight, it still is a tremendous improvement fromÂ the traditionalÂ management structure.
So then, what is the problem ?
These are great strategies to make software development more meaningful and value-adding to customers, and less painful to developers. Still, how come most companies (read ‘managers’) have not warmed up to the idea, let alone adopted it ? In a recent blog, Allan Kelly says the adoption of Agile is probably in the range of 5%-15%. I would imagine the number more closer to lower bound than upper bound, and even then, I am little sceptic about it real implementation and actual effectiveness. How do we know that the Agile implementation in those companies is not just limited to peripheral pilot projects ? How do we know that when a 10,000 people company says they adopt Agile, they are not saying that only some projects use Agile ? Joe Little maintains list of firms practicing Scrum, clearly the favorite Agile methodology, but there are just 160+ companies in it. Definitely, many companies doing Agile are not listed here, but I also know of companies listed in that list who just pay lip service to Agile. There were also recent discussions on implementations of Agile in large companies in some Agile / Scrum lists, and even though the debate is inconclusive (which is not so relevant), the fact remains that there are real issues in adoption of Agile in larger world. My question is, why is it so?
I think there are several reasons, and in this blog, I will not include the most favorite punching bag 🙂 but rather focus internally. I believe the way Agile buffet has been laid out, it offers considerable challenges that self-limits any further growth, at least in its current avatar. For one, Agile is clearly portrayed as anti-establishment. Now, this is not necessarily bad, but an outright contempt of management, so much so that they are not even wildly considered in the category of ‘chickens’, in my view, makes ‘management’ uninterested in knowing about it any further. If the so-called command and control was one-sided (towards management) and bad, Agile is equally one-sided (towards teams) and equally bad. I am pro-empowerment, so this statement needs to be understood well before I am misunderstood. There is all merit in the team owning up its own decisions and being accountable for its decisions, but they also owe to the management to be answerable and accountable for their decisions, progress and results. In traditional projects, the management had a grip on the overall project scope and schedule, even if there was an element of uncertainty in those commitments. In Agile projects, we took away the big picture, denied the sense of control to management (again, neither good nor bad – just stating the facts as they are) and worse, we are in no better condition to give them a sense of the overall project. Yes, we have a better grip in the current iteration, but are we any better informed with similar margin of errors as before, in predicting the overall projects ? I doubt.
So, if the result of an Agile adoption is that management is not involved, is getting no better commitment from teams on the final project delivery, is kept away from periodic progress, is not allowed to be part of project meetings, and has no say on the work being undertaken or when dropped from a sprint without any prior two-way discussion, I don’t see how that same management is going to bless the adoption of Agile in his teams? If it doesn’t help him having a better confidence about his business’s collective ability to solve problems, howsomuch it might help his teams, I don’t think any human being is going to be enthusiastic about it. So, it is not surprising that the Agile adoption continues to be so low, even when the benefits clearly outweigh the costs of adopting Agile. I think the bottleneck is not management, we are the bottleneck. We have failed to include management as part of the solution team. We just conveniently decided (unilaterally, I must add) that management is the classic villian who must be kept away if the team has to achieve something worthwhile.
I think in our zeal to fix the ‘problem’, we just let the pendulum swing to the other side, and it must eventually settle down somewhere in-between to be of use because that is the real world. Perched on extremes, it has no value because of the extreme positions it must take to stay put there which are not closely related to the real world. I believe that a New Agile will evolve (in my view, it is already there – just that some people are so steadfastly holding on to the remnants of the old Agile) to become a better fit in the real world and include all parts of the workforce as respectable members of the solution team. Smart among us will freely drop the dogmatic aspects of Old Agile and quickly adopt things that work for them, whether prescribed or not.
We already see people experimenting with Lean, Kanban, Scrumban…the smart ones will not limit their potential with someone else’s prescription.
The rest will perish together.