Tag Archives: Software Development

Is Planning an old idea whose time is up?

In the age of nano attention-spans of people, the tendency and respect for planning things upfront has taken a serious beating. The mainstream logic is to simply “play by the ear” because there are far too many moving parts to be completely accounted for and properly factored-in – and in any case, by the time the plan goes to execution, ground realities would have changed beyond recognition, thereby rendering the plan completely useless by that time. In software development, an inaccurate predictive long-range model such as Waterfall has been replaced by more accurate adaptive short-range Agile methods that solve the line-of-sight problem but don’t address the original problem – that of planning a large project with its own share of uncertainties.

While none of those arguments might be wrong par se, we conveniently ignore the fact that that is the nature of the beast, and we need to understand that planning is not a single-pass 100% process that stays current forever. Further, just because things will eventually change is not a good enough reason to abandon any systematic efforts to understand it, if not to tame it altogether!

Pillar on a railway trackWhile preparing my lecture notes for a recent class on planning, I decided to explore the subject of planning through the ages using various quotations over a period of time. I sorted them in chronological order in order to understand how human mind has evolved the thought process behind planning. What came out from this exercise surprised me – it seems like the values associated with planning are as timeless as the fundamental human nature. The age-old wisdom was that a “stitch in time saves nine” and hence there was a focus that “prevention is better than cure”. Some might argue their world was not so complex and dynamic. I disagree.

There world was highly uncertain, and hence inherently complex. With perhaps 95% of reasons behind mortality unexplainable, with perhaps 95% of scientific events being attributed to the Gods (mostly of the angry variety!) and no protection from animals, predators and even their own tribal enemies, how much more complexity do you need in a day? When faced such levels of uncertainty, the modern man is more likely to live life one day at a time. And yet, the thought process from yesteryears seems to indicate that they didn’t simply abandon long-range planning, on the contrary, they reinforced the values of such long-range planning as the only real way to deal with such uncertainties!

Here are some of the quotations that I discovered during my research:

  • If you are planning for a year, sow rice; if you are planning for a decade, plant trees; if you are planning for a lifetime, educate people – Chinese Proverb
  • A man who does not plan long ahead will find trouble at his door – Confucius (551-479 BC)
  •  Plan for what is difficult while it is easy, do what is great while it is small. The difficult things in this world must be done while they are easy, the greatest things in the world must be done while they are still small. For this reason sages never do what is great, and this is why they achieve greatness – Sun Tzu, Chinese General, The Art of War, 400 BC
  •  Before beginning, plan carefully – Marcus Tulius Cicero, Roman statesman and orator (106-43 BC)
  • Long-range planning works best in the short term – Euripides, poet (480-406 AD)
  • Forewarned, forearmed; to be prepared is half the victory – Miguel de Cervantes Saavedra, Spanish writer and author of Don Quixote (1547-1616). 
  • By failing to prepare, you are preparing to fail – Benjamin Franklin
  • In preparing for battle I have always found that plans are useless, but planning is indispensable – Dwight D. Eisenhower, general and president (1890 – 1969)
  • A plan which succeeds is bold, one which fails is reckless – General Karl von Clauswitz
  • Always plan ahead. It wasn’t raining when Noah built the ark – Richard Cushing, novelist
  • Whatever failures I have known, whatever errors I have committed, whatever follies I have witnessed in private and public life have been the consequence of action without thought – Bernard Baruch, A  stock broker, advisor to presidents Woodrow Wilson, Harry S. Truman, (1870-1965)
  • Those who plan do better than those who do not plan even thou they rarely stick to their plan – Winston Churchill, British Prime Minister
  • In the space of two days I had evolved two plans, wholly distinct, both of which were equally feasible. The point I am trying to bring out is that one does not plan and then try to make circumstances fit those plans. One tries to make plans fit the circumstances – General George Patton (1947).
  • A good plan today is better that a perfect plan tomorrow – George S. Patton
  • Planning is a process of choosing among those many options. If we do not choose to plan, then we choose to have others plan for us – Richard I. Winwood
  • Plans are only good intentions unless they immediately degenerate into hard work – Peter Drucker
  • Prediction is difficult, especially about the future – Yogi Berra, baseball catcher (1925-present)

Another interesting discovery was that planning didn’t mean people simply fell in love with their original plans. On the contrary, there are instances that people advocated not only adapting to changes, but rather incorporating mechanism to accommodate them upfront. Look at some of the quotations on this:

 

  • It’s a bad plan that admits of no modification – Publilius Syrus, Roman slave and poet (circa 100 BC)
  • Observe always that everything is the result of change, and get used to thinking that there is nothing Nature loves so well as to change existing forms and make new ones of them – Marcus Aurelius, emperor of Rome (121-180 AD)
  • It is not the strongest of the species that survive, not the most intelligent, but the one most responsive to change – Charles Darwin, scientist

So, is the concept of ‘planning’ an overrated and old idea who time is up? I don’t think so. It only means that there is more need to refresh our understanding and explore how to evolve long-range planning methods to incorporate short-term changes such that agility is not lost at the tactical level, while near-sight changes don’t create a bullwhip effect on strategic plans.

Do you think inaccurate and uncertain proactive planning should be completely abandoned in favor of a highly accurate and definite reactive ‘planning’ (if that can still be called as ‘planning’, that is)?

Why are we in this mess?

Prior to the industrial age, the world was essentially an agrarian and a trading economy. Production methods were often a craft and top secret, fiercely protected within a family and handed down from a master craftsman to his sons, and with no machinery for mass production, pretty much every product was handmade and unique, perhaps also customized, for its intended user. Industrial revolution made mass production and rapid movement of goods possible, and among other things, catapulted Britain into forefront of global economies. Gutenburg’s printing press was perhaps the first mass production system built by man. Subsequent inventions like harnessing of steam power made railways possible, spinning machines, and other advances in iron founding and chemicals pushed the envelope. However, a lot of these advances were limited to Europe and even more within the UK, which thrived on these advances and became the economic (and imperial) superpower well until the start of twentieth century.

However, by the start of twentieth century, America too woke up to industrial advancements and contributed to some of the most important advancements that still continue to touch our lives. The pioneering work by Eli Whitney on ‘interchangeable parts’ on his now classic cotton gin introduced the concept of modular design, followed up by Frederick Winslow Taylor’s groundbreaking work on scientific management led to the concepts of standard work and division of labor  (even if somewhat questionable and controversial in today’s context) and created the foundation for Henry Ford to envisage a mass production system with a moving assembly line where finished goods could be assembled from standard parts by semi-trained operators, e.g. Ford’s most famous Model T car in black color (“Any customer can have a car painted any colour that he wants so long as it is black”).

In essense, a production run, say in an automative parts production or even a car assembly line, is the repetition of a process that produces similar (or similar-looking) objects. Once the ‘process’ is ‘designed’, the job entails repeating the process till the desired number of objects have been produced. Clearly, the faster you produce those objects, the sooner you can put them up in the market for sale and start getting money. The more you produce, the more you are able to amortize the capex, and get lower per unit price over the long run. Intuitively, if you have to produce objects that are exactly alike in properties, shape, size, color or any other physical attributes achieved by the production process, you can ensure that your production machinery will need to be ‘programmed’ once and re-used several times later. So, if you run the paint shop (which seems to be the largest bottleneck in terms of time in a modern production setup) and need to produce purple colored chasis, once you set the process, stock the paint to desired levels, you are pretty much all set. Now imagine if you have to produce first 200 cars in purple, then next 30 in wine red, then next 70 in pearl white colors. Surely there needs to be some way (manual or otherwise) to alter the production process to suit such job order. Similarly, instead of producting 300 sedans, if you have to produce a mix of automobiles – say, 50 SUVs, 50 compacts, 150 sedans and 50 hybrids, your process will have to be different from the one that just produces 300 sedans in a single production run. While the customer desires options (don’t we all?), the manufacturer incurs additional time, money and resources in creating such options. From the manufacturer’s point of view, producing each piece exactly similar as the last one makes such great economic sense that he can create huge economies of scale made possible by principles of mass production. It simplifies the machine (and machine operations) required in plants, it standardizes the components required, there is no downtime to alter the production process, people don’t have to be retrained every now and then on different type of products, and all this make the entire production process very ‘controllable’ from throughput and quality perspective, and hence highly predictable. Elaborate statistical charts can be created based on prior experience on how much time it takes for a given production run, how much men (and women) and materials are needed to meet a given production target, and what levels of quality can be achieved based on statistical experiences. 

After WWII, an economically-broken Japan trying to rebuild itself too threw its hat in the ring and set out to Europe and US to learn about principles of mass production. For reasons that are well-research and well-chronicled in books starting from “The Machine that Changed the World” by Womack et al, companies like Toyota created a brand-new way of mass production that focused on just-in-time production as opposed to utilizing the production machinary as the way to achieve economices of scale. But we are getting ahead of ourselves here a bit, so let’s just ignore it for a while (because the world would not ‘discover’ these techniques until late 80s or early 90s).

Around the same time, computer science was born, and the earliest of software started to be written. Writing software was an extremely hard job, what with enough complexities of huge (and very costly) machines. Software had to be written in the so-called low-level or machine language and required very high level of cognitive abilities. With large endeavors, software creation soon became a techno-managerial problem involving several dozens or even hundred of people. However, unlike the semi-trained operators in Ford’s factories, these were highly educated and perhaps the first generation of knowledge workers of the digital economy.

At that time, there were perhaps the two best methods to organize men and machines: military was where men were employed in thousands and mass production was where machines were deployed. So, what happened next was only the most logical way to extend the thinking: software was treated as a ‘production’ problem and the techniques of mass production were used to develop a ‘waterfall’ model that allowed for enough (?) time and effort at the start of a project to gather all requirements and do the ‘design’ and then rest of the development was contrued as the ‘production’ and hence principles of mass production could be deployed. To organize men, the traditional command and control model was pereived as the best way to separate decision-making executive from the heavy-lifting workers.

Today we debate and critisize these as the worst things that could have happened to software development, but I don’t want to be unfair to what was done more than five decades back based on the state of art knowledge, tools and practices back then. I am sure people back then also wanted to do the best thing – as do the people today claim :).

This was a short overview of how some of the important advancement had impact on thought process on software development (and there are many more important ones that have influenced our world). In the next article, we will discuss how it actually impacted the software development, both positively and negatively.

Our methodology is 100% pure, our result is another thing!

What is worse then an anarchy ? You might say that is the absolute abyss, but I think blind allegiance is even more dangerous (and that includes following the letter but tweaking the spirit – things like ‘creative accounting‘ or its parallels in every field). Anarchy at least allows for things to become ‘better’ in order to survive – whether it is the idealogy, resistance, or even musclepower, or any other ills (and hopefully at some point, social forces of constructive destruction take over). But in a land where unquestionable compliance and blind allegiance rule the roost, IMNSHO, is like a terminal patient off the ventilator support. When people are on their deathbed, they don’t regret things that they did but much rather the things they did not do!

In project management, life is no less colorful. We have process pundits (read “prescription police”) shouting from the rooftop with a megaphone on how heavens will strike them bone dead with lighting if they ever as much as strayed from the ‘standards’. When projects are being postmortemed, we don’t often ask what or why the project did something that they did, but why they did not do things that they did not do. And quite often, you find answer in the map itself – because the map did not factor-in those conditions that were actually encountered on the terrain, the blind followers just followed the Pied Piper and danced their way into the river of death. What a terrible waste of human talent.

Why do we get stuck with methods so much that our result-orientation takes a back seat ? I think there might be many answers, but some that deserve merit:

We fall in love with ‘our’ methods

Let’s face it – introducing a new methodology in an organization, or a team, is not easy. People have their favorites, and it already is a fairly serious political game to get such groups to a decision (whether or not there is a consensus). After you have sold the executive sponsor to go for a particular methodology, you have to now somehow sell to middle-managers and the teams – without their support, you are toast. Sometimes you might have legitimate power (“position power”) to thrust the decision down their unwilling throats, but most often that doesn’t work anymore (even in jails!). So, you have to use your charm offensive to somehow make the other stakeholders believe that this methodology is the god’s greatest gift to mankind, and how it will take their next project to unprecendented glories. If only that were half-true! What follows next is a qunitessentially Bollywood storyline – the project is in tatters and the project manager and teams are exhausted and some even ready to bail-out. You go and put some more pressure on them, ask for weekly reports to be given on daily basis and take away even more productive hours into endless meetings to discuss yesterday’s weather. In short, we do everything but realize that our methods might be wrong…perhaps…for a change? Most people don’t get it, but sometimes, the smarter ones do get it. But by then, they are so much neck-deep in love with their methods that they can’t afford to take most obvious and the only correct decision – take a U turn, and admit their mistakes. Our professional ego checkmates us. Falling in a bad relation is bad, staying in it even after realizing one’s folly is worse.

We believe blindly in marketing claims

These days, there are dozens of marketing gurus, each extolling the virtues of their methods. In a way, it is like entering a village market – each vendor selling the same portfolio of vegetables, but each has to ‘differentiate’ from his neighboring vendor, and hence some make wild performance claims, quite often unbacked by any reliable data, and almost always trusted by gullible buyers at face value. So, some of us just go by the tall claims in glossy marketing brochures. You don’t believe me – tell me, how else you can explain those knowledgeable investors getting duped by Madoff’s elaborate Ponzi scheme in this age and day! Tell me how most banks fell for easy money in the subprime market. And if we were to assume, for a moment, that everything these marketing brochures said was indeed true, how come rest of the world had not quite figured it out already ? I think intelligence is not the elite preserve of a learned few – the rest of us lesser mortals also deserve to be treated with some professional respect that we can figure out what works and what doesn’t. In a previous blog, I once did an interesting analysis of one such marketing claim Blame your flaccid developers and your flaccid customers for your poor quality products !.

We are are too scared to experiment and take sensible risks

This is a real classic. Many organizations, certainly many more than we think there are, are so stuck in ‘command and control’ that they create a system where compliance is rewarded and creativity is shunned. This might work well in some industries or companies, but certainly doesn’t work for most of us. A beauty about such companies is that such outright disdain to new ideas might not just start and end at the top – depending on how deep-rooted the indoctrination is, it might be running in every employee at all levels. In the zest to display unflinching loyalty and to project oneself as a corporate citizen second to none, many employees (and many managers) simply abort new ideas because that might threaten the ‘status quo’ and makes them look very bad in front of the powers that be.

Because everyone else is doing it

This is groupthink at its best. Just because every Tom, Dick and Harry about town is doing it, I must also adopt (simply continue following mindlessly) the latest management fad. For example, we have all heard of (often unverified) stories how investors in the bay area during the dotcom boom era would only look at business plans that had an India component. Just because everyone is doing offshoring might not be good enough why I should also do it. Similarly, just because everyone I know is into Agile, does that make a strong case for my business also ? I think we should stop and think before succumbing to trade journals that routinely publish such forecasts and doomsday prophecies.

We are looking for easy cookbook solutions

Let’s accept it – some people are just plain lazy, or just too risk averse to do any meaningful research on what works for them. Instead of investing in figuring out what’s best for them, they are looking for some quick wins, some jumpstart, some fast food, some room service, some instant karma. They believe they can learn from other’s mistakes (which is definitely a great way to learn – definitely a lot better than not learning from anyone else’s mistakes!) and sometimes they might be successful, if they have copied the right recipe, but very often, that only results in wrong meal to wrong people at wrong time. What people don’t realize is that every problem is cast in a different mold, and whatever one says, there simply is no way one could airlift a solution and drop on the face of a second problem – in its totality – and expect the solution to work. Similarly, there is no way a cookboo solution might work. For example, Agile was designed around small collocated teams that have so high communication among itself that trust replaces formal contracts and communication replaces documentation – I mean they thrive on such a high-energy environment. But, sadly, simply to prove that Agile can fix any problem in the world, management consultants have stretched (should I say ‘diluted’) those very Agile princinples and they now try to forcefit those methods on a distributed, large team, that often has a heterogeneous ‘salad bowl’ culture than a homogeneous ‘melting pot’. So, you land in the situation that just because it worked for some random cases, some among us just naively believe it could work for our random case too. Does it get any more smarter than this 🙂

We believe compliance leads to repeatable success

Standards are often treated like insurance against catastrophes – both the termites and the tornadoes types (think of tornadoes as something that just comes out of the left field and kills a project overnight – like a meteor hits a neighborhood, and think of termites as slow and steady erosion that goes on and on and goes undetected until the time the damage is complete, comprehensive and beyong damage control). They ensure that no one deviates from the beaten track and tries something adventerous (without getting rapped on the knuckles), because that could lead to unpredictable results. And we all look for standards and certifications – ISO-this and ISO-that, CMM and CMMI (and the sunsequent obsession with all other 5-level models), Scrum, PMP, PRINCE2, MBA, PhD, IEEE, …so much so that even in the field of creative arts, there are schools that specify what creativity is allowed and what is out! There are success formulas for Bollywood movies – rich girl meets poor boy and the anti-social forces strike boy’s family. Eventually, our hero saves the day. Similarly, in Hollywood movies, it has to be the hero saving the nation from external threat (very often, coming from the space). In software development, ISO championed the cause of non-negotiable compliance and blind CMM practitioners only perfected it. Agilists were born out of the promise to create a world free of compliance, but it seems they have also ended up growing their tribes with their own mini-rules that give an instant gratification by useless things like Scrum Test to massage one’s ego that my Scrum is more Agile than your Scrum! Do, if you score a certain score in the Scrum Test, you are almost ‘guaranteed’ to get some x level of productivity improvement! Does that remind you of yesteryears, or does it make the future look more scarier?

Conclusion

Human behavior never ceases to amaze. For every one rational thinker, the world has to produce thousands of blind followers. Our schools and colleges teach us to learn the knowledge but they don’t always teach us how to convert it into wisdom. So, when we reach workplace, we are deeply apprehensive of trying out new stuff. We are excellent followers, but simply shudder at the mere thought of questioning status quo. We often behave like the monkey whose hand is stuck in the cookie-jar but refuses to release the cookies even when it knows that the only way he can extricate his hand out of the jar is without cookies. When those workplaces ignore result-orientation and only worship the compliance, the story only gets murkier. Think of a state where compliance is handsomely rewarded and questioning it seen as full and frontal attack, and its timid citizens are only too happy to oblige. They think a lifetime of blind obedience to methodology is far more superior than a moment of experimentation, even if leads to bad results.

After all…our methodology is 100% pure, our result is another thing!

(Inspired by a slogan on a tee: Our vodka is 90% pure, our reputation is another thing. very inspiring, indeed :))

Codifying Agile Skills or creating more checklists ?

Did you check out the Agile Skills Projects yet ? It seems to be a new and interesting initiative to “… establish a common baseline of the skills an Agile developer needs to have, including a shared vocabulary and understanding of fundamental practices”.

They talk about Agile Skills Matrix that has seven essential skills, or the Seven Pillars, organized into five skill levels.

Seven Pillars include:

  1. Product: Agile teams must share a vision of what they are working to achieve; the business problem they are trying to solve.  When a developer understands the problem domain, they can help the Product Owner evaluate upcoming features.  Understanding of the product is the first step to creating a fully cross-functional team.

  2. Collaboration: Teamwork is the heart of Agile software development.  A truly collaborative team shares all its information freely.  They share their individual knowledge and skills by working closely with their teammates.  Teammates who fully rely on one another become much more effective as a whole, yet simultaneously increase individual skill levels.

  3. Business Value: The purpose of any software development effort is to create features that deliver business value.  A good Agile developer will focus on delivering that value.  They do not add complexity just to make their program ‘cooler’.  Instead they will base their work on business priority.  They produce a steady stream of running, tested software with real business value.  They will build only what is needed to solve today’s problem, knowing that building too little or too much is waste. 

  4. Supportive Culture: Any highly productive enterprise is founded in learning. Practitioners in an effective agile team view everything they do as an opportunity to learn. Every step is an experiment intended to make real progress, and to clear our vision of what to do next. We accept and embrace the fact that when a task is done, we’ll see better what we should have done. That helps us see what to do next.

  5. Confidence: Those on an Agile project strive to know the state of their code and the state of their project.  They do not share their work until they can prove it functions properly and is well designed and implemented.  They report progress based upon the actual rate at which fully tested features of real business value are being created.

  6. Technical Excellence: Developers understand, and choose from, many possible technical ways to satisfy business needs–choices that reflect a craft that balances design, use, and support.  They provide the technical underpinnings that enable us always to move forward at a steady pace.  They do this using principles of truly simple design, combined with a grasp of technical debt and the means to keep it under control. They use the best techniques for keeping the design under control without excessive work or rework.

  7. Self Improvement: An Agile team member seeks new ways of doing things, while keeping existing skills as sharp as possible.  They know that ours is an ever changing world and they strive to be prepared to take advantage of anything new.  They are both introspective and aware of what’s going on around them, be it in the team, or the larger business context. They will take action to fix things that aren’t right, and to help those who are in trouble. 

And the skill levels are:

  1. Learning: Individuals at this level have been exposed to a skill, but do not yet have firsthand experience with it.  This may come through reading or conversation, through a formal class or casual presentation at a local user’s group. A training course should provide at least this level of attainment.

  2. Practitioner: At this level, one has practiced the skill in a ‘safe’ environment.  They may have taken a course with hands-on exercises, or recreated the examples from a book.  The Novice level represents the first big step from abstract to concrete.  A wise organization will not let a Practitioner loose on their own.  They do not yet know what they do not know. They do not yet know what they do not know how to do. A training course with a sufficient hands-on component could provide this level of attainment.

  3. Journeyman: A Journeyman is known to possess the specific skill.  They have demonstrated a practical knowledge of the skill in various environments.  They can work on their own, or to increase the competence of a team.  A Team member of a lower skill level will learn from them, a team member of a higher skill level will recognize their expertise. One cannot be taught to be a Journeyman, one can only learn it.  Regardless of the skill category, maintaining Journeyman status, one must practice their current techniques and seek out new and better ones.

  4. Master: A Master must possess unquestioned competence in the particular skill.  They know it cold; it is second nature to them. But, it does not stop there.  To be accepted as a Master, they must also accept the additional job of bringing up the skill level of their team mates.  They do this in several ways. They serve as an example of proper practice.  If you want to know how it is done, observe a Master.  They partner with other team members and actively share their knowledge and experience. They seek out teachable moments.  When asked a question, they prefer to engage in a dialog that will lead the questioner to an answer, rather than making a pronouncement.

  5. Contributor: To be a Contributor, one must have added to our community’s understanding of how to practice our craft.   Contributors bring new ideas to light.  They develop and evaluate new techniques.  They are the silverbacks and young turks, who advance the state of the art.  They may be seen as today’s heretics.

I think this is a good effort to break-down the craft into little more tangible traits that developers need to acquire, thus making the body of knowledge available, one step at a time. Further, it is also refreshing to see things other than technical excellence being part of the Seven Pillars. I have long believed that Sociology should be part of the computer science curriculum now that software development is a serious team effort, and eventual success being largely influenced by human dynamics in the team than individual programming effort alone, howsoever supreme and important that might be.

However, all good things have a shelf life, and all good ideas become fertile grounds for next set of consulting and assessment industry to take-off (and Agile Skills Project makes no secret of its desire to support any effort to certify developers, even though they won’t officially endorse nor condemn any particular certification). I suspect it won’t be long before people will be certified on this (yet another!) 5-level scale of individual proficiency. Look at how successfully we have reduced Scrum to a series of checklist items in the so-called Nokia Test. Then there is this perpetual debate raging in the Agile / Scrum community on merits and demerits of certification, with everyone sort of nodding their heads that certification can’t guarantee competence, but even then everyone continues to offer some form of certification. I even suspect this might soon find its way into job descriptions. However, my bigger question is: will having a bunch of certified developers on your next Agile team ensure, or at least improve chances of success ? Are we unwittingly creating a class-hierarchy in Agile teams where developers will have to be of a certain class (or caste, if you like that way) to be part of the team ? How does one judge things like ‘Confidence’ or ‘Self Improvement’ ? Does high rating guarantee domain competence, or portability of skills? What is better in a 5-people team: 5 Masters, or 3 Masters and 2 Practitioners? Does someone at the skill level ‘Learning’ stand a chance to be in the team of ‘Contributors’ without adversely impacting the team productivity? Further, this is by no means an exhaustive checklist of critical success factors, and hence, chances are that people will fill up some checklists and declare themselves some 86% compliant (on some arbitrary and controversial scale) and their managers will expect the next project to be a resounding success. However, you and I know better than that :).

I think these are highly contentious issues that could further create more checklists and thus degenerate a good intent into a big mess.

The best way to use this tool will be to self-use it and set personal targets to acquire higher levels of competency as one goes by. Any effort to use this as a yardstick to measure and compare individuals is, IMHO, against the basic ethos of a human endeavor, let alone software development. So, shun the checklist-driven software development. 

How Lean thinking improves productivity in software teams?

At its core, productivity for a software team (often wrongly termed as programming productivity because software development is much more than mere programming) looks like a great idea. In its simplest form, it compares output of the team (amount of useful and usable software created, amount of unnecessary rework done, number of defects produced, etc.) to the input (time, effort and resources invested in producing software) factored-in by the uniqueness of the given software  endeavor and other external environmental factors (complexity of the software being produced, impact due to team sizes, nature of team spread and its familiarity with the problem at hand, problem domain, and so on). Unless you are willing to discount software development as a non-business critical activity undertaken purely for the labor of love that should not be ‘measured’ lest it dilutes the very ethos of software development as a creative cognitive endeavor, one might agree, albeit reluctantly, that some measure of how well we are doing is clearly in order. Can you think of a single business activity where such a measure should not be done?

In last few decades, researchers and statisticians have tried to create quantitative measures of productivity. However, due to an often unilaterally percieved inherent ‘uniqueness’ of every software endeavor, practitioners have consistently rejected such measures contending that there are either far too many assumptions in its definition and far too many variations in practice, or both. Instead of focusing on the ‘vital few’ commonalities, it is only unfortunate that practitioners have virtually rejected productivity measures in all forms by considering the ‘trivial many’ differences. What could have gained the industry by considering a big-grain definition of productivity was unfortunately lost because of focusing on decimal-digit precision of productivity measures.

In this blog, I will discuss the subject of productivity from a conceptual plane, and explore how Lean thinking helps us understand it little better by establishing a correlation between wastes in software development and productivity of the team. Lean thinking is much more than identification and elimination of the seven types of wastes, but I will only take up the issue of wastes in this blog.

In the simplest form, we can call a process or an activity productive if it creates an output higher than its input. Though the input and output are often in two different and inequitable currencies, we are still mostly able to compare them. What that means at a conceptual level is that:

  • If Output < Input, we can say that the productivity is low
  • If Output > Input, we can say that the productuvuty is high
  • If rework is high, we can say that the productivity is low
  • If rework is low, we can say that the productivity is high 

These are highly generic views of productivity that most of us understand, but unfortunately they are not very usable to understand the factors that impact an output to become low, or the rework to become high. Developers might give one view, and testers might give another view as to why the output is low. There is no common vocabulary that can explain at a fundamental level what is it that is causing the productivity to dip. Context-specific variations (like differences in programming environment, percieved differences in complexities, developer competency, tester efficiency, etc.) might be too many to be usable to form any meaningful conclusions.

Lean thinking offers a more specific understanding into the wastes that typically make a process costlier (in terms of time or effort or both) than what it ought to be. Lean identifies seven types of wastes in a manufacturing setup. These wastes are known to introduce avoidable wastes, and Toyota pioneered a production system and an organizational culture that seeks to sysmetically eliminate root causes behind these wastes. Software practitioners, led by Mary and Tom Poppendieck, have investigated borrowing from Lean thinking into software development in last couple of years. Let’s examine how Lean encourages eliminations of following seven types of wastes in software development, and how they individually impact productivity of a team:

  1. Overproduction: Implementing a feature is an irreversible decision that takes lots of time and effort from the product engineering team, and delivery of a feature must wait until the next release cycle (whether the big-bang release in conventional software development, or the next real shippable release in agile development). If there are extra features in a software that remain unused because they are not of high priority but the engineering team is anyway spending effort to develop and test it, it is probably lowering the productivity of the team that could have used its effort elsewhere to develop other important features. The customer is also not benefiting from such overproduction of features, because he not only must wait for the desired features to be delivered, he must also pay the delivery team for those unimportant features that are not being used.
  2. Inventory: Anything that has not yet been delivered to the customer is work-in-progress (WIP) or inventory. The higher the inventory in a system, the higher is the number of resources locked-up and unavailable for any other activity. Even the work that has been completed but still sitting on the shelf is an inventory that is not creating any tangible value either to the team or the customer. If there are too many requirements yet to be implemented, it involves doing requirements analysis, design, development and testing without any of that effort being delivered to the customer, and hence increasing the work in progress and thus lowering the productivity at a business level. Even if a critical feature has beem completed, but it waiting simply because several other not-so-important features must be included alongwith in the next release, it is also adding to inventory because that important feature is not immediately delivered to the customer.
  3. Extra Processing: If the software is implemented in a complex manner that requires extra processing (e.g., a design that requires more storage, or duplicates data or code needlessly, etc.), it is either increasing the effort required to develop and test the software, or increasing the usability complexity, or both. Either ways, it is lowering the productivity. However, the opposite of extra processing is not smart programming, which might solve the problem in short run, but create maintenance problems in the long run.
  4. Motion: If the information is not readily available to the engineer, he/she has to spend unnecessary and avoidable time and effort that it takes to fetch the required information (which could anything – clarification about requirements, latest design specs, or locating the correct module from another team for integration, etc.), thus effectively lowering the productivity.
  5. Defects: Any defect that is passed on to the customers requires the engineering team to once again open up the code and spend effort to fix the problem. Since no new output is generated in this process, and the only output is fixing the functionality that should not have been broken in the first place, the net productivity over the entire product engineering lifecycle gets impacted adversely.
  6. Waiting: Perhaps no other project resource is as irrecoverable as lost time. Brooke’s Law reminds us how difficult, if not impossible, it is to recover delay from a project. Waiting for any resource for any reason creates unpredictability in software development apart from creating activities of zero or low output, which again reduces the productivity.
  7. Transportation: Handoffs are perhaps a Fordism legacy to the waterfall development where a team of business analysts would complete their work in isolation and handover the specs to dev team. Dev teams in turn would work in isolation and pass on the software to test team for testing, and so on. If all these teams are working in silos, such practice of working in isolation would often result in information distortion (or loss) down the stream, which could result in missed requirements, misunderstood requirements, unclear design, and so on. All these will eventually require extra (and unplanned) effort to implement the right functionality or fix the broken functionality, thus lowering net productivity over the entire engineering lifecycle.

 

As we can see, analyzing wastes in software development using the Lean thinking provides us a technology-independent and function-neutral view of the inefficiencies of the software process that can be used to achieve substantial improvements in productivity of the teams. None of these ‘attack’ an individual for lower productivity, and only look at the systemic deficiencies that must be eliminated for a team to perform at higher level.  However, of what possible business value would be the process of elimination of such wastes if we can’t picture ‘before’ and ‘after’ ? It is extremely important to assess the state before and after introducing a change – how else do we know if our efforts are yielding the desired results ? So, measuring the productivity in some absolute numbers is like the feedback whether an improvement is happening in the right direction (or not).

Lean thinking or not, do you measure productivity of your software teams? If yes, what for, and if not, why not?

Some ‘universal truths’ of software development

 

Over the years, I have had the good fortune of stumbling upon several universal truths of software development. that have stood the test of time. While some of them were gratefully borrowed from other more competent professionals, several of them have been earned first-hand 🙂

I offer them here for your critique:

  1. Good methodologies  are never at crossroads
  2. “One size, fit all” doesn’t fit any size
  3. Every good thing has a shelf life – and everything was good once
  4. Good engineers and great teams make a bad reference point for future estimations
  5. There are no pre-conditions to performance – especially when you are a manager
  6. People don’t just want to make good software – they also want to build a career alongside
  7. Nothing is new – especially the outside world is far more evolved than we believe
  8. A poor workman blames his tools – and a fool with a tool is still a fool
  9. Software development is not the goal – solving the problem is always the goal
  10. All silver bullets are made of clay
  11. A shaky take-off is better than not starting at all
  12. Best engineers self-train
  13. Project planning is a misnomer – but do it anyway
  14. Leadership is just another name for the response to a stimuli – and hence nobody has a monopoly over it
  15. Not everyone constantly working overtime might be a project’s best friend
  16. Most crisis managers were responsible for that snafu in the first place
  17. Invest in rookies – they will surprise you and everyone else

What is the Inventory of your Software Development project ?

 

Traditional software development follows a quintessential Waterfall model. Among its many limitations, originally discussed by Winston Royce in his 1970 classic paper “Managing the Development of Large Software Systems” and many more thereafter, it also forces a build-up of ‘work-in progress’, or inventory of unfinished work that has not yet been delivered to the customer, and even if it has been presented to the customer, it doesn’t add value until the final working software has been delivered. I define ‘value’ here as originally defined by Womack and Jones in their classic ‘Lean Thinking’, “…The critical starting point for lean thinking is value. Value can only be defined by the ultimate customer. And it’s only meaningful when expressed in terms of a specific product (a good or a service, and often both at once), which meets the customer’s needs at a specific price at a specific time”.  

 

Agile software development, as originally envisaged in the Agile Manifesto, aims to address several of the challenges faced in a Waterfall-type of software development by working on shorter delivery cycles using only a subset of requirements at a time. This subset of requirements (“Sprint Backlog” in Scrum) is prioritized by the customer or customer’s representative (“Product Owner”), so at the end of this short delivery cycle (“Sprint”), the customer gets ‘potentially shippable software’ having exactly those features that add ‘value’ to her. Frequent delivery of working software also has an additional impact on reducing ‘inventory’ levels in a software project, however, Agile doesn’t specifically address how is improves it over traditional software development. We can use concepts from Lean manufacturing to understand and explain inventory build-up in a software development, and how Agile practices help us manage them better compared to the traditional software development practices.

 

Little’s Law states that inventory in a process is the multiplication of throughput and the flow-time. In traditional manufacturing, there is a strong emphasis on plant capacity utilization as a core driver in cost management. However, a high plant capacity utilization requires (or rather leads to) high inventory to ensure the production doesn’t slow down for want of raw materials. High inventory in turn leads to a low inventory turnover, signifying poor sales, thus having high economic implications. Inventory is also identified as one of the seven wastes in Lean. I have discussed the concept of inventory in a typical manufacturing process and how Little’s Law can be used to analyze mathematical relationship between inventory, manufacturing lead time and cycle time in my article “Applying Little’s Law to Agile Project Management – part 1”.

 

In software development, inventory can be thought of as the development costs that continue to be incurred while the development activities are underway. Only when a successful delivery has been made and accepted by the customer, the development team can realize its developments costs, and free-up the ‘inventory’. Until such time, this ‘inventory’ accumulates and remains locked-up.

 

In traditional software development, there is no concept of interim releases to the customer. Some teams might provision for a proof of concept or a prototype, but most often, they are only meant to work under ‘test conditions’, meaning they are not fit for deployment in real world. At best, they give a feel of what it likely to come out of the development lab at the end of a generally long development cycle. In that long cycle, it is possible that some requirements become obsolete, or get clubbed with some other requirements, or just get de-prioritized. In effect, a ‘BRUF’ approach often might end up the team with a software built to original specs, but unfortunately for them, the customer’s view of the world has changed while they were busy developing the software in isolation. So, the ‘effort’ that doesn’t add to ‘value’ also becomes part of an already bloated inventory of development costs. Mary and Tom Poppendieck have discussed the subject of ‘inventory’ in software development in great details in their seminal work on Lean Software Development. They have taken a much wider view of the inventory, while I have limited my discussion in this article only to the development costs.

 

When we use Agile practices, we work on a prioritized subset of requirements for each increment, and at the end of each such increment, we deliver a working software (not a prototype) to the customer. The customer is in immediate position to test-drive the delivery and come back what any necessary changes that could be incorporated in the very next increment. At the end of each such successful increment that delivers ‘value’ to the customer, it is possible for the development teams to realize their development costs, thereby resetting the inventory levels to zero before starting the next increment. I have discussed this scenario taking a near-real-life example in my featured paper “Applying Little’s Law to Agile Project Management – part 2”.

 

Conclusions

Agile software development is a radical approach to improve productivity in software development teams while improving the value delivered to the customer by using shorter delivery cycles (and a lot of engineering practices, though not discussed in this article). Using Agile practices, it is possible for the development organization to create ‘value’ for the customer and also manage its own inventory levels. This is true for ISVs, or services companies working for a customer. It is also equally applicable for start-ups that often work for a prolonged gestation period in a ‘stealth mode’ and their only performance metric during that phase being the notorious ‘cash burn rate’. In today’s tough economic times, virtually every firm is trying to reduce its cash cycles, and Agile software development just provides one more financial incentive by reducing the inventory and shortening the cash cycle. 

 

 

 

 

Is Scrum serving your Software Development, or the other way round ?

There was a question on the group Scrum Practitioners on LinkedIn if “…implementing Scrum as a whole should be our goal or would you use aspects of the Scrum methodology to realise an agile culture change ?”

Looking at the disproportionately large number (with an increasing trend) of posts on popular mailing lists on Scrum and Agile software development, I am alarmed that most energy and thought is being spent on figuring out “how Agile you are”, how should the user stories be worded, and whether one uses planning poker or not ? I mean…does it really matter whether you use ‘deal hours’ or ‘story points’ whatever that means ? If your team members are not speaking out ‘impediments’ in daily scrum…come on…that was the case in good old days also…expecting that Scrumifying the process will make everyone speak up honestly, do their job on-time and love thy neighbors is little too much of wishful thinking in my honest, brutal and uncharitable view. 

Scrum exists to serve its master, which is Software Development and not the other way round. So, implementing Scrum can never be the goal by itself (isn’t that where we went wrong with the CMMs and the ISO9000s of the world ?). Timely delivery of on-specs and high-quality software is almost always the goal. Anything that helps in that direction is a only a means towards that goal.

I would take whatever is applicable and makes sense to MY business situation – which is a unique combination of:

  • my organizational maturity +
  • organizational culture +
  • type of problem being solved +
  • management style +
  • … +
  • another one hundred and one reasons that must be analyzed before prescribing a one-size-fit-all solution.

To that end, I will not stop at Scrum but explore other Agile methods, Six Sigma practices, Lean principles, Kaizen philosophy, TOC / Critical Chain..even borrow from CMM / CMMI and ISO…and…why not…the good old Waterfall (sometimes, the old wisdom is not so bad after all !)….just about anything under the sun that makes my business look better. In fact, for more complex problems, I might look up to other industries like construction, shipping, logistics, etc. to see how they solve such problems and learn from there. After all, my business is far more important than an academic satisfaction of having implemented a one hundred percent bookish definition of Scrum without knowing whether it is working for me or not, and completely depriving myself an opportunity to know if there are better ways to solve my problems.

Scrum doesn’t have the monopoly over good software development practices. If we all started blindly following Scrum in toto, there won’t be any better solutions to newer problems. There must be a healthy effort to question Scrum practices and make continuous improvements in it instead of treating it like holy scriptures which can’t be challenged.

My prescription for myScrum test would not to measure it by inputs (i.e., do you do 2 week or 4 week sprints, are your specification complete before the next iteration, etc.) but as follows:

  1. Are your new practices improving employee motivation ?
  2. Are your new practices improving team productivity ?
  3. Is frequent software delivery improving customer satisfaction ?
  4. Are you able to accommodate late requirements with increasing ease ?

I think if you can answer all of these questions in affirmative, you have a winner in your lap – doesn’t matter if it doesn’t measure up to the ‘stringent’ criteria of Nokia Test or some other activity-based assessment of your software development religion. You don’t even need to call it Agile or Scrum…just call it Photosynthesis, Metamorphism, Evolution, Oxidation…whatever you feel like…after all, your Scrum should serve your software development needs, rather than you serving Scrum.

Is Scrum serving your Software Development needs, or the other way round ?

When you still can’t develop quality software from Requirement Specs, how do you build it right using Test Specs ?

Test-Driven Development (TDD) is the new silver bullet in town. While several ‘obvious’ advantages are being cited in several articles, I fail to understand one simple fact: when we have not yet quite figured out how to develop good-quality software from requirement specs, how do you really use TDD to develop it using Test Specs ?

Bertrand Meyer has a point in case in a recent article, “Seven Principles of Software Testing” in IEEE Computer, August 2008:

…Test-driven development, given prominence by agile methods, has brought tests to the center stage, but sometimes with the seeming implication that tests can be a substitute for specifications. They cannot. Tests, even a million of them, are instances; they miss the abstraction that only a specification can provide.

While there might be several ‘right’ situations where TDD might cut down the effort and time to test, especially regression test, a software, I refuse to believe it is the panacea for all maladies. For one, if a test case passes, there could be two scenarios:

1. the software behaved correctly as per the specs

2. the test case was wrongly designed and did not quite check the functionality as per the specs

Any student of reliability engineering is familiar with Triple Modular Redundancy (TMR): when you use two time clocks, you will always get two different times, and to figure out the correct time, you need an arbiter, or a third clock. In case of software development, we are more known to produce anything faulty than correct, at least in the first try. So, when a test case passes, if is equally possible (if not more) that someone wrote a bad test case than someone writing a correct software ! Why ? Well, I have at least two arguments to support it:

1. when a software engineer can’t write an engineering code 100% correctly, how can you believe his test code will not have statistically similar number of defects ?

2. a software engineer is trained to write good engineering code and not to write good test code. Just like there is a sea of difference between an average developer and a world-class developer, there is also a huge difference between an average developer and world-class tester. Remember, you hired him on his programming skills – if you wanted him to write test code, you probably would have hired someone else !

So, this somehow doesn’t make sense, or it is likely that I have completed misunderstood this subject (more likely, but possible !). Time willing, I plan to write a paper for the upcoming Software Testing conference to explore this subject further. If you have any opinion, experience, anecdotal data, or rocksolid eveidence one way or other, I would love to hear from you.

Meanwhile, stay tuned…