When the London Garage meet with customers, one of the most frequently asked questions is “how much is my project going to cost?”.
It’s generally accepted that - without a working time machine - at least one corner of the project management triangle needs to be flexible. Trying to nail down scope, cost, and schedule at the beginning of a project is a notorious anti-pattern; delivering a pre-determined list of features on-schedule within a given budget may give the illusion of success, but quality will inevitably suffer. This isn’t malicious or a sign of a bad team; tired people make mistakes.
In the Garage, we’re convinced that time-boxing iterations while keeping scope flexible is the way to go. We work at a sustainable pace, instead of rushing to meet a deadline, injecting lots of exhaustion-induced defects, and then collapsing in a burned-out pile to recover until the next rush. In order to enable this sustainable pace, we can’t commit to a detailed feature list at the beginning of the project. This makes products way better, for two important reasons. No one is cutting corners to meet the letter of a contractual obligation. More importantly, the beginning of a project is when you know least about what the actual requirements are. Requirements will to change as the project progresses; if they don’t, that means you’re not learning as you go. Writing the wrong requirements in stone into a contract at the beginning of a project is pretty self-defeating. The end result is that the development team are too busy implementing requirements that no one really wants, just because they’re in the contract, to be able to pivot and work on the stuff that we’ve learned does have value. Over-specifying feature lists at the beginning of a project is a terrible risk, both technically and in terms of the user value of the finished product.
Should we be estimating?
So far, so good - until we get back to that “how much is this project going to cost?” question. The garage developers have been watching the #NoEstimates movement with interest, because much of it aligns well with our thinking. We want to be as lean and reflective as possible and only do things which add user value (lean startup). We release often and checkpoint even more often (these are extreme programming values). We know the real measure of value is user experience, not metrics on a spreadsheet or feature lists (that’s one of the reasons we do design thinking).
#NoEstimates suggests that trying to estimate project duration is inaccurate and wasteful. In the Garage, we know that we can’t know how much it will cost to build something until we fully understand the technical landscape, and we’ll only get that knowledge once we’re deep into the implementation (and even then, there will always be surprises). More fundamentally, we won’t know what we really need to build for our users until we’re getting regular feedback from them about what we’ve built so far. Does that mean estimation is always a bad idea?
In the Garage, we proud of what we do. I know my team is awesome, and they deliver great value. However, if we’re working with a customer we’ve never worked with before, they don’t necessarily already know we’re awesome. There’s no pre-existing trust relationship, and they’re possibly comparing our value proposition to that of another vendor. It’s difficult for a customer to evaluate value-for-money of the Bluemix Garage unless we give some kind of estimate about how much a project will cost (the money), as well as describing how we can provide unique solutions for business problems (the value). There’s another aspect, too. In general businesses will have capped budgets, determined either by how much seed funding they’ve received, or by internal budgeting decisions. A business will need to decide if they can afford to build the minimum viable product that allows them to start testing business hypotheses, before starting that build. Building half a product, that never reaches viability, and then throwing it out because the money’s run out, is bad business. In other words, a customer needs enough information to be able to decide whether to go ahead with a project, and by definition that information is needed at the beginning of a project. Our job in the garage is to give them that viability assessment, so that we can then help them build a great project.
There’s an growing mountain of academic and industry research about the optimum technique for sizing, including a whole range of models for software cost, from the simple to the super complex. Some are surprisingly old (back to the 1980s), and some are impressively mathematical. We aim for continual improvement in the Garage, so we’re experimenting with a couple of different sizing methodologies that have the potential to be best practice. We need something low cost, because experience has taught us that spending more time on estimates doesn’t actually reduce project risk. On the other hand, the estimate needs to have enough predictive value to allow a customer to make a sound go/no-go decision.
These are the ones we’ve tried so far:
- Our starting point is the process described in Kent Beck and Martin’s Fowler’sPlanning Extreme Programming. The basic principle is to break the project down into smaller pieces of work, estimate them, and then add back up.
- A much faster approach is to estimate projects based on our experience of similar projects. This is surprisingly effective - when we’ve tried the two techniques side by side, we’ve found that ‘gut feel’ estimates line up pretty well with ‘break down and add up’ estimates, and of course they’re way faster to produce. The gut feel approach falls down when we do a project which is the first of its kind, and since the Garage specialises in innovation, that’s actually pretty often.
- One way of adding a bit more rigour to the ‘gut feel’ estimate, is to lay out a straw-man architecture with CRC cards. Since we’re thinking at the architectural level, instead of “Class-Responsibility-Collaborator”, we do something more like “Component-Responsibility-Collaborator”. Actually, it’s “Component-or-Actor-Responsibility-Collaborator”, but that acronym gets pretty messy. We use different colour post-its for “things we write”, things we interact with”, and “actors”. Our aim is to get some of these relationships out on paper and get a sense of the shape of the project, rather than to produce a final architecture. What the rough architecture gives us is a “landscape” that we can then compare to other projects we’ve done in the past, to produce an experience-based effort estimate.
- Another approach is to make the gut feel less rigorous. In other words, the best way to handle the huge uncertainty in cost estimation is just to acknowledge it. What the ‘real’ project ends up costing will land somewhere on a spectrum, and there are a whole bunch of business and technical factors that can influence that. So rather than trying to guess those factors away, we can simply advise a customer to plan for both a likely-best and likely-worst case: “we think this project will take somewhere between two and four three week phases.” I call this spectrum-sizing.
- Agile methodologies emphasise the importance of estimating in terms of points, rather than weeks, so we decided to try a points-based project estimation technique. Break a project down into hills and goals, and then for each one assess the complexity of its user interface, data store, business logic, and integration. We don’t want to get hung up on the precise complexity score. Instead we use score to guide categorising the goal into a Fibonacci complexity bucket, and we can sum up how many goals are in each bucket, and derive an estimate in weeks. As we use the technique more, our calibration between ‘bucket count’ and pair-weeks should improve. This technique has the potential to be extremely accurate, but we found it a bit too labour-intensive.
- Some of our mobile development colleagues have developed a quite extraordinary spreadsheet which encodes years of team experience about how long stuff takes into Excel formulas. Starting with inputs about the number of user screens, supported platforms, and backend interactions, along with a whole bunch of experience-led (but adjustable) assumptions about the other activities that will be required, it produces a detailed cost estimate. It even takes into account things like re-use efficiencies for subsequent stories. The thing I like about it is that it stores our collective experience, and it’s detailed enough to act as an aide-memoire about some things that take time, and that which we should be doing, but which are easy to forget. However, for most of our projects it assumes too much up-front knowledge about the final UX design.
To date, we’ve had the best success with the first three, but we’re still looking out for improvements. We haven’t tried them yet, but we’re intrigued by some of the Monte Carlo modelling and randomised branch sampling. They leverage advanced mathematics to offer the promise of more accurate estimates for less effort, which is pretty awesome. They do assume that a reasonably detailed breakdown of a project into epics and at least some stories has already been done, so we’ll need to decide if they’re appropriate at the very-very beginning of a project.
As I was writing this blog, Kent Beck, the father of extreme programming, posted a blog on the same subject. I won’t try and reproduce it here (you should just go read it, because it’s really good!), but I was pleased to see that some of his arguments line up with what I’d already written. Kent points out that in an ideal world one would do everything, but in a world where resources are finite, and doing one thing means not doing another thing, estimates help us make informed choices about where we should put our resources. He summarises his position as “Estimate Sometimes”. “Estimate Sometimes” isn’t the catchiest strap line, but it’s the right thing to do, for us and our customers. We need to make sure, though, that our estimates are not turned into prescriptions about duration or commitments about detailed feature lists, because we don’t want to be making those sorts of decisions at the point in the project cycle where we know least. So estimate sometimes, and then leave that estimate aside and use all the feedback we can get over the course of the project to make sure we deliver the right thing.