Sucked back into the waterfall

This is a tale of a small company, we'll call it David Software Development DSD, and a huge telecommunications company we'll call GT, Goliath Telecommunication.

DSD were a small company working on large project called NewVis. It was a replacement for their current product OldVis, only better, faster, more flexible and generally fantastic. They didn't know how large the project was but it was big. Bigger than anything the company had attempted before. The guestimates kicking around the company were 5 years of new development with 50 programmers. Many other companies had tried projects as large as this before but haven't always been successful at delivering a solution, when it was expected and without spending more than originally planned. In fact only 15% of projects of this size were successful according to a well known industry report.

Undaunted by the plight of the industry DSD embarked on this project. They were different after all and wouldn't fall for the same traps their peers did. The first thing they did was look at what they had to do. The project was so huge DSD management didn't know what to ask the developers to build first. They scratched their heads and decided not to speak to the people who would ultimatly use the product straight away but instead put together a crack team of people who had worked on making the previous version, OldVis, to find out what to build. They called this team the "Customer team". This team was made up of trainers and contributors to OldVis. Who better than folk that had trained people up and down the country on how to use OldVis to know what was wrong with it and how to make the new version, NewVis, better than ever before.

Not knowing what to build and therefore knowing how long that was going to take, and hence how much it was going to cost didn't put DSD management off. NewVis was really important to the company. If NewVis was successful it would mean an awful lot of money coming into the company and this was always in the back of their minds.

They had heard of a different way of developing applications, something new and shiny called Extreme Programming which was always appearing in the news and magazines. Management got a book, any book would do right, and read as much as they could in a whole weekend. This seemed like the answer to all their problems. A super fast way to develop applications where you didn't need to know what you were doing before you started writing the code, which always took the longest time to do so it was good to get it started as early as possible, right?

With only 10 people in the company they tried to get Extreme Programming to work for NewVis. The customer team wrote stories and tests and the developers worked in 2 week long chunks of time showing the customer team what they had managed to develop at the end of each two week iteration. NewVis was so huge that the stories were grouped into areas called themes which collectively described large sections of the application. The teams were organised so that each team was responsible for one of these themes. Each team was cross functional and could work on any technology to get their stories for the current iteration completed.

Everyone would work at the same pace on the same routine but staggered by one iteration. The customer team would be working one iteration ahead of development, writing stories for developers to do next iteration. Developers wouldn't know what was coming up only the stories that they had been assigned in the current iteration. When they were finished they would hand over the code to testers who would test the code that was just implemented. After a year doing these cycles and hiring some more developers DSD settled into a semi comfortable routine. Although this wasn't what had been described in the book management knew that they had to adapt Extreme Programming to fit they're own project and they're team was different, right? They didn't name their own process but as it was now different from Extreme Programming they just told anyone who asked that they were Agile and that was good enough.

As DSD hired more developers (Development, that's the bit that takes the longest) the customer team had to write more stories each iteration to keep up with the increase in demand from developers without being given any more time to do it. They started to rush the stories. Relying on quantity rather than quality to keep the developers from whining about having no work to do. The developers would often have to rewrite sections of NewVis as the customer team learned more and changed their mind. They did this a lot. Often flip flopping between two conflicting options. The customer were wasting time! Time DSD management couldn't afford to lose. They didn't know how much time they had to make NewVis but they were damn sure they dind't have any to waste. The wasted time had meant that DSD management had missed a few of their deadlines. To be honest more than a few. In fact all of them.

GT being a huge experienced telecommunications company were the company given responsibility for providing the service that the real customers of NewVis would use. They were the face of the NewVis product that the customer would see and ultimatly responsible for making sure it was good. They didn’t know how to build NewVis so hired DSD to build the product and they would make sure it was good enough for their customers. They had to be sure it was good enough from the ground up so they wanted to see what DSD planned to deliver and sure enough DSD was able to demo a piece of working product that GT could see and their engineers could use. GT didn't know anything about building NewVis or even what it needed to do but they had a wealth of experience building products so asked DSD the question "When will it be finished?" only to be given the answer "We don't know what we are going to build yet so we don't know how long it's going to take".

But GT had seen the demos of NewVis so they new there was some product already build. How couldn't DSD know what they were building if they had already started. In GT's experioence to build a product you must know exactly what you are going to build and have it all designed before you even start building it. Having a plan and a design before you started building something was just good Engineering sense. This situation was very unsettling to GT and to be frank smarted of amateurism. GT started to probe deeper and told DSD "Our experts will have a look at your project plan".

DSD had been using Agile. There was no project plan to give GT. DSD management paniced. There was all that money at stake and making GT happy was the only way DSD management were going to get their hands on any of it. They had better do something and fast. They got all the developers to stop what they were doing and concentrate on helping them generate a plan.

For months developers worked with the Customer team creating stories, drawing out features and turning those grey ideas into a design for features that weren't going to be worked on for years from now. Each feature had to have an estimate, how long the developer was going to take to implement that feature. Each had to have a design, a plan for how the developer was going to build that feature. All the documents generated had to be reviewed and changes to one caused a change to the others and every one of them had to be kept in sync. The task was immense and it was all hands to the deck everyone worked on any theme even if they weren't going to be implementing it. The developers were unhappy as they liked to write code rather than documentation but management told them this had to be done so they struggled on.

After a few months All the features and the estimates were done. Management put all the features into a Gantt chart, a type of plan that GT liked, and looked at the end date. It was 10 years in teh future. This wasn't good enough for GT, "That's too long" they said even though they dind't know what NewVis had to do. DSD management went back and had a think. They knew, by just thinking about it, that developers working in cross functional teams was very inefficient. The work done on the front end always took 2-3 times longer than that done on the back end. If they could use the developers writing the back end while the front end developers were finishing their stories they could be more efficient and could save time on the plan. They immediately split the developers into three teams organised around the technologies and each story into three so it could be given to each developer in each separate technology team when they were ready to do something new.

DSD management collected all the story thirds and totalled them up and grouped them by theme. Each task had three lines one for each technology and when the last line finished the three pieces would have to be put together before the theme could be tested. They would draw 3 lines on their project plan to represent how long it was going to take to finish each technology third, another for integrating them together, another for how long to test and another for how long to debug. All in all 6 lines per theme. They didn't know how long it would take to debug a piece but figured that the larger the piece the longer it would take to debug so say 10%.

They added all the developers names into the plan as resources and they assigned each resource to something all the time from the start of the project to the end of the project. If everyone was working on something all the time then their would be less waste and the things would get done a lot faster. They played with the plan, sweating over it and trying hundreds of combinations to make the end date move. They threw out lots of features that the customer team had written to make the plan finish on the date agreed with GT. They even cut out paper and made a wall chart to show the plan to get a better look but in the end they couldn't make the date and had to ask GT for an extension. When they did they didn’t want to add in anything that wasn't useful to GT. They had heard o the idea of a project buffer and of taking into account risk in a project but they ignored that advice. They were different after all.

A third of a story isn't much for a developer to work with so when they came to work on a new story they asked for more and the Customer team were only too hapy to give them all the stories thirds for a theme because now they had them ready. This could be 2-3 months work handed over from the Customer team to a developer in one go. This was very different than before. Developers stopped working on the unit of a story and started concentrating on a third of a theme at a time. This didn't work well because mistakes in the design of one third would have an impact on another third and no one knew who to talk to get it fixed any more.

DSD still marked the passage of each two week iteration but they came and went without anyone taking much notice any more. A theme could only now be demoed when the last technology team finished instead of every two weeks so there wasn't much the developers could show the Customer team every two weeks and eventually these meetings were cancelled. When customers did finally see what they asked for as before they had often changed their minds. It was natural when everyone was so forced to produce stories in such a short time that not all the stories were correct and when an incorrect one was found it was changed. This change in a story had to be re-estimated and the impact on the design checked. If the design changed then the other two technology teams had to be notified and their work, which they thought was complete, had to be changed. If the estimate changed, which it often did as the people working on the stories now were not the same people that estimated them and everyone had learnt more since they estimated the story initially the plan had to be updated. This was done by raising a project exception.

There were loads of project exceptions and the DSD management didn't know why there were so many. The plan they had spent months slaving over was in danger of missing it's date every time an exception came about. They must be stopped. A change control board was put into place to make anyone raising a project exception think twice. It was a heavy process involving justifying any change by filling in lots of forms and attending a meeting. Fewer and fewer changes got raised, even the ones that would make the future users of NewVis very happy. It just wasn't worth the hassle to anyone at DSD.

After a years had passed the deadline for NewVis that GT had set and DSD agreed to came about. GT had what they asked for and could start their formal test process. The Customer team was unhappy because they didn’t get the project they wanted and most of the developers who started on the project had left and the faces that were here now were younger and less experienced. DSD wasn't the same place it had been and Agile anything was just a memory.

Slouching Towards Waterfall

Slouching Towards Waterfall

I found this very interesting article by
Dave Nicolette which discusses how organizations slide from either Agile or Waterfall to the Anti-Pattern Staggered-Iterative-Waterfall. I found the discussion of cultural differences between Agile and Lean very interesting but you may take a different view after reading Mary Poppendieck's book Lean Software Development (http://www.poppendieck.com/).

An iteresting read.

TDD Demo

I presented the Test Driven development in Excel demo at Agile Scotland last night. It went very well. The mistakes I made only served to get the audience more involved in the presentation by jumping in to help me out. Here are the slides and a review which have kindly been posted on Clarke Ching's site. Thanks Clarke.

Pair Programming

I wanted to tell you story a I recently heard about Pair Programming in the field:

A friend of mine, we'll call her Alice, has recently started working in a the offices of a small financial firm here in Dundee. She was asked to work on creating a financial report in Crystal Reports which would access the company database. The database schema is huge containing over 1000 tables and views and my friend knew that there was no way she was going to be able to learn enough about the database to produce the report on time. She is the only person in the organisation with enough Crystal experience to produce the report so I sugested she pair with someone who knows the database. She was hesitant at first but I explained that the combination of skills would help her meet her deadline.

Everything started well. Alice managed to get one of the database guys from another team, we'll call him Bob, to agree to pair with her. Alice and Bob sat at the same desk and sailed through the issues that came up. Alice could quickly sort out any Crystal report problems and was clear on the business need behind the report and Bob could sort out any problems with the logic in the query and the data that was returned. It was a perfect pairing combination of technical and business knowlege with just enough overlapping skills to be able to communicate together effectively.

Bob's manager wasn't so impressed. When she looked over all she saw were two people doing the same task. Two people that could be doing their own work and not wasting time sitting together. That night she decided to speak to Bob and sort this out. She asked Bob how Alice and he were getting on. Was he going to be doing the same thing tomorrow. Was he sure that it was necessary to sit with Alice. She suggested he install Crystal on his own machine to make it easier to work seperatly. She even suggested that Alice needed time by herself to learn on her own and implied that by pairing he was somehow holding her back.

Bob did what he thought was right, he wanted to please his manager, and told Alice that they couldn't pair any more. He told her how she had to learn for herself and that he couldn't waste the time pairing. Alice was hurt. The previous day had been a real sucess and she had come in full of enthusiasm to finish the report. The report suffered. Alice spent all the next morning struggelling with the subtle database issues and making little progress on the report itself. She finally got fed up and went to speak to Bob. She told him about the problems she had run into and asked him again if they could pair. Bob relented and they moved to the same desk. Progress on the report  leapt forward. They easily overcame any issues that came up and finished the report by the end of the day well before deadline. Pairing does work.

Bob's Manager, the customer of the report, is still unconvinvced about pairing but can't deny the obvious difference in progress. She was acting in the what she felt was in the best for the business, trying help make progress on the report faster but her actions had the exact opposite effect. There is a lack of understanding of the nature of the work on her behalf. If the report is the product of the work then this was a product development task not a product production taks. She failed to understand that.
Bob's manager isn't alone. This a reaction I've come across again and again when explaining Pair Prograqmming.

The problem is one of education, how to educate managers on the benifits of Pair programming. I've seen live demos but It's a difficult practise to demonstrate because you almost have to be in the pair to understand what's happening. I've seen some exercises on the web such as Industrial Logic's Pair Draw which may help in getting the point across and the articles at PairProgramming.com but I'm interested to hear what have you tried? How have you convinced both programmers and managers that Pairing is a good idea. What gave you the "Aha" moment when you learnt to Pair?

I'm going to try the pair draw exerceise with a group next week. I'll let you know how it goes.

"Project Backlog" is negative

A friend of mine recently pointed out that the term "Project Backlog" from scrum for the work still to be done in an Agile project has negative connitations. It sounds to non technical people like the project is behind, that there has been a build up of work and you're having to catch up. I think this is probably why Ken Schwaber chose that term, the project is only up to date when all work outstanding has been done, but I'm having a difficult time selling the idea to my customer and I think it is because of this negative connitation I hadn't previously thought about.

In the project I work on are in the enviable position to have spare development capacity and not enough customer capacity. Customers write stories for an iteration days, sometimes hours, before the iteration planning meeting and only have time to write one iterations worth of stories. The customers look at creating stories for the "backlog" as unncessesary because we're always crying out for more stories, why bother store them up. This is a big problem for us in development. We can't see what's coming up so design decisions often have to be reworked in the next iteration. Prioritising work to be done is missing becasue we just do the work when we get it. Design has to happen in the iteration where the work is done and we sometimes do work that wouldn't be necessary if we had a slightly longer term view of the upcoming work.

So I think it's time to rebrand the list of work still to do. Suggestions so far are "Work item list" or "Work Pool". I really llike the work pool name because it creates an image of something that has the same properties as I want from my list of "work to do". Diving into an empty pool is obviously dangerous. If a developer runs out of work they can dip into the work pool and many other useful management metophors. I'm sure you can think of others.

Let me know what you think and any other names you can think of.

Pete

Test Driven Development Example

I recently presented Clarke Ching's TDD in Excel example to upper management at the company I work for. The example follows through the development of an Integer to Roman numeral converter using Excel as the test harness. It took about 35 minutes to get to 15 fully refactored and 10 minutes for questions and discussion. I was nervous as hell but managed to get through the code with out too many problems. I thought I was clunky in places but one of the managers said the presentation was really slick so it must have looked better from the outside. I volunteered to present it again for any team that was interested so I should have some more experience for when I do it at Agile Scotland (http://agilescotland.blogspot.com/) later this month.


All the feedback so far has been really positive. One of the other managers said it was excellent at getting the point of TDD across. We talked about clean, beautiful code and how it motivates programmers. I think it was the first time my boss had heard of that or even considered that producing high quality code could motivate programmers. I don't think he's ever heard off that before but many of the managers listening started chipping in examples of their own. It was really good.


As a demo for managers I think that it really stood up. It got the point across without losing them. I think some of the programming was beyond my boss (he's not a programmer after all) but the frequent switches back to the tests kept him interested and he could see the power when I made a mistake, got instant feedback, fix it and ran the tests again. I think 15 was enough for this demo. When I was doing 1 – 8 there is a slight stall before you refactor at 8 and I think next time I'll try to fill that somehow. Apart from that it went swimmingly


Here is the example if you'd like to try it for yourself



http://www.clarkeching.com/2006/04/test_driven_dev.html

Agile Scotland

Last night I attended an Agile Scotland prescentation by Ben Fuchs and Joseph Pelrine(http://www.metaprog.com/blogs/)on the Social and Psycological Challenges of Agile Development. It was a very interesting talk taking the form of an Agile project. Questions were gained requested from the floor, costed by the speakers and prioritised by the audience with a show of hands. The topic covered everything from "How to deal with problem personalities" to "Agile Communism". The whole talk was good but the two highlights for me were the story of the bees and Joseph berating Agile by the book.

Joseph said that if you're doing Agile by the book then you're not doing Agile at all. Agile is about continuiosly looking at and improving your process. The book is just a starting point. If you're doing 10, 11 or 12 of the practices it doesn't matter, what matters is if you're using these techniques to ship high software faster and better than ever before.

The story of the bees was told by Ben Fuchs. I'm not sure where he got it but it's a story about organisations and their model of sustainability. It goes like this. When a Bee finds pollen it returns to the hive and does a dance to let the other bees know where the pollen is. Most of the bees, about 80%, head off to collenct the pollen from the new source. Most but not all. 20% fly off in random directions scouting for new sources of pollen. If those 20% didn't look for future sources then the hive would starve. If 100% of the bees gathered the pollen there would be no new sources of pollen and the hive would die.

Here is another blog covering the talk that Ben and Joseph are giving throughout the country http://jw.fi/2006/03/17/agile-complexity-and-friends/

Thanks to Clarke and Agile Scotland for arranging this event.