Sucked back into the waterfall

This is a tale of a small company, we'll call it David Software Development DSD, and a huge telecommunications company we'll call GT, Goliath Telecommunication.

DSD were a small company working on large project called NewVis. It was a replacement for their current product OldVis, only better, faster, more flexible and generally fantastic. They didn't know how large the project was but it was big. Bigger than anything the company had attempted before. The guestimates kicking around the company were 5 years of new development with 50 programmers. Many other companies had tried projects as large as this before but haven't always been successful at delivering a solution, when it was expected and without spending more than originally planned. In fact only 15% of projects of this size were successful according to a well known industry report.

Undaunted by the plight of the industry DSD embarked on this project. They were different after all and wouldn't fall for the same traps their peers did. The first thing they did was look at what they had to do. The project was so huge DSD management didn't know what to ask the developers to build first. They scratched their heads and decided not to speak to the people who would ultimatly use the product straight away but instead put together a crack team of people who had worked on making the previous version, OldVis, to find out what to build. They called this team the "Customer team". This team was made up of trainers and contributors to OldVis. Who better than folk that had trained people up and down the country on how to use OldVis to know what was wrong with it and how to make the new version, NewVis, better than ever before.

Not knowing what to build and therefore knowing how long that was going to take, and hence how much it was going to cost didn't put DSD management off. NewVis was really important to the company. If NewVis was successful it would mean an awful lot of money coming into the company and this was always in the back of their minds.

They had heard of a different way of developing applications, something new and shiny called Extreme Programming which was always appearing in the news and magazines. Management got a book, any book would do right, and read as much as they could in a whole weekend. This seemed like the answer to all their problems. A super fast way to develop applications where you didn't need to know what you were doing before you started writing the code, which always took the longest time to do so it was good to get it started as early as possible, right?

With only 10 people in the company they tried to get Extreme Programming to work for NewVis. The customer team wrote stories and tests and the developers worked in 2 week long chunks of time showing the customer team what they had managed to develop at the end of each two week iteration. NewVis was so huge that the stories were grouped into areas called themes which collectively described large sections of the application. The teams were organised so that each team was responsible for one of these themes. Each team was cross functional and could work on any technology to get their stories for the current iteration completed.

Everyone would work at the same pace on the same routine but staggered by one iteration. The customer team would be working one iteration ahead of development, writing stories for developers to do next iteration. Developers wouldn't know what was coming up only the stories that they had been assigned in the current iteration. When they were finished they would hand over the code to testers who would test the code that was just implemented. After a year doing these cycles and hiring some more developers DSD settled into a semi comfortable routine. Although this wasn't what had been described in the book management knew that they had to adapt Extreme Programming to fit they're own project and they're team was different, right? They didn't name their own process but as it was now different from Extreme Programming they just told anyone who asked that they were Agile and that was good enough.

As DSD hired more developers (Development, that's the bit that takes the longest) the customer team had to write more stories each iteration to keep up with the increase in demand from developers without being given any more time to do it. They started to rush the stories. Relying on quantity rather than quality to keep the developers from whining about having no work to do. The developers would often have to rewrite sections of NewVis as the customer team learned more and changed their mind. They did this a lot. Often flip flopping between two conflicting options. The customer were wasting time! Time DSD management couldn't afford to lose. They didn't know how much time they had to make NewVis but they were damn sure they dind't have any to waste. The wasted time had meant that DSD management had missed a few of their deadlines. To be honest more than a few. In fact all of them.

GT being a huge experienced telecommunications company were the company given responsibility for providing the service that the real customers of NewVis would use. They were the face of the NewVis product that the customer would see and ultimatly responsible for making sure it was good. They didn’t know how to build NewVis so hired DSD to build the product and they would make sure it was good enough for their customers. They had to be sure it was good enough from the ground up so they wanted to see what DSD planned to deliver and sure enough DSD was able to demo a piece of working product that GT could see and their engineers could use. GT didn't know anything about building NewVis or even what it needed to do but they had a wealth of experience building products so asked DSD the question "When will it be finished?" only to be given the answer "We don't know what we are going to build yet so we don't know how long it's going to take".

But GT had seen the demos of NewVis so they new there was some product already build. How couldn't DSD know what they were building if they had already started. In GT's experioence to build a product you must know exactly what you are going to build and have it all designed before you even start building it. Having a plan and a design before you started building something was just good Engineering sense. This situation was very unsettling to GT and to be frank smarted of amateurism. GT started to probe deeper and told DSD "Our experts will have a look at your project plan".

DSD had been using Agile. There was no project plan to give GT. DSD management paniced. There was all that money at stake and making GT happy was the only way DSD management were going to get their hands on any of it. They had better do something and fast. They got all the developers to stop what they were doing and concentrate on helping them generate a plan.

For months developers worked with the Customer team creating stories, drawing out features and turning those grey ideas into a design for features that weren't going to be worked on for years from now. Each feature had to have an estimate, how long the developer was going to take to implement that feature. Each had to have a design, a plan for how the developer was going to build that feature. All the documents generated had to be reviewed and changes to one caused a change to the others and every one of them had to be kept in sync. The task was immense and it was all hands to the deck everyone worked on any theme even if they weren't going to be implementing it. The developers were unhappy as they liked to write code rather than documentation but management told them this had to be done so they struggled on.

After a few months All the features and the estimates were done. Management put all the features into a Gantt chart, a type of plan that GT liked, and looked at the end date. It was 10 years in teh future. This wasn't good enough for GT, "That's too long" they said even though they dind't know what NewVis had to do. DSD management went back and had a think. They knew, by just thinking about it, that developers working in cross functional teams was very inefficient. The work done on the front end always took 2-3 times longer than that done on the back end. If they could use the developers writing the back end while the front end developers were finishing their stories they could be more efficient and could save time on the plan. They immediately split the developers into three teams organised around the technologies and each story into three so it could be given to each developer in each separate technology team when they were ready to do something new.

DSD management collected all the story thirds and totalled them up and grouped them by theme. Each task had three lines one for each technology and when the last line finished the three pieces would have to be put together before the theme could be tested. They would draw 3 lines on their project plan to represent how long it was going to take to finish each technology third, another for integrating them together, another for how long to test and another for how long to debug. All in all 6 lines per theme. They didn't know how long it would take to debug a piece but figured that the larger the piece the longer it would take to debug so say 10%.

They added all the developers names into the plan as resources and they assigned each resource to something all the time from the start of the project to the end of the project. If everyone was working on something all the time then their would be less waste and the things would get done a lot faster. They played with the plan, sweating over it and trying hundreds of combinations to make the end date move. They threw out lots of features that the customer team had written to make the plan finish on the date agreed with GT. They even cut out paper and made a wall chart to show the plan to get a better look but in the end they couldn't make the date and had to ask GT for an extension. When they did they didn’t want to add in anything that wasn't useful to GT. They had heard o the idea of a project buffer and of taking into account risk in a project but they ignored that advice. They were different after all.

A third of a story isn't much for a developer to work with so when they came to work on a new story they asked for more and the Customer team were only too hapy to give them all the stories thirds for a theme because now they had them ready. This could be 2-3 months work handed over from the Customer team to a developer in one go. This was very different than before. Developers stopped working on the unit of a story and started concentrating on a third of a theme at a time. This didn't work well because mistakes in the design of one third would have an impact on another third and no one knew who to talk to get it fixed any more.

DSD still marked the passage of each two week iteration but they came and went without anyone taking much notice any more. A theme could only now be demoed when the last technology team finished instead of every two weeks so there wasn't much the developers could show the Customer team every two weeks and eventually these meetings were cancelled. When customers did finally see what they asked for as before they had often changed their minds. It was natural when everyone was so forced to produce stories in such a short time that not all the stories were correct and when an incorrect one was found it was changed. This change in a story had to be re-estimated and the impact on the design checked. If the design changed then the other two technology teams had to be notified and their work, which they thought was complete, had to be changed. If the estimate changed, which it often did as the people working on the stories now were not the same people that estimated them and everyone had learnt more since they estimated the story initially the plan had to be updated. This was done by raising a project exception.

There were loads of project exceptions and the DSD management didn't know why there were so many. The plan they had spent months slaving over was in danger of missing it's date every time an exception came about. They must be stopped. A change control board was put into place to make anyone raising a project exception think twice. It was a heavy process involving justifying any change by filling in lots of forms and attending a meeting. Fewer and fewer changes got raised, even the ones that would make the future users of NewVis very happy. It just wasn't worth the hassle to anyone at DSD.

After a years had passed the deadline for NewVis that GT had set and DSD agreed to came about. GT had what they asked for and could start their formal test process. The Customer team was unhappy because they didn’t get the project they wanted and most of the developers who started on the project had left and the faces that were here now were younger and less experienced. DSD wasn't the same place it had been and Agile anything was just a memory.