Monday, August 1, 2011

The Mythical Man-Month

The Mythical Man-Month : Essays on Software Engineering
By Frederick P Brooks , Jr

Schopenhauer warned that, “Any book that is important should be reread immediately”. While I did follow his advice eventually, the gap between first read on a dog eared library book and now on a good silver jubilee edition was more than a decade and the impact is vastly different.

Author was overseeing one of the biggest ever software development undertaking in the history. Hence, the nuggets of wisdom he delivers ought to be taken seriously. But all the technologies he talks about (e.g.: Microfiche) can be found only in museums and the processes for controls are also less applicable in the current scheme of things. It goes to prove writing an enduring classic in this area is close to impossible.

Code Base Size : Large and beyond

Quad-4

Quad-1

Code Base size : Small to Medium

Quad-3

Quad-2

Short

Long

Time Horizon

Lines-of-Code (LOC). Want to avoid quoting any specific language for LOC. Ball park division would be: Small = up to 500K LOC; Medium: Up to Million LOC; Large and Beyond: Greater than Million LOC (Disclaimer: Small, Medium and Large definitions are flimsy at best).

Author is essentially talking about a project that clearly qualifies for Quad-1. By the way, I could not think of any Quad-4 projects. Now days, most of the people are involved in Quad-2 and Quad-3 projects and hence the applicability of the book’s advices would have to be taken in that light. Most of the mammoth projects are already in shape (like huge new OS development etc) and hence what you would witness is using them for our end application goals.

Some books stand the test of time since it deals with basic problems. I would say only a portion this book would stand that test. Let me summarize the main points of this book with my comments (disclosure: not exhaustive).

1. Man-Month is not an interchangeable unit. That is, by putting more people you cannot reduce the time taken – if you try to add more people, it can only exacerbate the situation. (May be in the next edition, author would change the reference as person month in tune with the spirit of times).

2. Importance of documentation and communication are well stressed. While there is nothing to be gainsaid on them, proposed solutions may not suit. Yet, the concept needs to be taken to heart during execution.

3. He rightly stresses the need for conceptual integrity in product. However, in the world of rapid changes, it is difficult to come up with enduring ones. Yet one (or the core team?) would have to try. Previously, users adapted to the devices more, but now market drives the specifications and design. Hence, it is bound to be fragmented and maintaining it is going to be even more daunting. In any product, as changes start creeping into each version of the launch or upgrade, the systems start accumulating what is known as “design debt”. It starts innocuously at first and over a long period (if changes are not frequent and well done) becomes eventually overwhelming such that it would justify a total redevelopment.

4. Small time lapses eventually add up to large delays. He advises us to be very vigilant on them. I would surely agree since each one in isolation would be very negligible but collectively it can deliver a catastrophe. He quotes in the beginning of the chapter “How does a project get to be a year late?” and answers “One day at a time”. That Q&A alone would provide the snappiest summary of the message.

5. He talks in detail about the structure of the team. So long as we take them in the light of Quad-1 kind of projects, they are fine. If you are handing Quad-2 or 3 projects, most of the suggestions are too overwhelming to implement, but one can look into the spirit of it and make sure it can be accomplished in some manner. Take away would be, think thoroughly on team structures before the journey.

6. Finally, author quotes from a restaurant in New Orleans. “Good cooking takes time. If you are made to wait, it is to serve you better and to please you”. In effect, advising “courteous stubbornness” like that of chef while handling change notes or modification request or rework etc for the project. Well said, but I would be apprehensive of this point in a market driven world - should be attempted with caution.

Let us go back to Schopenhauer’s warning where we started. Now, I know at least one reason as to why he said what he said. If you don’t read the book again immediately, the contents may become outdated faster than we all wish.

Thanks for reading this far.

Regards,

madhu

21 comments:

Mukund Srinivasan said...

To be honest Madhu-san, I definitely liked your commentary here more than the POV of the author. Primarily so, because of the concept that I think he is driving at. To take the chef/cooking analogy further, a chef is said to be only as good as his last preparation. Unfortunately, we cannot take the same focus to a programmer and his code - after all, if there are no defects, there is no notion of six sigma quality :-)

Suresh S said...

Madhu,

Yes, it is indeed true that you cannot write an enduring classic in this area. Things are changing so fast, new tools emerge and goal posts are shifted. So it is always better to re-evaluate the classics and change them a bit to suit our current needs.

Mohanakrishnan Gopalakrishnan said...

Madhu,

I have always wanted to read this book for the first time, but never got around to doing it. Now from my note, I feel that I have saved myself the trouble.

On the nature of software development, many things have changed over the years. Some of the large projects today are not about writing code at all. Tools and packages have introduced a layer of abstraction. So developers implement packages (like ERP packages) or generate code using tools. So only a fraction of projects write code from ground up.
On the life cycle side, there are new life cycles like agile that are getting popular.
On team composition side, as you nicely pointed out, we don't have teams made of only men any more :-)

Writing books on such fast changing field has advantages as well as disadvantages. You pointed out the disadvantages. But there are advantages as well. As things constantly change, there are so many new experiences and learning that almost anyone will have something to write about :-)

Finally, should we blame the changing environment for making our 'classics' outdated? Or should we blame our inability to formulate fundamental principles? Are we trapped on the froth that we are not able to dive any deeper?
Someone was able to philosophize on rights and wrongs of life some 3000 years back in a way that it is relevant today. Can't say that the change in software development in the last 50 years is more than the change in lifestyle during the last 3000 years :-) I know that the example is a bit of an extreme. But my point is this: as life has started changing quickly around us, it is possible that we lost the inclination and ability to think deeply on subjects.

Thanks Madhu for the note - I guess one of your best so far.

Srinivasan Desikan said...

Thanks Madhu for your excellent comments. Mythical man month is one of the SE books we can read many times and still get value in every reading. I only wish that LOC as a measure differentiated between what is "New" code, what is "Re-used" code in a product Quality. In today's scenario new code development is very small and so much of re-use (or unstated cut-and-paste) constiture 80% overall code.

vivisa said...

Hello Madhu-san,

It was a lively post again. Having worked (as part of your team) on maintaining a huge OS as well as designing a micro kernel I can appreciate the size-vs-time matrix you have drawn.

While time outdates everything at its own pace, some of what Brooks says seem to be very fundamental. For instance, statistically, it still seems be true that adding humans in the later phases of a delayed project causes further *non-linear* delays. The statistical outliers / counter examples to this theory, would no doubt be of academic interest too (either explained by tales of heroism and individual brilliance or by a comically flawed/non-sequitur logic that fails to account for randomness).

While, TMMM is a practical treatise on software project management, I would like to see your review(?) of Brooks' "No Silver Bullet" too. It appeals a lot to the programmer in me. And also happens to be one of the earliest papers I had read while I was in REC, Warangal, thanks to my CSE professor who asked me to present it to our class.
--
sAgar

Rajagopal said...

Madhu,
good one even though I cannot relate to software development. I have seen it from a general project perspective. I wud like to buy the book & read it but afraid it should not contain software jargons.

Jujubax said...

vivisa has left a new comment on your post "The Mythical Man-Month":
(re-posting on behalf of vivasa since his comments are not getting recorded)

Hello Madhu-san,

It was a lively post again. Having worked (as part of your team) on maintaining a huge OS as well as designing a micro kernel I can appreciate the size-vs-time matrix you have drawn.

While time outdates everything at its own pace, some of what Brooks says seem to be very fundamental. For instance, statistically, it still seems be true that adding humans in the later phases of a delayed project causes further *non-linear* delays. The statistical outliers / counter examples to this theory, would no doubt be of academic interest too (either explained by tales of heroism and individual brilliance or by a comically flawed/non-sequitur logic that fails to account for randomness).

While, TMMM is a practical treatise on software project management, I would like to see your review(?) of Brooks' "No Silver Bullet" too. It appeals a lot to the programmer in me. And also happens to be one of the earliest papers I had read while I was in REC, Warangal, thanks to my CSE professor who asked me to present it to our class.
--
sAgar

Jujubax said...

Hi Mohan:
Thanks for your insightful comments.

I think,humans ability to think deeply is still there very much. That said, the need to do so is less these days since basic or fundamental problems stands resolved - and well.
Hence, now a days, only tougher and complex ones are left behind to be resolved (which means we would have to think even more deeply but they can wait!).
I think, complexity can come in two ways - one is depth intensive ( a few parameters or variables) or breadth intensive (lots of variables each changing a little bit and deliver similar complexity). We seem to be moving to that era.
So from "principles" to "data driven" models. That is from causation era to correlation era.
regards
madhu

Mohanakrishnan Gopalakrishnan said...

Madhu,

I agree with you on that observation. It is just that I don't consider all changes that happen over time as progress. As you observed we have not lost the capability to think deeply, but we sure seem to have lost the will.

Nimmy said...

Interesting! Thank you for the lovely summary, Madhu. Not being a hard-core (not my main job and have managed only two software/coding projects on my own so far) "software project manager", I guess I ought to be happy I have the pleasure of reading your accurate interpretation and summary without having to read the full book. :-)

You're right. The rate at which the software ecosystem is changing, some of these books begin to look less worthwhile. :-)

Pratap R C said...

I have enjoyed reading such books - being with your team. I am just a programmer and I can only say that "programming is art", although I can enjoy the abstract ideas discussed by Fred brooks and Tom De marco.

Ramesh N Raghavan said...

Madhu-san,
I was not sure if I should post a comment or not, but from the other comments I thought that some were coming to the conclusion that they can skip reading this book. Actually, your post does highlight a lot of the time invariant truths, but Schopenhauer’s quote seemed to have had a halo effect overall. So, I thought I will write a detailed note on why I think this is still a must read. Due to size limitations for comments, I am making it a 4-part comment..
I would like to begin by saying that this book is probably one of the best books I have read on Software engineering in practice. After reading it now again after about 15 years (Thanks to you), I think I empathize with it much more than when I read it as a very young programmer. I saw Mohan’s comment about why we have not been able to arrive at some fundamental principles even after so many years. I sincerely feel Brooks has outlined quite a few such principles in his book, and I would strongly recommend a good read for anyone who has not read it before, you will certainly not be disappointed.
A chapter-by-chapter view of what Brooks has to offer in ‘The Mythical Man Month’ follows here:

* In his first chapter, The Woes of the Craft, he talks about the need for error free coding, ‘machine expectation of human programmers to be perfect’, dependence on other’s outputs, the 80:20 rule with respect to debugging where the last few defects take an enormously long time to fix, risk of obsolescence before the product is done, and an ever advancing technological base on which one builds – I think most of these are still relevant today and I don’t think we have come far from the situation Brooks outlines in his first chapter ‘The Tar Pit’. By the same token, the joys of the craft that he describes is also still true, which is what still attracts some creative folks to choose this as a profession.

* Chapter 2, the main message of the book, which is ‘The mythical man-month’ which says men and month are not interchangeable, is still very true. Every day I am seeing large projects that are slipping and it is extremely difficult to pull back a large slipping software project back. Madhu-san, I think you have also agreed that this is quite true.
(Rest in Part 2…)

Ramesh N Raghavan said...
This comment has been removed by the author.
Ramesh N Raghavan said...

(Part 4)…
* In Chapter 12, Brooks discusses the need and critical use of various tools, especially from a system software point of view, like simulators, emulators, the need to standardize tool usage across the team, explains how a version control system should be used (differentiates between a programmers personal playpen versus the sub-library where he submits his finished work for integration). He also talks about need for programming time on actual hardware in blocks of time that helps the programmer to debug issues much faster and not have to wait for another 6 hours before he can try his fix again. Though normal desktop resources are not an issue these days, getting time on actual newly minted hardware can still be scarce and his advice on how to allocate time on such systems makes a lot of sense even today. The last section is about Interactive programming which was new then, and Brooks correctly predicts that it is there to stay and we are to this day in that model.

* Chapter 13 is all about integration and debugging, and I did not find even one suggestion out of place even today.

* Chapter 14 is about project management, the need to have the patience to not intervene at the first bad news and demoralize your subordinate, and decide when you must step in. He says one should distinguish between Status information and Action information. He also talks about the need for PERT charts and identify which tasks are in the critical path. Though the industry by now should have advanced to have all the Project managers understand the need to identify dependencies and lay them in PERT charts, we often find many small projects being run out of the head, with no such chart in evidence. He also stresses the need to define sharp measurable milestones that one can test for. I am reminded of some of the serious debates that we have with customers on whether the milestone has been met ;)

* The final chapter, Chapter 15 is about external documentation on how a program should be used, and also other ways to document a program, like self-documenting code (by which he means reasonably well documented code with comments in the source file), the challenges of using Flow charts (he is quite strong when he says that the detailed blow-by-blow flow chart is an obsolete nuisance, suitable only for initiating beginners into algorithmic thinking). His analysis of why Flow charts are obsolete with modern high-level languages being available is very educative. He also talks about naming conventions and also code commenting guidelines which all still makes sense. At the time of writing what he suggested were difficult to implement, but he rightly concludes saying that those limitations are on the way out and it will probably be the way forward which is quite true today.

Thanks to anyone who had the patience to read this far. My only hope is that you find it intriguing enough to seek out for a copy of the book.

Best Regards,
N.R. Ramesh.

Ramesh N Raghavan said...

(Part 2)….

* In Chapter 3, Brooks gives a possible solution that he feels is appropriate for the extreme productivity differences amongst experienced programmers. Various studies have shown that it often varies by an order of magnitude. So, he proposes that each module of a large program be structured like a small surgical team, with a master surgeon who is the chief programmer whose experience level and mastery over programming is the best, a co-pilot who is almost equally capable but acts like the alter-ego of the surgeon and is the insurance if the surgeon is not available for some reason, an administrator who handles issues such as people, space, machines etc. (which Brooks says can often be a part-time responsibility and can serve multiple such teams), the editor who is responsible for editing the documentation produced by the Surgeon, reviews it, adds references etc. and helps in producing it, couple of secretaries, a program clerk for maintaining all the technical documentation for the team and making it available to everyone, the tool smith who makes available all the specialized tools needed for the team, the tester for ensuring everything gets tested adequately, and a language lawyer who helps choose the right language for the job, and this again can be a part-time job and can serve multiple teams. This may at the outset look a little outdated, especially when we mention secretaries, administrators etc. who are generally not found in our IT firms, but if one looks at the job division I think it is quite neat. With modern text editing and source code and document control systems, the need for one person to manage these may not be there, but still these roles like a Build Manager etc., documentation specialist etc. do exist. In practice, in most teams, there is a “surgeon” who is driving the entire technical design and development, often coding a large portion of the system himself. Sometimes it is by choice, or often due to the apparent inability of the rest of the team to scale up.

* In Chapter 4, Brooks tries to reason why Architectural integrity is very critical for the success of the project, and how there is enough creative work left for the implementers after the Architect has completed specifying the system. The way Brooks defines the Architect is probably different from what we use now. For Brooks, the Architect is the one who specifies the entire system, especially all the user visible portions of systems. One also has to remember Brooks’ descriptions are mostly around System software components like OS, Compilers, and Translators etc. The examples he gives in this context are quite good and I think there is something to learn from this as well.

* In Chapter 5, Brooks describes a specific phenomenon that he has observed called ‘The Second-System Effect’. He says that in the second large system that they work on, the Architects tend to go a little overboard trying to cram in huge amount of features with the result being huge delays and probably the product never making it to the market. I am not able to comment much on this per se.

* Chapter 6, called ‘Passing the word’ is all about how to communicate the design of the different modules, the interfaces across to a large team. He talks about the difficulties of using formal definitions to communicate and the challenges they pose. He also talks about the issue of documentation and the implementation diverging, and what gets fixed under what circumstances. An excellent chapter and the fact that formal definitions are yet to gain very wide spread acceptance and usage probably has roots in what Brooks describes here.
(Continued in Part 3 )…

Ramesh N Raghavan said...

(Part 2)….

* In Chapter 3, Brooks gives a possible solution that he feels is appropriate for the extreme productivity differences amongst experienced programmers. Various studies have shown that it often varies by an order of magnitude. So, he proposes that each module of a large program be structured like a small surgical team, with a master surgeon who is the chief programmer whose experience level and mastery over programming is the best, a co-pilot who is almost equally capable but acts like the alter-ego of the surgeon and is the insurance if the surgeon is not available for some reason, an administrator who handles issues such as people, space, machines etc. (which Brooks says can often be a part-time responsibility and can serve multiple such teams), the editor who is responsible for editing the documentation produced by the Surgeon, reviews it, adds references etc. and helps in producing it, couple of secretaries, a program clerk for maintaining all the technical documentation for the team and making it available to everyone, the tool smith who makes available all the specialized tools needed for the team, the tester for ensuring everything gets tested adequately, and a language lawyer who helps choose the right language for the job, and this again can be a part-time job and can serve multiple teams. This may at the outset look a little outdated, especially when we mention secretaries, administrators etc. who are generally not found in our IT firms, but if one looks at the job division I think it is quite neat. With modern text editing and source code and document control systems, the need for one person to manage these may not be there, but still these roles like a Build Manager etc., documentation specialist etc. do exist. In practice, in most teams, there is a “surgeon” who is driving the entire technical design and development, often coding a large portion of the system himself. Sometimes it is by choice, or often due to the apparent inability of the rest of the team to scale up.
(Continued in Part 2b)

Ramesh N Raghavan said...

(Part 2b)

* In Chapter 4, Brooks tries to reason why Architectural integrity is very critical for the success of the project, and how there is enough creative work left for the implementers after the Architect has completed specifying the system. The way Brooks defines the Architect is probably different from what we use now. For Brooks, the Architect is the one who specifies the entire system, especially all the user visible portions of systems. One also has to remember Brooks’ descriptions are mostly around System software components like OS, Compilers, and Translators etc. The examples he gives in this context are quite good and I think there is something to learn from this as well.

* In Chapter 5, Brooks describes a specific phenomenon that he has observed called ‘The Second-System Effect’. He says that in the second large system that they work on, the Architects tend to go a little overboard trying to cram in huge amount of features with the result being huge delays and probably the product never making it to the market. I am not able to comment much on this per se.

* Chapter 6, called ‘Passing the word’ is all about how to communicate the design of the different modules, the interfaces across to a large team. He talks about the difficulties of using formal definitions to communicate and the challenges they pose. He also talks about the issue of documentation and the implementation diverging, and what gets fixed under what circumstances. An excellent chapter and the fact that formal definitions are yet to gain very wide spread acceptance and usage probably has roots in what Brooks describes here.
(Continued in Part 3 )…

Ramesh N Raghavan said...

(Part 3…)

* Chapter 7, talks about the management challenges in a large programming project and offers suggestions on how some of them could be managed. This is where he discusses how they switched to microfiche to manage the huge volume of documentation and also talks about the disadvantages they had to endure due to that. Though Microfiche looks outdated today, at the end of that description he says how one could do it with the latest technology available when he wrote the book, which is using the then available word processing technology with “change bars” to highlight changes, LIFO filing of changes etc., which is quite similar to what we use today. He also talks about shielding programmers from the internal details of other module’s implementation (and also about the risks of depending only on interface definitions not having sufficient communication). He stresses the need for effective communication in large programs, which is as relevant today as then.
Chapter 8 is about programmer productivity and he shares some early data available at that time. The fact that productivity in terms of LOC drops as the size of the system gets larger is still true. In fact Brooks offers an insight which rings quite true ‘Productivity in terms of number of elementary statements of a programming language seems constant, a conclusion which he feels reflects the amount of thought a single statement requires and the errors that it may include’. This leads to the conclusion that higher-level languages that allow more “functionality” per statement help in productivity improvement by a large factor. This was written at a time when High-level languages were just emerging. I think this and his remark about the Language lawyer in an earlier chapter together makes a lot of sense. We often find that certain languages are optimal for certain tasks.
(continued in Part 3b)

Ramesh N Raghavan said...

(Part 3b)

* Chapter 9 is all about space-time trade offs and I think it is very relevant today as it was then. May be, for a while memory costs and disk costs plummeting down allowed us to not worry so much about this. With problem sizes increasing, and the whole buzz around “Big Data”, this is attracting a lot of attention towards these issues and I think any CS person should read this chapter.

* Chapter 10 is all about documentation and he stresses the need to write things down. This is a very valuable piece of advice. Often when we think in our mind, we feel we have got the full solution, only to find the gaps when we put it down to paper or attempt to code it right away. I think this is certainly an advice that will ring true for a long time given the way our brains have evolved.

* Chapter 11 talks about the need for pilot implementations a la chemical plants and why it is better to plan to throw a prototype away. Agree that this is probably not done often in practice, especially when you are working on a customer defined system, but the fact that it may be a better idea to throw out something along the way is still true. We often struggle with ill-designed modules for a long time, because we are not sure if a redesign can be done in time for the next release. ☺ Interestingly in this chapter he talks about the challenges of creating a technical growth ladder and the need to evangelize it, and give the same degree of prestige to the roles people play there. I found this excellent advice even today, especially for my current role ☺.

(Continued in Part 4)..

Ramesh N Raghavan said...

(Part 4)…

* In Chapter 12, Brooks discusses the need and critical use of various tools, especially from a system software point of view, like simulators, emulators, the need to standardize tool usage across the team, explains how a version control system should be used (differentiates between a programmers personal playpen versus the sub-library where he submits his finished work for integration). He also talks about need for programming time on actual hardware in blocks of time that helps the programmer to debug issues much faster and not have to wait for another 6 hours before he can try his fix again. Though normal desktop resources are not an issue these days, getting time on actual newly minted hardware can still be scarce and his advice on how to allocate time on such systems makes a lot of sense even today. The last section is about Interactive programming which was new then, and Brooks correctly predicts that it is there to stay and we are to this day in that model.

* Chapter 13 is all about integration and debugging, and I did not find even one suggestion out of place even today.

* Chapter 14 is about project management, the need to have the patience to not intervene at the first bad news and demoralize your subordinate, and decide when you must step in. He says one should distinguish between Status information and Action information. He also talks about the need for PERT charts and identify which tasks are in the critical path. Though the industry by now should have advanced to have all the Project managers understand the need to identify dependencies and lay them in PERT charts, we often find many small projects being run out of the head, with no such chart in evidence. He also stresses the need to define sharp measurable milestones that one can test for. I am reminded of some of the serious debates that we have with customers on whether the milestone has been met ;)

(continued in Part 4b)

Ramesh N Raghavan said...

(Part 4b)

* The final chapter, Chapter 15 is about external documentation on how a program should be used, and also other ways to document a program, like self-documenting code (by which he means reasonably well documented code with comments in the source file), the challenges of using Flow charts (he is quite strong when he says that the detailed blow-by-blow flow chart is an obsolete nuisance, suitable only for initiating beginners into algorithmic thinking). His analysis of why Flow charts are obsolete with modern high-level languages being available is very educative. He also talks about naming conventions and also code commenting guidelines which all still makes sense. At the time of writing what he suggested were difficult to implement, but he rightly concludes saying that those limitations are on the way out and it will probably be the way forward which is quite true today.

Thanks to anyone who had the patience to read this far. My only hope is that you find it intriguing enough to seek out for a copy of the book.

Best Regards,
N.R. Ramesh.