Metaphysics of Software Development
Some insights on the software development craft, from a philosophical viewpoint.Tuesday, August 23, 2011
Silos, impossible models and cognitive dissonance
When I was in college, I worked on several small-scale exercise projects, from toy compilers to automation of mechanical devices, using old microcontrollers. All those projects were developed by small teams, composed of about two to five members; there was no hierarchical distinction among team members nor functional barriers or predefined responsibilities. Basically, everyone was responsible for “delivering the product”, whatever that would entail. Of course, in order to allow for tasks to be executed in parallel, we would try to break the work up into small, independent pieces, that were synchronized quite frequently. In addition to that, the “customer” (i.e. the course professor) would be consulted as soon as any problem arose or any critical decision needed to be made. Pair programming was very common and little to no paperwork was done.
The description above sounds very much like the “new methodology” called agile, that was being formalized precisely during my college years. At that time, however, I had never heard the word “agile” used in this sense. The way we worked on our projects was felt as just the natural way of doing things. This so called “methodology” is nothing more than plain old common sense; a set of simple rules that every normal human being would abide by under similar circumstances. And, in fact, I think this is what all those great developers meeting at Snowbird had in mind when they wrote the agile manifesto. After all, they had been working like that for several years before.
Evidently, in an enterprise environment, the reality is different from college experience. There is more money involved, the stakes are higher, deadlines are stricter and so on. But apart from those accidental complexities, the work itself is pretty much the same. And most start-ups actually develop software the way college students do. Then, some time later, as the company grows, this way of developing software starts being perceived by management as “not professional enough”. And it becomes common to hear statements along the lines of “we need to have a process in place to organize this mess.” The next step is to hire project managers who sign their names followed by a bunch of three-letter acronyms. Their mission is to set up a rigid and detailed process.
The company then decides that the next thing to do is to separate people and confine them to functional silos. Each of these instituted functional areas gets its own hierarchy and its own agenda. Once this arrangement has taken place, it is just a matter of time for each of them to develop its own culture, values, prejudices, incentive systems etc. Communication and coordination across functional areas then becomes increasingly difficult, as well as a big source of conflicts between their managers. Such conflicts can only be settled by an even higher hierarchical level.
As if this state of affairs were not troublesome enough, there is an even bigger problem. Underlying this “division of labor”, there is a theoretical model (whether consciously perceived or not), according to which each functional area is a self-sufficient black box, and that by juxtaposing the right boxes in the right arrangement, work products will flow through the system so that, eventually, a higher goal will be attained. A fundamental piece of this model is the enforcement of barriers between the boxes. No one is allowed to cross the border and do someone else’s work. In many instances, not even suggestions coming from another functional area are much welcome. Likewise, once a set of tasks is completed by a team, those tasks should be considered someone else’s problem.
More often than not, this model is not presented as explicitly as I describe it here. But people are smart and can intuitively grasp it. So, understanding how the rewards and punishments work, the rational reaction is to behave accordingly. Let’s take one of the many examples commonly found when a company decides do adopt such process: the barrier between “development” and “test” — by the way, this division does not work out that way in the real world. Working under the assumption that there is such a barrier, the development team will work hard within their limits until the product reaches the point where integrated tests need to be done. This sequence of events (design > implementation > tests) is usually prescribed by the process itself. When the time comes, developers throw the unfinished product over the wall, so that the test team can start working on their own. Testers will possibly throw some bugs back, but that is where the communication ends.
I do not mean to say that there are not “heroes” out there. Teams members that work closely with each other, despite the useless barriers imposed by management. But this does not change the overall picture: the incentive system leads people to do exactly the opposite. And it does not take much time for this model to prove itself very ineffective. When this happens, the red alert rings and developers and testers, who are working on the trenches, get reprimanded for not communicating well enough, not seeing the big picture and most of all, not working for the greater interest of the company. While managers are right in demanding that attitude of a shared responsibility, they do not realize that the policy of separating people according to their specialties is precisely what has caused the problem, in the first place.
Leaving aside the obvious point that the company is wasting money by trying to enforce an impossible model, there is also a serious psychological consequence to all this: people receive two contradictory sets of stimuli, leading to a situation that psychologists call cognitive dissonance. On the one hand, according to the official discourse, communication, knowledge sharing and out-of-the-box thinking are the kinds of behavior expected from everyone. In practice, however, the rules, rites and procedures adopted by the company work against that theory.
The description above sounds very much like the “new methodology” called agile, that was being formalized precisely during my college years. At that time, however, I had never heard the word “agile” used in this sense. The way we worked on our projects was felt as just the natural way of doing things. This so called “methodology” is nothing more than plain old common sense; a set of simple rules that every normal human being would abide by under similar circumstances. And, in fact, I think this is what all those great developers meeting at Snowbird had in mind when they wrote the agile manifesto. After all, they had been working like that for several years before.
Evidently, in an enterprise environment, the reality is different from college experience. There is more money involved, the stakes are higher, deadlines are stricter and so on. But apart from those accidental complexities, the work itself is pretty much the same. And most start-ups actually develop software the way college students do. Then, some time later, as the company grows, this way of developing software starts being perceived by management as “not professional enough”. And it becomes common to hear statements along the lines of “we need to have a process in place to organize this mess.” The next step is to hire project managers who sign their names followed by a bunch of three-letter acronyms. Their mission is to set up a rigid and detailed process.
The company then decides that the next thing to do is to separate people and confine them to functional silos. Each of these instituted functional areas gets its own hierarchy and its own agenda. Once this arrangement has taken place, it is just a matter of time for each of them to develop its own culture, values, prejudices, incentive systems etc. Communication and coordination across functional areas then becomes increasingly difficult, as well as a big source of conflicts between their managers. Such conflicts can only be settled by an even higher hierarchical level.
As if this state of affairs were not troublesome enough, there is an even bigger problem. Underlying this “division of labor”, there is a theoretical model (whether consciously perceived or not), according to which each functional area is a self-sufficient black box, and that by juxtaposing the right boxes in the right arrangement, work products will flow through the system so that, eventually, a higher goal will be attained. A fundamental piece of this model is the enforcement of barriers between the boxes. No one is allowed to cross the border and do someone else’s work. In many instances, not even suggestions coming from another functional area are much welcome. Likewise, once a set of tasks is completed by a team, those tasks should be considered someone else’s problem.
More often than not, this model is not presented as explicitly as I describe it here. But people are smart and can intuitively grasp it. So, understanding how the rewards and punishments work, the rational reaction is to behave accordingly. Let’s take one of the many examples commonly found when a company decides do adopt such process: the barrier between “development” and “test” — by the way, this division does not work out that way in the real world. Working under the assumption that there is such a barrier, the development team will work hard within their limits until the product reaches the point where integrated tests need to be done. This sequence of events (design > implementation > tests) is usually prescribed by the process itself. When the time comes, developers throw the unfinished product over the wall, so that the test team can start working on their own. Testers will possibly throw some bugs back, but that is where the communication ends.
I do not mean to say that there are not “heroes” out there. Teams members that work closely with each other, despite the useless barriers imposed by management. But this does not change the overall picture: the incentive system leads people to do exactly the opposite. And it does not take much time for this model to prove itself very ineffective. When this happens, the red alert rings and developers and testers, who are working on the trenches, get reprimanded for not communicating well enough, not seeing the big picture and most of all, not working for the greater interest of the company. While managers are right in demanding that attitude of a shared responsibility, they do not realize that the policy of separating people according to their specialties is precisely what has caused the problem, in the first place.
Leaving aside the obvious point that the company is wasting money by trying to enforce an impossible model, there is also a serious psychological consequence to all this: people receive two contradictory sets of stimuli, leading to a situation that psychologists call cognitive dissonance. On the one hand, according to the official discourse, communication, knowledge sharing and out-of-the-box thinking are the kinds of behavior expected from everyone. In practice, however, the rules, rites and procedures adopted by the company work against that theory.
Labels: functional silos
posted by Otavio Macedo # 9:28 AM 
Archives
January 2010 February 2010 March 2010 August 2011 November 2011
Subscribe to Posts [Atom]
Post a Comment