Metaphysics of Software Development

A mature process?

2011-11-09T14:12:00.000-02:00

"We have a mature software process. We are now going to invest on the quality of our products."

There is an irony in intellectual debates. Fallacies and incoherent reasoning are much easier to express in a concise way than the truth. Take the statement above. There are several fallacies compacted inside these two nice little sentences. But it will take many more words than that to refute them. So, let’s do our job:

The process does not really matter. People are the important element of a successful (or unsuccessful thereof) software project. The Agile movement has been emphasizing this for over a decade now and managers still do not get it. But even if there were no movement shouting this truth out loud, it would take only a bit of common sense to realize it. Think of it: the most complicated problems in a software project can only be solved with a good dose of creativity. After all, a project is, by definition, the effort to develop new things. As such, new problems will arise and will require dedicated, talented team members to solve. No process could possible take care of this.
Similarly, no process can ever be mature. The specific conditions will always change from project to project, from team to team. Technologies (like programming languages, database systems, operating systems etc), performance requirements, the application domain all affect how a certain team will work on a certain project. And this will change over time, too. As a team’s knowledge increases, their approaches to problems will change accordingly.
The whole idea of a process is to have a fixed set of procedures and rules in place, so that no matter who gets to be working on the project, the results will always be the same. This is completely illusory, since it is people that are important (cf. item 1), but that is, at least, the reasoning behind the idea of a process. So, the question that must be answered at this point is: if the process is mature, why does it not automatically produce quality products?
The order is inverted. The first concern should be exactly the quality of the products. This is what the customer pays for and this is the end of the software development activity, after all. Should the need arise to implement some kind of process, so be it. But keep in mind that the process is the means and the product is the end.

Silos, impossible models and cognitive dissonance

2011-08-23T09:28:00.000-03:00

When I was in college, I worked on several small-scale exercise projects, from toy compilers to automation of mechanical devices, using old microcontrollers. All those projects were developed by small teams, composed of about two to five members; there was no hierarchical distinction among team members nor functional barriers or predefined responsibilities. Basically, everyone was responsible for “delivering the product”, whatever that would entail. Of course, in order to allow for tasks to be executed in parallel, we would try to break the work up into small, independent pieces, that were synchronized quite frequently. In addition to that, the “customer” (i.e. the course professor) would be consulted as soon as any problem arose or any critical decision needed to be made. Pair programming was very common and little to no paperwork was done.

The description above sounds very much like the “new methodology” called agile, that was being formalized precisely during my college years. At that time, however, I had never heard the word “agile” used in this sense. The way we worked on our projects was felt as just the natural way of doing things. This so called “methodology” is nothing more than plain old common sense; a set of simple rules that every normal human being would abide by under similar circumstances. And, in fact, I think this is what all those great developers meeting at Snowbird had in mind when they wrote the agile manifesto. After all, they had been working like that for several years before.

Evidently, in an enterprise environment, the reality is different from college experience. There is more money involved, the stakes are higher, deadlines are stricter and so on. But apart from those accidental complexities, the work itself is pretty much the same. And most start-ups actually develop software the way college students do. Then, some time later, as the company grows, this way of developing software starts being perceived by management as “not professional enough”. And it becomes common to hear statements along the lines of “we need to have a process in place to organize this mess.” The next step is to hire project managers who sign their names followed by a bunch of three-letter acronyms. Their mission is to set up a rigid and detailed process.

The company then decides that the next thing to do is to separate people and confine them to functional silos. Each of these instituted functional areas gets its own hierarchy and its own agenda. Once this arrangement has taken place, it is just a matter of time for each of them to develop its own culture, values, prejudices, incentive systems etc. Communication and coordination across functional areas then becomes increasingly difficult, as well as a big source of conflicts between their managers. Such conflicts can only be settled by an even higher hierarchical level.

As if this state of affairs were not troublesome enough, there is an even bigger problem. Underlying this “division of labor”, there is a theoretical model (whether consciously perceived or not), according to which each functional area is a self-sufficient black box, and that by juxtaposing the right boxes in the right arrangement, work products will flow through the system so that, eventually, a higher goal will be attained. A fundamental piece of this model is the enforcement of barriers between the boxes. No one is allowed to cross the border and do someone else’s work. In many instances, not even suggestions coming from another functional area are much welcome. Likewise, once a set of tasks is completed by a team, those tasks should be considered someone else’s problem.

More often than not, this model is not presented as explicitly as I describe it here. But people are smart and can intuitively grasp it. So, understanding how the rewards and punishments work, the rational reaction is to behave accordingly. Let’s take one of the many examples commonly found when a company decides do adopt such process: the barrier between “development” and “test” — by the way, this division does not work out that way in the real world. Working under the assumption that there is such a barrier, the development team will work hard within their limits until the product reaches the point where integrated tests need to be done. This sequence of events (design > implementation > tests) is usually prescribed by the process itself. When the time comes, developers throw the unfinished product over the wall, so that the test team can start working on their own. Testers will possibly throw some bugs back, but that is where the communication ends.

I do not mean to say that there are not “heroes” out there. Teams members that work closely with each other, despite the useless barriers imposed by management. But this does not change the overall picture: the incentive system leads people to do exactly the opposite. And it does not take much time for this model to prove itself very ineffective. When this happens, the red alert rings and developers and testers, who are working on the trenches, get reprimanded for not communicating well enough, not seeing the big picture and most of all, not working for the greater interest of the company. While managers are right in demanding that attitude of a shared responsibility, they do not realize that the policy of separating people according to their specialties is precisely what has caused the problem, in the first place.

Leaving aside the obvious point that the company is wasting money by trying to enforce an impossible model, there is also a serious psychological consequence to all this: people receive two contradictory sets of stimuli, leading to a situation that psychologists call cognitive dissonance. On the one hand, according to the official discourse, communication, knowledge sharing and out-of-the-box thinking are the kinds of behavior expected from everyone. In practice, however, the rules, rites and procedures adopted by the company work against that theory.

Imaginary constructions

2010-03-25T20:30:00.002-03:00

Ludwig von Mises — probably the greatest economist of the 20^th century — describes in his book "Human Action", an extremely useful method, employed by economic science: the method of imaginary constructions. According to Mises, "an imaginary construction is a conceptual image of a sequence of events logically evolved from the elements of action employed in its formation. It is a product of deduction, ultimately derived from the fundamental category of action, the act of preferring and setting aside. In designing such an imaginary construction the economist is not concerned with the question of whether or not it depicts the conditions of reality which he wants to analyze."

Before we go about understanding this seemingly obscure definition, we should first understand what the term action means in this context, since this is the essential fact underlying every economic phenomenon. Human action is purposeful behavior, whose end is always the relief from a felt uneasiness. Acting man, through a rational effort, attempts to substitute a more satisfactory state of affairs for a less satisfactory one.

With the concept of action in mind it is easier to grasp the concept of imaginary constructions, using examples. Let's consider the state of rest. It refers to a hypothetical market in which no exchange will be performed at all, either because every uneasiness has been removed or because it is no longer possible to remove them.

Two special cases are distinguished: the plain state of rest, a momentary state attained when buyers and sellers do not agree on the prices or when the conditions are not favorable for trade; and the final state of rest, in which the market activities come to an irreversible halt, no action being performed any longer. The prices of products in this final state are called final prices. The plain state of rest is real and is reached again and again (possibly everyday) in every market, but in the next instant the conditions change and the market starts "working" again. The final state of rest, on the other hand, is only hypothetical.

The evenly rotating economy — another imaginary construction — is a "fictitious system in which the market prices of all goods and services coincide with the final prices." This construction is different from the final state of rest in that buyers and sellers are in perpetual activity, but the economic data do not change. Everything stays constant, from market prices to population figures. In this hypothetical world, any medium of exchange would be of no use, since there is no uncertainty about the future. Supply would equal demand and every human dissatisfaction would be automatically removed. There would not be preferring nor setting aside, i.e., there would be no action.

Several other constructions like these are employed by economics. But it is important to remember that imaginary constructions are useful only as a limiting concept. Every one of those concepts lacks some feature of the real economy. This difference allows us to compare the real world with the imaginary world to better understand the nature of the missing feature. For instance, the evenly rotating economy is very useful for the economist trying to understand the nature of money.

Therefore, it is a serious mistake to deal with these constructions as if they were real. Take an example given by Mises himself: "the socialist scheme is logically compatible with the unrealizable imaginary constructions of an evenly rotating economy and of a stationary economy". But the socialist scheme is not compatible with the real world of human action. Indeed, all the socialist projects for a new society are based on a gross economic fallacy. No socialist government in history was (and never will be) able to succeed.

The waterfallish processes are the software development counterparts of these economic fallacies. As in economics, we might make a mental experiment and produce an imaginary construction of a software development environment with some desired characteristics. I am not saying that this method is actually suitable to software development. Anyway, the logical consequences derived from this approach are of great importance to the present discussion.

Following the procedure adopted by economics, we need to choose the features of the real world that are missing in this imaginary world. For example, we could create a world in which there would be no uncertainties about the requirements, so that they could be completely, correctly and unambiguously gathered before writing a single line of code. The requirements, moreover, would stay constant over time (as well as the business circumstances that usually cause requirements changes). Additionally, in that imaginary world, every software activity (requirements, design, test etc) could be performed in isolation, at the point of being executed by different people at different times. In other words, the inherent coupling between all the activities would not exist.

By making some additional abstractions like these from the real world, we could arrive at an imaginary construction that would be logically compatible with every flavor of waterfallish process. But this logical compatibility is meaningful as long as it is treated as a mental tool for the understanding of some aspect of software development. Unfortunately, the champions of heavyweight processes often forget that this is just an imaginary construct and starts treating it as real. As a consequence, developers are told to follow a set of rules that comply only with an imaginary and unrealizable world. So, it is easy to see that their appeal to the logical consistency of their rules is deceiving. Logical consistency means nothing if detached from reality. This is the reason why all such processes fail miserably. In a sense, they are not much different from socialist regimes.

Intertwined concerns

2010-02-17T09:32:00.000-02:00

As I have been writing about the ontological nature of software development, it is becoming more and more clear that there is central thesis in this blog: that the division of software development in activities is methodological, and not ontological. All other themes seem to gravitate around this one. So, I would like to go a little deeper into that discussion, by presenting a few examples.

Let's start with requirements. Several authors and processes advise their readers to gather requirements from the customer at the initial stages of a project, register them in requirements documents, then declare that phase completed, handing all the resulting artifacts to other developers to continue the work from that point. Note that in this advise there are some hidden premises: first, that this is a time-framed activity, precisely scheduled by some project manager. Second, the duration of that work is supposed to be relatively short, compared to the whole project schedule. After all — goes the advice — once the requirements have been collected, there is no longer a need for customer feedback. And third, there is the idea of avoiding the overlapping of activities. "Don't move on to the next phase until the customer has approved the final version of the requirements documents."

This advice is based on the assumption that there is an independent entity in the real world called "software requirements". Being an entity in and of itself, it could be tackled separately from all the other activities. Since every other activity depends on requirements, the latter should be done first and completed as soon as possible.

But it is important to recognize that the requirements aren't just wishlists dictated by the customer. The requirements are behavioral and structural expectations for a software application. The whole purpose of any application is to solve a particular problem set for some particular domain. So, some kind of conceptual domain model — albeit a rough one — must be constructed while the requirements are collected. In addition, as Eric Evans points out, "new systems almost always have to be integrated with legacy or other systems", so it is inevitable to think of some form of high level architecture. But these requirements, when implemented, will have to be tested. So test is another concern the requirements analysts should not overlook. In fact, automated tests are an excellent tool to register requirements.

Because of all this, it's easy to see that requirements don't exist in a vacuum. They can only be separated by abstraction, a mental process that is necessarily done a posteriori. A similar analysis could be done for all the other activities. For instance, software design, as discussed above, is intertwined with requirements. Likewise, professional developers write unit tests for every piece of production code (preferably written before the production code itself). So, one more time, tests come into play. Since the best tool to represent a domain model for software applications is the source code itself, software design and implementation are indistinguishable. And don't forget that the application will eventually be deployed (another software activity), which will spark a torrent of ideas from the end users, resulting in more requirements, starting it all over. The interconnections are virtually endless and have only been superficially addressed here.

It doesn't take much effort to realize that all concerns involved in a software project are intertwined. It doesn't make sense to set them apart and expect to get better results from such a strategy. Using an OO-inspired terminology, one could say that these concerns are highly coupled. High coupling is usually solved with encapsulation. But, as Martin Fowler puts it, "encapsulation is for objects, not for people".

On the explicit propositions

2010-02-11T16:02:00.000-02:00

Let reasonable debate mean the quest for the truth in a discussion, i.e., a sincere devotion, from all debaters, to get closer to the truth, regarding some subject matter. This attitude implies occasional changes of opinion and acknowledgment of errors. In other words, it implies the refutation of the false opinions that were already in people's minds before the discussion had started. False opinions may also naturally arise along the process, which are ideally detected and discarded in the dialectical confrontation.

Since every opinion (either true or false) is a proposition, there will certainly be refutations for some propositions in a reasonable debate. In order to be possible to refute any proposition, it is essential to recognize it as such. The debaters must be aware that they are uttering propositions, that is, sentences able of truth or falsehood. They should not take it for granted for the sole reason they ignore that possibility.

When stating something, a conscious debater is implicitly saying: "believe in what I say because this statement is true, even though I could say something false, instead. Put another way, the truth comes from the proposition that underlies my statement and not from the fact that I am stating it". On the other hand, for debaters who are unaware of their own propositions, the statements are merely followed by an implicit "believe in what I say". Period. Therefore, everyone involved in a discussion should recognize the propositions formulated by themselves. Anyone who is unable to do that cannot engage in any reasonable debate whatsoever.

Consider, for instance, someone who believes that the more textual documentation is produced, the better it is for a software project. The problem here is not that such a person is absolutely certain of that theory. A certainty, in this case, would be equivalent to affirming its truth, which thus stands opposite to falsehood. The problem is that they cannot even entertain the possibility of truth or falsehood, as far as the proposition is concerned. And it would hopeless, for refutation purposes, to cut the documentation down to a hundredth and yet achieve better and faster results. Before such an evidence, an unconscious debater would keep on repeating their initial assertions (after all, that is what they have known for several years, they learned that in school and so on and so forth). "Things have got to be that way".

Such people are invulnerable to any argument or evidence, regardless of how strong or convincing the argument or evidence may be. Whenever these debaters are recoiled by the power of their opponent's logical reasoning, it is very common to see them changing the subject of the discussion or presenting some fool objection. By constantly misguiding the discussion, those individuals can always escape from embarrassment, but cannot provide any contribution at all, either to the clarification of a question or to the pursuit of truth. The discussion is useless both for them and for the interlocutors.

The four discourses - part II

2010-02-08T19:03:00.001-02:00

Software development is a creative process, in which ingenuity and aesthetic concerns play a major role. Using an OO language, for instance, developers are free to choose how to distribute the behavior among classes, how these classes interact with each other and which layers will be used, just to name a few. But, although the set of possibilities is very large, it is not limitless. There are several technological and theoretical constraints, such as memory consumption, turnaround time, computational complexity, design complexity, coupling, cohesion, computability and formal correctness, among others.

According to the scale of four discourses proposed by prof. Olavo de Carvalho (see part I), we could say that, during the course of a project, developers go through all the levels, starting at the poetical one, when a great number of possible solutions are devised. The next step is to select a subset of those solutions that could possibly work (the rhetorical level). By confronting these apparently suitable solutions to one another, it is possible to identify which of them are inconsistent or incomplete and which ones are satisfactory. Sometimes, this confrontation of ideas can lead to the creation of a whole new set of acceptable solutions to the problem. This is what we could call a dialectical practice. Finally, the most likely solutions the team came up with are further refined and formalized, in order to be implemented them in some programming language. Evidently, this is not a linear process, and developers usually go back and forth, as the circumstances demand it.

Eventually some artifacts are produced as a result of the proceedings above, including use case specifications, UML diagrams, test reports, quality control reports and the like. This is especially true in the most bureaucratic, waterfallish companies. Every one of those artifacts, being instances of human discourse, can also be classified according to the four levels. But note that there is a key difference between a creative process and its resulting artifacts. The process, although it spans throughout all levels, is just a means to an end. What really matters is delivering a valuable product to the end user. It does not matter whether developers underwent a poetical stage early on; at the end, we all wish to deliver an executable and correct source code, amenable to lexical and syntactical analysis, mathematical proofs of correctness, style checking and so on. In other words, the final cause of this process is the logical discourse.

The artifacts, on the other hand, are crystallized forms of discourse, registered during the early stages, that are expected (a naive expectation, of course) to remain valid long after it has been produced. As a consequence, some artifacts stand at the level of high probability, such as the design diagrams and test result reports. Others fall at a lower level, the rhetorical one: examples include the process body of rules (usually a distorted version of RUP) and quality management reports (usually a bunch of finger-pointing paperwork). And there are those artifacts that could be put in the lowest level of the scale: use case documents are an outstanding specimen of the kind. Everything written down in such documents is nothing but poetical formulation. It represents just a possible world, very useful for hypothetical considerations, but with nothing that binds it to the actual world.

Actually, I think this is the ultimate criterion for classifying software artifacts according to the four discourses scale: how much it is bound by reality. In the case of software development, the reality forces come from the technological and theoretical constraints I mentioned above. After all, it is possible to write pretty much everything you want in a use case. But only a small subset of that is actually implementable. Consider this dramatic scenario: a requirements document which states that the application should divide integers by zero. It is even possible to get some client to "approve it", but no well-implemented compiler in the world would accept a tentative implementation of that requirement. This is a rather contrived example, but it is helpful to get the point: software development is a creative process, but it ultimately seeks to produce source code, a reality-bound artifact, which should be treated as the most important asset in the project.

The four discourses - part I

2010-02-07T09:38:00.001-02:00

One of the most important and well-known theories published by Olavo de Carvalho is that of the Four Discourses [book excerpt in Portuguese]. The theory states that in the works of Aristotle there is a core idea: that every human discourse may be classified as belonging to one the following four categories: poetical, rhetorical, dialectical or analytical (logical). Each of these is associated with a level of credibility that is expected from the audience.

Roughly speaking, the poetical discourse is targeted at the level of possibilities: people watching a play know that everything being presented at the stage is not true (they are "pretending", so to say). But nonetheless they realize that all those events could be true. The spectators can even feel the pain and enjoy the good fortunes experienced by the characters. This is what Samuel Taylor Coleridge called "suspension of disbelief".

The rhetorical discourse aims at creating a strong belief in the audience, in order to persuade them to take some decision. The rhetorical discourse is that of lawyers at courts and politicians in election campaigns, for example. The level of credibility intended here is similitude. It suffices to seem to be true, even though there is no certainty if what is being said is actually true.

The dialectical discourse is an attempt to reach a level of high probability. Beliefs and theories are tested and confronted with each other, so as to reduce the margin of error to a minimum acceptable threshold. This is the method employed by science and philosophy. As an investigation progresses (a process that could last for centuries), scientists come up with theories that are most likely to be true.

The last discourse — analytical — addresses absolute certainty. A mathematical proof is the canonical example of the analytical discourse; a (possibly long) chain of reasoning that begins with the first principles (axioms, definitions) and leads to conclusions that are necessarily true. Not the slightest error or doubt is accepted.

In the next post, I will be talking about the relationship between these levels of discourse and the artifacts produced in a software development project.

Principles and practices

2010-02-04T09:06:00.000-02:00

Every software development method aims to achieve certain goals, even if such goals are only implicitly stated. In order to achieve the desired objectives, each development method relies on a set of abstract principles, which are taken for granted. Theoretically, these principles can be applied to every particular case, without leading to too much contradiction. However, these general principles may not lend themselves to direct application in all the concrete cases, given their own abstract nature. To circumvent this problem, development methods usually have also a body of practical rules, which are more concrete and can be directly implemented with little or no adjustment.

The practical rules, therefore, are a means to applying the general principles to specific problems, such that developers abiding by the method get closer to their intended targets. As a consequence, the practical rules are subordinated to the principles and are only useful as long as they promote the adherence to the principles. But sometimes the practices happen to work against the principles themselves, which leads the developers astray, instead of getting them closer to the aimed targets.

Let's take a particular practice of a particular method to illustrate this. Consider the well-known ritual of daily Scrum. It is based on the first value and on the sixth and twelfth principles of the Agile Manifesto. As a practice, it states that developers should stand up together for 15 minutes every day to publicly update the project status. During the meeting, each team member speaks at a time, answering the following three questions:

What did you do yesterday?

What will you do today?

Are there any impediments in your way?

It is expected that, at the end of the meeting, everyone will be aware of all that is going on on the project. But, in teams with too many members (let's say, more than seven), the daily Scrum may lose its value altogether. Very large teams tend to commit to a large number of user stories. As user stories are not always strongly tied to each other, the team members become inclined — perhaps unconsciously so — to form "sub-teams" of 3 or 4 people. Each sub-team takes charge of a small subset of stories and has no time to get acquainted with the problems and solutions of the other sub-teams. Each sub-team has only a rough idea of what the others are doing.

So, during the daily meeting, when a member reports the status of their work, only the other members of their own sub-team are able to completely understand what it is all about. Hence, the meeting proves to be almost useless, since none of the members will get any new piece of information: those belonging to the same sub-team had already known all the details, while the others couldn't grasp most of what was being said.

In such cases, abiding by the rule of the daily meeting leads to a poorer communication, instead of a better one. Obviously, the problem in the above example is the team size and that is the real problem that needs to be tackled. But, supposing this is not possible (because of "orders from above", for example), it is important to recognize that the practice is not delivering the benefits it should. So, other forms of conveying information should be chosen and implemented, in spite of what the method prescribes. In summary: principles are more important than practices. If there is a better way to apply the principles, go for it.

Levels of abstraction

2010-02-02T09:10:00.001-02:00

Martin Fowler classifies the usages of UML diagrams in three modes: UML as sketch, UML as programming language and UML as blueprint. As sketches, UML diagrams are just great! They are a nice way to convey information to other developers or simply to help a developer focus on some specific parts of a system. As there is no maintenance involved, no need to use complicated tools (pencil and paper are enough) and no excessive formalism, the costs associated with this usage are nearly zero. As for the second mode, UML as programming language, I will not comment anything about, since I am not familiar with that usage.

So, I would like to write about the third mode: UML as blueprint. There is a common belief — especially in waterfall companies — that different software activities can be performed in isolation from one another. In particular, managers believe that "software design" is one thing and "implementation" is a completely different thing. So, theoretically, different people can be assigned different tasks and everything will just fit together perfectly. This theory leads to a strange role division: "systems analysts", who are in charge of producing a software design in UML and "coders", responsible for mechanically transforming the diagrams in executable code.

In that situation, what could be the "ideal" level of abstraction for the UML diagrams? If the level of abstraction is too high, the coders will have to fill in the gaps with their own knowledge or just make assumptions. But since they are not expected to understand anything about the business or the context of what they are producing, high-level designs are not an option. Indeed, the only useful level of abstraction in this case is the same as the source code's.

But what is the point in producing such low-level designs? I can hardly see any. First of all, if the activity of coding is just a manual copy from one language to another, it means that the effort needed to build (or maintain) anything is literally doubled, while providing no additional benefit. Another damaging consequence of that copy is that it violates the DRY principle. The likelihood that the two representations will become inconsistent increases as the time passes and the system gets larger. UML as blueprint has another major disadvantage: it is extremely hard to represent behavior using it and there is no way to "test" or statically verify the design, in some reliable way.

Modern programming languages are far better to represent a software design than any visual modeling language. In addition, there is a wide range of IDEs that help a lot in navigating through the code, checking style, performing formal verification, unit testing and so on. In summary, as the common sense teaches, the best way to design software is to use the source code itself.

Reasons behind functional separation

2010-01-31T10:30:00.001-02:00

As I have already commented, functional separation provides little benefit compared to the high costs it entails. But what are the causes of the functional separation mindset? What reason managers have in defining such an anti-natural policy? I can think of at least three factors that influence the creation of functional silos: enterprise political reasons, methodological misconception and lack of skills.

Let's start with the political reasons; people in an enterprise environment usually wish to climb up the hierarchy, gain power, status and higher wages. This is a cultural trait (at least in Brazil) that seems to be inescapable and widespread. To some extent, there is no problem with those aspirations. But there is an unfortunate consequence: a pressure to create and maintain management titles within the enterprise. People long to be called "bosses". But in order to have commanders there must be subordinates. And to establish a team of commanders and subordinates, some justification must be provided. One of the best ways to do that is to create functional silos, since at least in theory, those silos have a big justification of existence: they are comprised of specialists in this or that activity.

Regarding the second reason, I have written more about it elsewhere. Here it suffices to say that clueless managers and directors make crucial decisions without knowing how the reality works. Well, at least the software development reality, as witnessed by those working on the trenches.

The third reason I could figure out is that in some organizations, a number of developers lack a great deal of necessary skills for real software development. These skills include some algorithmic proficiency, knowledge in programming logic, testing, domain modeling, object-orientation (where applicable) and so on. But unfortunately, there is a lot of people who consider themselves as software developers, yet have very little knowledge about the topics mentioned above. So, many of them sit behind a desk holding the title of, let's say, "requirements analyst". That means that their daily routine is to write useless text documents while staying as far as possible from what really adds value to the project: source code, databases, etc. Fortunately, this attitude is less common, since most companies push individuals to perform better an better, which results in real commitment, most of the time.

Notice that none of the aforementioned reasons has anything to do with software development itself. Considerations of efficiency, quality and creativity, for example, have no influence at all on the decision to separate professionals according to their specialties.

Use cases are broken windows

2010-01-28T21:31:00.001-02:00

The French classical liberal economist Frédéric Bastiat wrote in 1850 that a large number of economic fallacies are derived from the bad habit of only paying attention to "that which is seen", while disregarding "that which is not seen". This happens too often in the real world, as a result of laws and political decisions.

To illustrate the fallacy, Bastiat tells a story about a broken window: a boy carelessly breaks a shop's square of glass. People witnessing the anger of the shopkeeper try to comfort him. They remind him that the event was a good thing, since now the glazier will have work to do and the money he will earn from that work will be used to buy other things and so on and so forth, making the economic wheel spin. In other words, people are saying that the boy is an agent of construction, not of destruction.

This is a fallacy because it takes into account only what is seen — the glazier's being paid for his work —, but ignores what is not seen — the fact that the shopkeeper is now poorer. If the window had not been broken, he could spend that money to buy a new pair of shoes, for example, so that he would have a new pair of shoes and an intact window. And the economic wheel would spin all the same.

In software development, the same kind of fallacy can be found in the justification for writing those long, detailed use cases. The argument often used is that use cases are a good tool to understand what the application does or, at least, what it should do. So, the more detailed it is, the more information will be available. Although this is true, it accounts only for the immediate consequences. But there are several other (negative) consequences that are usually overlooked.

The most important one is the cost of producing the use cases, in the first place. Writing detailed use cases is a time-consuming activity, and depending on how much time is spent, the benefits the use cases provide may not pay off. Besides, use cases (detailed as they may be) are incomplete and error-prone documents by their own nature. Alistair Cockburn estimates that they constitute about only a third of all the requirements you need to collect, since business rules, data formats and external interfaces are not specified in use cases. But this is not the only reason why they are incomplete:

In complex domains, it is virtually impossible to foresee all interaction scenarios between the application and the actors.
Developers and customers learn about the domain as the system is developed. So, new and better ways of doing things might be discovered after the use cases had been written.
There is a large portion of critical information about the domain that remains as tacit knowledge in experts' minds, despite the best efforts to write them down during the "requirements phase".
Sometimes, especially when there is a need to interact with legacy systems, some requirements specified in the use cases cannot be implemented, because important data is missing in the old system, data cannot be exchanged the way it is expected, among other interaction issues. In fact, this is not an incompleteness issue, but rather a "error-proneness" one.

Of course, one can think of projects in which writing detailed use cases may be an appropriate strategy, even if the above negative consequences are considered. In other circumstances, it may be the opposite. But the point is that the negative consequences are seldom weighed against the obvious benefits. To be fair, this fallacy is not restricted to use cases. The justifications for nearly every waterfallish kind of documentation fall under this category.

Functional separation

2010-01-27T08:24:00.004-02:00

As I have said in another post, the division of software development in activities is methodological and not ontological. As such, there is little point in creating functional silos, each one responsible for a specific part of the whole process. Besides being a brutal violation of common-sense (because it tries to force a theoretical model upon the real world), there is at least one harmful consequence: the task oriented mindset.

When tasks are assigned to different, non-communicating teams, according to their "specialties", some people tend to perform dull tasks, while others do all the hard thinking, yet becoming somewhat limited to put their ideas into practice. Consider the far too common separation between developers and deployers; in theory, developers should be focused on creating a running application while deployers are supposed to adapt the application, as needed, to the working environments. In practice, however, the deploy task is usually assigned to system administrators, who are often very good at keeping the infrastructure up and running. But system administrators in general have no knowledge whatsoever of the details of the application and how it interacts with its environment.

Since someone has to take care of this kind of issues, the burden usually falls in developers' hands. But, to complicate things further, according to a common policy, developers do not have acesss to the working environments (not even the development environment, in some cases). The consequence is ridiculous: developers sitting side-by-side with deployers, with ther formers telling the latters step by step what to do. And this ritual is performed everytime some deployment work needs to be done. Even if some deployers learn to do the job, they will not be able to cope with the slightest change in the interaction between the application and the environment.

Variations on this example can be produced for data administrators, testers, webdesigners and so on. When people are told that their main concern should be executing a certain task, they will naturally ignore the context in which the task is performed. In a software project, people should be focused on the project goals, not on the tasks that are needed for achieving it. People working under the task oriented mindset tend to behave as labourers, not professionals, as Uncle Bob would wisely say. As such, they cannot contribute to the project with creative ideas, better solutions for known problems and so on.

This consequence seems to be fairly predictable, since there is a tacit assumption among most software project managers that software development is a manufacturing activity. But, as every developer working on the trenches know, developing software is a creative process, more like writing a book than executing repetitive tasks on an assembly line. The communication between people working on the project (no matter what tasks they are performing) is essential to its sucess. And functional separation presents a huge barrier to effective communication. Despite all these obvious observations, several companies still insist is separating people according to the tasks they are supposed to execute and encouraging them to be short-sighted, believing that separation will speed up the development.

Common sense agile

2010-01-26T13:11:00.003-02:00

"Agile is mostly common sense", reads the title of a blog post written by Brad Feld a few years ago. I completely agree and go further: the agile principles, as published in the Agile Manifesto, are self-evident truths. There is simply no other way of efficiently developing software, as every college student may easily acknowledge.

I use to think of it by an analogy: everyone knows by their own experience that if you bathe a piece of cloth in water, it will come out wet. This is obvious! Nevertheless, one can come up with all sorts of scientific proofs of that fact. All these proofs, however, are not meant persuasion. Instead — supposing someone has actually produced such proofs — they are mere scientific puzzles; little games with the sole intent of testing one's own fluency in the matter.

And why is this so? For two basic reasons: the first one (as I already mentioned) is that every single person in the world has known that fact ever since they were born. The second one is that if anyone who has never had that simple experience (supposing there is one such person) could not even start following a complicated chemical explanation. Knowing a fact is a prerequisite for understanding its proof.

The same holds true for software development. Every time I see an eloquent presenter quoting the Agile Manifesto, showing the tree swing cartoon or exposing statistical data from the 1995 Standish Group report, it is as if they were saying that the water makes clothes wet. Those in the audience who do not know that instinctively are not able to understand a word of what is being said. And those who are already acquainted with the presentation contents are just losing their times. I myself have spent some time trying to convince people that agile software development is important. But now I recognize how vain it is.

In spite of the obviousness of Agile, we all know people who advocate for those heavy, complicated, waterfallish processes as the panacea for software projects. They do it by writing those boring and completely unreal academic books, speaking in conferences, teaching at university courses about the matter and so on. Worse, some of them are senior managers who impose those processes on projects under their supervision.

Since what they propose is so anti-natural and obviously flawed, we can come up with some hypotheses for such phenonemon: either these people are 1) extremely stupid, 2) evil-minded or 3) totally laymen as far as software development is concerned. The last hypothesis, though, is far more likely than the previous ones. In either case, developers should be wary of those harmful ideas, that start off as academic vices and then spread out to the IT community, leading to miserable project failures.

Methodological division

2010-01-25T12:05:00.002-02:00

Every science approaches its study object by making an abstractive clipping: some aspects of the studied object are taken into account while others are ignored. So, the same object may be of interest to a number of different sciences. Take, for instance, an orange: for Chemistry, what matters in an orange are the substances that compose it, the reactions that occur under given circumstances and so on. For Biology, other aspects of an orange are considered: its cells, its anatomy, how it grows and a few other things. Economics would try to make a forecast of the orange juice prices in the near future, considering current demands for the product and its quality, among other assessments. This list of sciences could grow almost indefinitely.

In spite of all the clippings we can make on the object, we will never be able to find a purely biological orange (or a purely economical orange thereof). Every orange has all those aspects at the same time. When studying a complex object (and all scientifically interesting objects are complex) it makes sense to divide it into several "pieces" and focus our attention on one or two of those pieces at a time and ignore the others. Or, at least, not dive into much detail.

What is most important to bear in mind is that when scientists ignore something, they are not saying that that thing does not exist. Similarly, taking only one object into account at a time does not mean that such object exists independently of the others. More formally, we can state that the aforementioned division is only a methodological precept and not an ontological one. According to the philosopher Olavo de Carvalho, "a methodological precept teaches you how to investigate things, an ontological principle establishes how things really are" [Article in Portuguese].

In the study of software development processes, there is a conventional division in activities (or disciplines): business modeling, requirements, analysis & design, implementation, test and deployment. So, the same thinking that applies to the orange also applies here. That set of disciplines is a methodological division, made a posteriori and intended to assist the study of software development. It does not mean that there is one such thing in the reality called, say, "software design", that exists in itself, indepedent of the other activities.

Those who have been developing software for some time now acknowledge this fact intuitively, even though people usually do not think in those philosophical terms. However, it is unfortunately very common to see companies trying to develop software as if this division was ontological. It is not rare to hear a project manager say something along the lines: "Alice, you will do the analysis & design of the system and you, Bob, will implement it, based on the result of Alice's work".

This is an example of a little philosophical misunderstanding that leads to all sorts of absurdities and failures in software development projects. People try to force the reality to behave according to the theoretical models they create, and not the other way around. I intend to write about the consequences of this inversion in the following posts.