Metaphysics of Software Development

Some insights on the software development craft, from a philosophical viewpoint.

Wednesday, November 9, 2011

 

A mature process?


"We have a mature software process. We are now going to invest on the quality of our products."
There is an irony in intellectual debates. Fallacies and incoherent reasoning are much easier to express in a concise way than the truth. Take the statement above. There are several fallacies compacted inside these two nice little sentences. But it will take many more words than that to refute them. So, let’s do our job:
  1. The process does not really matter. People are the important element of a successful (or unsuccessful thereof) software project. The Agile movement has been emphasizing this for over a decade now and managers still do not get it. But even if there were no movement shouting this truth out loud, it would take only a bit of common sense to realize it. Think of it: the most complicated problems in a software project can only be solved with a good dose of creativity. After all, a project is, by definition, the effort to develop new things. As such, new problems will arise and will require dedicated, talented team members to solve. No process could possible take care of this.
  2. Similarly, no process can ever be mature. The specific conditions will always change from project to project, from team to team. Technologies (like programming languages, database systems, operating systems etc), performance requirements, the application domain all affect how a certain team will work on a certain project. And this will change over time, too. As a team’s knowledge increases, their approaches to problems will change accordingly.
  3. The whole idea of a process is to have a fixed set of procedures and rules in place, so that no matter who gets to be working on the project, the results will always be the same. This is completely illusory, since it is people that are important (cf. item 1), but that is, at least, the reasoning behind the idea of a process. So, the question that must be answered at this point is: if the process is mature, why does it not automatically produce quality products?
  4. The order is inverted. The first concern should be exactly the quality of the products. This is what the customer pays for and this is the end of the software development activity, after all. Should the need arise to implement some kind of process, so be it. But keep in mind that the process is the means and the product is the end.

Tuesday, August 23, 2011

 

Silos, impossible models and cognitive dissonance

When I was in college, I worked on several small-scale exercise projects, from toy compilers to automation of mechanical devices, using old microcontrollers. All those projects were developed by small teams, composed of about two to five members; there was no hierarchical distinction among team members nor functional barriers or predefined responsibilities. Basically, everyone was responsible for “delivering the product”, whatever that would entail. Of course, in order to allow for tasks to be executed in parallel, we would try to break the work up into small, independent pieces, that were synchronized quite frequently. In addition to that, the “customer” (i.e. the course professor) would be consulted as soon as any problem arose or any critical decision needed to be made. Pair programming was very common and little to no paperwork was done.

The description above sounds very much like the “new methodology” called agile, that was being formalized precisely during my college years. At that time, however, I had never heard the word “agile” used in this sense. The way we worked on our projects was felt as just the natural way of doing things. This so called “methodology” is nothing more than plain old common sense; a set of simple rules that every normal human being would abide by under similar circumstances. And, in fact, I think this is what all those great developers meeting at Snowbird had in mind when they wrote the agile manifesto. After all, they had been working like that for several years before.

Evidently, in an enterprise environment, the reality is different from college experience. There is more money involved, the stakes are higher, deadlines are stricter and so on. But apart from those accidental complexities, the work itself is pretty much the same. And most start-ups actually develop software the way college students do. Then, some time later, as the company grows, this way of developing software starts being perceived by management as “not professional enough”. And it becomes common to hear statements along the lines of “we need to have a process in place to organize this mess.” The next step is to hire project managers who sign their names followed by a bunch of three-letter acronyms. Their mission is to set up a rigid and detailed process.

The company then decides that the next thing to do is to separate people and confine them to functional silos. Each of these instituted functional areas gets its own hierarchy and its own agenda. Once this arrangement has taken place, it is just a matter of time for each of them to develop its own culture, values, prejudices, incentive systems etc. Communication and coordination across functional areas then becomes increasingly difficult, as well as a big source of conflicts between their managers. Such conflicts can only be settled by an even higher hierarchical level.

As if this state of affairs were not troublesome enough, there is an even bigger problem. Underlying this “division of labor”, there is a theoretical model (whether consciously perceived or not), according to which each functional area is a self-sufficient black box, and that by juxtaposing the right boxes in the right arrangement, work products will flow through the system so that, eventually, a higher goal will be attained. A fundamental piece of this model is the enforcement of barriers between the boxes. No one is allowed to cross the border and do someone else’s work. In many instances, not even suggestions coming from another functional area are much welcome. Likewise, once a set of tasks is completed by a team, those tasks should be considered someone else’s problem.

More often than not, this model is not presented as explicitly as I describe it here. But people are smart and can intuitively grasp it. So, understanding how the rewards and punishments work, the rational reaction is to behave accordingly. Let’s take one of the many examples commonly found when a company decides do adopt such process: the barrier between “development” and “test” — by the way, this division does not work out that way in the real world. Working under the assumption that there is such a barrier, the development team will work hard within their limits until the product reaches the point where integrated tests need to be done. This sequence of events (design > implementation > tests) is usually prescribed by the process itself. When the time comes, developers throw the unfinished product over the wall, so that the test team can start working on their own. Testers will possibly throw some bugs back, but that is where the communication ends.

I do not mean to say that there are not “heroes” out there. Teams members that work closely with each other, despite the useless barriers imposed by management. But this does not change the overall picture: the incentive system leads people to do exactly the opposite. And it does not take much time for this model to prove itself very ineffective. When this happens, the red alert rings and developers and testers, who are working on the trenches, get reprimanded for not communicating well enough, not seeing the big picture and most of all, not working for the greater interest of the company. While managers are right in demanding that attitude of a shared responsibility, they do not realize that the policy of separating people according to their specialties is precisely what has caused the problem, in the first place.

Leaving aside the obvious point that the company is wasting money by trying to enforce an impossible model, there is also a serious psychological consequence to all this: people receive two contradictory sets of stimuli, leading to a situation that psychologists call cognitive dissonance. On the one hand, according to the official discourse, communication, knowledge sharing and out-of-the-box thinking are the kinds of behavior expected from everyone. In practice, however, the rules, rites and procedures adopted by the company work against that theory.

Labels:


Thursday, March 25, 2010

 

Imaginary constructions

Ludwig von Mises — probably the greatest economist of the 20th century — describes in his book "Human Action", an extremely useful method, employed by economic science: the method of imaginary constructions. According to Mises, "an imaginary construction is a conceptual image of a sequence of events logically evolved from the elements of action employed in its formation. It is a product of deduction, ultimately derived from the fundamental category of action, the act of preferring and setting aside. In designing such an imaginary construction the economist is not concerned with the question of whether or not it depicts the conditions of reality which he wants to analyze."

Before we go about understanding this seemingly obscure definition, we should first understand what the term action means in this context, since this is the essential fact underlying every economic phenomenon. Human action is purposeful behavior, whose end is always the relief from a felt uneasiness. Acting man, through a rational effort, attempts to substitute a more satisfactory state of affairs for a less satisfactory one.

With the concept of action in mind it is easier to grasp the concept of imaginary constructions, using examples. Let's consider the state of rest. It refers to a hypothetical market in which no exchange will be performed at all, either because every uneasiness has been removed or because it is no longer possible to remove them.

Two special cases are distinguished: the plain state of rest, a momentary state attained when buyers and sellers do not agree on the prices or when the conditions are not favorable for trade; and the final state of rest, in which the market activities come to an irreversible halt, no action being performed any longer. The prices of products in this final state are called final prices. The plain state of rest is real and is reached again and again (possibly everyday) in every market, but in the next instant the conditions change and the market starts "working" again. The final state of rest, on the other hand, is only hypothetical.

The evenly rotating economy — another imaginary construction — is a "fictitious system in which the market prices of all goods and services coincide with the final prices." This construction is different from the final state of rest in that buyers and sellers are in perpetual activity, but the economic data do not change. Everything stays constant, from market prices to population figures. In this hypothetical world, any medium of exchange would be of no use, since there is no uncertainty about the future. Supply would equal demand and every human dissatisfaction would be automatically removed. There would not be preferring nor setting aside, i.e., there would be no action.

Several other constructions like these are employed by economics. But it is important to remember that imaginary constructions are useful only as a limiting concept. Every one of those concepts lacks some feature of the real economy. This difference allows us to compare the real world with the imaginary world to better understand the nature of the missing feature. For instance, the evenly rotating economy is very useful for the economist trying to understand the nature of money.

Therefore, it is a serious mistake to deal with these constructions as if they were real. Take an example given by Mises himself: "the socialist scheme is logically compatible with the unrealizable imaginary constructions of an evenly rotating economy and of a stationary economy". But the socialist scheme is not compatible with the real world of human action. Indeed, all the socialist projects for a new society are based on a gross economic fallacy. No socialist government in history was (and never will be) able to succeed.

The waterfallish processes are the software development counterparts of these economic fallacies. As in economics, we might make a mental experiment and produce an imaginary construction of a software development environment with some desired characteristics. I am not saying that this method is actually suitable to software development. Anyway, the logical consequences derived from this approach are of great importance to the present discussion.

Following the procedure adopted by economics, we need to choose the features of the real world that are missing in this imaginary world. For example, we could create a world in which there would be no uncertainties about the requirements, so that they could be completely, correctly and unambiguously gathered before writing a single line of code. The requirements, moreover, would stay constant over time (as well as the business circumstances that usually cause requirements changes). Additionally, in that imaginary world, every software activity (requirements, design, test etc) could be performed in isolation, at the point of being executed by different people at different times. In other words, the inherent coupling between all the activities would not exist.

By making some additional abstractions like these from the real world, we could arrive at an imaginary construction that would be logically compatible with every flavor of waterfallish process. But this logical compatibility is meaningful as long as it is treated as a mental tool for the understanding of some aspect of software development. Unfortunately, the champions of heavyweight processes often forget that this is just an imaginary construct and starts treating it as real. As a consequence, developers are told to follow a set of rules that comply only with an imaginary and unrealizable world. So, it is easy to see that their appeal to the logical consistency of their rules is deceiving. Logical consistency means nothing if detached from reality. This is the reason why all such processes fail miserably. In a sense, they are not much different from socialist regimes.

Labels: , ,


Wednesday, February 17, 2010

 

Intertwined concerns

As I have been writing about the ontological nature of software development, it is becoming more and more clear that there is central thesis in this blog: that the division of software development in activities is methodological, and not ontological. All other themes seem to gravitate around this one. So, I would like to go a little deeper into that discussion, by presenting a few examples.

Let's start with requirements. Several authors and processes advise their readers to gather requirements from the customer at the initial stages of a project, register them in requirements documents, then declare that phase completed, handing all the resulting artifacts to other developers to continue the work from that point. Note that in this advise there are some hidden premises: first, that this is a time-framed activity, precisely scheduled by some project manager. Second, the duration of that work is supposed to be relatively short, compared to the whole project schedule. After all — goes the advice — once the requirements have been collected, there is no longer a need for customer feedback. And third, there is the idea of avoiding the overlapping of activities. "Don't move on to the next phase until the customer has approved the final version of the requirements documents."

This advice is based on the assumption that there is an independent entity in the real world called "software requirements". Being an entity in and of itself, it could be tackled separately from all the other activities. Since every other activity depends on requirements, the latter should be done first and completed as soon as possible.

But it is important to recognize that the requirements aren't just wishlists dictated by the customer. The requirements are behavioral and structural expectations for a software application. The whole purpose of any application is to solve a particular problem set for some particular domain. So, some kind of conceptual domain model — albeit a rough one — must be constructed while the requirements are collected. In addition, as Eric Evans points out, "new systems almost always have to be integrated with legacy or other systems", so it is inevitable to think of some form of high level architecture. But these requirements, when implemented, will have to be tested. So test is another concern the requirements analysts should not overlook. In fact, automated tests are an excellent tool to register requirements.

Because of all this, it's easy to see that requirements don't exist in a vacuum. They can only be separated by abstraction, a mental process that is necessarily done a posteriori. A similar analysis could be done for all the other activities. For instance, software design, as discussed above, is intertwined with requirements. Likewise, professional developers write unit tests for every piece of production code (preferably written before the production code itself). So, one more time, tests come into play. Since the best tool to represent a domain model for software applications is the source code itself, software design and implementation are indistinguishable. And don't forget that the application will eventually be deployed (another software activity), which will spark a torrent of ideas from the end users, resulting in more requirements, starting it all over. The interconnections are virtually endless and have only been superficially addressed here.

It doesn't take much effort to realize that all concerns involved in a software project are intertwined. It doesn't make sense to set them apart and expect to get better results from such a strategy. Using an OO-inspired terminology, one could say that these concerns are highly coupled. High coupling is usually solved with encapsulation. But, as Martin Fowler puts it, "encapsulation is for objects, not for people".

Labels: ,


Thursday, February 11, 2010

 

On the explicit propositions

Let reasonable debate mean the quest for the truth in a discussion, i.e., a sincere devotion, from all debaters, to get closer to the truth, regarding some subject matter. This attitude implies occasional changes of opinion and acknowledgment of errors. In other words, it implies the refutation of the false opinions that were already in people's minds before the discussion had started. False opinions may also naturally arise along the process, which are ideally detected and discarded in the dialectical confrontation.

Since every opinion (either true or false) is a proposition, there will certainly be refutations for some propositions in a reasonable debate. In order to be possible to refute any proposition, it is essential to recognize it as such. The debaters must be aware that they are uttering propositions, that is, sentences able of truth or falsehood. They should not take it for granted for the sole reason they ignore that possibility.

When stating something, a conscious debater is implicitly saying: "believe in what I say because this statement is true, even though I could say something false, instead. Put another way, the truth comes from the proposition that underlies my statement and not from the fact that I am stating it". On the other hand, for debaters who are unaware of their own propositions, the statements are merely followed by an implicit "believe in what I say". Period. Therefore, everyone involved in a discussion should recognize the propositions formulated by themselves. Anyone who is unable to do that cannot engage in any reasonable debate whatsoever.

Consider, for instance, someone who believes that the more textual documentation is produced, the better it is for a software project. The problem here is not that such a person is absolutely certain of that theory. A certainty, in this case, would be equivalent to affirming its truth, which thus stands opposite to falsehood. The problem is that they cannot even entertain the possibility of truth or falsehood, as far as the proposition is concerned. And it would hopeless, for refutation purposes, to cut the documentation down to a hundredth and yet achieve better and faster results. Before such an evidence, an unconscious debater would keep on repeating their initial assertions (after all, that is what they have known for several years, they learned that in school and so on and so forth). "Things have got to be that way".

Such people are invulnerable to any argument or evidence, regardless of how strong or convincing the argument or evidence may be. Whenever these debaters are recoiled by the power of their opponent's logical reasoning, it is very common to see them changing the subject of the discussion or presenting some fool objection. By constantly misguiding the discussion, those individuals can always escape from embarrassment, but cannot provide any contribution at all, either to the clarification of a question or to the pursuit of truth. The discussion is useless both for them and for the interlocutors.

Labels:


Monday, February 8, 2010

 

The four discourses - part II

Software development is a creative process, in which ingenuity and aesthetic concerns play a major role. Using an OO language, for instance, developers are free to choose how to distribute the behavior among classes, how these classes interact with each other and which layers will be used, just to name a few. But, although the set of possibilities is very large, it is not limitless. There are several technological and theoretical constraints, such as memory consumption, turnaround time, computational complexity, design complexity, coupling, cohesion, computability and formal correctness, among others.

According to the scale of four discourses proposed by prof. Olavo de Carvalho (see part I), we could say that, during the course of a project, developers go through all the levels, starting at the poetical one, when a great number of possible solutions are devised. The next step is to select a subset of those solutions that could possibly work (the rhetorical level). By confronting these apparently suitable solutions to one another, it is possible to identify which of them are inconsistent or incomplete and which ones are satisfactory. Sometimes, this confrontation of ideas can lead to the creation of a whole new set of acceptable solutions to the problem. This is what we could call a dialectical practice. Finally, the most likely solutions the team came up with are further refined and formalized, in order to be implemented them in some programming language. Evidently, this is not a linear process, and developers usually go back and forth, as the circumstances demand it.

Eventually some artifacts are produced as a result of the proceedings above, including use case specifications, UML diagrams, test reports, quality control reports and the like. This is especially true in the most bureaucratic, waterfallish companies. Every one of those artifacts, being instances of human discourse, can also be classified according to the four levels. But note that there is a key difference between a creative process and its resulting artifacts. The process, although it spans throughout all levels, is just a means to an end. What really matters is delivering a valuable product to the end user. It does not matter whether developers underwent a poetical stage early on; at the end, we all wish to deliver an executable and correct source code, amenable to lexical and syntactical analysis, mathematical proofs of correctness, style checking and so on. In other words, the final cause of this process is the logical discourse.

The artifacts, on the other hand, are crystallized forms of discourse, registered during the early stages, that are expected (a naive expectation, of course) to remain valid long after it has been produced. As a consequence, some artifacts stand at the level of high probability, such as the design diagrams and test result reports. Others fall at a lower level, the rhetorical one: examples include the process body of rules (usually a distorted version of RUP) and quality management reports (usually a bunch of finger-pointing paperwork). And there are those artifacts that could be put in the lowest level of the scale: use case documents are an outstanding specimen of the kind. Everything written down in such documents is nothing but poetical formulation. It represents just a possible world, very useful for hypothetical considerations, but with nothing that binds it to the actual world.

Actually, I think this is the ultimate criterion for classifying software artifacts according to the four discourses scale: how much it is bound by reality. In the case of software development, the reality forces come from the technological and theoretical constraints I mentioned above. After all, it is possible to write pretty much everything you want in a use case. But only a small subset of that is actually implementable. Consider this dramatic scenario: a requirements document which states that the application should divide integers by zero. It is even possible to get some client to "approve it", but no well-implemented compiler in the world would accept a tentative implementation of that requirement. This is a rather contrived example, but it is helpful to get the point: software development is a creative process, but it ultimately seeks to produce source code, a reality-bound artifact, which should be treated as the most important asset in the project.

Labels: , , ,


Sunday, February 7, 2010

 

The four discourses - part I

One of the most important and well-known theories published by Olavo de Carvalho is that of the Four Discourses [book excerpt in Portuguese]. The theory states that in the works of Aristotle there is a core idea: that every human discourse may be classified as belonging to one the following four categories: poetical, rhetorical, dialectical or analytical (logical). Each of these is associated with a level of credibility that is expected from the audience.

Roughly speaking, the poetical discourse is targeted at the level of possibilities: people watching a play know that everything being presented at the stage is not true (they are "pretending", so to say). But nonetheless they realize that all those events could be true. The spectators can even feel the pain and enjoy the good fortunes experienced by the characters. This is what Samuel Taylor Coleridge called "suspension of disbelief".

The rhetorical discourse aims at creating a strong belief in the audience, in order to persuade them to take some decision. The rhetorical discourse is that of lawyers at courts and politicians in election campaigns, for example. The level of credibility intended here is similitude. It suffices to seem to be true, even though there is no certainty if what is being said is actually true.

The dialectical discourse is an attempt to reach a level of high probability. Beliefs and theories are tested and confronted with each other, so as to reduce the margin of error to a minimum acceptable threshold. This is the method employed by science and philosophy. As an investigation progresses (a process that could last for centuries), scientists come up with theories that are most likely to be true.

The last discourse — analytical — addresses absolute certainty. A mathematical proof is the canonical example of the analytical discourse; a (possibly long) chain of reasoning that begins with the first principles (axioms, definitions) and leads to conclusions that are necessarily true. Not the slightest error or doubt is accepted.

In the next post, I will be talking about the relationship between these levels of discourse and the artifacts produced in a software development project.

Labels: , , ,


Archives

January 2010   February 2010   March 2010   August 2011   November 2011  

This page is powered by Blogger. Isn't yours?

Subscribe to Posts [Atom]