I’m leaving behind a career as an Agile consultant. I used to call me “Agile coach” but nowadays I see Jerry Weinberg’s definition of consulting – “influencing people at their request” – a much better fit for what I had been trying to achieve during the last 10 years. Which gets me in the middle of my frustration with the idea of bringing Agile to the masses.

During the first years of Agile development, Agile promotion and Agile influencing was a source of joy and positive fulfilment for me. Most of the time I had to deal with developers and teams who actually embraced the “extreme” ideas I brought them. Sure, their managers were initially unwilling to accept two people working on one computer or the team insisting on doing the estimations themselves. But in the end every team member knew they personally were much better off – even if they had to make a few compromises to interface with the non-agile rest of the company.

Then the nature of my assignments changed. Suddenly it was the CEO or at least s.o. from upper management who initiated the “Agile transition” in order to “adapt faster to a changing market”. At the beginning I deemed that a good thing, since I had always been complaining about the lack of management support. However, the work itself got ever more frustrating and ever less rewarding, especially when dealing with teams who were pushed towards Agile and hadn’t had the chance to choose for themselves. From then on, I spent most of my time trying to convince somebody of something that he or she didn’t actually want to do. If this sounds to you like a depressing way to earn your money, it was. Even when showing very hands-on techniques – like TDD or refactoring – I felt the lack of enthusiasm; first in my clients but then more and more in myself.

Slowly it dawned on me: We, as Agile consultants, have been abused by management to do all the convincing and motivation for them. In my opinion it should be the most essential part of their job to bring the right people together for the new challenge, might it be a new product, a leaner approach for the old product or a complete turn around of the company. Instead they hired us to do the impossible: Change their employees in a way that suits their latest business strategy. Accepting such an assignment often made us violate the “first Agile commandment”: People over process.

“People over process” can – and often should – mean: not doing an agile transition at all. Human beings have a right to choose which changes they want to go through and when. There are many valid personal reasons for not doing TDD, not taking accountability and not moving into a common team room. Let’s accept those reasons without being contemptuous and without trying to manipulate. Heck! – it’s the managers’ task to align their employees’ personal goals with those of the company. The few successes and the many failures during my time as Agile consultant have taught me one thing: It needs different people to make a different company. Rare exceptions will only prove the rule.

So, my dear fellow-coaches, my dear friends, we had a good time together. Sometimes even a great time. We brought teams from being overwhelmed by bugs to a zero-defect continuous feature flow. We turned around a company who was under severe legal pressure by its customers to a +50 net promoter score. We convinced senior managers that giving up control will enhance their teams’ productivity. But, on the way, we made some people unhappy, maybe a few very unhappy, and sometimes didn’t even notice.

I hope to see you all around, but I won’t be back to the party.

class Shelf {
    final products = []
    final lock = new ReentrantLock()
    void putIn(Product product) {
        lock {
            if (isFull())
                throw new StorageException("shelf is full.")
            products << product
        }
    }
    boolean takeOut(Product product) {
        lock {
            return products.remove(product)
        }
    }
}

It looks like lock was a new keyword but actually I achieved that with a tiny bit of Groovy meta programming:

Lock.metaClass.useFor = { Closure operation ->
    lock()
    try {
        operation()
    } finally {
        unlock()
    }
}
Lock.metaClass.call = { Closure operation -> delegate.useFor(operation) }

Now that I used it in a couple of examples I suggest it should be considered for inclusion in GDK. What do YOU think?

P.S.: Don’t get me wrong about the usefulness of explicit locking. In most cases other concepts – like parallel collections, data flows and agents – should be preferred.

Let’s look at a classical example: Calculating the factorial of a number. A recursive – but not tail recursive! – solution is this:

int factiorial(int number) {
     if (number == 1)
          return 1;
     return number * factorial(number - 1);
}

We can see here that tail recursion is not about the recursive call being the last thing in the function’s source code but in the function’s control flow!

For many recursive problems there’s an easy way to transform it to a tail recursive solution. Often the trick is to introduce an aggregator, i.e. an additional parameter that takes the intermediate result of a calculation. Doing that our function becomes:

int factorial(int number, int aggregator) {
    if (number == 1)
        return aggregator;
    return factorial(number - 1, number * aggregator);
}

Using this function requires a start value for the aggregator, in this case factorial(10, 1) would evaluate to 3628800 which sound reasonable. So how can we translate this into an iterative version? Let’s see:

int iFactorial(int number, int aggregator = 1) {
    int _number = number;
    int _aggregator = aggregator;
    while(true) {
        if (number == 1)
            return _aggregator;
        _aggregator = _number * _aggregator;
        _number = _number - 1;
    }
}

The trick is to wrap our function into an endless loop, save the function args into local temps, replace all usages of the args with temps and replace recursive calls by setting the temps to the recursive call args. Sounds easy, eh? If the JVM won’t do that for usautomaticall why not build a recursion to iteration translator ourselves? Since Groovy is my favourite language on the JVM it seemed the perfect time and target to give my first shot at AST transformations. AST transformations are @nnotations that will be invoked at compile time to let you analyze and manipulate a programs abstract syntax tree. AST transformations can be used on different levels, in my case a local transformation seemed to suffice and I started my dive into the world of ASTNodes, expressions, statements and other weired things…

Actually, I don’t want to bother you with the details of how to test-drive AST transformations; I’m still at the very beginning of figuring things out. What I want to show you is my first prototype: tailrec 0.1. Look what we can do now:

import groovyx.transform.*
@TailRecursive
int factorial(int number, int aggregator = 1) {
    if (number == 1)
        return aggregator;
    return factorial(number - 1, number * aggregator);
}

assert factorial(10, 1) == 3628800
assert factorial(10) == 3628800

The code runs as is in Groovy’s console as soon as you add tailrec jar file to your classpath. As you can see, when making use of Groovy’s default arguments we even get rid of the clumsy aggregator initialization. To prove that tailrec really does what it promises try the next example with and without @TailRecursive:

import groovyx.transform.*
@TailRecursive //remove line to get StackOverflowError
def countDown(long from) {
    if (from == 0)
        return 0
    countDown(from - 1)
}

countDown(1000000)

So far tailrec only handles non static functions, i.e. instance methods that have some real (non-void) return type. Moreover, there are a few cases I can think of that tailrec should and could handle correctly but does not (think: ternary operator in return statement). And there are a few difficult problems where I have no current idea how to solve; consider for example the use of recursive calls within closures.

The current version is a proof of concept. I want you to experiment with it and give me your feedback. In my opinion tailrec should become part of standard GDK if it’s worth doing it at all.

Update:
The code has now moved to github. I’m actually planning to enhance and stabilize it since the groovy team encouraged me to. It meanwhile supports static functions as well and has many more tests to earn your trust.

• For the first time in my career I’m trying to organize a public workshop completely on my own. The topic is Advanced TDD and I’m very thrilled about the simulations and exercices I want to do there. The first run will be in German.
• I’m auctioning one of the seats for this workshop on ebay. Let’s see how this turns out.
• Since I’m very eager to reach the minimum participation of 4 as soon as possible, I’ll give a commission of 20 percent to anyone who’ll send me a customer until 4 seats are taken. Just tell the interested person to refer to you when they contact me.

Update
The first auction ended at 401 EUR. So I started a second one which resulted in 512 EUR. Not bad I’d say..

Update 2
I’m sold out 6 weeks in advance.

This one shows how code samples are currently being displayed in my blog:

The second shows how code would be displayed if I wasn’t to renew my custom css option:

Is the difference worth 15 USD per year?

The exercises you are supposed to work on in my workshops are not very complicated but they stress one or two essential points. If you just rush over them, you can solve the task, but most of the effect lies in thinking about and discussing the subtleties of different solutions and approaches. That’s why I make participants work in pairs, especially in Pair-Programming workshop! Watching people work reveals that nowadays many – if not most – don’t do this carefull pondering and discussing anymore. They just hurry to a solution, going back to their email-reader or smart phone as soon as a solution to the task at hand seems to be found. When doing it this way most of the benefits of Pair-Programming, Test-Code-Refactor & Specifiaction by Example are lost. It’s all about thinking, discussing, experimenting, rethinking, rediscussing and improving.

During my first years of giving talks and presenting tutorials I always stopped on ringing cell phones, and I made sure the workshop computers had no internet access. I asked people to either switch their phones off or to leave. In short: I required attendants to be attentive! I guess I’ll have to reestablish these rules in workshops to come. Not in order to patronize people, just to become effective again.

Recently, on my ongoing quest for the wholly grail of acceptance testing, I examined and reconsidered the available AT frameworks such as FitNesse, Concordion, Cucumber and some of their siblings. At some point or other all of them have to solve the task of parsing text snippets from the application domain in order to extract some information which will then be used to execute the real code, aka the system under test. Consider for example the following step definition from Cucumber’s introductory tutorial:

Given /I have entered (.*) into the calculator/ do |n|
     calculator = new Calculator()
     calculator.push(n.to_i)
end

What this code does is  straightforward: It checks if the line in the test scenarios, ehm sorry, in the behaviour specification matches “I have entered…”, extracts the value of the number and tries to feed a calculator object with it. This allows for nearly natural language specifications of requirements, if – and only if – you stick exactly to the given pattern. Slight variations, like using “put in” instead of “entered into” will either break the spec execution or require you to write more and more complex (and unreadable) regular expressions. What if – so my line of thoughts – if we could use parsing technology to make text snippet interpretation in testing tools more robust and thus more liable to be used by people from the application domain?

No sooner said than done. Being mostly located in the JVM world of Java & Groovy, I figured that ANTLR would be a safe bet to use. Since buying (and sometimes reading) books has been one of my favourite coping strategies for a couple of years, I ordered “The definitive ANTLR Reference”, installed the Eclipse Antlr plugin and dived in. Well, to tell you the truth, it was a short – and cold – swim since I couldn’t find any really simple examples, had to use special editors, go through a code generation step, look for where the generated classed had been placed and so on and so on.

This meant it was time for thinking yet again: If I really wanted parsing technology to be an intergrated part of a testing framework this technology would have to be lighter, meaning at least: no code generation required, no extra tooling required except what the language requires anyway. As a consequence, the ANTLR Reference has joined the fate (the shelf that is) of a dozen other unread books, but I rememered a book I read in 2001: “Building Parsers with Java” by Steven John Metsker. Therein John introduces a framework to build left recursive parsers by programming only. This was exactly what I needed, despite the fact that the library is in  fin de siecle Java style, i.e. JDK 1.1 without the new collections and without generics and with enumerators instead of iterators… You get the picture. That’s why I tried to find out if Steve had maintained and modernized his library only to learn the sad truth that he had passed way in 2008. The book is still available and his website is still up; you can download the original version of his library and the examples there.

Since the topic fascinated me so much I eventually ended up modernizing Steve’s original library. Moreover, I added an additional layer to  facilitate the creation of grammars. The whole thing is called Parse4SJM in order to give continuous credit to Steve John Metsker. The library is basically an implementation of a recursive descent parser with backup; this type of parser has the comfortable property that you do not need to specify any look ahead since all possible interpretations of a grammar will be tried – if necessary. Of course, this feature comes with a price: potentially exponential execution time.

Let’s consider a simple example. You want to create a language to parse and evaluate simple math expression with addition, e.g. “3 + 5″ or “1.0 + 2  + 3.1″. Your final grammar should look like that:

expr = term ('+' term)*;
term = Num;

Being agile we get to the end result in small steps, of course. A simple step is to start with the term rule:

import sjm.grammar.Grammar
Grammar mathGrammar = new Grammar("simple math");
mathGrammar.defineRule("term = Num;", new IParserMatched() {
  public void apply(List<Object> matches, 
        Stack<Object> stack) {
    stack.push(((Token) matches.get(0)).value());
  }
});

IParsingResult result = mathGrammar.parse("5.1");
assert result.getStack().pop() == new BigDecimal("5.1");

This is as easy as it gets – in Java. In Groovy you can get rid of a bit ceremony:

    
def mathGrammar = new Grammar("simple math")
mathGrammar.defineRule("term = Num;") 
  { matches, stack ->
    stack.push(matches[0].value())
  }
def result = mathGrammar.parse("5.1")
assert result.stack.pop() == 5.1

What the code in the closure does is fairly simple: Whenever the rule matches, take the token from the stack and put its (mathematical) value on the stack instead. This kind of combined parsing and evaluation works quite often; if it doesn’t, Parse4SJM has other options as well. The full example looks like that:

def mathGrammar = new Grammar("simple math")
mathGrammar.discardAllConstants()
mathGrammar.defineRule("expr = term ('+' term)*;") 
  { matches, stack ->
    def sum = matches.inject(0) {sum, each -> sum + each}
    stack.push(sum)
  }
mathGrammar.defineRule("term = Num;") 
  { matches, stack ->
    stack.push(matches[0].value())
  }
result = mathGrammar.parse("5.1 + 2.0 + 1.8")
assert result.stack.pop() == 8.9

What’s my conclusion, then? Creating parsers can be easier than you think – and it doesn’t even require a code generation step. Maybe this approach can open the door for DSLs even a bit more. If you’re interested, just grab the jar or make your on clone from github. And then give me feedback. If you don’t the project will probably fall into oblivion, which should give you pangs of remorse.

I have been wondering for a while now why software design principles, heuristics and trade-offs are being described in so many different terms: coupling, cohesion, SOLID, LSP, DRY, LoD, you name it. It has always seemed obvious to me that there is only one concept that covers all the others: dependencies.

Here is my definition:

Party A depends on party B means: If party B changes one or more of its observable aspects, there is some probability (greater than zero) that party A will have to adapt.

Let’s examine the building blocks of this definition and its implications more closely:

• I use the word “party” instead of some more code-centric expression like “module” or “component”. Rationale: There are dependencies between pieces of my software (functions, classes, modules, subsystems, components etc) but there are also dependencies onto the outer world – the domain. For instance, within DRE I would consider the method who implements the logic for withdrawing money from an account as dependent on the domain rule how money should be withdrawn. Thus, DRE dependencies are a superset of static code dependencies.
• Change, the probability of the change and the probability of the change to affect the dependent together define the strength of a dependency. If the dependee can change (eg. disappear) without forcing the dependent to adapt there is no dependency worth mentioning. In that sense loose coupling has the goal to lower the probability of a change affecting the dependent and static typing usually raises the probability, e.g. by enforcing the number of arguments to a function call.
• I am only interested in the observables of a party like interface, timing behaviour, error conditions, availability, cost. To put it differently: I don’t care about unobservable implementation details or anything that can change without the dependent noticing.
• A dependency between individual parts results in a dependency between the respective aggregates. The more and the stronger the dependencies between the parts, the stronger the dependency between the aggregates.
• Dependencies are directed and that’s a good thing. On an atomic level there must not be bidirectional dependencies since otherwise change would result in an unstable state. On an aggregate level bidirectional dependencies arise much too easy if you are not careful.
• For most cases the dependency relation is not transitive. In some programming languages it seems like they are, but that’s usually because public visibility of inner parts is the default and the Law of Demeter strikes. One notable exception to the rule is facilitated by C/C++ compilers, which enforce recompilation of statically dependent components and thus propagate change all the way up.
• There exist at least three different major species of dependencies: compile-time dependencies, runtime dependencies and domain-rule dependencies. Statically typed and dynamically typed languages do different trade-offs between compile-time and runtime dependencies; type information is one kind of explicit dependency, which is obviously preferable over implicit – i.e. potentially unknown – dependencies. Speaking of explicitness: unit tests and acceptance tests are a different way of making dependencies explicit. Many of the techniques that promise  you a loosely coupled design do nothing else than going from explicit dependencies (a statically enforced method call) to implicit dependency by putting some obfuscating mechanism (e.g. XML serializing) in-between. This does not help you a thing, it just makes the system slower and more complex by introducing additional dependencies to libraries and structured documents. Unless, of course, if you really really really need it for cross-platform, cross-version, cross-process communication.

What is Good Design then?

A good design – or architecture for that matter – in DRE is one in which the overall of dependencies and their strengths is minimized. Since strength is defined as the probability that a change will hit you there is quite some fortune telling involved in finding “the best” of all designs. In other words: if you cannot agree on a probable future you cannot agree on a good design. That’s why Agile designers use a simple heuristic to foretell the future: Everything will change but we don’t know how. In DRE-terms that means: optimizing dependencies within the system and assuming that the world outside (the domain and its rules) has zero probability to change. In practice this leads to lots of design changes in early stages of a project until the typical domain changes have been incorporated in internal design elements and will only affect the outer-most “parties” (e.g. the adapter class, the configuration file, the business rules database).

What follows from that is: If you really know the future, the agile approach of evolutionary design is not the best. But be honest, who the hell does?

Reformulating OO Design Principles

For most heuristics of good OO design it’s quite easy to see why they work at least as well in the DRE universe. I leave it to the astute reader to translate the SOLID principles into DRE speak. One central rule is not so easy to reconcile with DRE, namely Don’t Repeat Yourself. I’ll try it anyway:

Consider a simple case of duplication:

def calculationOne() {
  ...
  def salesTax = amount * 0.19
  ...
}
def calculationTwo() {
  ...
  def salesTax = amount * 0.19
  ...
}

Given a probability of p that the sales tax rate will change during the system’s life cycle, you have two dependencies with a strength of p, so your total dependency number is 2p. Let’s now apply DRY in a straightforward matter:

def calculationOne() {
  ...
  def salesTax = calculateSalesTax(amount)
  ...
}
def calculationTwo() {
  ...
  def salesTax = calculateSalesTax(amount)
  ...
}
def calculateSalesTax(amount) {
  return amount * 0.19
}

Given a probability of r that we will have to change the interface of calculateSalesTax later on, the overall dependency number is now 2r + p; if we have chosen our abstraction wisely, this figure will be lower than 2p. Thus, the code with less duplication is better design in DRE theory. Quod erat demonstrandum.

So what?

There are two striking reasons why the unified theory appears attractive to me:

1. It gives me the vocabulary to talk about the relation between different design approaches and thereby tackle questions like “What has loose coupling to do with the SOLID principles?” and “Does this decoupling technique really decouple anything?”.
2. It might provide me with a quantifiable means to compare different designs. Any tool vendors, contact me for licensing the approach!

Now it’s up to you all to tell me why this is utter nonsense or perfectly useless or both. And I do appreciate congratulations for my future nobel prize on computer science. Shoot!

Update:

To make that clear, it was not my intention to suggest that simply adding up propabilities would make for a mathematically sound model for a “design quality number”; it was just the simplest thing to do and it felt intuitive enough.

Basically one new feature got added. No bugs fixed since 1.2.0 beta – because no one has found any.

I added a new feature, too.