In the previous episode you’ve already seen jqwik in action. One of the interesting aspects of this PBT library is the fact that it’s not a standalone framework but that it hooks into JUnit 5 in order to “inherit” IDE and built-tool support.

Jqwik and the JUnit Platform

The fifth generation of JUnit does not only come with a modernized approach to write and execute tests, but it is based on the idea of providing a platform for a large spectrum of different test engines. An engine provides two entry points: One entry point is for discovering tests and test suites - e.g. through scanning parts of the classpath for methods with a certain annotation. The other entry point is used to run tests and test suites, usually the ones you have discovered during the discovery step. The rest, like filtering or selecting subsets of tests, is done by the platform itself.

The big advantage of such an approach is that any IDE and any build tool only has to integrate the platform and not the individual engines. It’s also a big plus for engine developers who don’t have to bother with aspects like public APIs for discovering and running their test specifications.

Moreover, the platform allows to have any number of engines in parallel. That’s how JUnit 5 provides full backwards-compatibility to JUnit 4 and how a smooth migration path can be realized. Using JUnit 4 (called Vintage), JUnit 5 (called Jupiter) and jqwik in a single project is not only possible, it’s really simple.

IntelliJ has been an early platform adopter for over a year now. As of March 2018, we also see native support from Eclipse, Gradle and Maven-Surefire. If you’re already using JUnit 5, using jqwik as additional engine requires a single additional dependency. If jqwik is your first contact with the platform you should check out this part in jqwik’s user guide.

Wildcards and Type Parameters

Let’s get back to the concrete property test of the previous episode:

@Property
@Report(Reporting.GENERATED)
boolean reverseWithWildcardType(@ForAll List<?> original) {
    return reverse(reverse(original)).equals(original);
}

You might miss the tiny change: Instead of a concretely typed List<Integer> I used the wildcard variant: List<?>. Actually, this reflects the precondition better, since the method under test - Collections.reverse() - should work with any element type. Under the hood jqwik will create instances of a special subtype of Object. Just run the property with reporting switched on.

This would, by the way, also work with a type variable instead of a wildcard. Upper or lower bounds, however, are not fully supported yet.

Many Parameters

You might have guessed that parameter generation is not restricted to a single one but works for as many as you need:

@Property
boolean joiningTwoLists(
    @ForAll List<String> list1,
    @ForAll List<String> list2
) {
    List<String> joinedList = new ArrayList<>(list1);
    joinedList.addAll(list2);
    return joinedList.size() == list1.size() + list2.size();
}

If you look at the generated lists you will notice that the variance in list size and string length is quite high. You can also see that now and then an empty list and an empty string is generated. This is due to the fact that value generation is not purely random and not equally distributed across the allowed domain. Instead, jqwik tries to be “smarter”:

This smart generation approach aims at raising the probability to detect not so obvious specification gaps and implementation bugs.

Automatic Parameter Generation

Out of the box jqwik is able to generate objects of the most common JDK types:

You might notice that Map and all calendar related classes are not covered (yet). It’s quite easy, though, to provide and register generators yourself.

What’s also not covered are functional types. They are on the backlog but it’s not easy to come up with a good variance of functions to try. Constant functions are obvious, everything else not so much.

Influencing Automatic Parameter Generation

The easiest way to influence and constrain the domain of values considered for generation is to use additional annotations provided for many of the default types. Here are a few examples:

Here is the full list of built-in constraining annotations.

Programmatic Generation

Sometimes we’re dealing with classes that cannot be generated by default. On other occasions the domain-specific constraints of a primitive type is so specific that the existing annotations are not powerful enough. In these cases you can delegate provision of parameter generators to another method in your test container class. The following example shows how to generate German zip codes:

@Property
void letsGenerateGermanZipCodes(@ForAll("germanZipCodes") String zipCode) {
}

@Provide
Arbitrary<String> germanZipCodes() {
    return Arbitraries.strings().withCharRange('0', '9').ofLength(5);
}

The String value of the @ForAll annotation serves as a reference to a method within the same class (or one of its superclasses or owning classes). This reference refers to either the method’s name or the String value of the method’s @Provide annotation. The providing method has to return an object of type @Arbitrary<T> where T is the static type of the parameter to be provided. Arbitrary is a bit more than a value generator; think of it as a “configurator for value generators”.

Parameter provision methods usually start with a static method call to Arbitraries, maybe followed by one or more filtering, mapping or combining actions as described in the next section.

Filter and Map

As the core type of all value generation Arbitrary has quite a few default methods that can be used to modify generating behaviour. You usually start with one of the static basic generator functions in class Arbitraries. Most base generators return a specific subtype of Arbitrary that gives you additional constraining possibilities.

Let’s say we want to generate integers between 1 and 300 that are multiples of 6. Here are two alternatives to do that:

Which way is better? Sometimes it’s only a matter of style or readability. Sometimes, however, the way you choose can influence performance. When comparing the two options above, the former is close to the given spec but it will - through filtering - throw away five sixths of all generated values. The latter is therefore more efficient but also less comprehensible when coming from the spec. Usually generating primitive values is so fast that readability trumps efficiency.

Combining Arbitraries

Real domain objects often have several distinct and mostly unrelated parts. That’s why - when you generate them - you want to start from unrelated base generators.

Given our domain class Person:

class Person {
    private final String firstName, lastName;

    Person(String firstName, String lastName) {
        this.firstName = firstName;
        this.lastName = lastName;
    }

    public String fullName() {
        return firstName + " " + lastName;
    }

    @Override
    public String toString() {
        return String.format("Person(%s:%s)", firstName, lastName);
    }
}

We want to write a property which checks that any person has a full name. The person generator method combines three arbitraries into the needed one:

@Property
boolean anyValidPersonHasAFullName(@ForAll("validPerson") Person aPerson) {
    return aPerson.fullName().length() >= 5;
}

@Provide
Arbitrary<Person> validPerson() {
  Arbitrary<String> firstName = Arbitraries.strings()
      .withCharRange('a', 'z')
      .ofMinLength(2).ofMaxLength(10)
      .map(this::capitalize);
  Arbitrary<String> lastName = Arbitraries.strings()
      .withCharRange('a', 'z')
      .ofMinLength(2).ofMaxLength(20);
  return Combinators.combine(firstName, lastName).as(Person::new);
}

Fighting Indeterminism

The values generated during a test run are random - at least to a large degree. That way you enhance the chance of hitting upon bugs and specification gaps that no one considered during conception and implementation. The downside of introducing chance into testing is inherent indeterminism: A falsified property in this run could succeed in the next since different test data might lead to different test results.

There are two things jqwik does to keep this problem in check:

  @Property(seed = "424242")
  @Report(ReportingMode.GENERATED)
  void alwaysTheSameValues(@ForAll int aNumber) { ... }

Other PBT Frameworks and Libs for Java

jqwik being a JUnit 5 test engine requires you to use the JUnit platform. If you cannot or do not want to use JUnit 5 yet, there are a few alternatives for doing PBT on the JVM:

Next Episode

The next article will focus on a crucial feature of mature Property-based Testing libraries: Shrinking.