Listeners¶
Eris provide the possibility to pass in Listeners to be notified of events happening during a test run.
Listeners implement the Eris\Listener interface and are advised to extend the Eris\EmptyListener abstract base class to get an empty implementation for all the methods corresponding to events they don’t need to listen to.
Consider that Eris performs (by default) 100 iterations for each forAll() instance, each corresponding to a different set of generated values. The following methods can be overridden to receive an event:
startPropertyVerification()is called before the first iteration starts.endPropertyVerification($ordinaryEvaluations, $iterations, Exception $exception = null)is called when no more iterations will be performed, both in the case of test success and failure. The$ordinaryEvaluationsparameter provides the actual number of evaluations performed. This number may be less than than the number of target$iterationsdue to failures orwhen()filters not being satisfied. The$exception, when not null, indicated that the test has finally failed and corresponds to the error that is actually bubbling up rather than the original, unshrunk error.newGeneration(array $generation, $iteration)is called after generating a new iteration, and is passed the tuple of values along with the 0-based index of the iteration.failure(array $generation, Exception $e)is called after the failure of an assertion (and not for generic exceptions). The method can be called only once perthen()run, and is called before any shrinking takes place.shrinking(array $generation)is called before each shrinking attempt, with the values that will be used as the simplified input.
$generation is always an array of the same form as the arguments passed to then(), without any Eris class wrapping them.
Collect Frequencies¶
The collectFrequencies() Listener allows to gather all generated values in order to display their statistical distribution.
<?php
use Eris\Generator;
use Eris\TestTrait;
use Eris\Listener;
class CollectTest extends PHPUnit_Framework_TestCase
{
use TestTrait;
public function testGeneratedDataCollectionOnScalars()
{
$this
->forAll(Generator\neg())
->hook(Listener\collectFrequencies())
->then(function ($x) {
$this->assertTrue($x < $x + 1);
});
}
public function testGeneratedDataCollectionOnMoreComplexDataStructures()
{
$this
->forAll(
Generator\vector(2, Generator\int()),
Generator\char()
)
->hook(Listener\collectFrequencies())
->then(function ($vector) {
$this->assertEquals(2, count($vector));
});
}
public function testGeneratedDataCollectionWithCustomMapper()
{
$this
->forAll(
Generator\seq(Generator\nat())
)
->withMaxSize(10)
->hook(Listener\collectFrequencies(function ($array) {
return count($array);
}))
->then(function ($array) {
$this->assertEquals(count($array), count(array_reverse($array)));
});
}
}
testGeneratedDataCollectionOnScalars collects integers:
12% -1
6% -2
4% -5
4% -18
4% -11
3% -4
...
testGeneratedDataCollectionOnMoreComplexDataStructures shows how by default more complex structures are encoded into a JSON value, to be used as the bin key in the map of values to counters:
1% [[-19,-16],"m"]
1% [[-3,-30],";"]
1% [[-9,1],"\f"]
1% [[-7,-1],"P"]
1% [[-1,-9],"^"]
1% [[1,18],"8"]
1% [[-53,-1],"."]
...
testGeneratedDataCollectionWithCustomMapper shows how to provide a custom callable to map the generated values into a bin key. Arguments are passed to the callable in the same way as then(). In this example, we are discovering that 10% of the generated arrays have length 3.
39% 0
26% 1
10% 3
5% 4
5% 2
4% 5
3% 7
3% 6
3% 8
1% 10
1% 9
Log¶
The log() Listener allows to write a log file while particularly long tests are executing, showing the partial progress of the test.
<?php
use Eris\Generator;
use Eris\TestTrait;
use Eris\Listener;
class LogFileTest extends PHPUnit_Framework_TestCase
{
use TestTrait;
public function testWritingIterationsOnALogFile()
{
$this
->forAll(
Generator\int()
)
->hook(Listener\log(sys_get_temp_dir().'/eris-log-file-test.log'))
->then(function ($number) {
$this->assertInternalType('integer', $number);
});
}
public function testLogOfFailuresAndShrinking()
{
$this
->forAll(
Generator\int()
)
->hook(Listener\log(sys_get_temp_dir().'/eris-log-file-shrinking.log'))
->then(function ($number) {
$this->assertLessThanOrEqual(42, $number);
});
}
}
A file will be written during the test run with the following contents:
...
[2016-03-24T09:14:20+00:00][2593] iteration 12: [-9]
[2016-03-24T09:14:20+00:00][2593] iteration 13: [-59]
[2016-03-24T09:14:20+00:00][2593] iteration 14: [-51]
[2016-03-24T09:14:20+00:00][2593] iteration 15: [-52]
[2016-03-24T09:14:20+00:00][2593] iteration 16: [-83]
[2016-03-24T09:14:20+00:00][2593] iteration 17: [78]
[2016-03-24T09:14:20+00:00][2593] failure: [78]. Failed asserting that 78 is equal
to 42 or is less than 42.
[2016-03-24T09:14:20+00:00][2593] shrinking: [77]
[2016-03-24T09:14:20+00:00][2593] shrinking: [76]
[2016-03-24T09:14:20+00:00][2593] shrinking: [75]
[2016-03-24T09:14:20+00:00][2593] shrinking: [74]
[
...
It is not advised to rely on this format for parsing, being it only oriented to human readability.
Minimum Evaluations —
The @eris-ratio {ratio} annotation or the``minimumEvaluations($ratio)`` API method instantiate and wire in a Listener that checks that at least ratio``% of the total number of inputs being generated is actually evaluated. This Listener is only needed in case of an aggressive use of ``when().
Management of this Listener is provided through this method instead of explicitly adding a Listener object, as there is a default Listener instantiated with a threshold of 0.5 that has to be replaced in case a new minimum is chosen.
<?php
use Eris\Generator;
class MinimumEvaluationsTest extends PHPUnit_Framework_TestCase
{
use Eris\TestTrait;
public function testFailsBecauseOfTheLowEvaluationRatio()
{
$this
->forAll(
Generator\choose(0, 100)
)
->when(function ($n) {
return $n > 90;
})
->then(function ($number) {
$this->assertTrue($number * 2 > 90 * 2);
});
}
public function testPassesBecauseOfTheArtificiallyLowMinimumEvaluationRatio()
{
$this
->minimumEvaluationRatio(0.01)
->forAll(
Generator\choose(0, 100)
)
->when(function ($n) {
return $n > 90;
})
->then(function ($number) {
$this->assertTrue($number * 2 > 90 * 2);
});
}
/**
* @eris-ratio 1
*/
public function testPassesBecauseOfTheArtificiallyLowMinimumEvaluationRatioFromAnnotation()
{
$this
->forAll(
Generator\choose(0, 100)
)
->when(function ($n) {
return $n > 90;
})
->then(function ($number) {
$this->assertTrue($number * 2 > 90 * 2);
});
}
}
Both tests generate inputs in the range from 0 to 100, and since the condition of them being greater than 90 is rare, most of them will be discarded. By default Eris will check that 50% of the inputs are actually evaluated; therefore testFailsBecauseOfTheLowEvaluationRatio will fail with this message:
...
There was 1 error:
1) MinimumEvaluationsTest::testFailsBecauseOfTheLowEvaluationRatio
OutOfBoundsException: Evaluation ratio 0.05 is under the threshold 0.5
...
The actual ratio may vary depending on the inputs being generated and may not be 0.05.
In testPassesBecauseOfTheArtificiallyLowMinimumEvaluationRatio, we accept a lower minimum evaluation ratio of 1%; therefore the test does not ordinarily fail. Its coverage will still be very poor, so the user is advised to precisely specify the inputs rather than generating a lot of them and discarding a large percentage with when().