At CrowdFlower, I spend a lot of my time developing and executing performance scripts. I either run the test using Jenkins or my laptop. Last week, I realize that if I combine the various scripts I wrote, we have a very nice end to end functional test.
In setting it all up to run in Jenkins, I realized that I needed a way to mark the build as a failure if an error occurs during the execution of a sampler. By default, jmeter will exit properly and Jenkins will mark the build as green.
However, looking at the console output of the job:
Starting the test @ Tue Mar 07 23:45:22 GMT 2017 (1488930322080)
Waiting for possible Shutdown/StopTestNow/Heapdump message on port 4445
summary = 3 in 00:00:01 = 2.4/s Avg: 256 Min: 29 Max: 407 Err: 1 (33.33%)
Tidying up ... @ Tue Mar 07 23:45:23 GMT 2017 (1488930323430)
... end of run
JMeter has outputted an error, but jenkins marked as pass. So, how do we resolve this?
Luckily, there's a Jenkins plugin called TextFinder. After installing the plugin, we just add the following Post Build action to the job:
After the build runs, the TextFinder will scan the console output (which is checked). If it detects a line with an error greater than 0.00%, then it will mark the build as unstable.