This is something that should have been done through the SLA but on NeoLoad controller we can't stop a test based on an SLA failure.

However using the neoload-cli tool described here it is something available.

Basically it's based on NeoLoad Web API. So you could also do it by yourself directly from NeoLoad UI, enabling the export to NeoLoad Web and query the NeoLoad Web SLA's endpoint to get the value of your SLA's at runtime. Based on the result you could stop your test in NeoLoad using the "Stop" action.

If you do not use NeoLoad Web then maybe another way would be to create two Javascripts one before and after your transaction to compute the duration time of your transaction and decide to stop or not the test based on that result.

For example the first JS could be:

var startTime = Packages.java.lang.System.currentTimeMillis();
context.variableManager.setValue("startTime",startTime);
context.variableManager.setValue("continue", "true");

And the one after your transaction:

// Get the start time.
var startTime = context.variableManager.getValue("startTime");
var endTime = Packages.java.lang.System.currentTimeMillis();
var duration=(endTime-startTime)/1000;
if (duration>5) {
    context.variableManager.setValue("continue", "false");
}
logger.debug("duration = " +duration + " s");

So based on the value inside the "continue" variable you can decide to stop your test or not.