No, the point is you said magic isn't an instant win button, except yeah it is. Rory has already used magic effectively in battle in a much more threatening and stressful situation than this one. Yet here he burns a guy's sleeve and seems pleased with himself.
Also, there's nothing stopping them from going after them. Their horses are war steeds which are trained to follow explicit commands which these bandits wouldn't know. I'd tell them to halt. There must be safeguards to prevent thr enemy from using their own horses against them.