Do you really think Nelson's mom would have been so welcoming and allowing if Milo had walked in through the front door when she's been finding out, first hand, from Uncle Ray, that Tard Trust doesn't want Milo having anything to do with Nelson? Not likely. As an adult she'd be stupid and say, you're kids, you don't know what you're doing, we understand, but still, your dad doesn't want you here, so you need to leave.
Especially when the sheriff (uncle Ray) got called in twice, already...
Sorry, but it makes perfect sense to me.