Look at E Hunter. He came in skinny, left skinny. Never put in any mass. Some of it might be genetics.Edey. He's significantly stronger and better conditioned.
I don't think anyone is expecting Loyer to become jacked or super athletic, but he gets ragdolled out there more than any player I've seen in a long time. I don't think it's crazy to put on ten pounds of muscle and at least reach Smith levels of ability to fight throw contact.