For what it's worth, this is basically the learning model fast.ai works on. You start by just applying pre-built models to things, then learn how to tweak them, then learn the theory that makes the tweaks work.