> How close would this be to GPT 2

Here's a post from 2015 doing something a bit like this [1]

[1] https://nbviewer.org/gist/yoavg/d76121dfde2618422139