Interesting benchmarks. One thing I'd love to see separated out: how much of the gap is syntax vs. how much is the model just having seen far less of the "fixed

Interesting benchmarks. One thing I'd love to see separated out: how much of the gap is syntax vs. how much is the model just having seen far less of the "fixed" form in training. A simpler grammar with zero corpus behind it could easily perform worse in practice.