I'm kinda surprised that the mixture of depths paper didn't come up here. It approaches the other direction of sometimes dropping layers: https://arxiv.org/abs

I'm kinda surprised that the mixture of depths paper didn't come up here. It approaches the other direction of sometimes dropping layers: https://arxiv.org/abs/2404.02258 https://arxiv.org/abs/2404.02258