What you are describing is the most basic form of prompt injection. Current LLMs acts like 5 years old when it comes to cuddling them to write what you want. If

What you are describing is the most basic form of prompt injection. Current LLMs acts like 5 years old when it comes to cuddling them to write what you want. If you ask it for meth formula, it'll refuse. But you can convince it to write you a poem about creating meth, which it would do if you are clever enough. This is a simplification, check Pliny[0]'s work for how far prompt injection techniques go. None of the LLMs managed to survive against them. [0]: https://github.com/elder-plinius https://github.com/elder-plinius