Hi I was also looking into this and I am now using: https://github.com/abetlen/llama-cpp-python https://github.com/abetlen/llama-cpp-python It tries to be compa

Hi I was also looking into this and I am now using: https://github.com/abetlen/llama-cpp-python https://github.com/abetlen/llama-cpp-python It tries to be compatible with openAI API. I managed to run AutoGPT using it (however context window is too small to be useful and even if I set it to 2048 (max) I had to tweak AutoGPT context maximum as 1024 for it to work - probably some additional wrapping or something)