6 ms·
I wonder if FauxPilot's models (Salesforce Codegen family) can be quantized and run on the CPU. I was able to run the 350M model on my machine but it wasn't abl
by phantom32 3y ago
I wonder if FauxPilot's models (Salesforce Codegen family) can be quantized and run on the CPU. I was able to run the 350M model on my machine but it wasn't able to compete with Copilot in any way. Salesforce claims their model is competitive with OpenAI Codex their github description[1]. Maybe their largest 16B model is, but I haven't been able to try it.
[1] https://github.com/salesforce/CodeGen https://github.com/salesforce/CodeGen
- ayushkaushal 3y agoWe will add quantized CodeGen for fast inference on CPUs up on cformers (https://github.com/NolanoOrg/cformers/ https://github.com/NolanoOrg/cformers/) by later today.
- underlines 3y ago4bit GPTQ maybe?
- syntaxing 3y agoWhoa is there a PR or wiki about this
- meghan_rain 3y ago> by later today Wow, that's the timeframe things are moving at right now, we better get used to it!