7 ms·
Phi 4 available on Ollama
- gnabgib 2y agoRelated Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning (439 points, 24 days ago, 144 comments) https://news.ycombinator.com/item?id=42405323 https://news.ycombinator.com/item?id=42405323 Also on hugging face https://huggingface.co/microsoft/phi-4 https://huggingface.co/microsoft/phi-4
- deleted 2y ago[deleted]
- andhuman 2y agoI’ve seen on the localllama subreddit that some GGUFs have bugs in them. The one recommended was by unsloth. However, I don’t know how the Ollama GGUF holds up.
- compumetrika 2y agoOllama can pull directly from HF, you just provide the URL and add to the end :Q8_0 (or whatever) to specify your desired quant. Bonus: use the short form url of `hf` instead of `huggingface` to shorten the model name a little in the ollama list table. Edit: so for example of you want the unsloth "debugged" version of Phi4, you would run: `$ollama pull hf.co/unsloth/phi-4-GGUF:Q8_0` (check on the right side of the hf.co/unsloth/phi-4-GGUF page for the available quants)
- jimmySixDOF 2y agoYou still need to make sure the modelfile works so this method will not run out of the box on a vision GGUF or anything with special schemas. Thats why mostly a good idea to pull from ollama directly.
- wruza 2y agoIs it true that non-gguf models are basically all Q4-equivalent? I'm always not sure which one to download to get the "default score".
- magicalhippo 2y agoHere's[1] a recent submission on that. [1]: https://news.ycombinator.com/item?id=42660335 https://news.ycombinator.com/item?id=42660335 Phi-4 Bug Fixes
- jmorgan 2y agoPhi-4's architecture changed slightly from Phi-3.5 (it no longer uses a sliding window of 2,048 tokens [1]), causing a change in the hyperparameters (and ultimately an error at inference time for some published GGUF files on Hugging Face, since the same architecture name/identifier was re-used between the two models). For the Phi-4 uploaded to Ollama, the hyperparameters were set to avoid the error. The error should stop occurring in the next version of Ollama [2] for imported GGUF files as well In retrospect, a new architecture name should probably have been used entirely, instead of re-using "phi3". [1] https://arxiv.org/html/2412.08905v1 https://arxiv.org/html/2412.08905v1 [2] https://github.com/ollama/ollama/releases/tag/v0.5.5 https://github.com/ollama/ollama/releases/tag/v0.5.5
- mythz 2y agoWas disappointed in all the Phi models before this, whose benchmark results scored way better than it worked in practice, but I've been really impressed with how good Phi-4 is at just 14B. We've run it against the top 1000 most popular StackOverflow questions and it came up 3rd beating out GPT-4 and Sonnet 3.5 in our benchmarks, only behind DeepSeek v3 and WizardLM 8x22B [1]. We're using Mixtral 8x7B to grade the quality of the answers which could explain how WizardLM (based on Mixtral 8x22B) took 2nd Place. Unfortunately I'm only getting 6 tok/s on NVidia A4000 so it's still not great for real-time queries, but luckily now that it's MIT licensed it's available on OpenRouter [2] for a great price of $0.07/$0.14M at a fast 78 tok/s. Because it yields better results and we're able to self-host Phi-4 for free, we've replaced Mistral NeMo with it in our default models for answering new questions [3]. [1] https://pvq.app/leaderboard https://pvq.app/leaderboard [2] https://openrouter.ai/microsoft/phi-4 https://openrouter.ai/microsoft/phi-4 [3] https://pvq.app/questions/ask https://pvq.app/questions/ask
- KTibow 2y agoInteresting eval but my first reaction is "using Mixtral as a judge doesn't sound like a good idea". Have you tested how different its results are from GPT-4 as a judge (on a small scale) or how stuff like style and order can affect its judgements? Edit: they have a blog post https://pvq.app/posts/individual-voting-comparison https://pvq.app/posts/individual-voting-comparison although it could go deeper
- mythz 2y agoYeah we evaluated several models for grading ~1 year ago and concluded Mixtral was the best choice for us, as it was the best model yielding the best results that we could self-host and distribute the load of grading 1.2M+ answers over several GPU Servers. We would have liked to pick a neutral model like Gemini which was fast, reliable and low cost, unfortunately it gave too many poor answers good grades [1]. If we had to pick a new grading model now, hopefully the much improved Gemini Flash 2.0 might yield better results. [1] https://pvq.app/posts/individual-voting-comparison#gemini-pro-1.0-speed-vs-performance https://pvq.app/posts/individual-voting-comparison#gemini-pr...
- crorella 2y agoIt’s odd that MS is releasing models they are competitors to OA. This reinforce the idea that there is no real strategic advantage in owning a model. I think the strategy is now offer cheap and performant infra to run the models.
- easton 2y agoI think they want/need a plan b in case OpenAI falls apart like it almost did when Sam got fired.
- PittleyDunkin 2y ago> It’s odd that MS is releasing models they are competitors to OA. > I think the strategy is now offer cheap and performant infra to run the models. Is this not what microsoft is doing? What can microsoft possibly lose by releasing a model?
- lolinder 2y agoThat's exactly what they're saying: it's interesting that Microsoft came to the same conclusion that Meta did, that models are generally not worth keeping locked down. It suggests that OpenAI has a very fragile business model, given that they're wholly dependent on large providers for the infra, which is apparently the valuable part of the equation.
- sumedh 2y ago> It suggests that OpenAI has a very fragile business model That is the reason they are making products so that people stay on the platform.
- lolinder 2y agoTheir big risk there as I see it is that the market for "I need an AI" is much much smaller than they thought it would be. People don't generally need or want to pay for "AI", they want to pay for solutions to specific problems. This means that in a world where AWS/Azure/GCP all compete in the compute and the models themselves are commodities, AI isn't a product, it's a feature of every product. In that world, what is OpenAI doing besides being an unnecessary middleman to Azure?
- hbcondo714 2y agoFWIW, Phi-4 was converted to Ollama by the community last month: https://ollama.com/vanilj/Phi-4 https://ollama.com/vanilj/Phi-4
- Patrick_Devine 2y agoWe ended up not publishing it as a library model just because it was leaked and not the official weights.
- smallerize 2y agoAnd adopted unsloth's bug fixes a few days ago. https://ollama.com/vanilj/phi-4-unsloth https://ollama.com/vanilj/phi-4-unsloth
- summarity 2y agoThe template doesn't match Unsloth's recommendation: https://news.ycombinator.com/item?id=42662106 https://news.ycombinator.com/item?id=42662106
- behnamoh 2y ago[flagged]
- digdugdirk 2y agoI thought ollama was just a convenience wrapper around llama.cpp?
- magicalhippo 2y agoThat might be how it started, but there are differences. For example support for LLama 3.2 Vision was added to Ollama[1], but not upstreamed[2] to llama.cpp due to image processing requirements AFIAK. [1]: https://github.com/ollama/ollama/releases/tag/v0.4.0 https://github.com/ollama/ollama/releases/tag/v0.4.0 [2]: https://github.com/ggerganov/llama.cpp/issues/9643 https://github.com/ggerganov/llama.cpp/issues/9643
- chamomeal 2y agoLooks like you’re being downvoted. It’d be nice if somebody could explain the difference, cause I’m also kinda out of the loop on this
- noodletheworld 2y agoThe ollama application has zero value; it’s just an easy to use front end to their model hosting which is both what this is and why they’re important. Only having one model host (hugging face) is bad for obvious reasons (and good in others, yes, but still) Ollama offering an alternative as a model host seems quite reasonable and quite well implemented. The frontend really is nothing; it’s just llama.cpp in a go wrapper. It has no value and it’s not really interesting, it’s simple stable technology that is perfectly fine to rely on and be totally unexcited or interested in, technically. …but, they do a lot more than that; and I think it’s a little unfair to imply that trivial piece of their stack is all they do.
- mythz 2y agoThe software that controls the front-end has enormous value, it becomes the central point and brand to manage and self-host LLMs that's used to manage 100 GB catalog of models which acts like a moat inhibiting switching to alternatives. Awareness and user-base are the hardest things to obtain with new Software products and it has both - right now it doesn't look it's monetizing its user base, but it could easily attract millions in VC funding to spin off a company to sell support contracts and "higher value" SaaS hosting or enterprise management features. Whilst it's now a UX friendly front-end for llama.cpp, it's also working on adding support for other backends like MLX [1]. [1] https://github.com/ollama/ollama/issues/1730 https://github.com/ollama/ollama/issues/1730
- sgk284 2y agoOver the holidays, we published a post[1] on using high-precision few-shot examples to get `gpt-4o-mini` to perform similar to `gpt-4o`. I just re-ran that same experiment, but swapped out `gpt-4o-mini` with `phi-4`. `phi-4` really blew me away in terms of learning from few-shots. It measured as being 97% consistent with `gpt-4o` when using high-precision few-shots! Without the few-shots, it was only 37%. That's a huge improvement! By contrast, with few-shots it performs as well as `gpt-4o-mini` (though `gpt-4o-mini`'s baseline without few-shots was 59% – quite a bit higher than `phi-4`'s). [1] https://bits.logic.inc/p/getting-gpt-4o-mini-to-perform-like https://bits.logic.inc/p/getting-gpt-4o-mini-to-perform-like
- yard2010 2y agoThis is really nice. I loved the detailed process and I'm definitely gonna use it. One nit though: I didn't understand what the graphs mean, maybe you should add the axes names.
- vincent_s 2y agoHave you also tried using the large model as FSKD model?
- sgk284 2y agoWe have, and it works great! We currently do this in production, though we use it to help us optimize for consistency between task executions (vs the linked post, which is about improving the capabilities of a model). Phrased differently, when a task has many valid and correct conclusions, this technique allows the LLM to see "How did I do similar tasks before?" and it'll tend to solve new tasks by making similar decisions it made for previous similar tasks. Two things to note: - You'll typically still want to have some small epsilon where you choose to run the task without few-shots. This will help prevent mistakes from propagating forward indefinitely. - You can have humans correct historical examples, and use their feedback to improve the large model dynamically in real-time. This is basically FSKD where the human is the "large model" and the large foundation model is the "small model".
- raybb 2y agoI was going to ask if this or other Ollama models support structured output (like JSON). Then a quick search revealed you can as of a free weeks ago https://ollama.com/blog/structured-outputs https://ollama.com/blog/structured-outputs
- porker 2y agoFor structured output from anywhere I'm finding https://github.com/BoundaryML/baml https://github.com/BoundaryML/baml good. It's more accurate than gpt-04-mini will do on its own, and any of the other JSON schema approaches I've tried.
- svachalek 2y agoYeah it's not as strong as constrained beam search like OpenAI uses (at least afaik) but it works on any models that support tool calling. Just keep it simple, don't have a lot of deep nested structures or complicated rules. Lots of other models will work nearly as well though if you just give them a clear schema to follow and ask them to output json only, then parse it yourself. Like I've been using gemma2:9b to analyze text and output a json structure and it's nearly 100% reliable despite it being a tiny model and not supporting tools or structured output officially.
- v3ss0n 2y agoTranslation, Phi-4 available on llmacpp
- OhioMan2943 2y agoIs anyone blown away by how fast we got to running something this powerful locally? I know it's easy to get burnt out on llms but this is pretty incredible. I genuinely think we're only 2 years away from full custom local voice to voice llm assistants that grow with you like JOI in BR2049 and it's going to change how we think about being human and being social, and how we grow up.
- yeahwhatever10 2y agoHow can a model "grow with you"? Do current models do this other than adding the full conversation to the context window?
- simonw 2y agoIt's incredible. I've been experimenting with running local LLMs for nearly two years now, ever since the first LLaMA release back in March 2023. About six months ago I had mostly lost interest in them. They were fun to play around with but the quality difference between the ones I could run on my MacBook and the ones I could access via an online API felt insurmountable. This has completely changed in the second half of 2024. The models I can run locally had a leap in quality - they feel genuinely GPT-4 class now. They're not as good as the best hosted models (GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet) but they're definitely good enough to be extremely useful. This started with the Qwen 2 and 2.5 series, but I also rate Llama 3.3 70B and now Phi-4 as GPT-4 class models that run on my laptop. I wrote more about this here: https://simonwillison.net/2024/Dec/31/llms-in-2024/#some-of-those-gpt-4-models-run-on-my-laptop https://simonwillison.net/2024/Dec/31/llms-in-2024/#some-of-...
- OhioMan2943 2y agoI'm in complete agreement with your more recent timeline piece (the negative one), and as a younger user (22 year old student) I'm actively relocating this year to somewhere slightly more rural with a focus on physical/knowledge combined work to secure a good quality of life nearly solely because of how fast our timelines are. A 'word calculator' this effective is the best substitute that we have for a logic calculator. And the fact that it's enough in 90% of situations is terrifying as it is transformative, as is the fact no one is awake to it. Exponential power scaling in an unstable world feels like it only makes it exponentially more unstable though.
- summarity 2y agoDoes it include the unsloth fixes yet?
- buyucu 2y agoI have unfortunately been disappointed with the llama.cpp/ollama ecosystem of late, and thinking about moving my things to vllm instead. llama.cpp basically dropped support for multimodal visual models. ollama still does support them, but only a handful. Also ollama still does not support vulkan eventhough llama.cpp had vulkan support for a long long time now. This has been very sad to watch. I'm more and more convinced that vllm is the way to go, not ollama.
- jacooper 2y agoBut can you run llms that easily with vllm? do you have to fiddle with formats to get it to run?
- buyucu 2y agoI'm still in early stages of exploration, but vllm seems to be compatible with most models on huggingface.
- mistercheph 2y agoOllama maintainers seem a bit burned out
- k__ 2y ago"built upon a blend of synthetic datasets, data from filtered public domain websites, and acquired academic books and Q&A datasets" Does this mean the model was trained without copyright infringements?
- redcobra762 2y agoThis is a presumptive question, as training AI models may fall under fair use.
- k__ 2y agoJust because some laws define fair use in some kind of way, it doesn't mean potential customers see it that way.
- dartos 2y agoDoes this include some of the config fixes that the sloth guys pointed out?
- ionwake 2y agoCan this run on a macbook m1? What is the performance like? Or would I need an m3? Thanks
- svachalek 2y agoYeah as long as it has 16GB+ RAM. I've got a newer cpu and it's very fast, so I expect on an M1 it would be at least bearable.
- mercer 2y agoIt's good enough for me on an M1, 16Gb, and slow but good enough as a background job on my older intel mbp with 16Gb. I somehow expected it to not work on intel macs at all, so that's a freebie.
- kuatroka 2y agoI’ve pulled and ran it. It launches fine, but when I actually ask it anything I constantly get just a blank line. Does anyone else experience this?
- mettamage 2y agoHow come models can be so small now? I don't know a lot about AI, but is there an ELI5 for a software engineer that knows a bit about AI? For context: I've made some simple neural nets with backprop. I read [1]. [1] http://neuralnetworksanddeeplearning.com/ http://neuralnetworksanddeeplearning.com/
- blharr 2y agoYou can find the phi-4 technical report [here](https://www.microsoft.com/en-us/research/uploads/prod/2024/12/P4TechReport.pdf https://www.microsoft.com/en-us/research/uploads/prod/2024/1...) The brief of it is by curating a smaller synthetic dataset of high quality from textbooks, problem sets, etc. instead of dumping a massive dataset with tons of information.
- sega_sai 2y agoI've just tried to make it run something, and I just could not force to include the python code inside ``` ``` quotation marks. It always wants to put word python after three quotes, like this: ```python .. code.. ``` I wonder if that's the result of training. (I use the LLM output to then run the resulting code)
- XCSme 2y agoDoes this have an "instruct" version? Or it's already sort-of like that, as it was trained more on Q&A scenarios?