8 ms·

Ollama releases Python and JavaScript Libraries

607 points by adamhowell 3y ago

deepsquirrelnet 3y ago
I love the ollama project. Having a local llm running as a service makes sense to me. It works really well for my use. I’ll give this Python library a try. I’ve been wanting to try some fine tuning with LLMs in the loop experiments.
rgbrgb 3y ago
Are these libraries for connecting to an ollama service that the user has already installed or do they work without the user installing anything? Sorry for not checking the code but maybe someone has the same question here. I looked at using ollama when I started making FreeChat [0] but couldn't figure out a way to make it work without asking the user to install it first (think I asked in your discord at the time). I wanted FreeChat to be 1-click install from the mac app store so I ended up bundling the llama.cpp server instead which it runs on localhost for inference. At some point I'd love to swap it out for ollama and take advantage of all the cool model pulling stuff you guys have done, I just need it to be embeddable. My ideal setup would be importing an ollama package in swift which would start the server if the user doesn't already have it running. I know this is just js and python to start but a dev can dream :) Either way, congrats on the release! [0]: https://github.com/psugihara/FreeChat https://github.com/psugihara/FreeChat
- icyfox 3y ago
  Just for connecting to an existing service: https://github.com/ollama/ollama-python/blob/main/ollama/_client.py#L109 https://github.com/ollama/ollama-python/blob/main/ollama/_cl...
  - thrdbndndn 3y ago
    For the client API it's pretty clear: from ollama import Client client = Client(host='http://localhost:11434') But I don't quite get how the example in "Usage" can work: import ollama response = ollama.chat(model='llama2', messages=[ { 'role': 'user', 'content': 'Why is the sky blue?', }, ]) print(response['message']['content']) Since there is no parameter for host and/or port.
    - jmorgan 3y ago
      Once you have a custom `client` you can use it in place of `ollama`. For example: client = Client(host='http://my.ollama.host:11434') response = client.chat(model='llama2', messages=[...])
      - thrdbndndn 3y ago
        Thanks. I don't have the service installed on my computer RN, but I assume the former works because it by default uses a host (localhost) and port number that is also the default for ollma service?
        flakes 3y ago
        Exactly that. Client host options default, https://github.com/ollama/ollama-python/blob/main/ollama/_client.py#L610-L657 https://github.com/ollama/ollama-python/blob/main/ollama/_cl... Also overrideable with OLLAMA_HOST env var. The default imported functions are then based off of a no-arg constructed client https://github.com/ollama/ollama-python/blob/main/ollama/__init__.py https://github.com/ollama/ollama-python/blob/main/ollama/__i... # ollama-python/ollama/__init__.py _client = Client() generate = _client.generate chat = _client.chat embeddings = _client.embeddings ...
- SnowLprd 3y ago
  On the subject of installing Ollama, I found it to be a frustrating and user-hostile experience. I instead recommend the much more user-friendly LLM[0] by Simon Willison. Among the problems with Ollama include: * Ollama silently adds a login item with no way to opt out: <https://github.com/jmorganca/ollama/issues/162 https://github.com/jmorganca/ollama/issues/162> * Ollama spawns at least four processes, some persistently in the background: 1 x Ollama application, 1 x `ollama` server component, 2 x Ollama Helper * Ollama provides no information at install time about what directories will be created or where models will be downloaded. * Ollama prompts users to install the `ollama` CLI tool, with admin access required, with no way to cancel, and with no way to even quit the application at that point. Ollama provides no clarity that about what is actually happening during this step: all it is doing is symlinking `/Applications/Ollama.app/Contents/Resources/ollama` to `/usr/local/bin/` The worst part is that not only is none of this explained at install time, but the project README doesn’t tell you any of this information either. Potential users deserve to know what will happen on first launch, but when a PR arrived to at least provide that clarification in the README, Ollama maintainers summarily closed that PR and still have not rectified the aforementioned UX problems. As an open source maintainer myself, I understand and appreciate that Ollama developers volunteer their time and energy into the project, and they can run it as they see fit. So I intend no disrespect. But these problems, and a seeming unwillingness to prioritize their resolution, caused me to delete Ollama from my system entirely. As I said above, I think LLM[0] by Simon Willison is an excellent and user-friendly alternative. [0]: https://llm.datasette.io/ https://llm.datasette.io/
  - siquick 3y ago
    "User hostile experience" is complete hyperbole and disrespectful to the efforts of the maintainers of this excellent library.
    - config_yml 3y ago
      Indeed, I thought the user experience was great. Simple way to download, install and start: everything just worked.
    - SnowLprd 3y ago
- Abishek_Muthian 3y ago
  I used Ollama docker image to integrate with Gait Analyzer[1], a self-hosted gait analysis tool; all I had to do was to set up the docker compose file, I was able to get the setup done with a single script for the end user and I used langchain to interact with Ollama. [1] https://github.com/abishekmuthian/gaitanalyzer https://github.com/abishekmuthian/gaitanalyzer
Kostic 3y ago
I used this half a year ago, love the UX but it was not possible to accelerate the workloads using an AMD GPU. How's the support for AMD GPUs under Ollama today?
- mchiang 3y ago
  Hi, I'm one of the maintainers on Ollama. We are working on supporting ROCm in the official releases. If you do build from source, it should work (Instructions below): https://github.com/ollama/ollama/blob/main/docs/development.md#linux-rocm-amd https://github.com/ollama/ollama/blob/main/docs/development.... The reason why it's not in released builds is because we are still testing ROCm.
- accelbred 3y ago
  I'm using it on an AMD GPU with the clblast backend.
- brucethemoose2 3y ago
  Unfortunately "AMD" and "easy" are mutually exclusive right now. You can be a linux/python dev and set up rocm. Or you can run llama.cpp's very slow OpenCL backend, but with easy setup. Or you can run MLC's very fast Vulkan backend, but with no model splitting and medium-hard setup.
joaomdmoura 3y ago
So cool! I have bene using Ollama for weeks now and I just love it! Easiest way to run local LLMs, we are actually embedding them into our product right now and super excited about it!
- visarga 3y ago
  I am using ollama as LLM server + ollama-webui as chat app server. Great UI
- nbbaier 3y ago
  What's the product?
behnamoh 3y ago
What I hate about ollama is that it makes server configuration a PITA. ollama relies on llama.cpp to run GGUF models but while llama.cpp can keep the model in memory using `mlock` (helpful to reduce inference times), ollama simply won't let you do that: https://github.com/ollama/ollama/issues/1536 https://github.com/ollama/ollama/issues/1536 Not to mention, they hide all the server configs in favor of their own "sane defaults".
- jmorgan 3y ago
  Sorry this isn't easier! You can enable mlock manually in the /api/generate and /api/chat endpoints by specifying the "use_mlock" option: {“options”: {“use_mlock”: true}} Many other sever configurations are also available there: https://github.com/ollama/ollama/blob/main/docs/api.md#request-5 https://github.com/ollama/ollama/blob/main/docs/api.md#reque...
  - bestai 3y ago
    I think a faq with the answers of this kind of questions could be useful for users.
WhackyIdeas 3y ago
This is going to make my current project a million times easier. Nice.
sqs 3y ago
I posted about my awesome experiences using Ollama a few months ago: https://news.ycombinator.com/item?id=37662915 https://news.ycombinator.com/item?id=37662915. Ollama is definitely the easiest way to run LLMs locally, and that means it’s the best building block for applications that need to use inference. It’s like how Docker made it so any application can execute something kinda portably kinda safely on any machine. With Ollama, any application can run LLM inference on any machine. Since that post, we shipped experimental support in our product for Ollama-based local inference. We had to write our own client in TypeScript but will probably be able to switch to this instead.
- sqs 3y ago
  Also one feature request - if the library (or another related library) could also transparently spin up a local Ollama instance if the user doesn’t have one already. “Transparent-on-demand-Ollama” or something.
  - zenlikethat 3y ago
    That gets into process management which can get dicey, but I agree, a "daemonless" mode could be really interesting
  - chown 3y ago
    I have been working on something similar to that in Msty [1]. I haven’t announced the app anywhere (including my friends as I got a few things in pipeline that I want to get out first :) [1]: https://msty.app https://msty.app
- refulgentis 3y ago
  > Ollama is definitely the easiest way to run LLMs locally Nitro outstripped them, 3 MB executable with OpenAI HTTP server and persistent model load
  - jmorgan 3y ago
    Persistent model loading will be possible with: https://github.com/ollama/ollama/pull/2146 https://github.com/ollama/ollama/pull/2146 – sorry it isn't yet! More to come on filesize and API improvements
jdlyga 3y ago
Thanks Ollama
- gregorymichael 3y ago
  [flagged]
3Sophons 3y ago
The Rust+Wasm stack provides a strong alternative to Python in AI inference. * Lightweight. Total runtime size is 30MB as opposed 4GB for Python and 350MB for Ollama. * Fast. Full native speed on GPUs. * Portable. Single cross-platform binary on different CPUs, GPUs and OSes. * Secure. Sandboxed and isolated execution on untrusted devices. * Modern languages for inference apps. * Container-ready. Supported in Docker, containerd, Podman, and Kubernetes. * OpenAI compatible. Seamlessly integrate into the OpenAI tooling ecosystem. Give it a try --- https://www.secondstate.io/articles/wasm-runtime-agi/ https://www.secondstate.io/articles/wasm-runtime-agi/
- fillskills 3y ago
  Why would anyone downvote this? There is nothing against HN rules and the comment itself is adding new and relevant information.
  - coder543 3y ago
    From the HN Guidelines: “Please don't use HN primarily for promotion. It's ok to post your own stuff part of the time, but the primary use of the site should be for curiosity.” That user almost exclusively links to what appears to be their own product, which is self promotion. They also do it without clarifying their involvement, which could come across as astroturfing. Self promotion sometimes (not all the time) is fine, but it should also be clearly stated as such. Doing it in a thread about a competing product is not ideal. If it came up naturally, that would be different from just interjecting a sales pitch. I haven’t downvoted them, but I came close.
  - deleted 3y ago
    [deleted]
- anhldbk 3y ago
  Interesting. But the gguf file for llama2 is 4.78 GB in size. For ollama, llama2:7b is 3.8 GB. See: https://ollama.ai/library/llama2/tags https://ollama.ai/library/llama2/tags. Still I see ollama requires less RAM to run llama 2
bearjaws 3y ago
If you're using TypeScript I highly recommend modelfusion https://modelfusion.dev/guide/ https://modelfusion.dev/guide/ It is far more robust, integrates with any LLM local or hosted, supports multi-modal, retries, structure parsing using zod and more.
- kvz 3y ago
  This looks really nice but it’s good to point out that this project can use the Ollama HTTP API or any other API, but does not run models itself. So not a replacement to Ollama, but rather to the Ollama npm. Perhaps that was obvious because the post is about that, but I briefly thought this could run models too.
awongh 3y ago
Wow, I guess I wouldn’t have thought there would be GPU support. What’s the mechanism for this?
- brucethemoose2 3y ago
  Via llama.cpp's GPU support.
imrehg 3y ago
This should be nice to be easier to integrate with things like Vanna.ai, that was on HN recently. There a bunch of methods need to be implemented to work, but then usual OpenAI buts can be switched out to anything else, e.g. see the code stub in https://vanna.ai/docs/bigquery-other-llm-vannadb.html https://vanna.ai/docs/bigquery-other-llm-vannadb.html Looking forward to more remixes for other tools too.
palashkulsh 3y ago
Noob question, and may be probably being asked at the wrong place. Is there any way to find out min system requirements for running ollama run commands with different models.
- slawr1805 3y ago
  They have a high level summary of ram requirements for the parameter size of each model and how much storage each model uses on their GitHub: https://github.com/ollama/ollama#model-library https://github.com/ollama/ollama#model-library
- deleted 3y ago
  [deleted]
- mike978 3y ago
  I have a 11th gen intel cpu with 64gb ram and I can run most of big models slowly... so it's partly what you can put up with.
- nextlevelwizard 3y ago
  Rule of thumb I have used is to check the size and if it fits into your GPUs VRAM then it will run nicely. I have not ran into a llama that won't run, but if it doesn't fit into my GPU you have to count seconds per token instead of tokens per second
- mark_l_watson 3y ago
  On my 32G M2 Pro Mac, I can run up to about 30B models using 4 bit quantization. It is fast unless I am generating a lot of text. If I ask a 30B model to generate 5 pages of text it can take over 1 minute. Running smaller models like Mistral 7B is very fast. Install Ollama from https://ollama.ai https://ollama.ai and experiment with it using the command line interface. I mostly use Ollama’s local API from Common Lisp or Racket - so simple to do. EDIT: if you only have 8G RAM, try some of the 3B models. I suggest using at least 4 bit quantization.
- hellsten 3y ago
  Check out this guide for some recommendations: https://www.hardware-corner.net/guides/computer-to-run-llama-ai-model/ https://www.hardware-corner.net/guides/computer-to-run-llama... You can easily experiment with smaller models, for example, Mistral 7B or Phi-2 on M1/M2/M3 processors. With more memory, you can run larger models, and better memory bandwidth (M2 Ultra vs. M2 base model) means improved performance (tokens/second).
porridgeraisin 3y ago
Used ollama as part of a bash pipeline for a tiny throwaway app. It blocks until there is something on the mic, then sends the wav to whisper.cpp, which then sends it to llama which picks out a structured "remind me" object from it, which gets saved to a text file.
- killermouse0 3y ago
  Would you share that code? I'm not familiar with using the mic in Linux, but interested to do something similar!
- nbbaier 3y ago
  I'd also be really interested in seeing this
- awayto 3y ago
  I made something pretty similar over winter break so I could have something read books to me. ... Then it turned into a prompting mechanism of course! It uses Whisper, Ollama, and TTS from CoquiAI. It's written in shell and should hopefully be "Posix-compliant", but it does use zenity from Ubuntu; not sure how widely used zenity is. https://github.com/jcmccormick/runtts https://github.com/jcmccormick/runtts
pamelafox 3y ago
API wise, it looks very similar to the OpenAI python SDK but not quite the same. I was hoping I could swap out one client for another. Can anyone confirm they’re intentionally using an incompatible interface?
- WiSaGaN 3y ago
  There is an issue for this: [1]. I think it's more of priority issue. [1] https://github.com/ollama/ollama/issues/305 https://github.com/ollama/ollama/issues/305
- d4rkp4ttern 3y ago
  Same question here. Ollama is fantastic as it makes it very easy to run models locally, But if you already have a lot of code that processes OpenAI API responses (with retry, streaming, async, caching etc), it would be nice to be able to simply switch the API client to Ollama, without having to have a whole other branch of code that handles Ollama API responses. One way to do an easy switch is using the litellm library as a go-between but it’s not ideal. For an OpenAI compatible API my current favorite method is to spin up models using oobabooga TGW. Your OpenAI API code then works seamlessly by simply switching out the api_base to the ooba endpoint. Regarding chat formatting, even ooba’s Mistral formatting has issues[1] so I am doing my own in Langroid using HuggingFace tokenizer.apply_chat_template [2] [1] https://github.com/oobabooga/text-generation-webui/issues/5356 https://github.com/oobabooga/text-generation-webui/issues/53... [2] https://github.com/langroid/langroid/blob/main/langroid/language_models/prompt_formatter/hf_formatter.py https://github.com/langroid/langroid/blob/main/langroid/lang... Related question - I assume ollama auto detects and applies the right chat formatting template for a model?
- lhenault 3y ago
  I've built exactly this if you want to give it a try : https://github.com/lhenault/simpleAI https://github.com/lhenault/simpleAI
leansensei 3y ago
There is also an Elixir library: https://overbring.com/blog/2024-01-14-ollamex-ollama-api-embeddings/ https://overbring.com/blog/2024-01-14-ollamex-ollama-api-emb...
ivanfioravanti 3y ago
I posted about the Python library few hours after release. Great experience. Easy, fast and works well. I create a GIST with a quick and dirty way of generating a dataset for fine-tuning Mistral model using Instruction Format on a given topic: https://gist.github.com/ivanfioravanti/bcacc48ef68b02e9b7a4034161824287 https://gist.github.com/ivanfioravanti/bcacc48ef68b02e9b7a40...
- jumperabg 3y ago
  How does this fine-tuning work? I can see that you are loading a train.jsonl file and the some instructions but is the output model generated or this is some kind of a new way of training the models?
  - jerpint 3y ago
    The gist is only to create the dataset not to fine tune
- LoganDark 3y ago
  Gist isn't an acronym, it's a word. (e.g. "get the gist of things")
- pknerd 3y ago
  can we use it on cloud or I gotta download it locally? it might not work on my MacBook 2015 with 8GB ram
- eurekin 3y ago
  What's your observations about finetunes - are they really useful for anything practical? :)
- tinyhouse 3y ago
  Does olana support fine-tuning? I assume not. (Not asking about finetuned models that I know they support)
techn00 3y ago
Does Ollama support GBNF grammars?
- visarga 3y ago
  No, but it does support json formatting
malux85 3y ago
I love ollama, the engine underneath is llama.cpp, and they have the first version of self-extend about to me merged into main, so with any luck it will be available in ollama soon too!
- brucethemoose2 3y ago
  A lot of the new models coming out are long context anyway. Check out Yi, InternLM and Mixtral. Also, you really want to wait until flash attention is merged before using mega context with llama.cpp. The 8 bit KV cache would be ideal too.
reacharavindh 3y ago
Not directly related to what Ollama aims to achieve. But, I’ll ask nevertheless. Local LLMs are great! But, it would be more useful once we can _easily_ throw our own data for them to use as reference or even as a source of truth. This is where it opens doors that a closed system like OpenAI cannot - I’m never going to upload some data to ChatGPT for them to train on. Could Ollama make it easier and standardize the way to add documents to local LLMs? I’m not talking about uploading one image or model and asking a question about it. I’m referring to pointing a repository of 1000 text files and asking LLMs questions based on their contents.
- emmanueloga_ 3y ago
  I don’t know if Ollama can do this but https://gpt4all.io/ https://gpt4all.io/ can.
- reacharavindh 3y ago
  Basically, I want to do what this product does, but locally with a model running on Ollama. https://www.zenfetch.com/ https://www.zenfetch.com/
  - NetOpWibby 3y ago
    Ooh, I want this too.
  - camillomiller 3y ago
    Interactive smart knowledge bases is such a massively cool direction for LLMs. I’ve seen Chat with RTX at the NVIDIA preview at CES and it’s mindblowingly simple and cool to use. I believe that interactive search in limited domains is gonna be massive for LLMs
  - rex123 3y ago
    Hey - Akash from Zenfetch here. We’ve actually tested some of our features with local models and have found that they significantly underperform compared to hosted models. With that said, we are actively working on new approaches to offer a local version of Zenfetch. In the meanwhile, we do have agreements in place with all of our AI providers to ensure none of our users information is used for training or any other purpose. Hope that helps!
filleokus 3y ago
An off topic question: Is there such a thing as a "small-ish language model". A model that you could simple give instructions / "capabilities" which a user can interact with. Almost like Siri-level of intelligence. Imagine you have an API-endpoint where you can set the level of some lights and you give the chat a system prompt explaining how to build the JSON body of the request, and the user can prompt it with stuff like "Turn off all the lights" or "Make it bright in the bedroom" etc. How low could the memory consumption of such a model be? We don't need to store who the first kaiser of Germany was, "just" enough to kinda map human speech onto available API's.
- oblio 3y ago
  Speaking of, I imagine Alexa, Siri, etc, should now be replaced by LLMs? Or where they already implemented using LLMs?
- andy99 3y ago
  There are "smaller" models, for example tinyllama 1.1B (tiny seems like an exaggeration). PHI2 is 2.7B parameters. I can't name a 500M parameter model but there is probably one. The problem is they are all still broadly trained and so they end up being Jack of all trades master of none. You'd have to fine tune them if you want them good at some narrow task and other than code completion I don't know that anyone has done that. If you want to generate json or other structured output, there is Outlines https://github.com/outlines-dev/outlines https://github.com/outlines-dev/outlines that constrains the output to match a regex so it guarantees e.g. the model will generate a valid API call, although it could still be nonsense if the model doesn't understand, it will just match the regex. There are other similar tools around. I believe llama.cpp also has something built in that will constrain the output to some grammar.
  - nattaylor 3y ago
    https://pypi.org/project/languagemodels/ https://pypi.org/project/languagemodels/ can load some small models but forming JSON-reliably seems to require a larger-ish model (or fine tuning) Aside: I expect Apple will do exactly what you're proposing and that's why they're exposing more APIs for system apps
nextlevelwizard 3y ago
What is the benefit? Ollama already exposes REST API that you can query with whatever language (or you know, just using curl) - why do I want to use Python or JS?
- JrProgrammer 3y ago
  What’s the benefit of abstracting something?
  - nextlevelwizard 3y ago
    There is a reason why "leftpad" is followed by "incident".
- girvo 3y ago
  That one doesn’t have to write the glue code around your HTTP client library?
  - nextlevelwizard 3y ago
    Feels pretty bad to install dependency just so you can avoid making a HTTP request.
mfalcon 3y ago
I love Ollama's simplicity to download and consume different models with its REST API. I've never used it in a "production" environment, anyone knows how Ollama performs? or is it better to move to something like Vllm for that?
- jerpint 3y ago
  They all probably already use elements of deep learning but are very likely trained in a supervised way to output structured data (I.e. actions)
  - tripleo1 3y ago
    +1
- hellsten 3y ago
  The performance will probably be similar as long as you remember to tune the settings listed here: https://github.com/ollama/ollama/blob/main/docs/api.md https://github.com/ollama/ollama/blob/main/docs/api.md Try to, for example, set 'num_gpu' to 99 and 'use_mlock' to true.
deleted 3y ago
[deleted]
rezonant 3y ago
I wish JS libraries would stop using default exports. They are not ergonomic as soon as you want to export one more thing in your package, which includes types, so all but the most trivial package requires multiple exports. Just use a sensibly named export, you were going to write a "how to use" code snippet for the top of your readme anyway. Also means that all of the code snippets your users send you will be immediately sensible, even without them having to include their import statements (assuming they don't use "as" renaming, which only makes sense when there's conflicts anyway)
- deleted 3y ago
  [deleted]
deleted 3y ago
[deleted]
deleted 3y ago
[deleted]
hatmanstack 3y ago
Why does this feel like an exercise in the high priesting of coding. Shouldn't a python library have everything necessary and work out of the box?
Havoc 3y ago
What model format does ollama use? Or is one constrained to the handful of preselected models they list?
- mchiang 3y ago
  You can import GGUF, PyTorch or safetensors models into Ollama. I'll caveat that there are current limitations to some model architectures https://github.com/ollama/ollama/blob/main/docs/import.md https://github.com/ollama/ollama/blob/main/docs/import.md
  - Havoc 3y ago
    Thanks!
maswewe 3y ago
[dead]
sjwhevvvvvsj 3y ago
Literally wrote an Ollama wrapper class last week. Doh!
jquaint 3y ago
I'm a huge fan of Ollama. Really like how easy it makes local LLM + neovim https://github.com/David-Kunz/gen.nvim https://github.com/David-Kunz/gen.nvim
dchuk 3y ago
Is anyone using this as an api behind a multi user web application? Or does it need to be fed off of a message queue or something to basically keep it single threaded?
cranberryturkey 3y ago
`ollama serve` exposes an api you can query with fetch. why the need for a library?
lobocinza 3y ago
ollama feels like llama.cpp with extra undesired complexities. It feels like the former project is desperately trying to differentiate and monetize while the latter is where all the things that matter happens.

607 points by adamhowell 3y ago

I love the ollama project. Having a local llm running as a service makes sense to me. It works really well for my use. I’ll give this Python library a try. I’ve been wanting to try some fine tuning with LLMs in the loop experiments.

rgbrgb 3y ago

Are these libraries for connecting to an ollama service that the user has already installed or do they work without the user installing anything? Sorry for not checking the code but maybe someone has the same question here. I looked at using ollama when I started making FreeChat [0] but couldn't figure out a way to make it work without asking the user to install it first (think I asked in your discord at the time). I wanted FreeChat to be 1-click install from the mac app store so I ended up bundling the llama.cpp server instead which it runs on localhost for inference. At some point I'd love to swap it out for ollama and take advantage of all the cool model pulling stuff you guys have done, I just need it to be embeddable. My ideal setup would be importing an ollama package in swift which would start the server if the user doesn't already have it running. I know this is just js and python to start but a dev can dream :) Either way, congrats on the release! [0]: https://github.com/psugihara/FreeChat https://github.com/psugihara/FreeChat

icyfox 3y ago
Just for connecting to an existing service: https://github.com/ollama/ollama-python/blob/main/ollama/_client.py#L109 https://github.com/ollama/ollama-python/blob/main/ollama/_cl...
- thrdbndndn 3y ago
  For the client API it's pretty clear: from ollama import Client client = Client(host='http://localhost:11434') But I don't quite get how the example in "Usage" can work: import ollama response = ollama.chat(model='llama2', messages=[ { 'role': 'user', 'content': 'Why is the sky blue?', }, ]) print(response['message']['content']) Since there is no parameter for host and/or port.
  - jmorgan 3y ago
    Once you have a custom `client` you can use it in place of `ollama`. For example: client = Client(host='http://my.ollama.host:11434') response = client.chat(model='llama2', messages=[...])
    - thrdbndndn 3y ago
      Thanks. I don't have the service installed on my computer RN, but I assume the former works because it by default uses a host (localhost) and port number that is also the default for ollma service?
      - flakes 3y ago
        Exactly that. Client host options default, https://github.com/ollama/ollama-python/blob/main/ollama/_client.py#L610-L657 https://github.com/ollama/ollama-python/blob/main/ollama/_cl... Also overrideable with OLLAMA_HOST env var. The default imported functions are then based off of a no-arg constructed client https://github.com/ollama/ollama-python/blob/main/ollama/__init__.py https://github.com/ollama/ollama-python/blob/main/ollama/__i... # ollama-python/ollama/__init__.py _client = Client() generate = _client.generate chat = _client.chat embeddings = _client.embeddings ...
SnowLprd 3y ago
On the subject of installing Ollama, I found it to be a frustrating and user-hostile experience. I instead recommend the much more user-friendly LLM[0] by Simon Willison. Among the problems with Ollama include: * Ollama silently adds a login item with no way to opt out: <https://github.com/jmorganca/ollama/issues/162 https://github.com/jmorganca/ollama/issues/162> * Ollama spawns at least four processes, some persistently in the background: 1 x Ollama application, 1 x `ollama` server component, 2 x Ollama Helper * Ollama provides no information at install time about what directories will be created or where models will be downloaded. * Ollama prompts users to install the `ollama` CLI tool, with admin access required, with no way to cancel, and with no way to even quit the application at that point. Ollama provides no clarity that about what is actually happening during this step: all it is doing is symlinking `/Applications/Ollama.app/Contents/Resources/ollama` to `/usr/local/bin/` The worst part is that not only is none of this explained at install time, but the project README doesn’t tell you any of this information either. Potential users deserve to know what will happen on first launch, but when a PR arrived to at least provide that clarification in the README, Ollama maintainers summarily closed that PR and still have not rectified the aforementioned UX problems. As an open source maintainer myself, I understand and appreciate that Ollama developers volunteer their time and energy into the project, and they can run it as they see fit. So I intend no disrespect. But these problems, and a seeming unwillingness to prioritize their resolution, caused me to delete Ollama from my system entirely. As I said above, I think LLM[0] by Simon Willison is an excellent and user-friendly alternative. [0]: https://llm.datasette.io/ https://llm.datasette.io/
- siquick 3y ago
  "User hostile experience" is complete hyperbole and disrespectful to the efforts of the maintainers of this excellent library.
  - config_yml 3y ago
    Indeed, I thought the user experience was great. Simple way to download, install and start: everything just worked.
  - SnowLprd 3y ago
Abishek_Muthian 3y ago
I used Ollama docker image to integrate with Gait Analyzer[1], a self-hosted gait analysis tool; all I had to do was to set up the docker compose file, I was able to get the setup done with a single script for the end user and I used langchain to interact with Ollama. [1] https://github.com/abishekmuthian/gaitanalyzer https://github.com/abishekmuthian/gaitanalyzer

Kostic 3y ago

I used this half a year ago, love the UX but it was not possible to accelerate the workloads using an AMD GPU. How's the support for AMD GPUs under Ollama today?

mchiang 3y ago
Hi, I'm one of the maintainers on Ollama. We are working on supporting ROCm in the official releases. If you do build from source, it should work (Instructions below): https://github.com/ollama/ollama/blob/main/docs/development.md#linux-rocm-amd https://github.com/ollama/ollama/blob/main/docs/development.... The reason why it's not in released builds is because we are still testing ROCm.
accelbred 3y ago
I'm using it on an AMD GPU with the clblast backend.
brucethemoose2 3y ago
Unfortunately "AMD" and "easy" are mutually exclusive right now. You can be a linux/python dev and set up rocm. Or you can run llama.cpp's very slow OpenCL backend, but with easy setup. Or you can run MLC's very fast Vulkan backend, but with no model splitting and medium-hard setup.

joaomdmoura 3y ago

So cool! I have bene using Ollama for weeks now and I just love it! Easiest way to run local LLMs, we are actually embedding them into our product right now and super excited about it!

visarga 3y ago
I am using ollama as LLM server + ollama-webui as chat app server. Great UI
nbbaier 3y ago
What's the product?

behnamoh 3y ago

What I hate about ollama is that it makes server configuration a PITA. ollama relies on llama.cpp to run GGUF models but while llama.cpp can keep the model in memory using `mlock` (helpful to reduce inference times), ollama simply won't let you do that: https://github.com/ollama/ollama/issues/1536 https://github.com/ollama/ollama/issues/1536 Not to mention, they hide all the server configs in favor of their own "sane defaults".

jmorgan 3y ago
Sorry this isn't easier! You can enable mlock manually in the /api/generate and /api/chat endpoints by specifying the "use_mlock" option: {“options”: {“use_mlock”: true}} Many other sever configurations are also available there: https://github.com/ollama/ollama/blob/main/docs/api.md#request-5 https://github.com/ollama/ollama/blob/main/docs/api.md#reque...
- bestai 3y ago
  I think a faq with the answers of this kind of questions could be useful for users.

WhackyIdeas 3y ago

This is going to make my current project a million times easier. Nice.

sqs 3y ago

I posted about my awesome experiences using Ollama a few months ago: https://news.ycombinator.com/item?id=37662915 https://news.ycombinator.com/item?id=37662915. Ollama is definitely the easiest way to run LLMs locally, and that means it’s the best building block for applications that need to use inference. It’s like how Docker made it so any application can execute something kinda portably kinda safely on any machine. With Ollama, any application can run LLM inference on any machine. Since that post, we shipped experimental support in our product for Ollama-based local inference. We had to write our own client in TypeScript but will probably be able to switch to this instead.

sqs 3y ago
Also one feature request - if the library (or another related library) could also transparently spin up a local Ollama instance if the user doesn’t have one already. “Transparent-on-demand-Ollama” or something.
- zenlikethat 3y ago
  That gets into process management which can get dicey, but I agree, a "daemonless" mode could be really interesting
- chown 3y ago
  I have been working on something similar to that in Msty [1]. I haven’t announced the app anywhere (including my friends as I got a few things in pipeline that I want to get out first :) [1]: https://msty.app https://msty.app
refulgentis 3y ago
> Ollama is definitely the easiest way to run LLMs locally Nitro outstripped them, 3 MB executable with OpenAI HTTP server and persistent model load
- jmorgan 3y ago
  Persistent model loading will be possible with: https://github.com/ollama/ollama/pull/2146 https://github.com/ollama/ollama/pull/2146 – sorry it isn't yet! More to come on filesize and API improvements

jdlyga 3y ago

Thanks Ollama

gregorymichael 3y ago
[flagged]

3Sophons 3y ago

The Rust+Wasm stack provides a strong alternative to Python in AI inference. * Lightweight. Total runtime size is 30MB as opposed 4GB for Python and 350MB for Ollama. * Fast. Full native speed on GPUs. * Portable. Single cross-platform binary on different CPUs, GPUs and OSes. * Secure. Sandboxed and isolated execution on untrusted devices. * Modern languages for inference apps. * Container-ready. Supported in Docker, containerd, Podman, and Kubernetes. * OpenAI compatible. Seamlessly integrate into the OpenAI tooling ecosystem. Give it a try --- https://www.secondstate.io/articles/wasm-runtime-agi/ https://www.secondstate.io/articles/wasm-runtime-agi/

fillskills 3y ago
Why would anyone downvote this? There is nothing against HN rules and the comment itself is adding new and relevant information.
- coder543 3y ago
  From the HN Guidelines: “Please don't use HN primarily for promotion. It's ok to post your own stuff part of the time, but the primary use of the site should be for curiosity.” That user almost exclusively links to what appears to be their own product, which is self promotion. They also do it without clarifying their involvement, which could come across as astroturfing. Self promotion sometimes (not all the time) is fine, but it should also be clearly stated as such. Doing it in a thread about a competing product is not ideal. If it came up naturally, that would be different from just interjecting a sales pitch. I haven’t downvoted them, but I came close.
- deleted 3y ago
  [deleted]
anhldbk 3y ago
Interesting. But the gguf file for llama2 is 4.78 GB in size. For ollama, llama2:7b is 3.8 GB. See: https://ollama.ai/library/llama2/tags https://ollama.ai/library/llama2/tags. Still I see ollama requires less RAM to run llama 2

bearjaws 3y ago

If you're using TypeScript I highly recommend modelfusion https://modelfusion.dev/guide/ https://modelfusion.dev/guide/ It is far more robust, integrates with any LLM local or hosted, supports multi-modal, retries, structure parsing using zod and more.

kvz 3y ago
This looks really nice but it’s good to point out that this project can use the Ollama HTTP API or any other API, but does not run models itself. So not a replacement to Ollama, but rather to the Ollama npm. Perhaps that was obvious because the post is about that, but I briefly thought this could run models too.

awongh 3y ago

Wow, I guess I wouldn’t have thought there would be GPU support. What’s the mechanism for this?

brucethemoose2 3y ago
Via llama.cpp's GPU support.

imrehg 3y ago

This should be nice to be easier to integrate with things like Vanna.ai, that was on HN recently. There a bunch of methods need to be implemented to work, but then usual OpenAI buts can be switched out to anything else, e.g. see the code stub in https://vanna.ai/docs/bigquery-other-llm-vannadb.html https://vanna.ai/docs/bigquery-other-llm-vannadb.html Looking forward to more remixes for other tools too.

palashkulsh 3y ago

Noob question, and may be probably being asked at the wrong place. Is there any way to find out min system requirements for running ollama run commands with different models.

slawr1805 3y ago
They have a high level summary of ram requirements for the parameter size of each model and how much storage each model uses on their GitHub: https://github.com/ollama/ollama#model-library https://github.com/ollama/ollama#model-library
deleted 3y ago
[deleted]
mike978 3y ago
I have a 11th gen intel cpu with 64gb ram and I can run most of big models slowly... so it's partly what you can put up with.
nextlevelwizard 3y ago
Rule of thumb I have used is to check the size and if it fits into your GPUs VRAM then it will run nicely. I have not ran into a llama that won't run, but if it doesn't fit into my GPU you have to count seconds per token instead of tokens per second
mark_l_watson 3y ago
On my 32G M2 Pro Mac, I can run up to about 30B models using 4 bit quantization. It is fast unless I am generating a lot of text. If I ask a 30B model to generate 5 pages of text it can take over 1 minute. Running smaller models like Mistral 7B is very fast. Install Ollama from https://ollama.ai https://ollama.ai and experiment with it using the command line interface. I mostly use Ollama’s local API from Common Lisp or Racket - so simple to do. EDIT: if you only have 8G RAM, try some of the 3B models. I suggest using at least 4 bit quantization.
hellsten 3y ago
Check out this guide for some recommendations: https://www.hardware-corner.net/guides/computer-to-run-llama-ai-model/ https://www.hardware-corner.net/guides/computer-to-run-llama... You can easily experiment with smaller models, for example, Mistral 7B or Phi-2 on M1/M2/M3 processors. With more memory, you can run larger models, and better memory bandwidth (M2 Ultra vs. M2 base model) means improved performance (tokens/second).

porridgeraisin 3y ago

Used ollama as part of a bash pipeline for a tiny throwaway app. It blocks until there is something on the mic, then sends the wav to whisper.cpp, which then sends it to llama which picks out a structured "remind me" object from it, which gets saved to a text file.

killermouse0 3y ago
Would you share that code? I'm not familiar with using the mic in Linux, but interested to do something similar!
nbbaier 3y ago
I'd also be really interested in seeing this
awayto 3y ago
I made something pretty similar over winter break so I could have something read books to me. ... Then it turned into a prompting mechanism of course! It uses Whisper, Ollama, and TTS from CoquiAI. It's written in shell and should hopefully be "Posix-compliant", but it does use zenity from Ubuntu; not sure how widely used zenity is. https://github.com/jcmccormick/runtts https://github.com/jcmccormick/runtts

pamelafox 3y ago

API wise, it looks very similar to the OpenAI python SDK but not quite the same. I was hoping I could swap out one client for another. Can anyone confirm they’re intentionally using an incompatible interface?

WiSaGaN 3y ago
There is an issue for this: [1]. I think it's more of priority issue. [1] https://github.com/ollama/ollama/issues/305 https://github.com/ollama/ollama/issues/305
d4rkp4ttern 3y ago
Same question here. Ollama is fantastic as it makes it very easy to run models locally, But if you already have a lot of code that processes OpenAI API responses (with retry, streaming, async, caching etc), it would be nice to be able to simply switch the API client to Ollama, without having to have a whole other branch of code that handles Ollama API responses. One way to do an easy switch is using the litellm library as a go-between but it’s not ideal. For an OpenAI compatible API my current favorite method is to spin up models using oobabooga TGW. Your OpenAI API code then works seamlessly by simply switching out the api_base to the ooba endpoint. Regarding chat formatting, even ooba’s Mistral formatting has issues[1] so I am doing my own in Langroid using HuggingFace tokenizer.apply_chat_template [2] [1] https://github.com/oobabooga/text-generation-webui/issues/5356 https://github.com/oobabooga/text-generation-webui/issues/53... [2] https://github.com/langroid/langroid/blob/main/langroid/language_models/prompt_formatter/hf_formatter.py https://github.com/langroid/langroid/blob/main/langroid/lang... Related question - I assume ollama auto detects and applies the right chat formatting template for a model?
lhenault 3y ago
I've built exactly this if you want to give it a try : https://github.com/lhenault/simpleAI https://github.com/lhenault/simpleAI

leansensei 3y ago

There is also an Elixir library: https://overbring.com/blog/2024-01-14-ollamex-ollama-api-embeddings/ https://overbring.com/blog/2024-01-14-ollamex-ollama-api-emb...

ivanfioravanti 3y ago

I posted about the Python library few hours after release. Great experience. Easy, fast and works well. I create a GIST with a quick and dirty way of generating a dataset for fine-tuning Mistral model using Instruction Format on a given topic: https://gist.github.com/ivanfioravanti/bcacc48ef68b02e9b7a4034161824287 https://gist.github.com/ivanfioravanti/bcacc48ef68b02e9b7a40...

jumperabg 3y ago
How does this fine-tuning work? I can see that you are loading a train.jsonl file and the some instructions but is the output model generated or this is some kind of a new way of training the models?
- jerpint 3y ago
  The gist is only to create the dataset not to fine tune
LoganDark 3y ago
Gist isn't an acronym, it's a word. (e.g. "get the gist of things")
pknerd 3y ago
can we use it on cloud or I gotta download it locally? it might not work on my MacBook 2015 with 8GB ram
eurekin 3y ago
What's your observations about finetunes - are they really useful for anything practical? :)
tinyhouse 3y ago
Does olana support fine-tuning? I assume not. (Not asking about finetuned models that I know they support)

techn00 3y ago

Does Ollama support GBNF grammars?

visarga 3y ago
No, but it does support json formatting

malux85 3y ago

I love ollama, the engine underneath is llama.cpp, and they have the first version of self-extend about to me merged into main, so with any luck it will be available in ollama soon too!

brucethemoose2 3y ago
A lot of the new models coming out are long context anyway. Check out Yi, InternLM and Mixtral. Also, you really want to wait until flash attention is merged before using mega context with llama.cpp. The 8 bit KV cache would be ideal too.

reacharavindh 3y ago

Not directly related to what Ollama aims to achieve. But, I’ll ask nevertheless. Local LLMs are great! But, it would be more useful once we can _easily_ throw our own data for them to use as reference or even as a source of truth. This is where it opens doors that a closed system like OpenAI cannot - I’m never going to upload some data to ChatGPT for them to train on. Could Ollama make it easier and standardize the way to add documents to local LLMs? I’m not talking about uploading one image or model and asking a question about it. I’m referring to pointing a repository of 1000 text files and asking LLMs questions based on their contents.

emmanueloga_ 3y ago
I don’t know if Ollama can do this but https://gpt4all.io/ https://gpt4all.io/ can.
reacharavindh 3y ago
Basically, I want to do what this product does, but locally with a model running on Ollama. https://www.zenfetch.com/ https://www.zenfetch.com/
- NetOpWibby 3y ago
  Ooh, I want this too.
- camillomiller 3y ago
  Interactive smart knowledge bases is such a massively cool direction for LLMs. I’ve seen Chat with RTX at the NVIDIA preview at CES and it’s mindblowingly simple and cool to use. I believe that interactive search in limited domains is gonna be massive for LLMs
- rex123 3y ago
  Hey - Akash from Zenfetch here. We’ve actually tested some of our features with local models and have found that they significantly underperform compared to hosted models. With that said, we are actively working on new approaches to offer a local version of Zenfetch. In the meanwhile, we do have agreements in place with all of our AI providers to ensure none of our users information is used for training or any other purpose. Hope that helps!

filleokus 3y ago

An off topic question: Is there such a thing as a "small-ish language model". A model that you could simple give instructions / "capabilities" which a user can interact with. Almost like Siri-level of intelligence. Imagine you have an API-endpoint where you can set the level of some lights and you give the chat a system prompt explaining how to build the JSON body of the request, and the user can prompt it with stuff like "Turn off all the lights" or "Make it bright in the bedroom" etc. How low could the memory consumption of such a model be? We don't need to store who the first kaiser of Germany was, "just" enough to kinda map human speech onto available API's.

oblio 3y ago
Speaking of, I imagine Alexa, Siri, etc, should now be replaced by LLMs? Or where they already implemented using LLMs?
andy99 3y ago
There are "smaller" models, for example tinyllama 1.1B (tiny seems like an exaggeration). PHI2 is 2.7B parameters. I can't name a 500M parameter model but there is probably one. The problem is they are all still broadly trained and so they end up being Jack of all trades master of none. You'd have to fine tune them if you want them good at some narrow task and other than code completion I don't know that anyone has done that. If you want to generate json or other structured output, there is Outlines https://github.com/outlines-dev/outlines https://github.com/outlines-dev/outlines that constrains the output to match a regex so it guarantees e.g. the model will generate a valid API call, although it could still be nonsense if the model doesn't understand, it will just match the regex. There are other similar tools around. I believe llama.cpp also has something built in that will constrain the output to some grammar.
- nattaylor 3y ago
  https://pypi.org/project/languagemodels/ https://pypi.org/project/languagemodels/ can load some small models but forming JSON-reliably seems to require a larger-ish model (or fine tuning) Aside: I expect Apple will do exactly what you're proposing and that's why they're exposing more APIs for system apps

nextlevelwizard 3y ago

What is the benefit? Ollama already exposes REST API that you can query with whatever language (or you know, just using curl) - why do I want to use Python or JS?

JrProgrammer 3y ago
What’s the benefit of abstracting something?
- nextlevelwizard 3y ago
  There is a reason why "leftpad" is followed by "incident".
girvo 3y ago
That one doesn’t have to write the glue code around your HTTP client library?
- nextlevelwizard 3y ago
  Feels pretty bad to install dependency just so you can avoid making a HTTP request.

mfalcon 3y ago

I love Ollama's simplicity to download and consume different models with its REST API. I've never used it in a "production" environment, anyone knows how Ollama performs? or is it better to move to something like Vllm for that?

jerpint 3y ago
They all probably already use elements of deep learning but are very likely trained in a supervised way to output structured data (I.e. actions)
- tripleo1 3y ago
  +1
hellsten 3y ago
The performance will probably be similar as long as you remember to tune the settings listed here: https://github.com/ollama/ollama/blob/main/docs/api.md https://github.com/ollama/ollama/blob/main/docs/api.md Try to, for example, set 'num_gpu' to 99 and 'use_mlock' to true.

deleted 3y ago

[deleted]

rezonant 3y ago

I wish JS libraries would stop using default exports. They are not ergonomic as soon as you want to export one more thing in your package, which includes types, so all but the most trivial package requires multiple exports. Just use a sensibly named export, you were going to write a "how to use" code snippet for the top of your readme anyway. Also means that all of the code snippets your users send you will be immediately sensible, even without them having to include their import statements (assuming they don't use "as" renaming, which only makes sense when there's conflicts anyway)

deleted 3y ago
[deleted]

deleted 3y ago

[deleted]

deleted 3y ago

[deleted]

hatmanstack 3y ago

Why does this feel like an exercise in the high priesting of coding. Shouldn't a python library have everything necessary and work out of the box?

Havoc 3y ago

What model format does ollama use? Or is one constrained to the handful of preselected models they list?

mchiang 3y ago
You can import GGUF, PyTorch or safetensors models into Ollama. I'll caveat that there are current limitations to some model architectures https://github.com/ollama/ollama/blob/main/docs/import.md https://github.com/ollama/ollama/blob/main/docs/import.md
- Havoc 3y ago
  Thanks!

maswewe 3y ago

[dead]

sjwhevvvvvsj 3y ago

Literally wrote an Ollama wrapper class last week. Doh!

jquaint 3y ago

I'm a huge fan of Ollama. Really like how easy it makes local LLM + neovim https://github.com/David-Kunz/gen.nvim https://github.com/David-Kunz/gen.nvim

dchuk 3y ago

Is anyone using this as an api behind a multi user web application? Or does it need to be fed off of a message queue or something to basically keep it single threaded?

cranberryturkey 3y ago

`ollama serve` exposes an api you can query with fetch. why the need for a library?

lobocinza 3y ago

ollama feels like llama.cpp with extra undesired complexities. It feels like the former project is desperately trying to differentiate and monetize while the latter is where all the things that matter happens.