rshemet

6 ms·

1.
by rshemet 1mo ago
Did you give it a tool to increase temperature, or only one that sets temperature to an absolute value? Either way, setting temperature to 5° is obviously wrong - even if it knew the current temperature - but models of this size can't
2.
by rshemet 1mo ago
hey Kenny, Roman from Cactus here - could you say more? What kind of home assistant / what stack
3.
by rshemet 1mo ago
it stands for Lets-not-be-sarcastic :)
4.
by rshemet 1mo ago
Roman from Cactus here - yes you're right, there's only so much a 14MB model can do. Needle excels at in-conext inference, with tightly defined environments. In our experience: accurate descriptions + narrow tool scope = success
5.
by rshemet 1mo ago
there are android binaries you can ship in your own app - https://huggingface.co/Cactus-Compute/needle2/tree/main but if you're just looking for somewhere to try the model, use our in-browser playground!
6.
by rshemet 1mo ago
Hey! Roman here from Cactus - yes, we're putting putting together a detailed guide for ESP32. In the meantime, if you have enough RAM for the current model (≈28MB), our repo will get you up & running: https://github.com&
7.
Show HN: Cactus v2 – On-device AI with cloud fallback
1 points by rshemet 2mo ago|0 comments
8.
by rshemet 1y ago
Yes! Cactus is optimized for mobile CPU inference and we're finishing internal testing of hybrid kernels that use the NPU, as well other chips. We don't advise using GPUs on smartphones, since they're very energy-inefficient.
9.
by rshemet 1y ago
indeed, this is exactly the goal! The license grants rights to commercial use, unlocks additional hardware acceleration, includes cloud telemetry, and offers significant savings over using cloud APIs. In our deployments, we've seen ope
10.
by rshemet 1y ago
you can run it in Cactus Chat (download from the Play Store)
11.
by rshemet 1y ago
you can also run it on Cactus - either in Cactus Chat from the App/Play Store or by using the Cactus framework to integrate it into your own app
12.
by rshemet 1y ago
THIS IS THE BOMB!!! So excited for this one. Thanks for putting cool tech out there.
13.
by rshemet 1y ago
if you ever end up trying to take this in the mobile direction, consider running on-device AI with Cactus – https://cactuscompute.com/ Blazing-fast, cross-platform, and supports nearly all recent OS models.
14.
by rshemet 1y ago
https://play.google.com/store/apps/details?id=com.rshemetsub...
15.
by rshemet 1y ago
thank you! Very kind feedback, and we'll add your feedback to our to-dos. re: "question would get stuck on the last phrase and keep repeating it without end." - that's a limitation of the model i'm afraid. Smaller m
16.
by rshemet 1y ago
say more about "community tools"?
17.
by rshemet 1y ago
in the app you mean? Adding shortly!
18.
by rshemet 1y ago
that's our mission! if you are passionate about the space, we look forward to your contributions!
19.
by rshemet 1y ago
no, good observation - not hidden; we don't have a "clear conversation" button. to your previous point - Cactus fully supports tool calling (for models that have been instruction-trained accordingly, e.g. Qwen 1.7B) for &quot
20.
by rshemet 1y ago
looking forward to your feedback!
21.
by rshemet 1y ago
hot off the press in our latest feature release :) we support cloud fallback as an add-on feature. This lets us support vision and audio in addition to text.
22.
by rshemet 1y ago
great observation - this data is not from a controlled environment; these are metrics from our Cactus Chat use (we only collect tok/sec telemetry). S25 is an outlier that surprised us too. I got $10 on S25 climbing back up to the top o
23.
by rshemet 1y ago
thank you! We're continue to add performance metrics as more data comes in. A Qwen 2.5 500M will get you to ≈45tok/sec on an iPhone 13. Inference speeds are somewhat linearly inversely proportional to model sizes. Yes, speeds are
24.
by rshemet 1y ago
Great question. Currently, each app is sandboxed - so each model file is downloaded inside each app's sandbox. We're working on enabling file sharing across multiple apps so you don't have to redownload the model. With respec
25.
by rshemet 1y ago
reminds me of - "You are, undoubtedly, the worst pirate i have ever heard of" - "Ah, but you have heard of me" Yes, we are indeed a young project. Not two weeks, but a couple of months. Welcome to AI, most projects are y
26.
by rshemet 1y ago
love this. So many layers deep, we just had a good laugh.
27.
by rshemet 1y ago
Very good point - we've heard this before. We're restructuring the model initialization API to point to a local file & exposing a separate abstracted download function that takes in a URL. wrt downloading post-install: based o
28.
by rshemet 1y ago
True - but Cactus is not just an app. We are a dev toolkit to run LLMs cross-platform locally in any app you like.
29.
by rshemet 1y ago
Thanks for the feedback. You're right to point out that Google AI Edge is cross-platform and more flexible than our phrasing suggested. The core distinction is in the ecosystem: Google AI Edge runs tflite models, whereas Cactus is buil
30.
by rshemet 1y ago
So far, our focus is on supporting models with fully open-sourced weights. Providers who are sensitive about their weights typically lock those weights up in their cloud and don't run their models locally on consumer devices anyway. I

More ›