EarlyOom

6 ms·

1.
by EarlyOom 1y ago
Shocking how poor frontier models perform on simple visual tasks. Best-in-domain tool calling will Become the norm
2.
by EarlyOom 2y ago
VLM Run | Member of Technical Staff, ML Systems | Full-time | Hybrid Bay Area, CA | https://vlm.run | 150k-220k / yr + Equity VLM Run is a first-of-its-kind API dedicated to running Vision Language Models on Documents, Imag
3.
by EarlyOom 2y ago
This is the main focus of VLM Run and typed extraction more generally. If you provide proper type constraints (e.g. with Pydantic) you can dramatically reduce the surface area for hallucination. Then there's actually fine-tuning on you
4.
by EarlyOom 2y ago
You can try out some of our schemas with Ollama if you want: https://github.com/vlm-run/vlmrun-hub (instructions in Readme)
5.
by EarlyOom 2y ago
VLMs are able to take context into account when filling in fields, following either a global or field specific prompt. This is great for e.g. unlabeled axes, checking a legend for units to be suffixed after a number, etc. Also, you catch lo
6.
by EarlyOom 2y ago
You can! it works with Ollama https://github.com/vlm-run/vlmrun-hub At the end of the day its just schemas. You can decide for yourself if its work upgrading to a larger, more expensive model.
7.
by EarlyOom 2y ago
We can do bounding boxes too :) we just call it visual grounding https://github.com/vlm-run/vlmrun-cookbook/blob/main/noteboo...
8.
by EarlyOom 2y ago
We convert to a JSON schema, but it would be trivial to convert this to yaml. There are some minor differences in e.g. tokens required to output JSON vs yaml which is why we've opted for our strategy.
9.
Replace OCR with Vision Language Models(github.com)
292 points by EarlyOom 2y ago|125 comments
10.
by EarlyOom 2y ago
OCR seems to be mostly solved for 'normal' text laid out according to Latin alphabet norms (left to right, normal spacing etc.), but would love to see more adversarial examples. We've seen lots of regressions around faxed or
11.
Show HN: Visually parse an entire YouTube video frame by frame(github.com)
5 points by EarlyOom 2y ago|0 comments
12.
Ask HN: What are folks using to train/fine-tune Vision Language Models
1 points by EarlyOom 2y ago|0 comments
13.
A Node.js SDK for calling Vision Language Models(github.com)
6 points by EarlyOom 2y ago|0 comments
14.
by EarlyOom 2y ago
Would love to chat! reach out scott@vlm.run
15.
by EarlyOom 2y ago
That's one of our main focuses, yes: https://docs.vlm.run/api-reference/v1/fine-tuning/post-finet...
16.
Run structured extraction on documents/images locally with Ollama and Pydantic(github.com)
170 points by EarlyOom 2y ago|29 comments
17.
by EarlyOom 2y ago
We put together an open-source collection of Pydantic schemas for a variety of document categories (W2 filings, invoices etc.), including instructions for how to get structured JSON responses from any visual input with the model of your cho
18.
by EarlyOom 2y ago
VLM Run | Member of Technical Staff, ML Systems, Developer Relations | Full-time | Bay Area, CA | https://vlm.run | 150k-220k / yr + Equity VLM Run is a first-of-its-kind API dedicated to running Vision Language Models on D
19.
Show HN: Vlm Run, Extract JSON from images, videos and documents in a simple API(vlm.run)
2 points by EarlyOom 2y ago|0 comments
20.
by EarlyOom 2y ago
Curious how this compares to platforms like https://unstructured.io/
21.
by EarlyOom 2y ago
This is a bigger issue than folks realize, visual inputs to GPT4 are really expensive (like several cents per dozen images in some cases), which means that you can't just spam the API to iterate on HTML/webpages with a software ag
22.
by EarlyOom 2y ago
We've been building something similar with https://vlm.run/ : we're starting out with documents, but feel like the real killer app will involve agentic workflows grounded in visual inputs like websites. The challen
23.
by EarlyOom 2y ago
We've been trying to solve this with https://vlm.run : the idea is to combine the character level accuracy of an OCR pipeline (like Tesseract) with the flexibility of a VLM. OCR pipelines struggle with non-trivial text layou
24.
by EarlyOom 2y ago
Hi Arthur! There's a bit of confusion here. It looks like you're referring to _audio_ transcription; that is, passing the audio component into an ASR pipeline (like Whisper, Otter etc.) to generate a transcript of any spoken words
25.
Fine-grained Visual Transcription for YouTube videos(vlm-docs.nos.run)
9 points by EarlyOom 2y ago|3 comments
26.
by EarlyOom 2y ago
TLDR: There are dozens of audio transcription APIs, but nothing for video and visual transcriptions. So we built one. If you want visual chaptering, summarization, OCR / text-extraction, audio transcriptions, and sentiment analysis on
27.
by EarlyOom 2y ago
We're trying to do something similar with VLM-1 https://vlm-docs.nos.run/guides/guide-pdf-presentations . We've found that a lot of the peculiarities of LLMs for text parsing (hallucinations etc.) can be avoid
28.
"Ok Computer, why are you slow?"(scottloftin.substack.com)
2 points by EarlyOom 3y ago|0 comments
29.
Show HN: NOS – A fast, and ergonomic PyTorch inference server(github.com)
3 points by EarlyOom 3y ago|0 comments