janchorowski

5 ms·

1.
by janchorowski 2y ago
Yes, we hope that the YAMLs will provide a clear separation between configuration and code allowing easier deploys of many apps from same codebase and principled experiments. Thanks for the hint, we'll do a large summary of all options
2.
Show HN: Build Live AI and RAG Pipelines in Minutes with YAML Templates(pathway.com)
8 points by janchorowski 2y ago|2 comments
3.
by janchorowski 2y ago
An index that combines multiple indexing techniques, e.g. vector search with more classical information retrieval techniques such as BM25. We found that they have different strengths: vector indexes are very good at getting synonyms and ind
4.
Show HN: Multimodal Search Using GPT4o for Metadata Extraction and Hybrid Index(github.com)
4 points by janchorowski 2y ago|2 comments
5.
by janchorowski 2y ago
Thanks! Can you share some more details on the usecases and features used?
6.
by janchorowski 2y ago
Thanks for the kind words!
7.
Show HN: Pathway – Build Mission Critical ETL and RAG in Python (NATO, F1 Used)(github.com)
73 points by janchorowski 2y ago|19 comments
8.
A Drive in the Office: comma.ai self-driving tech works indoors too(blog.comma.ai)
1 points by janchorowski 2y ago|0 comments
9.
by janchorowski 2y ago
Hey, I'm Jan, a Machine Learning researcher, and CTO of Pathway. I've been working on RAG for the last year and I'm excited to share a new RAG optimization strategy that adapts the number of supporting documents to the LLM be
10.
by janchorowski 3y ago
Now you can use Airbyte source connectors to process data in memory with Python. We integrated Airbyte connectors with Pathway, a Python stream processing framework, using the airbyte-serverless project. We believe ETL pipelines are coming
11.
Python ETL with Airbyte and Pathway(pathway.com)
3 points by janchorowski 3y ago|2 comments
12.
by janchorowski 3y ago
Originally from 1998: https://web.archive.org/web/19980206223800/https://norvig.co... I didn't want to mislead with a newer date. Sorry if it confuses you.
13.
Teach Yourself Programming in Ten Years (1998)(norvig.com)
586 points by janchorowski 3y ago|302 comments
14.
by janchorowski 3y ago
That's a good idea, the deduplication criterion is easy to change, using an llm is faster to get started, but after a while a corpus of decisions is created and can be used to either select another mechanism, or e.g. train one on top o
15.
by janchorowski 3y ago
Sure: when a new response is produced because some source documents have changed we ask an LLM to compare the responses and tell if they are significantly different. Even a simplistic prompt, like the one used in the example would do:
16.
Show HN: Alerting in realtime RAG: spot changes to LLM answers, using few tokens(github.com)
8 points by janchorowski 3y ago|5 comments
17.
by janchorowski 3y ago
An index is a software component building block, which becomes a database when wrapped with the data management system. We will see more and more traditional databases to add a vector-search index, for instance pgvector makes a vector datab
18.
by janchorowski 3y ago
To quickly get to the application sources please go to: - https://github.com/pathwaycom/llm-app/blob/main/llm_app/path... for the simplest contextless app - https://github.com/pathwa
19.
by janchorowski 3y ago
This depends on the data source used. Some track updateable collections, some have a more "append-only" nature. For instance, tracing a database table using CDC+Debezium will support reacting to all document changes out of the box
20.
by janchorowski 3y ago
Pathway ( https://github.com/pathwaycom/pathway ) is a data processing framework we are developing that unifies stream and batch processing of large datasets. It lets developers concentrate on writing the data processing
21.
Show HN: LLM App – build a realtime LLM app in 30 lines, with no vector database(github.com)
11 points by janchorowski 3y ago|8 comments