shortishly

6 ms·

1. 1.▲
   by shortishly 8mo ago

   You could use the proxy to spread topics over a number of brokers. The broker and proxy share a number of services and layers, that could be used to route: https://blog.tansu.io/articles/route-layer-service My itch for
2. 2.▲
   by shortishly 8mo ago

   I agree that it isn't straight forward! Tansu uses the JSON protocol descriptors from Apache Kafka, generating ~60k LoC of Rust to represent the structures. It then uses a custom Serde encoder/decoder to implement the protocol: or
3. 3.▲
   by shortishly 8mo ago

   Yes: with a schema backed topic (AVRO, JSON or Protocol buffer) Tansu can write to Apache Iceberg, Delta or Parquet. You can use a Sink topic to write directly to an open table format (including Parquet) skipping (most of) the Kafka metadat
4. 4.▲
   by shortishly 8mo ago

   Tansu author here. Storage is a pluggable choice of: PostgreSQL, memory, SQLite or S3. There are others in the pipeline (SlateDB, ...).
5. 5.▲
   Show HN: Tansu – Kafka Compatible Broker Transforms Proto/Avro/JSON into Parquet(github.com)

   1 points by shortishly 1y ago|0 comments