11 ms·
Show HN: Trench – Open-source analytics infrastructure
Hey HN! I want to share a new open source project I've been working on called Trench (https://trench.dev https://trench.dev). It's open source analytics infrastructure for tracking events, page views, and identifying users, and it's built on top of ClickHouse and Kafka.
https://github.com/frigadehq/trench https://github.com/frigadehq/trench
I built Trench because the Postgres table we used for tracking events at our startup (http://frigade.com/ http://frigade.com/) was getting expensive and becoming a performance bottleneck as we scaled to millions of end users.
Many companies run into the same problem as us (e.g. Stripe, Heroku: https://brandur.org/fragments/events https://brandur.org/fragments/events). They often start by adding a basic events table to their relational database, which works at first, but can become an issue as the application scales. It’s usually the biggest table in the database, the slowest one to query, and the longest one to back up.
With Trench, we’ve put together a single Docker image that gives you a production-ready tracking event table built for scale and speed. When we migrated our tracking table from Postgres to Trench, we saw a 42% reduction in cost to serve on our primary Postgres cluster and all lag spikes from autoscaling under high traffic were eliminated.
Here are some of the core features:
* Fully compliant with the Segment tracking spec e.g. track(), identify(), group(), etc.
* Can handle thousands of events per second on a single node
* Query tracking data in real-time with read-after-write guarantees
* Send data anywhere with throttled and batched webhooks
* Single production-ready docker image. No need to manage and roll your own Kafka/ClickHouse/Nodejs/etc.
* Easily plugs into any cloud hosted ClickHouse and Kafka solutions e.g. ClickHouse Cloud, Confluent
Trench can be used for a range of use cases. Here are some possibilities:
1. Real-Time Monitoring and Alerting: Set up real-time alerts and monitoring for your services by tracking custom events like errors, usage spikes, or specific user actions and sending that data anywhere with Trench’s webhooks
2. Event Replay and Debugging: Capture all user interactions in real-time for event replay
3. A/B Testing Platform: Capture events from different users and groups in real time. Segment users by querying in real time and serve the right experiences to the right users
4. Product Analytics for SaaS Applications: Embed Trench into your existing SaaS product to power user audit logs or tracking scripts on your end-users’ websites
5. Build a custom RAG model: Easily query event data and give users answers in real-time. LLMs are really good at writing SQL
The project is open-source and MIT-licensed. If there’s interest, we’re thinking about adding support for Elastic Search, direct data integrations (e.g. Redshift, S3, etc.), and an admin interface for creating queries, webhooks, etc.
Have you experienced the same issues with your events tables? I'd love to hear what HN thinks about the project.
- Attummm 2y agoLooks great, but what is missing for me are use cases. Why should I use it? What are the unique selling points of your project?
- pancomplex 2y agoI looked around, but all the open source analytics projects I could find were bloated with all kinds of UI and unnecessary code paths. They also all seemed to use row-based RDMS as the data backbone (vs columnar stores like ClickHouse). I was looking for a backend-only solution that we could shape for our product use case that could scale. So TLDR, if you're at a smaller scale (<1M MAUs), you probably will be fine just using a table in MySQL or Postgres. If you have a lot of traffic and users, you will need something like Trench that uses Kafka and ClickHouse.
- Attummm 2y agoYou are selling the underlying technologies(Kafka/Clickhouse). I'm interested in your project can do for me, my project(s), team/company. There is a reason that most of the internet still uses PHP and old technologies. Because they focused not on the latest tech but on solving problems for others. The project looks cool, but tell us the usecases.
- dfltr 2y agoUse case #1: You have a problem table (e.g. a high-volume events table) that grows non-linearly as your business starts to scale up. A queue + columnar store package like Trench moves the problem table out to a system better equipped to deal with it and lets your DB server handle its relational business in relative peace and quiet.
- Attummm 2y agoMaybe I wasn't clear enough but my questions have been rhetorical. They were not for me. If one starts stating technologies, it is akin to describing the individual ingredients of a sandwich. The question remains: Why choose Trench over just using Kafka and Clickhouse or any other message queue and columnar database / big data base? If the goal of the post and the landing website is to entice people to use the tool, then answering these questions is important. If what is being discussed seems obvious, then who is the target demographic? Because they already know the space, use alternatives or have built their own.