9 ms·
Supabase Logs: open-source logging server
- nextaccountic 3y ago> Logflare was available under a BSL license prior to joining Supabase. We’ve since changed the license to Apache 2.0, aligning it with our open source philosophy. I wish more companies were like this! (note, the vector.dev link is broken)
- kiwicopple 3y agothanks for spotting that, fixed here: https://github.com/supabase/supabase/pull/13594 https://github.com/supabase/supabase/pull/13594
- simonw 3y agohttps://github.com/Logflare/logflare/tree/staging https://github.com/Logflare/logflare/tree/staging > Simply provide your BigQuery credentials and we stream logs into your BigQuery table while automatically managing the schema I didn't know BigQuery was capable of accepting streaming log data - in my mental model of the world it was the kind of database that you update using the occasional batch job, not from a streaming source of data. Looks like that's the tabledata.insertAll method which has been around for quite a few years - though it's now called the "legacy streaming API" on https://cloud.google.com/bigquery/docs/streaming-data-into-bigquery https://cloud.google.com/bigquery/docs/streaming-data-into-b... which suggests using the more recent Storage Write API instead: https://cloud.google.com/bigquery/docs/write-api https://cloud.google.com/bigquery/docs/write-api
- martin_ 3y agoIt's pretty awesome & very cost effective! When I was at Twilio we streamed logstash/fluentd (forget which) -> kinesis -> BigQuery and it worked great - certainly better than the days we were trying to manage ES ourselves
- harpratap 3y agoWere people using it as an actual ES replacement for logs? fluentd -> GCP log sink -> BQ can get awful if you need to work with it on daily basis
- chasers 3y agoCurious how was it awful for you? Really have not regretted any part of using BigQuery.
- swyx 3y agois kinesis in the middle for more durability/being able to handle spikes? when and how does one make that sort of decision to insert kinesis into things? (esp since you were gonna get the data from aws -> gcp so you dont save anything staying within aws anyway)
- lyziinc 3y agoI'm one of the Logflare devs @ supabase that made this a reality, feel free to AMA about the technical details.
- ignoramous 3y agoHi, You mentioned above that BigQuery reduces cost. I am surprised by that assertion, tbh. Can you point out ways in which Logflare uses it that makes it so (for ex, is it tiered-storage with a BQ front-end)? How does Logflare's approach contrast with other entrants like axiom.co/99 who are leveraging blob stores (Cloudflare R2) for storage and serverless for querying for lower costs? Multiple pluggable storage/query backends (like Clickhouse) is all good, but is there a default that Logflare is going to recommend / settle on? Are there plans to go beyond just APM with Logflare (like metrics and traces, for instance)? I guess, at some level, this product signals a move away from Postgres-for-everything stance? Thanks.
- chasers 3y ago> Can you point out ways in which Logflare uses it that makes it so (for ex, is it tiered-storage with a BQ front-end)? After 3 months BigQuery storage ends up being about half the cost of object storage if you use partitioned tables and don't edit the data. > How does Logflare's approach contrast with other entrants like axiom.co/99 who are leveraging blob stores (Cloudflare R2) for storage and serverless for querying for lower costs? Haven't really looked at their arch but BigQuery kind of does that for us. > Multiple pluggable storage/query backends (like Clickhouse) is all good, but is there a default that Logflare is going to recommend / settle on? tbd > Are there plans to go beyond just APM with Logflare (like metrics and traces, for instance)? Yes. You can send any JSON payload to Logflare and it will simply handle it. Official open telemetry support is coming, but it should just work if your library can send it over as JSON. And you can send it metrics. > I guess, at some level, this product signals a move away from Postgres-for-everything stance? Postgres will last you a very long time usually but at some point with lots of this kind of data you'll really want to use an OLAP store. With Supabase Wrappers you'll be able to easily access your analytics store from Postgres. https://supabase.com/blog/postgres-foreign-data-wrappers-rust https://supabase.com/blog/postgres-foreign-data-wrappers-rus...
- gvv 3y agoI weep a bit every time I see a new Supabase capability knowing I'm stuck on Firebase.
- wutania 3y agoGiven that Logflare is a feature independent from supabase seems like you can use the logs independently if you wish!
- wenbo 3y agoWhat's stopping you from coming over to Supabase?
- scottfr 3y agoFirebase uses Google Cloud Logging. Taking a quick look at the blog post here, Google Cloud Logging already seems to support everything it describes. Is there something in it that makes it a better solution in some way than what Google is already providing? (Note that Supabase Logs appears to rely on Google BigQuery so you'll be running on Google either way.)
- swyx 3y agonot for long: > Logflare currently supports a BigQuery backend. We plan to add support for other analytics-optimized databases, like Clickhouse. We will also support pushing data to other web services, making Logflare a good fit for any data pipeline. > > This will benefit the Supabase CLI: once Postgres support is available, Logflare will be able to integrate seamlessly, without the BigQuery requirement.
- madjam002 3y agoIt would be awesome if this could use Quickwit as a backend which is a new promising alternative to Elasticsearch, I’ve been using it internally and it’s much more lightweight and easier to run.
- francoismassot 3y agoCool :) We would need to support SQL though. Just putting the link here: http://github.com/quickwit-oss/quickwit http://github.com/quickwit-oss/quickwit Would love to have your feedback on ease of use/perf/whatever here/on twitter/discord (https://discord.quickwit.io/ https://discord.quickwit.io/).
- lyziinc 3y agoSupporting quickwit and its query language is definitely feasible and would not require quickwit to have SQL support. However, we've got SQL DBs on our roadmap at the moment, so it might be a while until we get to quickwit
- chasers 3y agoWow man Quickwit looks amazing. Probably would have reached for it when I started Logflare if it was around. I have not seen anything open source that really separates compute and storage like that in one package. You can setup Clickhouse to use an object store but there's a bunch of nuance there. People are starting to hack together something similar using DuckDB on Lambda, basically as a query engine on top of S3. I wonder if you can use DuckDb to get a SQL interface going? Going to have to take a close look for sure. SQL is definitely a blocker. With Supabase we give people the ability to query their logs with full SQL. And with Logflare Endpoints you create an API endpoint with a SQL query.
- francoismassot 3y agothanks :) I'm convinced that a combo supabase + quickwit can be quite powerful. It's possible to make Quickwit support simple SQL queries like "SELECT xxx FROM yyy GROUP BY something" quite fast. If you are eager to explore a POC with quickwit, let's have a quick chat, here is my email francois [at] quickwit [dot] io.
- kiwicopple 3y agohey hn, supabase ceo here this one is a long-time coming and it's a continuation of our acquisition of Logflare[0]. Since the acquisition we've be open-source-ing the server, which you can find here: https://github.com/Logflare/logflare https://github.com/Logflare/logflare Logflare handles about 1.5 billion log-events everyday on supabase. It's built with Elixir and has no problems with that workload. This is really just the start of the Logflare updates. All logs are currently ingested into BigQuery, and we are adding support for Clickhouse and other backends (focusing primarily on open source backend). Over time Logflare will function very much like an open source Sentry alternative, where you can ingest data from various sources. The team will be around if you have any questions about the technical implementation [0] acquision: https://supabase.com/blog/supabase-acquires-logflare https://supabase.com/blog/supabase-acquires-logflare
- ThePhysicist 3y agoSentry is open-source.
- kiwicopple 3y agoSimply drawing a comparison with that comment, so that readers know the direction of the product. I'll leave my comment above unedited, so that your comment makes sense - but perhaps I shouldn't have said "alternative", and simply left it a "Over time Logflare will function very much like Sentry."
- mdaniel 3y agoThat is no longer true, it is source available, becoming open source after a 3 year embargo period: https://github.com/getsentry/sentry/blob/23.3.1/LICENSE#L19 https://github.com/getsentry/sentry/blob/23.3.1/LICENSE#L19
- swyx 3y agothank you thank you. the longer i spend as a dev the more i have infinite appreciation for better logging. this is asking a lot but could we get a breakdown as to what you think are relevant decision factors for choice of logging server? (eg vs logstash/kibana, splunk and i guess people DIY with clickhouse but we can effectively ignore that)
- sidcool 3y agoIs self hosting possible and feasible?
- kiwicopple 3y agoYes, the docs are here: https://supabase.com/docs/reference/self-hosting-analytics/introduction https://supabase.com/docs/reference/self-hosting-analytics/i... One caveat: the only backend-store supported right now BigQuery. We will be releasing support for Clickhouse and Postgres in the coming months, we just couldn't fit it into this Launch Week
- mosselman 3y agoIs there somewhere I can subscribe for an update on this specifically?
- kiwicopple 3y agoIt's easiest to "watch releases" on either the Logflare repo[0] or the Supabase repo[1]. If you're signed up to Supabase we send 1 email per month with all of our releases. [0] Logflare: https://github.com/Logflare/logflare https://github.com/Logflare/logflare [1] Supabase: https://github.com/supabase/supabase https://github.com/supabase/supabase
- progx 3y agoExists a pino transport?
- kiwicopple 3y agoyes: https://github.com/Logflare/pino-logflare https://github.com/Logflare/pino-logflare
- lyziinc 3y agoSupabase-specific SDKs are still in the works. However, if you're using the Logflare service as is, there is a pino transport[0] available for sending events directly to Logflare. [0]: https://github.com/Logflare/pino-logflare https://github.com/Logflare/pino-logflare
- trialect 3y agoWhy is this better than loki?
- mdaniel 3y agohttps://github.com/Logflare/logflare/blob/staging/LICENSE.md https://github.com/Logflare/logflare/blob/staging/LICENSE.md for one thing
- cpursley 3y agoGreat job, Supabase! Can we use Logflare as a generic logging service?
- chasers 3y agoYes! You can sign up for Logflare separately actually.
- Nathanba 3y agoIf sentryio has a self hosted version and logflare has one too, why would I pick logflare? Are there any differences? I tried sentryio and it's really convenient, at least the hosted version.
- chasers 3y agoDepends on your use case. If the volume of data you're looking at is not much, Sentry is more feature rich and refined. If you want to expose a logging interface to your customers and/or easily integrate large volumes of structured event type data into the rest of your infra, then maybe look at Logflare.
- mtlynch 3y agoI tried LogFlare (which is now Supabase Logs) in January, but it didn't work well for what I wanted. Supabase Logs / Logflare seems primarily interested in creating graphs from logs rather than using logs for diagnostic purposes. I've been looking for a log solution that's good for the use case of high retention but low volume. I have a few small apps that generate a few MB of logs per month, so basically nothing. But I still want to have all my logs searchable in one place. Most logging solutions set retention based on time rather than data size. So regardless of how much you're logging, they throw away your logs within somewhere between 7-30 days unless you're on an insane Enterprise plan. I was excited about LogFlare because it supports unlimited retention, but I ran into too many issues and had to cancel my subscription: * To search your logs, you need to write a SQL-like query in LogFlare's DSL. You can't just put in a route (e.g. /api/auth) like you can with other log analytics. * Search only shows the matching lines. Usually, what I want to see is the log line in context. For example, if I search "catastrophic error" I want to see the log lines leading up to that, not just that specific line. * Search is limited to a maximum of 100 results. If you want to see more results, you need to rewrite your query rather than just scroll up or hit a "load more" button. * When you do adjust the query to a larger time window, the query will fail because it can't generate a graph unless you also adjust the group_by in your query to match the new time window's limits. This is an annoying obstacle if you don't care about graphing the results and are just trying to diagnose an issue in your logs. I found support lacking as well. I emailed support to ask if I was misunderstanding how to use Logflare or if it was just designed for a different use case. I was on a paid plan, but I still had to wait 3 business days for a response. When the response came, they just said that it was designed for me but didn't address any of the issues I brought up. I do like that Logflare/Supabase let you bring your own BigQuery. That's nice for customers like me who want low-volume, high retention. I hope they continue iterating because it has potential. In the meantime, I've found LogTail to be a pretty good alternative, but they're limited to 30 days of retention even on the highest tier plan.
- chasers 3y agoAppreciate the detailed feedback here. Frankly these are all things I've wanted to address. We've just had to prioritize other tasks. We're definitely going to keep iterating. Thanks for taking the time to try it!
- ralusek 3y agoWhat are some of the largest production applications built using Supabase? I know it's popular for whipping something up for a hackathon, but how battle tested is it? Also does anybody know what they're doing behind the scenes with the database? I know their storage uses s3, functions (I think) use Deno, this uses BigQuery. Is their db on RDS/Aurora? If so how do they claim max DB size of 1024 TB while Aurora is 128 TB?
- kiwicopple 3y ago- Storage: s3 - Database: EC2 t4g - Functions: Deno We have tens-of-thousands of applications using Supabase in production. "Size" is a hard one to answer because it means different things to different people. We have projects that have databases and storage in the hundreds-of-terrabytes, projects making millions of API requests, and logos from Fortune 100 companies.
- kiwicopple 3y ago*hundreds-of-gigabytes - not terabytes. apologies, I’m responding too fast this week, and don’t want to mislead anyone
- ralusek 3y agoHow do you claim max DB size of 1024TB, are you guys using Citus? Would the user need to handle partitioning etc at that size?
- kiwicopple 3y agoThat is a now incorrect, and I will get the team to correct it. The max is 64TB (the size of a EBS volume, https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/volume_constraints.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/volume_c...) Thank you for pointing it out
- notadeveloper 3y agoLooks great! Supabase is amazing product. Why not Bigtable though?