8 ms·

Supabase Storage now supports the S3 protocol

501 points by inian 2y ago

JoshTriplett 2y ago
You specifically say "for large files". What's your bandwidth and latency like for small files (e.g. 20-20480 bytes), and how does it compare to raw S3's bandwidth and latency for small files?
- fenos 2y ago
  You can think of the Storage product as an upload server that sits in front of S3. Generally, you would want to place an upload server to accept uploads from your customers, that is because you want to do some sort of file validation, access control or anything else once the file is uploaded. The nice thing is that we run Storage within the same AWS network, so the upload latency is as small as it can be. In terms of serving files, we provide a CDN out-of-the-box for any files that you upload to Storage, minimising latencies geographically
  - Rapzid 2y ago
    > Generally, you would want to place an upload server to accept uploads from your customers A common pattern on AWS is to not handle the upload on your own servers. Checks are made ahead of time, conditions baked into the signed URL, and processing is handled after the fact via bucket events.
    - fenos 2y ago
      That is also a common pattern I agree, both ways are fine if the upload server is optimised accordingly :)
- egorr 2y ago
  hey, supabase engineer here; we didn’t check that out with files that small, but thanks for the idea, i will try it out the only thing i can say related to the topic is that s3 multipart outperforms other methods for files larger than 50mb significantly, but tends to have similar or slightly slower speeds compared to s3 regular upload via supabase or simplest supabase storage upload for files with size about and less than 50mb. s3-multipart is indeed the fastest way to upload file to supabase with speeds up to 100mb/s(115 even) for files>500mb. But for files about 5mb or less you are not going to need to change anything in your upload logic just for performance cause you won’t notice any difference probably everything mentioned here is for upload only
- ovaistariq 2y ago
  S3 is not known for performing well with such small files. Is this primarily how your dataset on S3 looks like?
denysvitali 2y ago
Friendly reminder that Supabase is really cool, and if you haven't tried it out you should do it (everything can be self hosted and they have generous free tiers!)
- kiwicopple 2y ago
  thanks for taking time to make a comment. it's nice to get some kind words between the feedback (which we also like)
- isoprophlex 2y ago
  Plus, with all the inflated vc money fueled hype on vector databases, they seem to have the only offering in this space that actually makes sense to me. With them you can store your embeddings close to all the rest of your data, in a single postgres db.
  - BoorishBears 2y ago
    pgVector is literally everywhere: https://github.com/pgvector/pgvector/issues/54 https://github.com/pgvector/pgvector/issues/54
- joshxyz 2y ago
  their team is crazy good
avodonosov 2y ago
The article does not mention: do you support pre-signed URLs? https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-...
- fenos 2y ago
  Thanks for asking this, we do not support signed URLs just yet, but it will be added in the next iteration
  - avodonosov 2y ago
    Presigned URLs are useful because client app can upload/download directly from S3, saving the app server from this traffic. Does Row-Level Security achieve the same benefit?
    - fenos 2y ago
      Yes, I agree! Although I should have specified that we support signed URL https://supabase.com/docs/reference/javascript/storage-from-createsignedurl https://supabase.com/docs/reference/javascript/storage-from-... just not in the S3 protocol just yet :)
      - dmattia 2y ago
        Thanks for confirming! Could you maybe update https://supabase.com/docs/guides/storage/s3/compatibility https://supabase.com/docs/guides/storage/s3/compatibility to include s3-esque presigning as a feature to track?
  - arxpoetica 2y ago
    Can you give an indication of what "next iteration" means in terms of timeline (even just ballpark)?
kiwicopple 2y ago
hey hn, supabase ceo here For background: we have a storage product for large files (like photos, videos, etc). The storage paths are mapped into your Postgres database so that you can create per-user access rules (using Postgres RLS) This update adds S3 compatibility, which means that you can use it with thousands of tools that already support the protocol. I'm also pretty excited about the possibilities for data scientists/engineers. We can do neat things like dump postgres tables in to Storage (parquet) and you can connect DuckDB/Clickhouse directly to them. We have a few ideas that we'll experiment with to make this easy Let us know if you have any questions - the engineers will also monitor the discussion
inian 2y ago
Here is the example of the DuckDB querying parquet files directly from Storage because it supports the S3 protocol now - https://github.com/TylerHillery/supabase-storage-duckdb-demo https://github.com/TylerHillery/supabase-storage-duckdb-demo https://www.youtube.com/watch?v=diL00ZZ-q50 https://www.youtube.com/watch?v=diL00ZZ-q50
- cmollis 2y ago
  Yes. Duckdb works very well with parquet scans on s3 right now.
  - iamcreasy 2y ago
    Does it work well with Hive tables storing parquet files on s3?
jonplackett 2y ago
Dear supabase. Please don’t get bought out by anyone and ruined. I’ve built too many websites with a supabase backend now to go back.
- kiwicopple 2y ago
  we don't have any plans to get bought. we only have plans to keep pushing open standards/tools - hopefully we have enough of a track record here that it doesn't feel like lip service
  - robertlagrant 2y ago
    As long as you make it so if you do get bought a team of you can always fork and move on, it's about the best anyone can hope for.
  - jonplackett 2y ago
    Absolutely. I am so impressed with SB. It’s like you read my mind and then make what I’ll need before I realise. (This is not a paid promotion)
    - kiwicopple 2y ago
      >like you read my mind we receive a lot of community feedback and ultimately there are only a few "primitives" developers need to solve 90% of their problems I think we're inching closer to the complete set of primitives which can be used to combine into second-order primitives (eg: queues/search/maps can all be "wrappers" around the primitives we already provide)
      - deleted 2y ago
        [deleted]
      - jonplackett 2y ago
        That's a neat way of thinking about it. Thanks for an awesome product. Please also never get bought or make plans to in the future, or if you really, really, really have to then please not by google.
kaliqt 2y ago
I tried to migrate from Firebase once and it wasn't really straightforward and decided against doing it, I think you guys (if you haven't already) should make migration plugins a first class priority that "just works" as the amount of real revenue generating production projects on Firebase and similar are of a much higher number. It's a no-brainer that many of them may want to switch if it were safe and simple to do so.
- kiwicopple 2y ago
  we have some guides (eg: https://supabase.com/docs/guides/resources/migrating-to-supabase/firebase-storage https://supabase.com/docs/guides/resources/migrating-to-supa...) also some community tools: https://github.com/supabase-community/firebase-to-supabase https://github.com/supabase-community/firebase-to-supabase we often help companies migrating from firebase to supabase - usually they want to take advantage of Postgres with similar tooling.
gime_tree_fiddy 2y ago
Shouldn't it be API rather than protocol? Also my sympathies for having to support the so-called "S3 standard/protocol".
- fenos 2y ago
  Yes, both can be fine :) after all, a Protocol can be interpreted as a Standardised API which the client and server interact with, it can be low-level or high-level. I hope you like the addition and we have the implementation all open-source on the Supabase Storage server
- preommr 2y ago
  I think that protocol is appropriate here since s3 resources are often represented by a s3:// url where the scheme part of the url is often used to represent the protocol.
  - ovaistariq 2y ago
    S3 uses HTTP protocol for communication between client and server. The `s3://` paths make it seem like it’s a protocol but that’s just a way to represent the path on the s3 client to differentiate it from a local file path.
nextaccountic 2y ago
A question about implementation, is the data really stored in a Postgres database? Do you support transactional updates like atomically updating two files at once? Is there a Postgres storage backend optimized for storing large files?
- deleted 2y ago
  [deleted]
- fenos 2y ago
  We do not store the files in Postgres, the files are stored in a managed S3 bucket. We store the metadata of the objects and buckets in Postgres so that you can easily query it with SQL. You can also implement access control with RLS to allow access to certain resources. It is not currently possible to guarantee atomicity on 2 different file uploads since each file is uploaded on a single request, this seems a more high-level functionality that could be implemented at the application level
  - nextaccountic 2y ago
    Oh. So this is like, S3 on top of S3? That's interesting.
    - kiwicopple 2y ago
      in case it's not clear why this is required, some of the things the storage engine handles are: image transformations, caching, automatic cache-busting, multiple protocols, metadata management, postgres compatibility, multipart uploads, compatibility across storage backends, etc
    - fenos 2y ago
      Yes indeed! I would call it S3 on steroids! Currently, it happens to be S3 to S3, but you could write an adapter, let's say GoogleCloudStorage and it will become S3 -> GoogleCloudStorage, or any other type of underline Storage. Additionally, we provide a special way of authenticating to Supabase S3 using the SessionToken, which allows you to scope S3 operations to your users specific access control https://supabase.com/docs/guides/storage/s3/authentication#session-token https://supabase.com/docs/guides/storage/s3/authentication#s...
madsbuch 2y ago
This is really nice to see! I asked about that feature almost 2 years ago as we wanted to use Supabase for everything. Unfortunately there were no plans back then to support it, so we had to use another provider for object storage. Congrats on the release!
ntry01 2y ago
This is great news, and I agree with everyone in the thread - Supabase is a great product. Does this mean that Supabase (via S3 protocol) supports file download streaming using an API now? As far as I know, it was not achievable before and the only solution was to create a signed URL and stream using HTTP.
- fenos 2y ago
  Yes, absolutely! You can download files as streams and make use of Range requests too. The good news is that the Standard API is also supporting stream!
withinboredom 2y ago
Now we just need flutterflow to get off the firebase bandwagon.
- kiwicopple 2y ago
  (supabase team member) Firebase is an amazing tool for building fast. I want Supabase to be a "tool for everyone", but ultimately giving developers choices between various technologies is a good thing for developers. I think it's great that Flutterflow support both Firebase & Supabase. I know Flutterflow's Firebase integration is a bit more polished so hopefully we can work closer with the FF team to make our integration more seamless
filleokus 2y ago
Do you think Supabase Storage (now or in the future) could be an attractive standalone S3 provider as an alternative to e.g MinIO?
- kiwicopple 2y ago
  It's more of a "accessibility layer" on top of S3 or any other s3-compatible backend (which means that it also works with MinIO out-of-the-box [0]) I don't think we'll ever build the underlying storage layer. I'm a big fan of what the Tigris[1] team have built if you're looking for other good s3 alternatives [0] https://github.com/supabase/storage/blob/master/docker-compose.yml#L59 https://github.com/supabase/storage/blob/master/docker-compo... [1] Tigris: https://tigrisdata.com https://tigrisdata.com
  - masfuerte 2y ago
    I'm still confused. Who is paying Amazon? Is it 1. You buy the storage from Amazon so I pay you and don't interact with Amazon at all or 2. I buy storage from Amazon and you provide managed access to it?
    - kiwicopple 2y ago
      it's 1 - you pay us, we pay AWS. In the future we will likely provide "bring your own s3" (2)
  - ovaistariq 2y ago
    I appreciate the callout to Tigris. Now it’s super easy to use Tigris with Supabase for all those files.
yoavm 2y ago
This looks great! How easy is it to self host Supabase? Is it more like "we're open-source, but good luck getting this deployed!", or can someone really build on Supabase and if things get a little too expensive it's easy enough to self-host the whole thing and just switch over? I wonder if people are doing that.
- kiwicopple 2y ago
  self-hosting docs are here: https://supabase.com/docs/guides/self-hosting/docker https://supabase.com/docs/guides/self-hosting/docker And a 5-min demo video with Digital Ocean: https://www.youtube.com/watch?v=FqiQKRKsfZE&embeds_referring_euri=https%3A%2F%2Fsupabase.com%2F https://www.youtube.com/watch?v=FqiQKRKsfZE&embeds_referring... Anyone who is familiar with basic server management skills will have no problem self-hosting. every tool in the supabase stack[0] is a docker image and works in isolation. If you just want to use this Storage Engine, it's on docker-hub (supabase/storage-api). Example with MinIO: https://github.com/supabase/storage/blob/master/docker-compose.yml https://github.com/supabase/storage/blob/master/docker-compo... [0] architecture: https://supabase.com/docs/guides/getting-started/architecture https://supabase.com/docs/guides/getting-started/architectur...
  - replwoacause 2y ago
    I was pretty unhappy with the self hosted offering. It’s neutered compared to the cloud, which was disappointing.
    - kiwicopple 2y ago
      > neutered Can you share more? There is nothing missing, it’s all the same code that we run in production (besides some login changes to keep it decoupled from the platform)
      - ihateolives 2y ago
        From top of my head: reports and multiple projects. Also I kept getting weird "fetch failed" errors when creating rules or adding users from gui working behind reverse proxy so I gave up on the end.
devjume 2y ago
This is great news. Now I can utilize any CDN provider that supports S3. Like bunny.net [1] which has image optimization, just like Supabase does but with better pricing and features. I have been developing with Supabase past two months. I would say there are still some rough corners in general and some basic features missing. Example Supabase storage has no direct support for metadata [2][3]. Overall I like the launch week and development they are doing. But more attention to basic features and little details would be needed because implementing workarounds for basic stuff is not ideal. [1] https://bunny.net/ https://bunny.net/ [2] https://github.com/orgs/supabase/discussions/5479 https://github.com/orgs/supabase/discussions/5479 [3] https://github.com/supabase/storage/issues/439 https://github.com/supabase/storage/issues/439
- kiwicopple 2y ago
  > I can utilize any CDN provider that supports S3. Like bunny.net Bunny is a great product. I'm glad this release makes that possible for you and I imagine this was one of the reasons the rest of the community wanted it too > But more attention to basic features and little details This is what we spend most of our time doing, but you won't hear about it because they aren't HN-worthy. > no direct support for metadata Fabrizio tells me this is next on the list. I understand it's frustrating, but there is a workaround - store metadata in the postgres database (I know, not ideal but still usable). We're getting through requests as fast as we can.
  - fenos 2y ago
    This is indeed at the very top of the list :)
- giancarlostoro 2y ago
  I've not done a whole lot with S3 but is this due to it being easy to sync between storage providers that support S3 or something? I'm more used to Azure Blob Storage than anything, so I'm OOL on what people do other than store files on S3.
simonbarker87 2y ago
Supabase is great and I’ve used it for a number of projects over the years both with a backend alongside it or direct from the client with RLS. There are some weird edges (well really just faff) around auth with the JS library but if nothing else they are by far the cheapest hosted SQL offering I can find so any faff you don’t want to deal with there’s an excellent database right there to allow you to roll your own (assuming you have a backend server alongside it)
sgt 2y ago
Can Supabase host static content yet (in a decent way)?
- fenos 2y ago
  We don’t support static website hosting just yet - might happen in the future :)
  - kiwicopple 2y ago
    just a small elaboration: You can serve HTML content if you have a custom domain enabled. We don't plan to do anything beyond that - there are already some great platforms for website hosting, and we have enough problems to solve with Postgres
  - sgt 2y ago
    That would be cool, at least for someone building a small app with just one tiny SaaS bill to worry about. Even better; using the free tier for an app that is a very very small niche.
jimmySixDOF 2y ago
Supabase also announced this week Oriole (the team not just the table storage plugin) is joining them so I guess this is part of the same story. Anyway it's nice timing I was thinking about a hookup to Cloudflare R2 for something and this may be the way.
- kiwicopple 2y ago
  Oriole are joining to work on the OrioleDB postgres extension. That's slightly different to this release: - This: for managing large files in s3 (videos, images, etc). - Oriole: a postgres extension that's a "drop-in replacement" for the default storage engine We also hope that the team can help develop Pluggable Storage in Postgres with the rest of the community. From the blog post[0]: > Pluggable Storage gives developers the ability to use different storage engines for different tables within the same database. This system is available in MySQL, which uses the InnoDB as the default storage engine since MySQL 5.5 (replacing MyISAM). Oriole aims to be a drop-in replacement for Postgres' default storage engine and supports similar use-cases with improved performance. Other storage engines, to name a few possibilities, could implement columnar storage for OLAP workloads, highly compressed timeseries storage for event data, or compressed storage for minimizing disk usage. Tangentially: we have a working prototype for decoupled storage and compute using the Oriole extension (also in the blog post). This stores Postgres data in s3 and there could be some inter-play with this release in the future [0] https://supabase.com/blog/supabase-aquires-oriole https://supabase.com/blog/supabase-aquires-oriole
  - kabes 2y ago
    What's the point of acquiring them instead of just sponsering the project? I'm trying to understand supabase's angle here and if this is good or bad news for non-supabase postgres users.
    - kiwicopple 2y ago
      We sponsored/invested in them previously (over half a million USD). The benefit of acquiring is that the team can focus on the technology rather than the business/operations around it. I hope this is the best outcome for the industry. If there are other postgres companies that would like to use oriole, feel free to reach out. It’s 100% open source and we’d love more design partners
spacebanana7 2y ago
I wish supabase had more default integrations with CDNs, transactional email services and domain registrars. I'd happily pay a 50% markup for the sake of having everything in one place.
- kiwicopple 2y ago
  are there any specific integrations that you want? (eg: which companies?) we have a built-in CDN[0] and we have some existing integrations for transactional emails [1] [0] Smart CDN: https://supabase.com/docs/guides/storage/cdn/smart-cdn https://supabase.com/docs/guides/storage/cdn/smart-cdn [1] Integrations: https://supabase.com/partners/integrations https://supabase.com/partners/integrations
  - spacebanana7 2y ago
    Thanks for the link about integrations! I wasn't aware about the resend one for transactional email. It'd be nice to have an integration with a domain register, like Ghandi.net or Namecheap. Ideally with the cost coming through as an item in my Supabase bill.
    - kiwicopple 2y ago
      Thanks for the recommendations - I’ll see what this would involve
      - deleted 2y ago
        [deleted]
    - zenorocha 2y ago
      Resend founder here - feel free to try out the integration with Supabase and let me know if you have any questions
- tootie 2y ago
  At the same time, I worry about them drifting too far from their core mission. I think Vercel and Netlify kinda went that way and when you look at their suite of features, you just have to ask why would I not just use AWS directly.
Rapzid 2y ago
I like to Lob my BLOBs into PG's storage. You need that 1-2TB of RDS storage for the IOPS anyway; might as well fill it up. Large object crew, who's with me?!
- kiwicopple 2y ago
  that's not how this works. files are stored in s3, metadata in postgres
  - Rapzid 2y ago
    Sad. J/K. It could be a really good back-end option for Supabase's S3 front end. A lot of PG clients don't support proper "streaming" and looking at the codebase it's TypeScript.. postgres.js is the only client nearing "performant" I'm aware of(last I looked) on Node.js but it's not clear it supports streaming outside "Copy" per the docs. Support could be added to the client if missing. Edit: Actually it could be a good option for your normal uploads too. Docs talk about it being ideal for 6Mb or smaller files? Are you using bytea or otherwise needing to buffer the full upload/download in memory? Streaming with Lob would resolve that, and you can compute incremental hash sums for etags and etc. Lob has downsides and limitations but for a very large number of people it has many great benefits that can carry them very far and potentially all the way.
  - tln 2y ago
    Will the files get deleted with ON CASCADE DELETE somehow? That would be awesome. Then for GDPR, when you delete a user, the associated storage can be deleted. One could cobble this together with triggers, some kind of external process, and probably repetititious code so there is one table of metadata per "owning" id, although it would be nice to be packaged.
    - Rapzid 2y ago
      That's an interesting concept. Of course you don't want to delete your record of the blob until the blog is deleted.. A trigger could add a "job" to delete the blob into another table when the file record is deleted though..
jarpineh 2y ago
Hi, a question, but first some background. I've been looking at solutions to store columnar data with versioning, essentially Parquet. But, I'd also like to store PDFs, CSVs, images, and such for our ML workflows. I wonder if now, that Supabase is getting better for data science DuckDB crowd, could Supabase be that one solution for all this?
- kiwicopple 2y ago
  > Parquet. But, I'd also like to store PDFs, CSVs, images yes, you can store all of these in Supabase Storage and it will probably "just work" with the tools that you already use (since most tools are s3-compatible) Here is an example of one of our Data Engineers querying parquet with DuckDB: https://www.youtube.com/watch?v=diL00ZZ-q50 https://www.youtube.com/watch?v=diL00ZZ-q50 We're very open to feedback here - if you find any rough edges let us know and we can work on it (github issues are easiest)
  - jarpineh 2y ago
    Well, this is great news. I'll take "just works" guarantee any day ;) We have yet to make a commitment to any one product. Having Postgres there is a big plus for me. I'll have to see about doing a test or two.
- Bnjoroge 2y ago
  you should look at lance(https://lancedb.github.io/lance/ https://lancedb.github.io/lance/)
mritchie712 2y ago
Very cool and very good timing. We just released better support to directly query S3[0] in Definite[1]. It's powered by duckdb. 0 - https://www.youtube.com/watch?v=yrrCQnfKEig https://www.youtube.com/watch?v=yrrCQnfKEig 1 - https://www.definite.app/ https://www.definite.app/
- deleted 2y ago
  [deleted]
brap 2y ago
Always thought it’s kind of odd how the proprietary API of AWS S3 became sort of the de-facto industry standard
- mousetree 2y ago
  Because that's where most of the industry store their data.
  - brap 2y ago
    Yeah I understand how it came to be, it’s just kind of an uncommon situation
- moduspol 2y ago
  Yeah--though I guess kudos to AWS for not being litigious about it.
- garbanz0 2y ago
  Same thing seems to be happening with openai api
- bdcravens 2y ago
  S3 is one of the original AWS services (SQS predates it), and has been around for 18 years. The idea of a propriety API becoming the industry defacto standard isn't uncommon. The same thing happened with Microsoft's XMLHttpRequest.
  - crubier 2y ago
    Also, the S3 API is simple and makes sense, no need to reinvent something different just for the pleasure of it
    - ovaistariq 2y ago
      There are valid reasons for extending and redoing some parts of the API. I will give you one example. Suppose you want to extend list objects to support ordering by last modified or you want to support filtering of objects by user metadata. Right now doing this is quite clunky via headers.
foerster 2y ago
no feedback on this in particular, but I love supabase. I use it for several projects and it's been great. I was hesitant to use triggers and PG functions initially, but after I got my migrations sorted out, it's been pretty awesome.
- SOLAR_FIELDS 2y ago
  Do you manage your functions and triggers through source code? What framework do you use to do that? I like Supabase but it’s desire to default to native pg stuff for a lot of that has kind of steered me away from using it for more complex projects where you need to use sprocs to retrieve data and pgtap to test them, because hiding away business logic in the db like that is viewed as an anti pattern in a lot of organizations. I love it for simple CRUD apps though, the kind where the default postgrest functionality is mostly enough and having to drop into a sproc or build a view is rarely necessary. I think if there was a tightly integrated framework for managing the state of all of these various triggers, views, functions and sproc through source and integrating them into the normal SDLC it would be a more appealing sell for complex projects
  - inian 2y ago
    The Supabase CLI [1] provides a way for you to manage functions, triggers and anything else in your Postgres database as a migration. These migrations would be checked into your source control. You can then take it a step further but opting-in to use Branching [2] to better manage environments. We just opened up the Branching feature to everyone [3]. [1]: https://supabase.com/docs/guides/cli/local-development#database-migrations https://supabase.com/docs/guides/cli/local-development#datab... [2]: https://supabase.com/docs/guides/platform/branching https://supabase.com/docs/guides/platform/branching [3]: https://supabase.com/blog/branching-publicly-available https://supabase.com/blog/branching-publicly-available
    - SOLAR_FIELDS 2y ago
      Thanks for the response. I don’t think I was super clear about what I meant - I’m more talking about the following scenario: Let’s say that we are using the Typescript sdk to make our app and need to do some fancy aggregation on a table that isn’t supported by Postgrest natively (specifically we can’t retrieve the data with Postgrest out of the box with its typical setup). Postgrest tells us that in this case we can do two things: create a view or make a Postgres function. Each has their pros and cons, but with either choice now we have this problem: some of our business logic is in sproc/function/view and some of it is in Typescript. In a typical setup using an ORM it would all be in Typescript. The conventional wisdom is that db’s are dumb state holders and all of the business logic goes in the app - Supabase attempts to turn this on its head and say no actually it’s ok to store business logic in the db. But now if we do that we have a new problem: we don’t have a blessed path forward for where the line is on what goes where anymore. We don’t have good patterns for storing and managing this so other developers understand where to put things and how to work with our app anymore, because it no longer holds the principle of least astonishment. That’s what I mean by framework in this context. Maybe all that is necessary here is a battle tested example project that demonstrates the “correct” way to make this demarcation. But as-is it steers me away from using Supabase for more complex projects if I even think they will need something that Postgrest won’t support without making a view or sproc/function
WhitneyLand 2y ago
Is iOS support a priority for supabase?
- kiwicopple 2y ago
  Yes, we made it an official library this week: https://supabase.com/blog/supabase-swift https://supabase.com/blog/supabase-swift
tootie 2y ago
One of the big wins we get from AWS is that you can do things like load structured data files (csv, parquet) from S3 directly in Redshift using SQL queries. https://docs.aws.amazon.com/redshift/latest/dg/t_loading-tables-from-s3.html https://docs.aws.amazon.com/redshift/latest/dg/t_loading-tab...
- inian 2y ago
  This is indeed pretty cool. They also have the `aws_s3` extension [1] for doing the same thing inside Postgres. Unfortunately, the extension isn't open source. [1]: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PostgreSQL.S3Import.html https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_...
  - kiwicopple 2y ago
    we might be able to achieve the same thing now with our S3 Wrapper: https://supabase.github.io/wrappers/s3/ https://supabase.github.io/wrappers/s3/
poxrud 2y ago
Do you support S3 event notifications?
- bdcravens 2y ago
  "S3 protocol" typically refers to object storage read/write/delete, not additional service APIs. Support in other S3-compatible vendors varies, often with a different payload (though a translation wrapper shouldn't be too difficult to implement) https://developers.cloudflare.com/r2/buckets/event-notifications/ https://developers.cloudflare.com/r2/buckets/event-notificat... https://docs.digitalocean.com/reference/api/spaces-api/ https://docs.digitalocean.com/reference/api/spaces-api/
- inian 2y ago
  We don't support S3 event notifications directly, but you achieve similar functionality by using Database Webhooks [1]. You can trigger any HTTP endpoint or a Supabase Edge function by adding a trigger to the objects table [3] in the Storage schema. [1]: https://supabase.com/docs/guides/database/webhooks https://supabase.com/docs/guides/database/webhooks [2]: https://supabase.com/docs/guides/functions https://supabase.com/docs/guides/functions [3]: https://supabase.com/docs/guides/storage/schema/design https://supabase.com/docs/guides/storage/schema/design
mattgreenrocks 2y ago
Just commenting to say I really appreciate your business model. Whereas most businesses actively seek to build moats to maintain competitive advantage and locking people in, actions like this paint a different picture of Supabase. I'll be swapping out the API my app uses for Supabase storage to switch it to an S3 API this weekend in case I ever need to switch it. My only real qualm at this point is mapping JS entities using the JS DB API makes it hard to use camelcase field names due to PG reasons I can't recall. I'm not sure what a fix for that would look like. Keep up the good work.
- kiwicopple 2y ago
  > mapping JS entities using the JS DB API makes it hard to use camelcase field names due to PG reasons I can't recall Postgres requires you to "quoteCamelCase". There are some good JS libs to map between snake case and camel case. FWIW, I like a mix of both in my code: snake_case indicates it's a database property, and camelCase indicates its a JS property. Try it out - it might grow on you :)
animeshjain 2y ago
Is there any request pricing (I could not find a mention to it on the pricing page). Could be quite compelling for some use-cases if request pricing is free.
- inian 2y ago
  There is no per request pricing.
stephen37 2y ago
Cool to see that Supabase is adding S3 protocol! Nice to see more and more storage solutions available. We, at Milvus, I've integrated S3, Parquet and other ones to make it possible for developers to use their data no matter what they use. For those who have used both, how do you find the performance and ease of integration compares between Supabase and other solutions like Milvus that have had these features for some time?
- teaearlgraycold 2y ago
  Just make your own post if you’re selling your product
- jongjong 2y ago
  Nice. Will check out Milvus.
iamcreasy 2y ago
In this setup, can postgresql query data stored in object storage? i.e. Hive/Icebarg table.
fswd 2y ago
I just finished implementing S3 file upload in nextjs to cloudflare R2 with a supabase backend. Wish I had been lazy and waited a day!
pier25 2y ago
At $0.1/GB of egress it’s not super attractive compared to B2 or R2 for anything but trivial projects. I wish they would offer a plan with just the pg database. Any news on pricing of Fly PG?
- inian 2y ago
  We are hosted on aws and are just passing the cost over to our users. We make no margin on egress fees. Deploying storage on other clouds including Fly.io is planned. We are actively working on our Fly integration. At the start, the pricing is going to be exactly the same as our hosted platform on aws - https://supabase.com/docs/guides/platform/fly-postgres#pricing https://supabase.com/docs/guides/platform/fly-postgres#prici...
  - pier25 2y ago
    Thanks. So a user that only wants pg has to pay for storage etc?
Havoc 2y ago
What are the chances of Supabase doing a license change? Seems to be fashionable these days so always a little wary of building on these sort of platforms
- ezekg 2y ago
  Seeing as Neon, Nile, Citus, etc. are all open source, I highly doubt it. But who knows. In the end, most license changes are blown out of proportion by a vocal minority and largely have zero effect on 99.9% of users.
- kiwicopple 2y ago
  Unfortunately there isn't a way to prove our intentions to remain open source That said, I think Supabase is much more de-risked from this happening because we aim to support existing tools with a strong preference of tools that are controlled by foundations rather than commercial entities. For example, 2 of the main tools: - Postgres (PostgreSQL license) - PostgREST (MIT license) Every tool/library/extension that we develop and release ourselves is either MIT, Apache2, or PostgreSQL
  - Havoc 2y ago
    Thanks for the response. Seems reasonable
pull_my_finger 2y ago
Is there a formal s3 protocol spec or do these companies try to reverse engineer/feature match what AWS provides?
- starttoaster 2y ago
  The S3 API isn't unknown or anything, the client library SDKs all being open source. So I'd imagine a software developer writing a tool that aims to be S3 API-compliant would use one of the open source SDKs, and write their API while making requests to it locally through a client from one of the Amazon SDKs. Not trivial effort, but also pretty easy to imagine how you'd start off. If a client function from the SDK doesn't work with your API, you write your API handler to handle the HTTP request that function makes until it is supported. I have wondered if Amazon has some additional tooling for other software providers to make their own S3-compliant APIs, but I don't know what Amazon's motivation would be to help make it easier for people to switch between other vendors. Whereas the incentive is much more obvious for other software vendors to make their own APIs S3-compliant. So I've so far imagined it is a similar process to how I described above, instead.
- inian 2y ago
  The S3 API reference [1] is closest to a formal spec there is. The request, response and the error codes are pretty well documented. [1]: https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operations_Amazon_Simple_Storage_Service.html https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operatio...
  - snthpy 2y ago
    Thanks for this. Do you know how are redirects handled? I searched on that page and didn't find anything but I've seen it mentioned elsewhere that they are catered for but I haven't found any documentation for that. I'm particularly interested in the temporary ones - 302, 303 and 307.
  - swyx 2y ago
    perhaps you could open source the test suite as a standalone thing so that other s3 compatible apis can prove they match the supabase standard

501 points by inian 2y ago

You specifically say "for large files". What's your bandwidth and latency like for small files (e.g. 20-20480 bytes), and how does it compare to raw S3's bandwidth and latency for small files?

fenos 2y ago
You can think of the Storage product as an upload server that sits in front of S3. Generally, you would want to place an upload server to accept uploads from your customers, that is because you want to do some sort of file validation, access control or anything else once the file is uploaded. The nice thing is that we run Storage within the same AWS network, so the upload latency is as small as it can be. In terms of serving files, we provide a CDN out-of-the-box for any files that you upload to Storage, minimising latencies geographically
- Rapzid 2y ago
  > Generally, you would want to place an upload server to accept uploads from your customers A common pattern on AWS is to not handle the upload on your own servers. Checks are made ahead of time, conditions baked into the signed URL, and processing is handled after the fact via bucket events.
  - fenos 2y ago
    That is also a common pattern I agree, both ways are fine if the upload server is optimised accordingly :)
egorr 2y ago
hey, supabase engineer here; we didn’t check that out with files that small, but thanks for the idea, i will try it out the only thing i can say related to the topic is that s3 multipart outperforms other methods for files larger than 50mb significantly, but tends to have similar or slightly slower speeds compared to s3 regular upload via supabase or simplest supabase storage upload for files with size about and less than 50mb. s3-multipart is indeed the fastest way to upload file to supabase with speeds up to 100mb/s(115 even) for files>500mb. But for files about 5mb or less you are not going to need to change anything in your upload logic just for performance cause you won’t notice any difference probably everything mentioned here is for upload only
ovaistariq 2y ago
S3 is not known for performing well with such small files. Is this primarily how your dataset on S3 looks like?

denysvitali 2y ago

Friendly reminder that Supabase is really cool, and if you haven't tried it out you should do it (everything can be self hosted and they have generous free tiers!)

kiwicopple 2y ago
thanks for taking time to make a comment. it's nice to get some kind words between the feedback (which we also like)
isoprophlex 2y ago
Plus, with all the inflated vc money fueled hype on vector databases, they seem to have the only offering in this space that actually makes sense to me. With them you can store your embeddings close to all the rest of your data, in a single postgres db.
- BoorishBears 2y ago
  pgVector is literally everywhere: https://github.com/pgvector/pgvector/issues/54 https://github.com/pgvector/pgvector/issues/54
joshxyz 2y ago
their team is crazy good

avodonosov 2y ago

The article does not mention: do you support pre-signed URLs? https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-...

fenos 2y ago
Thanks for asking this, we do not support signed URLs just yet, but it will be added in the next iteration
- avodonosov 2y ago
  Presigned URLs are useful because client app can upload/download directly from S3, saving the app server from this traffic. Does Row-Level Security achieve the same benefit?
  - fenos 2y ago
    Yes, I agree! Although I should have specified that we support signed URL https://supabase.com/docs/reference/javascript/storage-from-createsignedurl https://supabase.com/docs/reference/javascript/storage-from-... just not in the S3 protocol just yet :)
    - dmattia 2y ago
      Thanks for confirming! Could you maybe update https://supabase.com/docs/guides/storage/s3/compatibility https://supabase.com/docs/guides/storage/s3/compatibility to include s3-esque presigning as a feature to track?
- arxpoetica 2y ago
  Can you give an indication of what "next iteration" means in terms of timeline (even just ballpark)?

kiwicopple 2y ago

hey hn, supabase ceo here For background: we have a storage product for large files (like photos, videos, etc). The storage paths are mapped into your Postgres database so that you can create per-user access rules (using Postgres RLS) This update adds S3 compatibility, which means that you can use it with thousands of tools that already support the protocol. I'm also pretty excited about the possibilities for data scientists/engineers. We can do neat things like dump postgres tables in to Storage (parquet) and you can connect DuckDB/Clickhouse directly to them. We have a few ideas that we'll experiment with to make this easy Let us know if you have any questions - the engineers will also monitor the discussion

inian 2y ago

Here is the example of the DuckDB querying parquet files directly from Storage because it supports the S3 protocol now - https://github.com/TylerHillery/supabase-storage-duckdb-demo https://github.com/TylerHillery/supabase-storage-duckdb-demo https://www.youtube.com/watch?v=diL00ZZ-q50 https://www.youtube.com/watch?v=diL00ZZ-q50

cmollis 2y ago
Yes. Duckdb works very well with parquet scans on s3 right now.
- iamcreasy 2y ago
  Does it work well with Hive tables storing parquet files on s3?

jonplackett 2y ago

Dear supabase. Please don’t get bought out by anyone and ruined. I’ve built too many websites with a supabase backend now to go back.

kiwicopple 2y ago
we don't have any plans to get bought. we only have plans to keep pushing open standards/tools - hopefully we have enough of a track record here that it doesn't feel like lip service
- robertlagrant 2y ago
  As long as you make it so if you do get bought a team of you can always fork and move on, it's about the best anyone can hope for.
- jonplackett 2y ago
  Absolutely. I am so impressed with SB. It’s like you read my mind and then make what I’ll need before I realise. (This is not a paid promotion)
  - kiwicopple 2y ago
    >like you read my mind we receive a lot of community feedback and ultimately there are only a few "primitives" developers need to solve 90% of their problems I think we're inching closer to the complete set of primitives which can be used to combine into second-order primitives (eg: queues/search/maps can all be "wrappers" around the primitives we already provide)
    - deleted 2y ago
      [deleted]
    - jonplackett 2y ago
      That's a neat way of thinking about it. Thanks for an awesome product. Please also never get bought or make plans to in the future, or if you really, really, really have to then please not by google.

kaliqt 2y ago

I tried to migrate from Firebase once and it wasn't really straightforward and decided against doing it, I think you guys (if you haven't already) should make migration plugins a first class priority that "just works" as the amount of real revenue generating production projects on Firebase and similar are of a much higher number. It's a no-brainer that many of them may want to switch if it were safe and simple to do so.

kiwicopple 2y ago
we have some guides (eg: https://supabase.com/docs/guides/resources/migrating-to-supabase/firebase-storage https://supabase.com/docs/guides/resources/migrating-to-supa...) also some community tools: https://github.com/supabase-community/firebase-to-supabase https://github.com/supabase-community/firebase-to-supabase we often help companies migrating from firebase to supabase - usually they want to take advantage of Postgres with similar tooling.

gime_tree_fiddy 2y ago

Shouldn't it be API rather than protocol? Also my sympathies for having to support the so-called "S3 standard/protocol".

fenos 2y ago
Yes, both can be fine :) after all, a Protocol can be interpreted as a Standardised API which the client and server interact with, it can be low-level or high-level. I hope you like the addition and we have the implementation all open-source on the Supabase Storage server
preommr 2y ago
I think that protocol is appropriate here since s3 resources are often represented by a s3:// url where the scheme part of the url is often used to represent the protocol.
- ovaistariq 2y ago
  S3 uses HTTP protocol for communication between client and server. The `s3://` paths make it seem like it’s a protocol but that’s just a way to represent the path on the s3 client to differentiate it from a local file path.

nextaccountic 2y ago

A question about implementation, is the data really stored in a Postgres database? Do you support transactional updates like atomically updating two files at once? Is there a Postgres storage backend optimized for storing large files?

deleted 2y ago
[deleted]
fenos 2y ago
We do not store the files in Postgres, the files are stored in a managed S3 bucket. We store the metadata of the objects and buckets in Postgres so that you can easily query it with SQL. You can also implement access control with RLS to allow access to certain resources. It is not currently possible to guarantee atomicity on 2 different file uploads since each file is uploaded on a single request, this seems a more high-level functionality that could be implemented at the application level
- nextaccountic 2y ago
  Oh. So this is like, S3 on top of S3? That's interesting.
  - kiwicopple 2y ago
    in case it's not clear why this is required, some of the things the storage engine handles are: image transformations, caching, automatic cache-busting, multiple protocols, metadata management, postgres compatibility, multipart uploads, compatibility across storage backends, etc
  - fenos 2y ago
    Yes indeed! I would call it S3 on steroids! Currently, it happens to be S3 to S3, but you could write an adapter, let's say GoogleCloudStorage and it will become S3 -> GoogleCloudStorage, or any other type of underline Storage. Additionally, we provide a special way of authenticating to Supabase S3 using the SessionToken, which allows you to scope S3 operations to your users specific access control https://supabase.com/docs/guides/storage/s3/authentication#session-token https://supabase.com/docs/guides/storage/s3/authentication#s...

madsbuch 2y ago

This is really nice to see! I asked about that feature almost 2 years ago as we wanted to use Supabase for everything. Unfortunately there were no plans back then to support it, so we had to use another provider for object storage. Congrats on the release!

ntry01 2y ago

This is great news, and I agree with everyone in the thread - Supabase is a great product. Does this mean that Supabase (via S3 protocol) supports file download streaming using an API now? As far as I know, it was not achievable before and the only solution was to create a signed URL and stream using HTTP.

fenos 2y ago
Yes, absolutely! You can download files as streams and make use of Range requests too. The good news is that the Standard API is also supporting stream!

withinboredom 2y ago

Now we just need flutterflow to get off the firebase bandwagon.

kiwicopple 2y ago
(supabase team member) Firebase is an amazing tool for building fast. I want Supabase to be a "tool for everyone", but ultimately giving developers choices between various technologies is a good thing for developers. I think it's great that Flutterflow support both Firebase & Supabase. I know Flutterflow's Firebase integration is a bit more polished so hopefully we can work closer with the FF team to make our integration more seamless

filleokus 2y ago

Do you think Supabase Storage (now or in the future) could be an attractive standalone S3 provider as an alternative to e.g MinIO?

kiwicopple 2y ago
It's more of a "accessibility layer" on top of S3 or any other s3-compatible backend (which means that it also works with MinIO out-of-the-box [0]) I don't think we'll ever build the underlying storage layer. I'm a big fan of what the Tigris[1] team have built if you're looking for other good s3 alternatives [0] https://github.com/supabase/storage/blob/master/docker-compose.yml#L59 https://github.com/supabase/storage/blob/master/docker-compo... [1] Tigris: https://tigrisdata.com https://tigrisdata.com
- masfuerte 2y ago
  I'm still confused. Who is paying Amazon? Is it 1. You buy the storage from Amazon so I pay you and don't interact with Amazon at all or 2. I buy storage from Amazon and you provide managed access to it?
  - kiwicopple 2y ago
    it's 1 - you pay us, we pay AWS. In the future we will likely provide "bring your own s3" (2)
- ovaistariq 2y ago
  I appreciate the callout to Tigris. Now it’s super easy to use Tigris with Supabase for all those files.

yoavm 2y ago

This looks great! How easy is it to self host Supabase? Is it more like "we're open-source, but good luck getting this deployed!", or can someone really build on Supabase and if things get a little too expensive it's easy enough to self-host the whole thing and just switch over? I wonder if people are doing that.

kiwicopple 2y ago
self-hosting docs are here: https://supabase.com/docs/guides/self-hosting/docker https://supabase.com/docs/guides/self-hosting/docker And a 5-min demo video with Digital Ocean: https://www.youtube.com/watch?v=FqiQKRKsfZE&embeds_referring_euri=https%3A%2F%2Fsupabase.com%2F https://www.youtube.com/watch?v=FqiQKRKsfZE&embeds_referring... Anyone who is familiar with basic server management skills will have no problem self-hosting. every tool in the supabase stack[0] is a docker image and works in isolation. If you just want to use this Storage Engine, it's on docker-hub (supabase/storage-api). Example with MinIO: https://github.com/supabase/storage/blob/master/docker-compose.yml https://github.com/supabase/storage/blob/master/docker-compo... [0] architecture: https://supabase.com/docs/guides/getting-started/architecture https://supabase.com/docs/guides/getting-started/architectur...
- replwoacause 2y ago
  I was pretty unhappy with the self hosted offering. It’s neutered compared to the cloud, which was disappointing.
  - kiwicopple 2y ago
    > neutered Can you share more? There is nothing missing, it’s all the same code that we run in production (besides some login changes to keep it decoupled from the platform)
    - ihateolives 2y ago
      From top of my head: reports and multiple projects. Also I kept getting weird "fetch failed" errors when creating rules or adding users from gui working behind reverse proxy so I gave up on the end.

devjume 2y ago

This is great news. Now I can utilize any CDN provider that supports S3. Like bunny.net [1] which has image optimization, just like Supabase does but with better pricing and features. I have been developing with Supabase past two months. I would say there are still some rough corners in general and some basic features missing. Example Supabase storage has no direct support for metadata [2][3]. Overall I like the launch week and development they are doing. But more attention to basic features and little details would be needed because implementing workarounds for basic stuff is not ideal. [1] https://bunny.net/ https://bunny.net/ [2] https://github.com/orgs/supabase/discussions/5479 https://github.com/orgs/supabase/discussions/5479 [3] https://github.com/supabase/storage/issues/439 https://github.com/supabase/storage/issues/439

kiwicopple 2y ago
> I can utilize any CDN provider that supports S3. Like bunny.net Bunny is a great product. I'm glad this release makes that possible for you and I imagine this was one of the reasons the rest of the community wanted it too > But more attention to basic features and little details This is what we spend most of our time doing, but you won't hear about it because they aren't HN-worthy. > no direct support for metadata Fabrizio tells me this is next on the list. I understand it's frustrating, but there is a workaround - store metadata in the postgres database (I know, not ideal but still usable). We're getting through requests as fast as we can.
- fenos 2y ago
  This is indeed at the very top of the list :)
giancarlostoro 2y ago
I've not done a whole lot with S3 but is this due to it being easy to sync between storage providers that support S3 or something? I'm more used to Azure Blob Storage than anything, so I'm OOL on what people do other than store files on S3.

simonbarker87 2y ago

Supabase is great and I’ve used it for a number of projects over the years both with a backend alongside it or direct from the client with RLS. There are some weird edges (well really just faff) around auth with the JS library but if nothing else they are by far the cheapest hosted SQL offering I can find so any faff you don’t want to deal with there’s an excellent database right there to allow you to roll your own (assuming you have a backend server alongside it)

sgt 2y ago

Can Supabase host static content yet (in a decent way)?

fenos 2y ago
We don’t support static website hosting just yet - might happen in the future :)
- kiwicopple 2y ago
  just a small elaboration: You can serve HTML content if you have a custom domain enabled. We don't plan to do anything beyond that - there are already some great platforms for website hosting, and we have enough problems to solve with Postgres
- sgt 2y ago
  That would be cool, at least for someone building a small app with just one tiny SaaS bill to worry about. Even better; using the free tier for an app that is a very very small niche.

jimmySixDOF 2y ago

Supabase also announced this week Oriole (the team not just the table storage plugin) is joining them so I guess this is part of the same story. Anyway it's nice timing I was thinking about a hookup to Cloudflare R2 for something and this may be the way.

kiwicopple 2y ago
Oriole are joining to work on the OrioleDB postgres extension. That's slightly different to this release: - This: for managing large files in s3 (videos, images, etc). - Oriole: a postgres extension that's a "drop-in replacement" for the default storage engine We also hope that the team can help develop Pluggable Storage in Postgres with the rest of the community. From the blog post[0]: > Pluggable Storage gives developers the ability to use different storage engines for different tables within the same database. This system is available in MySQL, which uses the InnoDB as the default storage engine since MySQL 5.5 (replacing MyISAM). Oriole aims to be a drop-in replacement for Postgres' default storage engine and supports similar use-cases with improved performance. Other storage engines, to name a few possibilities, could implement columnar storage for OLAP workloads, highly compressed timeseries storage for event data, or compressed storage for minimizing disk usage. Tangentially: we have a working prototype for decoupled storage and compute using the Oriole extension (also in the blog post). This stores Postgres data in s3 and there could be some inter-play with this release in the future [0] https://supabase.com/blog/supabase-aquires-oriole https://supabase.com/blog/supabase-aquires-oriole
- kabes 2y ago
  What's the point of acquiring them instead of just sponsering the project? I'm trying to understand supabase's angle here and if this is good or bad news for non-supabase postgres users.
  - kiwicopple 2y ago
    We sponsored/invested in them previously (over half a million USD). The benefit of acquiring is that the team can focus on the technology rather than the business/operations around it. I hope this is the best outcome for the industry. If there are other postgres companies that would like to use oriole, feel free to reach out. It’s 100% open source and we’d love more design partners

spacebanana7 2y ago

I wish supabase had more default integrations with CDNs, transactional email services and domain registrars. I'd happily pay a 50% markup for the sake of having everything in one place.

kiwicopple 2y ago
are there any specific integrations that you want? (eg: which companies?) we have a built-in CDN[0] and we have some existing integrations for transactional emails [1] [0] Smart CDN: https://supabase.com/docs/guides/storage/cdn/smart-cdn https://supabase.com/docs/guides/storage/cdn/smart-cdn [1] Integrations: https://supabase.com/partners/integrations https://supabase.com/partners/integrations
- spacebanana7 2y ago
  Thanks for the link about integrations! I wasn't aware about the resend one for transactional email. It'd be nice to have an integration with a domain register, like Ghandi.net or Namecheap. Ideally with the cost coming through as an item in my Supabase bill.
  - kiwicopple 2y ago
    Thanks for the recommendations - I’ll see what this would involve
    - deleted 2y ago
      [deleted]
  - zenorocha 2y ago
    Resend founder here - feel free to try out the integration with Supabase and let me know if you have any questions
tootie 2y ago
At the same time, I worry about them drifting too far from their core mission. I think Vercel and Netlify kinda went that way and when you look at their suite of features, you just have to ask why would I not just use AWS directly.

Rapzid 2y ago

I like to Lob my BLOBs into PG's storage. You need that 1-2TB of RDS storage for the IOPS anyway; might as well fill it up. Large object crew, who's with me?!

kiwicopple 2y ago
that's not how this works. files are stored in s3, metadata in postgres
- Rapzid 2y ago
  Sad. J/K. It could be a really good back-end option for Supabase's S3 front end. A lot of PG clients don't support proper "streaming" and looking at the codebase it's TypeScript.. postgres.js is the only client nearing "performant" I'm aware of(last I looked) on Node.js but it's not clear it supports streaming outside "Copy" per the docs. Support could be added to the client if missing. Edit: Actually it could be a good option for your normal uploads too. Docs talk about it being ideal for 6Mb or smaller files? Are you using bytea or otherwise needing to buffer the full upload/download in memory? Streaming with Lob would resolve that, and you can compute incremental hash sums for etags and etc. Lob has downsides and limitations but for a very large number of people it has many great benefits that can carry them very far and potentially all the way.
- tln 2y ago
  Will the files get deleted with ON CASCADE DELETE somehow? That would be awesome. Then for GDPR, when you delete a user, the associated storage can be deleted. One could cobble this together with triggers, some kind of external process, and probably repetititious code so there is one table of metadata per "owning" id, although it would be nice to be packaged.
  - Rapzid 2y ago
    That's an interesting concept. Of course you don't want to delete your record of the blob until the blog is deleted.. A trigger could add a "job" to delete the blob into another table when the file record is deleted though..

jarpineh 2y ago

Hi, a question, but first some background. I've been looking at solutions to store columnar data with versioning, essentially Parquet. But, I'd also like to store PDFs, CSVs, images, and such for our ML workflows. I wonder if now, that Supabase is getting better for data science DuckDB crowd, could Supabase be that one solution for all this?

kiwicopple 2y ago
> Parquet. But, I'd also like to store PDFs, CSVs, images yes, you can store all of these in Supabase Storage and it will probably "just work" with the tools that you already use (since most tools are s3-compatible) Here is an example of one of our Data Engineers querying parquet with DuckDB: https://www.youtube.com/watch?v=diL00ZZ-q50 https://www.youtube.com/watch?v=diL00ZZ-q50 We're very open to feedback here - if you find any rough edges let us know and we can work on it (github issues are easiest)
- jarpineh 2y ago
  Well, this is great news. I'll take "just works" guarantee any day ;) We have yet to make a commitment to any one product. Having Postgres there is a big plus for me. I'll have to see about doing a test or two.
Bnjoroge 2y ago
you should look at lance(https://lancedb.github.io/lance/ https://lancedb.github.io/lance/)

mritchie712 2y ago

Very cool and very good timing. We just released better support to directly query S3[0] in Definite[1]. It's powered by duckdb. 0 - https://www.youtube.com/watch?v=yrrCQnfKEig https://www.youtube.com/watch?v=yrrCQnfKEig 1 - https://www.definite.app/ https://www.definite.app/

deleted 2y ago
[deleted]

brap 2y ago

Always thought it’s kind of odd how the proprietary API of AWS S3 became sort of the de-facto industry standard

mousetree 2y ago
Because that's where most of the industry store their data.
- brap 2y ago
  Yeah I understand how it came to be, it’s just kind of an uncommon situation
moduspol 2y ago
Yeah--though I guess kudos to AWS for not being litigious about it.
garbanz0 2y ago
Same thing seems to be happening with openai api
bdcravens 2y ago
S3 is one of the original AWS services (SQS predates it), and has been around for 18 years. The idea of a propriety API becoming the industry defacto standard isn't uncommon. The same thing happened with Microsoft's XMLHttpRequest.
- crubier 2y ago
  Also, the S3 API is simple and makes sense, no need to reinvent something different just for the pleasure of it
  - ovaistariq 2y ago
    There are valid reasons for extending and redoing some parts of the API. I will give you one example. Suppose you want to extend list objects to support ordering by last modified or you want to support filtering of objects by user metadata. Right now doing this is quite clunky via headers.

foerster 2y ago

no feedback on this in particular, but I love supabase. I use it for several projects and it's been great. I was hesitant to use triggers and PG functions initially, but after I got my migrations sorted out, it's been pretty awesome.

SOLAR_FIELDS 2y ago
Do you manage your functions and triggers through source code? What framework do you use to do that? I like Supabase but it’s desire to default to native pg stuff for a lot of that has kind of steered me away from using it for more complex projects where you need to use sprocs to retrieve data and pgtap to test them, because hiding away business logic in the db like that is viewed as an anti pattern in a lot of organizations. I love it for simple CRUD apps though, the kind where the default postgrest functionality is mostly enough and having to drop into a sproc or build a view is rarely necessary. I think if there was a tightly integrated framework for managing the state of all of these various triggers, views, functions and sproc through source and integrating them into the normal SDLC it would be a more appealing sell for complex projects
- inian 2y ago
  The Supabase CLI [1] provides a way for you to manage functions, triggers and anything else in your Postgres database as a migration. These migrations would be checked into your source control. You can then take it a step further but opting-in to use Branching [2] to better manage environments. We just opened up the Branching feature to everyone [3]. [1]: https://supabase.com/docs/guides/cli/local-development#database-migrations https://supabase.com/docs/guides/cli/local-development#datab... [2]: https://supabase.com/docs/guides/platform/branching https://supabase.com/docs/guides/platform/branching [3]: https://supabase.com/blog/branching-publicly-available https://supabase.com/blog/branching-publicly-available
  - SOLAR_FIELDS 2y ago
    Thanks for the response. I don’t think I was super clear about what I meant - I’m more talking about the following scenario: Let’s say that we are using the Typescript sdk to make our app and need to do some fancy aggregation on a table that isn’t supported by Postgrest natively (specifically we can’t retrieve the data with Postgrest out of the box with its typical setup). Postgrest tells us that in this case we can do two things: create a view or make a Postgres function. Each has their pros and cons, but with either choice now we have this problem: some of our business logic is in sproc/function/view and some of it is in Typescript. In a typical setup using an ORM it would all be in Typescript. The conventional wisdom is that db’s are dumb state holders and all of the business logic goes in the app - Supabase attempts to turn this on its head and say no actually it’s ok to store business logic in the db. But now if we do that we have a new problem: we don’t have a blessed path forward for where the line is on what goes where anymore. We don’t have good patterns for storing and managing this so other developers understand where to put things and how to work with our app anymore, because it no longer holds the principle of least astonishment. That’s what I mean by framework in this context. Maybe all that is necessary here is a battle tested example project that demonstrates the “correct” way to make this demarcation. But as-is it steers me away from using Supabase for more complex projects if I even think they will need something that Postgrest won’t support without making a view or sproc/function

WhitneyLand 2y ago

Is iOS support a priority for supabase?

kiwicopple 2y ago
Yes, we made it an official library this week: https://supabase.com/blog/supabase-swift https://supabase.com/blog/supabase-swift

tootie 2y ago

One of the big wins we get from AWS is that you can do things like load structured data files (csv, parquet) from S3 directly in Redshift using SQL queries. https://docs.aws.amazon.com/redshift/latest/dg/t_loading-tables-from-s3.html https://docs.aws.amazon.com/redshift/latest/dg/t_loading-tab...

inian 2y ago
This is indeed pretty cool. They also have the `aws_s3` extension [1] for doing the same thing inside Postgres. Unfortunately, the extension isn't open source. [1]: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PostgreSQL.S3Import.html https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_...
- kiwicopple 2y ago
  we might be able to achieve the same thing now with our S3 Wrapper: https://supabase.github.io/wrappers/s3/ https://supabase.github.io/wrappers/s3/

poxrud 2y ago

Do you support S3 event notifications?

bdcravens 2y ago
"S3 protocol" typically refers to object storage read/write/delete, not additional service APIs. Support in other S3-compatible vendors varies, often with a different payload (though a translation wrapper shouldn't be too difficult to implement) https://developers.cloudflare.com/r2/buckets/event-notifications/ https://developers.cloudflare.com/r2/buckets/event-notificat... https://docs.digitalocean.com/reference/api/spaces-api/ https://docs.digitalocean.com/reference/api/spaces-api/
inian 2y ago
We don't support S3 event notifications directly, but you achieve similar functionality by using Database Webhooks [1]. You can trigger any HTTP endpoint or a Supabase Edge function by adding a trigger to the objects table [3] in the Storage schema. [1]: https://supabase.com/docs/guides/database/webhooks https://supabase.com/docs/guides/database/webhooks [2]: https://supabase.com/docs/guides/functions https://supabase.com/docs/guides/functions [3]: https://supabase.com/docs/guides/storage/schema/design https://supabase.com/docs/guides/storage/schema/design

mattgreenrocks 2y ago

Just commenting to say I really appreciate your business model. Whereas most businesses actively seek to build moats to maintain competitive advantage and locking people in, actions like this paint a different picture of Supabase. I'll be swapping out the API my app uses for Supabase storage to switch it to an S3 API this weekend in case I ever need to switch it. My only real qualm at this point is mapping JS entities using the JS DB API makes it hard to use camelcase field names due to PG reasons I can't recall. I'm not sure what a fix for that would look like. Keep up the good work.

kiwicopple 2y ago
> mapping JS entities using the JS DB API makes it hard to use camelcase field names due to PG reasons I can't recall Postgres requires you to "quoteCamelCase". There are some good JS libs to map between snake case and camel case. FWIW, I like a mix of both in my code: snake_case indicates it's a database property, and camelCase indicates its a JS property. Try it out - it might grow on you :)

animeshjain 2y ago

Is there any request pricing (I could not find a mention to it on the pricing page). Could be quite compelling for some use-cases if request pricing is free.

inian 2y ago
There is no per request pricing.

stephen37 2y ago

Cool to see that Supabase is adding S3 protocol! Nice to see more and more storage solutions available. We, at Milvus, I've integrated S3, Parquet and other ones to make it possible for developers to use their data no matter what they use. For those who have used both, how do you find the performance and ease of integration compares between Supabase and other solutions like Milvus that have had these features for some time?

teaearlgraycold 2y ago
Just make your own post if you’re selling your product
jongjong 2y ago
Nice. Will check out Milvus.

iamcreasy 2y ago

In this setup, can postgresql query data stored in object storage? i.e. Hive/Icebarg table.

fswd 2y ago

I just finished implementing S3 file upload in nextjs to cloudflare R2 with a supabase backend. Wish I had been lazy and waited a day!

pier25 2y ago

At $0.1/GB of egress it’s not super attractive compared to B2 or R2 for anything but trivial projects. I wish they would offer a plan with just the pg database. Any news on pricing of Fly PG?

inian 2y ago
We are hosted on aws and are just passing the cost over to our users. We make no margin on egress fees. Deploying storage on other clouds including Fly.io is planned. We are actively working on our Fly integration. At the start, the pricing is going to be exactly the same as our hosted platform on aws - https://supabase.com/docs/guides/platform/fly-postgres#pricing https://supabase.com/docs/guides/platform/fly-postgres#prici...
- pier25 2y ago
  Thanks. So a user that only wants pg has to pay for storage etc?

Havoc 2y ago

What are the chances of Supabase doing a license change? Seems to be fashionable these days so always a little wary of building on these sort of platforms

ezekg 2y ago
Seeing as Neon, Nile, Citus, etc. are all open source, I highly doubt it. But who knows. In the end, most license changes are blown out of proportion by a vocal minority and largely have zero effect on 99.9% of users.
kiwicopple 2y ago
Unfortunately there isn't a way to prove our intentions to remain open source That said, I think Supabase is much more de-risked from this happening because we aim to support existing tools with a strong preference of tools that are controlled by foundations rather than commercial entities. For example, 2 of the main tools: - Postgres (PostgreSQL license) - PostgREST (MIT license) Every tool/library/extension that we develop and release ourselves is either MIT, Apache2, or PostgreSQL
- Havoc 2y ago
  Thanks for the response. Seems reasonable

pull_my_finger 2y ago

Is there a formal s3 protocol spec or do these companies try to reverse engineer/feature match what AWS provides?

starttoaster 2y ago
The S3 API isn't unknown or anything, the client library SDKs all being open source. So I'd imagine a software developer writing a tool that aims to be S3 API-compliant would use one of the open source SDKs, and write their API while making requests to it locally through a client from one of the Amazon SDKs. Not trivial effort, but also pretty easy to imagine how you'd start off. If a client function from the SDK doesn't work with your API, you write your API handler to handle the HTTP request that function makes until it is supported. I have wondered if Amazon has some additional tooling for other software providers to make their own S3-compliant APIs, but I don't know what Amazon's motivation would be to help make it easier for people to switch between other vendors. Whereas the incentive is much more obvious for other software vendors to make their own APIs S3-compliant. So I've so far imagined it is a similar process to how I described above, instead.
inian 2y ago
The S3 API reference [1] is closest to a formal spec there is. The request, response and the error codes are pretty well documented. [1]: https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operations_Amazon_Simple_Storage_Service.html https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operatio...
- snthpy 2y ago
  Thanks for this. Do you know how are redirects handled? I searched on that page and didn't find anything but I've seen it mentioned elsewhere that they are catered for but I haven't found any documentation for that. I'm particularly interested in the temporary ones - 302, 303 and 307.
- swyx 2y ago
  perhaps you could open source the test suite as a standalone thing so that other s3 compatible apis can prove they match the supabase standard