5 ms·
Hello everyone, I'm the primary author for WAL-G and would be happy to answer any questions.
by kafkes 9y ago
Hello everyone, I'm the primary author for WAL-G and would be happy to answer any questions.
- whitepoplar 9y agoThanks for making this! To someone who's unfamiliar with Postgres tooling, what's the difference between WAL-G and Barman? What're the advantages of using one over the other?
- fdr 9y agoI was kafkes's guide on this project, and the WAL-E author and maintainer... I'd say the differences between WAL-G and Barman are similar to WAL-E and Barman, which comes up relatively frequently. https://news.ycombinator.com/item?id=13573481 https://news.ycombinator.com/item?id=13573481 In summary, WAL-E is simpler program all around that focuses on cloud storage, barman does more around inventories of backups and file-based backups and configuring Postgres. There are integrative downsides to its span. WAL-E also happens to predate Barman.
- ibotty 9y agoWAL-G (and WAL-E) are expected to run next to the main database, while barman is to be run on a separate machine. Barman can also backup many databases. It is essentially the difference between a central backup service and local backups.
- sehrope 9y agoWhat's the min version of the Postgres server this can be used with?
- bgentry 9y agoSeriously awesome work on this! I was expecting some solid improvement when I heard you were rewriting this in Go, but this is beyond what I could have expected. 7x improvement on high end instance types! Also, what an impressive project to have on the resume as a college intern. I don't think many interns get to tackle something so meaningful.
- andrewstuart2 9y agoNor would many choose to, given the opportunity. Way to pick an interesting and challenging project! :-)
- ocharles 9y agoCan you elaborate on how much automated testing is behind this? When it comes to backup tools, I am very cautious.
- kafkes 9y agoWAL-G has a number of unit tests, and has been tested manually in a staging environment for a number of weeks without issues. We are looking to implement more integration tests in the future.
- brianwawok 9y agoIs this production ready, or just an early dev snapshot?
- kafkes 9y agoWAL-G is not yet production ready, but it has been used in a staging environment for the past few weeks without any issues. Once fdr adds parallel WAL support, he plans to take it into production.
- brianwawok 9y agoCool cool. is google cloud storage on the roadmap?
- fdr 9y agoI answered this question in a couple of other places, but: unknown because I don't have use for that yet. https://news.ycombinator.com/item?id=15049527 https://news.ycombinator.com/item?id=15049527
- justinclift 9y agoWAL-G looks like it should be able to talk to Minio (minio.io, S3 compatible thing, also written in go) as a backend instead of S3 itself. Minio has an interesting feature where it can be a "gateway" to other cloud storage. Google Cloud Storage is one of their specific examples: https://docs.minio.io/docs/minio-gateway-for-gcs https://docs.minio.io/docs/minio-gateway-for-gcs So WAL-G would talk to Minio, and Minio would transparently proxy that to GCS.
- brianwawok 9y agoNeat. My concern out of the gate is what would be the perf hit. I assume I am switching from WAL-E to WAL-G for more perf. But WAL-E speaks GCS. If WAL-G needs an extra hop to do so, may lose some of the point of it..
- trojanowski 9y agoDo you have plans to support encrypted backups?
- fdr 9y agoMaybe. Depends what you mean by that: https://news.ycombinator.com/item?id=15049691 https://news.ycombinator.com/item?id=15049691
- bluetech 9y agoIt would be nice if you could sign the tarball with the binary. I see the tag is already signed so hopefully it's not much trouble: https://wiki.debian.org/Creating%20signed%20GitHub%20releases https://wiki.debian.org/Creating%20signed%20GitHub%20release...
- kafkes 9y agoDone.
- patrick_vbn 9y agoAny plans to support backup to Google Cloud Storage instead of just S3?
- timdorr 9y agoOr some sort of pluggable storage system. For now, since I'm also on GCP, I'm using PGHoard: https://github.com/ohmu/pghoard https://github.com/ohmu/pghoard
- fdr 9y agoUnknown, but I don't have an immediate plan to implement them: https://news.ycombinator.com/item?id=15049527 https://news.ycombinator.com/item?id=15049527
- ngrilly 9y agoThanks! Does WAL-G provides some kind of "continuous backup" where changes committed to the database are continuously streamed to the backup storage? Or does it work "step by step", for example by backing up every 5 minutes or every 10 MB?
- andruby 9y agoIt does continuous backup like WAL-E. Both back up PG's WAL files (Write Ahead Log) and allow restoring your database state as it was at a specific time or after a specific transaction committed. This is known as point-in-time recovery (PITR) [0] Users and admins make mistakes, and accidentally delete or overwrite data. With PITR you can restore in a new environment, just before the mistake occurred and recover the data from there. [0] https://www.postgresql.org/docs/9.6/static/continuous-archiving.html https://www.postgresql.org/docs/9.6/static/continuous-archiv...
- ngrilly 9y agoWhat I meant is that the archive_command is run only when a WAL segment is completed or when archive_timeout is reached. In the meantime, nothing is backed up. On a low traffic database, this can be a problem. I'm wondering if there is a way to continuously stream the WAL to an object storage like S3, without waiting to have a complete segment.
- daurnimator 9y agoS3 is a block store; not something you can really stream to. However it might be interesting to stream WAL logs to e.g. AWS Kinesis....