tdrhq

5 ms·

1.
SwiftUI Previews and Prefire: free snapshot tests(screenshotbot.io)
3 points by tdrhq 1y ago|0 comments
2.
Can Git LFS scale for screenshot tests?(screenshotbot.io)
2 points by tdrhq 1y ago|0 comments
3.
PixelDiff: A desktop app for image comparisons and screenshot tests(github.com)
2 points by tdrhq 1y ago|0 comments
4.
by tdrhq 1y ago
Yes, but our SaaS tool required our customers to not use sparse or shallow checkouts since we needed the git commit-graph. Using upload-pack allowed us to remove that constraint, since even in a shallow clone we can still get the commit gra
5.
by tdrhq 1y ago
Azure DevOps has an additional requirement that Git clients support a protocol feature called "multi-ack". We don't support it yet, and we didn't think we need it. Rather than blocking our roll-out on implementing multi-
6.
by tdrhq 1y ago
Thanks! (author here)
7.
Using Git-upload-pack for a simpler CI integration(blog.screenshotbot.io)
20 points by tdrhq 1y ago|8 comments
8.
by tdrhq 2y ago
> The paper that definitionally is Raft doesn't tell you how to interact with durable storage. That's being a bit pedantic. Yeah, I did mean that any respectable library implementing Raft would handle all of this correctly. &gt
9.
by tdrhq 2y ago
> Upgrades are especially challenging Yes indeed! But this doesn't apply to a startup in the Explore phase, where you don't need replication, and how we did it for a long time. This is the phase where this architecture is the m
10.
by tdrhq 2y ago
We used an existing library called bknr.datastore to handle this part, so we didn't have to reinvent the wheel :) I mentioned that at the end of the blog post, but I wanted to build up the idea for people who have no prior knowledge ab
11.
by tdrhq 2y ago
To clarify, as I think some people have misunderstood: we used an existing library called bknr.datastore to handle the "database" part of the in-memory store, so we didn't have to invent too much. Our only innovation here was
12.
by tdrhq 2y ago
Good catch on the metrics! We do use Preset for metrics and dashboards, and obviously Preset isn't going to talk to our in-memory database. So we do have a separate MySQL database where we just push analytics data. (e.g. each time an e
13.
by tdrhq 2y ago
Ah, sure: I did not consider Redis at all. My goal in the Explore phase was to keep the data in the same process as my code, and replacing MySQL with any other database doesn't really help here. This was a developer-productivity goal,
14.
by tdrhq 2y ago
Raft does do persistence and crash recovery, at least of the transaction logs. What you need from your side (and there are libraries that already do this): a) A mechanism to snapshot all the data b) An easy in-memory mechanism to create in
15.
by tdrhq 2y ago
This is fascinating, thanks for the data! I agree with the the other reply to this: I probably should've said that it's easy to get a machine with 100s of GB of RAM instead of saying it's "cheap".
16.
by tdrhq 2y ago
> It's quite odd that an argument grounded on performance claims I probably did a bad job then, because everything in the blog post was meant to be developer productivity claims, not performance claims. (I come from a developer prod
17.
by tdrhq 2y ago
Good question! And this comes to the difference between Explore phase and Expand phase. In the Explore phase, data migration was just running code on the production server via a REPL. Some migrations such as adding/removing fields are
18.
by tdrhq 2y ago
Absolutely. By the way, if it wasn't clear from my blog post, in the Explore phase, I used an existing library to do this. It was only in the Expand phase that I put this existing library behind a Raft replication.
19.
by tdrhq 2y ago
> Wait, so you’re blocking on a Raft round-trip to make forward progress? That’s the correct decision wrt durability, but… Yeah. I hope it was clear in my post that the goal was developer productivity, not performance. The round trip is
20.
by tdrhq 2y ago
I haven't seen either of these! But I have to say, my inspiration here came from existing libraries. (My only innovation here is taking an existing library that did the whole transaction log thing, and putting it behind a Raft cluster.
21.
by tdrhq 2y ago
There's a list of libraries here, which include a few Python libraries: https://raft.github.io/ I don't know if they're production grade. I was drawn to Braft because of Baidu's backing.
22.
by tdrhq 2y ago
> but checking his socials he is not really experienced person. I'm not sure what qualifies as experience if Meta/Google doesn't. ;)
23.
by tdrhq 2y ago
I think it's important to understand that every startup goes through three phases: Explore, Expand, Extract. What's simple in one phase isn't simple in the other. A transactional database is simple in Expand and Extract, but
24.
by tdrhq 2y ago
> This is actually not an easy thing to do. If your shutdowns are always clean SIGSTOPs, yes, you can reliably flush writes to disk. But if you get a SIGKILL at the wrong time, or don’t handle an io error correctly, you’re probably going
25.
by tdrhq 2y ago
I was certainly inspired by PG's writing (after all we do use Common Lisp, and it's hard to avoid PG in this space). But I don't think they did things like transaction logs like how bknr.datastore does, which makes the develo
26.
by tdrhq 2y ago
Good catch! Snapshotting was certainly a bottleneck that I chose not to write about. But we aren't really taking the snapshot of RAM, instead we're running some code asking each object to snapshot itself into a stream. If you do t
27.
by tdrhq 2y ago
[Author here] The transactions and snapshots are still logged to disk. So if the cluster goes down and comes back up, each one just reloads the state. Until at least two machines are back up, we won't be able to serve requests though.
28.
by tdrhq 2y ago
[Author here] Indeed, bknr.datastore was inspired by Prevayler and similar libraries
29.
by tdrhq 2y ago
Author here. I agree, the infrastructure required to make this happen eventually gets quite complicated. But the developer experience is what's super simple. If somebody had to take all our infrastructure and just use it to build their
30.
by tdrhq 2y ago
Haha, I totally hear you. But but, we didn't really build the raft consensus layer from scratch. We used an existing robust library for that: https://github.com/baidu/braft

More ›