5 ms·
What happened to Vivaldi Social?
- photoGrant 3y agoBad luck on timing? Feels like luck had little to do with it and migration testing wasn't fuzz'd enough?
- TheDong 3y agoWho are you suggesting fuzz what? The bug wouldn't have occurred in a normal mastodon installation since mastodon's recommended configuration is a single postgres database, or at the very least to use synchronous replication. Also, very typically, fuzzers intentionally use simplified configuration, so it seems even less likely fuzzing would have caught this interaction.
- yellowapple 3y agoOne of the better post-mortems I've read in a long while.
- pluto_modadic 3y agoI remember the hachyderm postmortems were also pretty good. I'm glad that folks are transparent.
- rsynnott 3y agoItems two and three not happening atomically feels like an issue, though I assume there's a reason that it's not trivial to do so (I haven't looked at the code; really should at some point.)
- TheDong 3y agoOne of the linked fixes is: https://github.com/mastodon/mastodon/commit/13ec425b721c959415921046d7a24ed8c9994cee https://github.com/mastodon/mastodon/commit/13ec425b721c9594... It seems like it was trivial to make it happen atomically. There just wasn't a need to before since them not being atomic isn't an issue, unless you have a poor configuration like someone pointing sidekiq at a stale database server (sorry, a replica), which I see as the primary issue here.
- rsynnott 3y agoI think even without that there was likely still at least theoretically a race condition.
- williamdclt 3y agoMaybe I’m missing something but I f it’s not atomic, it doesn’t matter whether there’s a replica or not: sidekiq (whatever that is) might do a read in-between step 2 and 3. I see several problems in their setup really - lack of strong consistency - using eventually consistent data, the replica, to take business decision - no concurrency control (pessimistic or optimistic) I don’t know much about mastodon but, while not trivial, that’s pretty basic systems design concepts
- afavour 3y ago> There just wasn't a need to before since them not being atomic isn't an issue I disagree: there clearly is an issue with a non-local account having a null URI. It’s unlikely but totally possible for the server to crash inbetween query 1 and query 2, irrespective of database replication stuff. This is a textbook example of why you use database transactions.
- psd1 3y agoOTOH, reading from a read-only copy reduces load on the master
- TylerE 3y agoThis make anyone elses eyebrows raise sky high at this? > Claire replied, asking for the full stacktraces for the log entries, which I was able to also extract from the logs. This is either deep voodoo magic, or the code or configuration is turning a Xeon into the equivalent of a 286. House is that not, like, megabytes on every single hit?
- k1t 3y agoRecording stacktraces of errors is a pretty reasonable thing to do. And ideally not every hit causes an error.
- TheDong 3y ago> HTTP 500 errors when viewing an account > Stacktrace for that 500 This is the default ruby on rails behavior. It prints a stacktrace on any 500 or unknown error, and it's just line numbers and filepaths. > megabytes on every single hit I run a rails app that's very poorly designed. I just checked, and the stack trace for a single 500 is 5KiB. It doesn't even add up to 1MiB a day since there's only a 500 error about every hour. > This is either deep voodoo magic, or the code or configuration is turning a Xeon into the equivalent of a 286 Having a call stack handy is is actually pretty performant. Java's default exception behavior is to bubble up a stack trace with every exception, whether you print it or not, and java applications run just fine. You have the call stack anyway since you have to know how to return, so the only extra information you need handy is the filename and line number debug symbols, and ruby needs that info anyway just by the nature of the language.
- SV_BubbleTime 3y ago>Java's default exception behavior is to bubble up a stack trace with every exception, whether you print it or not, Anyone who has spent 5 minutes in Java knows exactly what this looks like. And also how unwelcoming it is to new programmers.
- vore 3y agoSure, but imagine how unwelcoming to everyone it would be if you didn't get a stack trace.
- AtlasBarfed 3y agoHm, so a distributed twitter runs into the challenge that each independently managed node is ... and independently managed node. Backup problems etc. Centralized twitter improves its operations for all users over time. But can be purchased by a nutso billionaire on a whim, or subjected to the """"""national security"""""" directives of the US Government.
- olah_1 3y agoPerhaps better is decentralized twitter (Nostr). Your account doesn't live on a server and you send events to multiple servers if you want to. If one server goes down, it hardly impacts you.
- input_sh 3y agoI'm sure all 30 of Nostr users will benefit.
- olah_1 3y agoYeah, because Mastodon is the most happening place online lol. I can't believe people on hacker news talk like this. Embarrassing how far this community has declined. Can't even discuss protocols without these stupid comments.
- prmoustache 3y agoIn the grand scheme of things, even twitter is anecdotic as only a small and marginal fraction of internet users are using twitter. It makes the remark above even more idiotic.
- anigbrowl 3y agoA protocol is only as useful as the number of people that adopt it. Network effects and preferential attachment are real phenomena. I've been using Nostr almost since the beginning, so it's not that I am biased against it.
- 3y ago
- empathy_m 3y agoThe part that resonates here is saying "ah yes well we have a full database backup so we can do a full restore", then "the full restore will be tough and involve downtime and has some side effects," then "I bet we could be clever and restore only part of the data that are missing", then doing that by hand, which hits weird errors, then finally shipping the jury-rigged selective restore and cleaning up the last five missing pieces of data (hoping you didn't miss a sixth) Happens every time someone practices backup/restore no matter how hard they've worked in advance. It always ends up being an application level thing to decide what data to put back from the backup image.
- SV_BubbleTime 3y agoI agree with you. The phrase is you don’t have backups unless you test your backups. But in this case I don’t really get what the issue is. Restore everything from the last good backup and people miss some posts made in the meantime, sucks, but it’s an instant solution instead of hand work and uncertainty.
- NikkiA 3y agoWhen I worked as a VMS sysadmin full restore checks were one of the things I insisted on doing, sure, it used up a morning every couple of weeks, and tied up one of our microvaxes, but it was worth it. Especially three months after I finished being sysadmin and moved to development, and they had a disk failure. me: 'so you have backups?' the replacement: 'sure, but they didn't restore' me: 'what's the last good backup you have?' tr: 'august, the last one you did' me: 'welp' tr's boss: 'guess £390,000 for third party disk recovery is our only option...'
- NikkiA 3y agoTo add some context... Yes, it was documented in our ISO 9000 docs. But only 'strongly recommended' to perform a regular/routine test restore. I attempted to get it converted to a mandatory step, but since I was only a temporary sysadmin and an intern, it wasn't going to happen. I was told by my predecessor (who was a direct contractor to my employer) to perform it as routinely as I could. I would guess that he had attempted to get it put as a mandatory step, but his time was billed, mine wasn't, so shrug. My/the replacement was an external contractor as part of a 'company Y now provides system administration services' deal, who presumably ended up eating the liability of not having working backups that they were contracted to produce. As horrified as I was, 'it's not really my problem, I wasn't responsible' was the only attitude I could bear to take. Besides, I was busy with fortran.
- deleted 3y ago[deleted]
- INTPenis 3y agoI'll never forget the first time I had to restore a massive sql dump and realized that vim actually segfaults trying to read it. That's when I discovered the magic of spit(1) "split a file into pieces". I just split the huge dump into one file per table. Of course a table can also be massive, but at least the file is now more uniform which means you can easier run other tools on it like sed or awk to transform queries.
- Terr_ 3y agoI once had to administer a system where a particular folder had so many files that things stopped working, even the ls command would not complete. (It was probably on ext3 or ext2.) The workaround involved writing a python script that handled everything in a gradual manner, moving files into subdirectories based on shared prefixes.
- fragmede 3y agooh yes. ls uses 4k buffers for dirents, and in a directory with lots of entries, the time for userspace to hit the kernel to list the entities until that 4k buffer is full, back in the day, became noticable. In my dealings with a system like that, I had a hacked copy of ls that used bigger buffers so at least it wouldn't hang. Tab completion would also hang if there were too many entries.
- williamdclt 3y agoI'm surprised that vim segfaults! I had it slow to open huge files, but I always assumed it could handle anything, through some magic buffering mechanisms. I could be wrong! That being said, from the point that one has to edit the dump to restore data... something is very wrong in the restore process (the knowledge of which isn't helpful when you're actually faced with the situation, of course)
- INTPenis 3y agoYes you shouldn't be manually restoring sql dumps but I've been working in this field long before versioned source control or pgbackrest existed.
- chx 3y ago> And it just so happens that all local accounts in a Mastodon instance have a null value in their URI field, so they all matched. How? NULL = NULL evaluates to FALSE, SQL is a three value logic, specifically Kleene's weak three-valued logic, NULL anyoperator NULL is NULL.
- porridgeraisin 3y agoYeah, was wondering. Maybe they filter at the application level? And check equality with their language's null value?
- nateb2022 3y agoThis is the case, more or less. The fix for this issue boiled down to a one-liner: https://github.com/mastodon/mastodon/commit/13ec425b721c959415921046d7a24ed8c9994cee#diff-0bc8e960429e6c65ff6b5f31c40c67ed4cab1904dcf2dcb9f13d1503338c84ef https://github.com/mastodon/mastodon/commit/13ec425b721c9594... But basically, some object attributes (which should have been set by default) weren't set by default. This is a common oversight when dealing with data structures that are incomplete at one point or another, and it's easy to assume during programming that code will execute in a fixed order that allows for the necessary fields to be present when needed although sometimes it doesn't always work out that way. In my opinion, they were lucky to have caught this but a fix should include more than adding missing initialization. They should implement a sanity check to ensure that fields used are present and !NULL, and if things are undefined or missing for whatever reason, abort whatever process they are attempting to perform and log the issue.
- mollems 3y agoGreat writeup (including the human cost, e.g. loss / lack of sleep, which in my experience has a huge impact on complicated incident resolution). Here’s what jumped out at me: “The new account was created in our database with a null value in the URI field.” Almost every time I see a database-related postmortem — and I have seen a lot of them — NULL is lurking somewhere in the vicinity of the crime scene. Even if NULL sometimes turns out not to be the killer, it should always be brought in for questioning. My advice is: never rely on NULL as a sentinel value, and if possible, don’t allow it into the database at all. Whatever benefits you think you might gain, they will inevitably be offset by a hard-to-find bug, quite possibly years later, where some innocuous-seeming statement expects either NULL or NOT NULL and the results are unexpected (often due to drift in the semantics of the data model). Although this was a race condition, if the local accounts and the remote accounts were affirmatively distinguished by type, the order of operations may not have mattered (and the account merge code could have been narrowly scoped).
- kaoD 3y agoWhat's the alternative, an empty string? IMO the problem (at least in this case) is not NULL in the DB, but NULL at the application level. If NULL is some sort of Maybe monad and you're forced to deal with it, well, you're forced to deal with it, think about it, etc. Empty string, whatever NULL string is in your language of choice, or some sort of sigil value you invent... not much of a difference.
- Tempest1981 3y agoA separate Boolean column?
- kaoD 3y agoCan't wait to have a NULL url and has_url TRUE. Might or might not be based in production data I deal with on a daily basis.
- semiquaver 3y agoJoins are cheap. Wide tables are often a sign that a data-model is a bit too CRUDdy. Foreign key relationships often do a much better job modeling optionality/cardinality in relational systems. In this case, a `user_uris` table with non-nullable columns and a unique constraint on `user_id` is the first option that comes to mind.
- ChrisArchitect 3y agoThis is a tough and crappy situation but that was an engaging read surprisingly! Nice one
- martey 3y ago> To Renaud, Claire, and Eugen of the Mastodon developer team, who went above and beyond all expectations to help us out. You folks were amazing, you took our situation very seriously, and immediately jumped in to help us. I really could not have asked for anything more. Thank you! I don't know if Vivaldi provides financial support to Mastodon (I couldn't find their name on the sponsors page). If not, I hope this situation causes them (and other companies using Mastodon) to consider sponsorship or a support contract.
- progval 3y agoThey aren't on https://joinmastodon.org/sponsors https://joinmastodon.org/sponsors so probably not.
- kaoD 3y agoWell they provide the Mastodon federation with what seems to be a large instance and people working on it.
- renchap 3y agoWe (the Mastodon non-profit) do not offer support contracts at the moment, but this is a good idea, thanks :) But we indeed have sponsorships open, and they really have impact. Having full-time people working on the project is very impactful, but at the moment we only have 1 full-time developer in addition to Eugen (the founder) and a DevOps person on the technical side.
- notresidenter 3y ago> 6 Users with symbols in their usernames couldn’t log in. This turned out to be due to a mistake I’d made in the recovery script, and was very easily fixed. UTF-8 strikes again.
- ziml77 3y agoHow did accounts with a NULL value in the URI column match the query? NULL does not compare equal to NULL. Is this some awful Rails magic at work?
- account42 3y agoWhat kind of monster disables text selection on their blog. Absolutely unreadable.
- danillonunes 3y agoIt's not disabled, the selection color is the same as the background color.