points by mschuster91 12 hours ago | 0 comments
When the threat model is "a fire takes out the DC", sure. But "the servers are root-level compromised", whoops, that's worth rebuilding from scratch. No way to make sure whatever AI agent did the compromise didn't go and embed malware in some flash ROM.
[–] rafram 12 hours ago
[–] mschuster91 12 hours ago
There aren't that many scenarios which make a restoration time of weeks inevitable, even after taking into account the AO3 people are volunteers - given the large intersection between nerds and fanfiction or a loooot of furries working in IT, I'd assume you'll find the most competent people working on it, so I doubt the cause is incompetence.
They say the data is safe, which implies an offsite backup with a different architecture (or a managed provider). They said in the past they had a secondary site which they could switch over to if needed, which implies that the issue is not related to hardware or datacenter problems (e.g. your usual backhoe related issue or a physical destruction of the datacenter).
And that leaves only two options: a) a former staff member went fully rogue and everything needs to be rebuilt because everything might be backdoored, but there is no indication of that, or b) they got compromised really hard by one of the multitude of the vulnerabilities that cropped up the last days and chose to rebuild from scratch for a similar reason.
(And as far as I know AO3, like FFN, is limited to text and stuff like cover art, so the dataset should be small and easy enough to restore from a backup)