8 ms·

Not all high-traffic websites are equal. Youtube has way more visitors than Twitter, but is a far simpler site to scale. Twitter is a particularly difficult p

Not all high-traffic websites are equal. Youtube has way more visitors than Twitter, but is a far simpler site to scale. Twitter is a particularly difficult problem, because unlike most high-traffic sites the number of writes is pretty close to the number of reads. The latency expected by users is also very low - people are carrying on conversations over it after all, and these two factors combined mean that there's very little that can be done in the way of caching. Add this the complex relationships between all the different users and you've got a difficult site to scale to the sort of volumes they're experiencing. If the guys at Twitter want to offer me a job, I've got a few ideas though ;)

Retric 18y ago
IMO, it's not a hard problem, they a just incompetent. Edit: For more constructive criticism: 1) Don't require a DATABASE as part of a real time system. 2) If it's not in RAM your doing it wrong. PS: I have worked on a real time messaging system with million+ updates per day that used SMS for sending and receiving data. There where times that parts of the system where down, but I never had total failure for the 3.5 years I was there. Granted individual cell phone networks tended to crash and burn, but that was not really our issue.
gaius 18y ago
the number of writes is pretty close to the number of reads This is only a "problem" if your database is MySQL. People have been doing this kind of workload on the major commercial database for decades.
moe 18y ago
Twitter is a particularly difficult problem Excuse me? Twitter, at its core, is a large-scale pub/sub system with a bunch of frontends. Such systems are well understood and, frankly, quite trivial as far as the non-existant constraints of the twitter-app are concerned. After all no ordering guarantees or fancy routing are needed and obviously they don't give a damn about fault tolerance or latency either. Their constant failure to keep that thing online just screams incompetency very loudly... There are plenty of mature building blocks available in the OSS world to implement such a system. RabbitMQ and the spread toolkit come to mind. Or, if you have the dough, you could buy a shrinkwrapped solution from TIBCO, Solace or the like. The latter will even come with fancy functionality and availability guarantees in writing because those are normally deployed in "mission critical" industrial systems, such as logistics, banks, stock exchanges. Now, twitter is not SWIFT and doesn't need five nines of uptime. But saying that this is a "particularly difficult problem" is, sorry, ridiculous.
- gaius 18y ago
  There is an old saying, those who were asleep in CS 101 are doomed to be mocked on the Internet forever :-)

Retric 18y ago

IMO, it's not a hard problem, they a just incompetent. Edit: For more constructive criticism: 1) Don't require a DATABASE as part of a real time system. 2) If it's not in RAM your doing it wrong. PS: I have worked on a real time messaging system with million+ updates per day that used SMS for sending and receiving data. There where times that parts of the system where down, but I never had total failure for the 3.5 years I was there. Granted individual cell phone networks tended to crash and burn, but that was not really our issue.

gaius 18y ago

the number of writes is pretty close to the number of reads This is only a "problem" if your database is MySQL. People have been doing this kind of workload on the major commercial database for decades.

moe 18y ago

Twitter is a particularly difficult problem Excuse me? Twitter, at its core, is a large-scale pub/sub system with a bunch of frontends. Such systems are well understood and, frankly, quite trivial as far as the non-existant constraints of the twitter-app are concerned. After all no ordering guarantees or fancy routing are needed and obviously they don't give a damn about fault tolerance or latency either. Their constant failure to keep that thing online just screams incompetency very loudly... There are plenty of mature building blocks available in the OSS world to implement such a system. RabbitMQ and the spread toolkit come to mind. Or, if you have the dough, you could buy a shrinkwrapped solution from TIBCO, Solace or the like. The latter will even come with fancy functionality and availability guarantees in writing because those are normally deployed in "mission critical" industrial systems, such as logistics, banks, stock exchanges. Now, twitter is not SWIFT and doesn't need five nines of uptime. But saying that this is a "particularly difficult problem" is, sorry, ridiculous.

gaius 18y ago
There is an old saying, those who were asleep in CS 101 are doomed to be mocked on the Internet forever :-)