10 ms·

I think it'll be fine. You don't need 7000 people to run a micro blogging service. It's really that simple.

rockostrich 4y ago
I don't disagree that 7000 people is too many for what Twitter has become but Twitter has been at the bleeding edge in terms of building web and data systems that can handle scale (while also open sourcing most of that work).
duhast 4y ago
Why do you think so?
- reitanqild 4y ago
  Look at early WhatsApp until they were aquired. Vastly more innovative and scaled crazy fast without fail whales or anything. Or look to Telegram today. Delivering a vastly more complex product with a fraction of the company size it seems.
  - jameshart 4y ago
    Aren’t WhatsApp and telegram messaging services rather than microblogging services? The difference between a messaging app and a microblogging app is basically the difference between an O(n) problem and an O(n^2) problem. If you don’t think that adds some complexity, you’re kidding yourself.
    - scarmig 4y ago
      They're different problems, but messaging isn't some trivial thing. And a user having a single unified public view of their tweets is pretty much O(n).
    - reitanqild 4y ago
      I might be underestimating how hard it is to scale microblogging. I most certainly am. But have you looked at the scale of what Telegram provides both in width and at scale? Certainly there are celebrities with more followers on Twitter than the largest Telegram channels, but Telegram scales surprisingly far, and I haven't seen it struggle more than once or twice since the start.
    - Nimitz14 4y ago
      Telegram is also a microblogging service. With far more users. It's clear you are totally clueless about how many features telegram has.
      - steviesands 4y ago
        Maybe you could provide links, examples, or helpful context instead of snark?
    - vidarh 4y ago
      It adds some complexity, but consider that we know very well how to scale this type of service: E-mail + reflectors (mailing lists), and we know very well how to do parallel mass delivery for the small proportion of accounts with huge numbers of followers. Scaling this is easily done with decomposition and sharding coupled with a suitable key->value mapping of external id to current shard. I first sharded e-mail delivery and storage for millions of users 23 years ago. It was neither hard nor novel to do then, with hardware slower than my current laptop handling hundreds of thousands of users each.
      - jameshart 4y ago
        Those models are predicated on every user having an ‘inbox’ Do you believe that every Twitter user has an inbox stored on disk somewhere that just contains every tweet posted by someone they follow?
        vidarh 4y ago
        I have no idea if that is how Twitter ended up doing it. But building it that way is vastly easier to scale than trying to do some variation over joining the timelines of everyone you follow "live" on retrieval, because in models like this the volume of reads tends to massively dominate. You also don't need to store every tweet, you need to store the id's of the tweets (a KV store of the tweet id to full tweet is also easy to shard), and since they're reasonably chronological the id's can be compressed fairly efficiently (quite a few leading digits of tweet id's are chronological). You also have straightforward options for "hybrid" solutions, such as e.g. dealing with extreme outliers. Have someone followed by more than X% of total userbase? Cache the most recent N tweets from those accounts on that small set of timelinesyour frontends, and do joins over those few with users who follow them. Most importantly, it's an extensively well tested pattern in a multitude of systems with follower/following graphs whenever consumers/reads dominate over a period of decades at this point, so behaviours and failure modes are well understood with straightforward, well tested solutions for most challenges you'll run into, which matters in the context of whether it'd be possible to build with a small team. Put another way: I know from first hand experience you can scale this to millions of users per server on modern hardware, so the number of shards you'd need to be able to manage to deal with Twitter-level volume is lower than the number of servers I've had ops teams manage (you'd need more servers total, because your read load means you'd want extensive caching, as well as storage systems for e.g. images and the like - there's lots of other complexity, but scaling the core timeline functionality is not a complex problem)
  - efsavage 4y ago
    > Look at early WhatsApp until they were aquired. Look at what they didn't do, which was make any revenue, or support any customers, or moderate any content.
    - reitanqild 4y ago
      Wasn't WhatsApp on track to be cash flow positive? I know I at least was shouting at them to take my money: it was the perfect HN product, reasonably priced, technically superior and with no ads or tracking. And yes, I paid as well.
  - chillfox 4y ago
    Messaging services don’t need the same amount of moderation as micro blogging do.
  - jasmer 4y ago
    WhatsApp is not 'vastly more innovative', and they solved different kinds of problems. Twitter is a 'universe of 100M connected people'. WhatsApp mostly connected single entities together. So, for example, 'real time search' and 'relevant updates'. Imagine taking a firehose of 100M people's random thoughts, putting that into an index, making it instantly searchable. Now pull up the most relevant thoughts from those 100M to each and every other 100M user. Now moderate all of it in really subtle ways, whereupon most of the 'negative activity' is tantamount to spam or annoying behaviour, and not anything we might normally consider 'abuse'. That's an incredibly different challenge and that's only two small artifacts of what they are doing. Twitter is not rocket science, but it's not trivial either. Also consider that R&D is usually maybe on 20% of overhead - yes - it takes 'all those other jobs and expenses' to run a company. Some of these statements are a bit glib.
  - steviesands 4y ago
    Twitter has ads serving infra, recommendation systems (timeline, notifications, events, users), user generated events, prediction systems (ads), user graphs. The complexity is from processing and persisting exabytes of data in company owned datacenters. eg. Twitter stores images, videos, user events, user data, tweets/replies. WhatsApp has little persistence outside of metadata maybe? But your messages are not stored in a FB datacenter and if they are I'd be concerned. You can read about their infra in their blog. Comparing p2p messaging versus a distributed social media site with mountains of data and years of iteration in ML systems does not make sense. https://blog.twitter.com/engineering/en_us/topics/infrastructure/2017/the-infrastructure-behind-twitter-scale https://blog.twitter.com/engineering/en_us/topics/infrastruc...
deleted 4y ago
[deleted]
VerminOctopus1 4y ago
Not sure how this is relevant given that Twitter is not a “micro blogging service”
- CountSessine 4y ago
  What is it then? That seems like the best description to me.
  - travisgriggs 4y ago
    It’s a messaging service that has a broadcast feature. And for that multicast/broadcast feature you get to have a crappy peer to peer message experience, be limited to small messages, look at ads, surf through unknown algorithm manipulation of what you read, locked in a single system, and be told you might need to pay to either a) prove you’re “real” or b) not see the ads. Sign me up!! (A blog is also a broadcast messaging service, so I think you’re both right).
chasd00 4y ago
i think 7000 is too high as well but don't forget, in a business, the technology part is maybe 10% of the overall effort. There's a lot that goes into running a business beyond the tech.
- dclowd9901 4y ago
  Yes, where would businesses be without a fleet of energy sucking middle managers. I don’t like EM at all, but truth is internet companies are making so much goddamn money, they have employee pools absolutely full of turds.
falcolas 4y ago
> I think it'll be fine. You don't need 7000 people to run a micro blogging service. It's really that simple. When you're generating and distributing and moderating multiple TB of tweets in real time every day to billions of people and also feeding other corporations parts of that data... maybe you do.
- jlangenauer 4y ago
  Twitter has around 190m daily active users. You're going to struggle to run something like that with an engineering team of 20-30 people (although WhatsApp did exactly that for many years), sure. But I don't see why an company and an application of that size couldn't be run by a company of, say, 1500 people, instead of 7000. 250 engineering staff, 250 doing moderation/support, 800 doing sales & account management and 100 in management and 100 doing sundry tasks.
  - whimsicalism 4y ago
    You guys have never worked at a major social media platform and it shows. It takes around that many engineering staff just to run a passable advertising platform. There are so many technical nuances that you cannot even imagine.
    - zero_shift 4y ago
      For context, when I worked at _The Guardian_, roughly half our engineering staff worked in Commercial - doing things for advertisers
  - falcolas 4y ago
    250 people moderating AND supporting 190m daily active users?? Let's do some napkin math here: 500 million tweets, let's say 10% are reported, and evenly distributed. That would make for reviewing 200,000 tweets per employee per day. That's 7 tweets that need to be moderated per second for a standard 8 hour day. 190m active users, .1% of which need support daily. On top of 7 tweets per second, that's just shy of three user support tickets per second that they need to manage as well. And that's without weekends, holidays, sick days, etc. EDIT: Let's go the other way with napkin math too. Let's say each content moderator can review 6 tweets per minute. 50m tweets to be moderated at that velocity means you need around 136,000 man hours per day. For an 8 hour day, that's 17,000 employees. Too many. So you make an ML algorithm that processes the reported tweets, but needs backup and spot checking on 10% of those. 1,700 content moderators (and a team dedicated to building/maintaining the ML report checking algorithm). Now, about those support requests. Based on my experience from being a CSR for Amazon years ago, you'll probably want 2-3 minutes per support request, and that's if you're sending a form letter back. 1.9m requests at 20 per hour means some 95,000 man-hours per day, or 12k CSRs. Too many. Another ML algorithm (and team to maintain it) and we're down to 1,200 CSRs. About 3 thousand employees plus their support structure, just to handle tweet reports and CSR requests. Based on napkin math, 7,000 employees makes a lot of sense to me.
humanistbot 4y ago
You don't if you don't need any content moderation.
oneeyedpigeon 4y ago
It's possible you need 100 engineers and 6,900 moderators.
- idlewords 4y ago
  I believe a lot of the moderation is contracted out and isn't part of employee headcount.
deleted 4y ago
[deleted]
robotresearcher 4y ago
I've seen this statement a lot, and I think it's missing something. Public companies typically have growth as a goal. Twitter was not making a lot of money. Some of these people were working on building a future Twitter that would grow bigger and richer. I'm not saying they were on the path to success, just that I expect a lot of the activity there could be described that way. Five years ago you could say of Uber 'You don't need 5000 people to run a freelance taxi app', when they had hundreds (thousands?) of people working on autonomous driving. Amazon didn't seem to need scores of backend engineers to run a web store, but now AWS is a huge business. Google employs vast numbers of people but has a relatively small number of impactful products, only some of which make money. Again, not claiming they were doing it well, just that they were trying stuff beyond maintaining what we see.
- harlequinn77 4y ago
  To be frank, twitter hasn't changed in 10 years. So in this hypothetical, the people building the future of the company have achieved nothing during this time and should absolutely be removed or replaced.
  - robotresearcher 4y ago
    No argument there. Just pointing out that it's common for head count to be larger than the visible product justifies. Uber cut their losses. Twitter needed a change of owner to get there.

I don't disagree that 7000 people is too many for what Twitter has become but Twitter has been at the bleeding edge in terms of building web and data systems that can handle scale (while also open sourcing most of that work).

duhast 4y ago

Why do you think so?

reitanqild 4y ago
Look at early WhatsApp until they were aquired. Vastly more innovative and scaled crazy fast without fail whales or anything. Or look to Telegram today. Delivering a vastly more complex product with a fraction of the company size it seems.
- jameshart 4y ago
  Aren’t WhatsApp and telegram messaging services rather than microblogging services? The difference between a messaging app and a microblogging app is basically the difference between an O(n) problem and an O(n^2) problem. If you don’t think that adds some complexity, you’re kidding yourself.
  - scarmig 4y ago
    They're different problems, but messaging isn't some trivial thing. And a user having a single unified public view of their tweets is pretty much O(n).
  - reitanqild 4y ago
    I might be underestimating how hard it is to scale microblogging. I most certainly am. But have you looked at the scale of what Telegram provides both in width and at scale? Certainly there are celebrities with more followers on Twitter than the largest Telegram channels, but Telegram scales surprisingly far, and I haven't seen it struggle more than once or twice since the start.
  - Nimitz14 4y ago
    Telegram is also a microblogging service. With far more users. It's clear you are totally clueless about how many features telegram has.
    - steviesands 4y ago
      Maybe you could provide links, examples, or helpful context instead of snark?
  - vidarh 4y ago
    It adds some complexity, but consider that we know very well how to scale this type of service: E-mail + reflectors (mailing lists), and we know very well how to do parallel mass delivery for the small proportion of accounts with huge numbers of followers. Scaling this is easily done with decomposition and sharding coupled with a suitable key->value mapping of external id to current shard. I first sharded e-mail delivery and storage for millions of users 23 years ago. It was neither hard nor novel to do then, with hardware slower than my current laptop handling hundreds of thousands of users each.
    - jameshart 4y ago
      Those models are predicated on every user having an ‘inbox’ Do you believe that every Twitter user has an inbox stored on disk somewhere that just contains every tweet posted by someone they follow?
      - vidarh 4y ago
        I have no idea if that is how Twitter ended up doing it. But building it that way is vastly easier to scale than trying to do some variation over joining the timelines of everyone you follow "live" on retrieval, because in models like this the volume of reads tends to massively dominate. You also don't need to store every tweet, you need to store the id's of the tweets (a KV store of the tweet id to full tweet is also easy to shard), and since they're reasonably chronological the id's can be compressed fairly efficiently (quite a few leading digits of tweet id's are chronological). You also have straightforward options for "hybrid" solutions, such as e.g. dealing with extreme outliers. Have someone followed by more than X% of total userbase? Cache the most recent N tweets from those accounts on that small set of timelinesyour frontends, and do joins over those few with users who follow them. Most importantly, it's an extensively well tested pattern in a multitude of systems with follower/following graphs whenever consumers/reads dominate over a period of decades at this point, so behaviours and failure modes are well understood with straightforward, well tested solutions for most challenges you'll run into, which matters in the context of whether it'd be possible to build with a small team. Put another way: I know from first hand experience you can scale this to millions of users per server on modern hardware, so the number of shards you'd need to be able to manage to deal with Twitter-level volume is lower than the number of servers I've had ops teams manage (you'd need more servers total, because your read load means you'd want extensive caching, as well as storage systems for e.g. images and the like - there's lots of other complexity, but scaling the core timeline functionality is not a complex problem)
- efsavage 4y ago
  > Look at early WhatsApp until they were aquired. Look at what they didn't do, which was make any revenue, or support any customers, or moderate any content.
  - reitanqild 4y ago
    Wasn't WhatsApp on track to be cash flow positive? I know I at least was shouting at them to take my money: it was the perfect HN product, reasonably priced, technically superior and with no ads or tracking. And yes, I paid as well.
- chillfox 4y ago
  Messaging services don’t need the same amount of moderation as micro blogging do.
- jasmer 4y ago
  WhatsApp is not 'vastly more innovative', and they solved different kinds of problems. Twitter is a 'universe of 100M connected people'. WhatsApp mostly connected single entities together. So, for example, 'real time search' and 'relevant updates'. Imagine taking a firehose of 100M people's random thoughts, putting that into an index, making it instantly searchable. Now pull up the most relevant thoughts from those 100M to each and every other 100M user. Now moderate all of it in really subtle ways, whereupon most of the 'negative activity' is tantamount to spam or annoying behaviour, and not anything we might normally consider 'abuse'. That's an incredibly different challenge and that's only two small artifacts of what they are doing. Twitter is not rocket science, but it's not trivial either. Also consider that R&D is usually maybe on 20% of overhead - yes - it takes 'all those other jobs and expenses' to run a company. Some of these statements are a bit glib.
- steviesands 4y ago
  Twitter has ads serving infra, recommendation systems (timeline, notifications, events, users), user generated events, prediction systems (ads), user graphs. The complexity is from processing and persisting exabytes of data in company owned datacenters. eg. Twitter stores images, videos, user events, user data, tweets/replies. WhatsApp has little persistence outside of metadata maybe? But your messages are not stored in a FB datacenter and if they are I'd be concerned. You can read about their infra in their blog. Comparing p2p messaging versus a distributed social media site with mountains of data and years of iteration in ML systems does not make sense. https://blog.twitter.com/engineering/en_us/topics/infrastructure/2017/the-infrastructure-behind-twitter-scale https://blog.twitter.com/engineering/en_us/topics/infrastruc...

deleted 4y ago

[deleted]

VerminOctopus1 4y ago

Not sure how this is relevant given that Twitter is not a “micro blogging service”

CountSessine 4y ago
What is it then? That seems like the best description to me.
- travisgriggs 4y ago
  It’s a messaging service that has a broadcast feature. And for that multicast/broadcast feature you get to have a crappy peer to peer message experience, be limited to small messages, look at ads, surf through unknown algorithm manipulation of what you read, locked in a single system, and be told you might need to pay to either a) prove you’re “real” or b) not see the ads. Sign me up!! (A blog is also a broadcast messaging service, so I think you’re both right).

chasd00 4y ago

i think 7000 is too high as well but don't forget, in a business, the technology part is maybe 10% of the overall effort. There's a lot that goes into running a business beyond the tech.

dclowd9901 4y ago
Yes, where would businesses be without a fleet of energy sucking middle managers. I don’t like EM at all, but truth is internet companies are making so much goddamn money, they have employee pools absolutely full of turds.

falcolas 4y ago

> I think it'll be fine. You don't need 7000 people to run a micro blogging service. It's really that simple. When you're generating and distributing and moderating multiple TB of tweets in real time every day to billions of people and also feeding other corporations parts of that data... maybe you do.

jlangenauer 4y ago
Twitter has around 190m daily active users. You're going to struggle to run something like that with an engineering team of 20-30 people (although WhatsApp did exactly that for many years), sure. But I don't see why an company and an application of that size couldn't be run by a company of, say, 1500 people, instead of 7000. 250 engineering staff, 250 doing moderation/support, 800 doing sales & account management and 100 in management and 100 doing sundry tasks.
- whimsicalism 4y ago
  You guys have never worked at a major social media platform and it shows. It takes around that many engineering staff just to run a passable advertising platform. There are so many technical nuances that you cannot even imagine.
  - zero_shift 4y ago
    For context, when I worked at _The Guardian_, roughly half our engineering staff worked in Commercial - doing things for advertisers
- falcolas 4y ago
  250 people moderating AND supporting 190m daily active users?? Let's do some napkin math here: 500 million tweets, let's say 10% are reported, and evenly distributed. That would make for reviewing 200,000 tweets per employee per day. That's 7 tweets that need to be moderated per second for a standard 8 hour day. 190m active users, .1% of which need support daily. On top of 7 tweets per second, that's just shy of three user support tickets per second that they need to manage as well. And that's without weekends, holidays, sick days, etc. EDIT: Let's go the other way with napkin math too. Let's say each content moderator can review 6 tweets per minute. 50m tweets to be moderated at that velocity means you need around 136,000 man hours per day. For an 8 hour day, that's 17,000 employees. Too many. So you make an ML algorithm that processes the reported tweets, but needs backup and spot checking on 10% of those. 1,700 content moderators (and a team dedicated to building/maintaining the ML report checking algorithm). Now, about those support requests. Based on my experience from being a CSR for Amazon years ago, you'll probably want 2-3 minutes per support request, and that's if you're sending a form letter back. 1.9m requests at 20 per hour means some 95,000 man-hours per day, or 12k CSRs. Too many. Another ML algorithm (and team to maintain it) and we're down to 1,200 CSRs. About 3 thousand employees plus their support structure, just to handle tweet reports and CSR requests. Based on napkin math, 7,000 employees makes a lot of sense to me.

humanistbot 4y ago

You don't if you don't need any content moderation.

oneeyedpigeon 4y ago

It's possible you need 100 engineers and 6,900 moderators.

idlewords 4y ago
I believe a lot of the moderation is contracted out and isn't part of employee headcount.

deleted 4y ago

[deleted]

robotresearcher 4y ago

I've seen this statement a lot, and I think it's missing something. Public companies typically have growth as a goal. Twitter was not making a lot of money. Some of these people were working on building a future Twitter that would grow bigger and richer. I'm not saying they were on the path to success, just that I expect a lot of the activity there could be described that way. Five years ago you could say of Uber 'You don't need 5000 people to run a freelance taxi app', when they had hundreds (thousands?) of people working on autonomous driving. Amazon didn't seem to need scores of backend engineers to run a web store, but now AWS is a huge business. Google employs vast numbers of people but has a relatively small number of impactful products, only some of which make money. Again, not claiming they were doing it well, just that they were trying stuff beyond maintaining what we see.

harlequinn77 4y ago
To be frank, twitter hasn't changed in 10 years. So in this hypothetical, the people building the future of the company have achieved nothing during this time and should absolutely be removed or replaced.
- robotresearcher 4y ago
  No argument there. Just pointing out that it's common for head count to be larger than the visible product justifies. Uber cut their losses. Twitter needed a change of owner to get there.