8 ms·

Scribe: Transporting petabytes per hour via a distributed, buffered queueing

124 points by gluegadget 7y ago

londons_explore 7y ago
Keep in mind the network cost of petabytes per hour cross continent. Those of us who don't own our own cross-ocean fiber can't afford to design systems like this.
- gnulinux 7y ago
  Sounds like a startup idea.
  - liveoneggs 7y ago
    Level 3 and Vodafone have a bit of a first mover advantage here
- carapace 7y ago
  Never underestimate the bandwidth of a cargo ship full of SSDs. (I'm paraphrasing an old, old joke.)
  - noir_lord 7y ago
    Fairly high latency though I guess.
    - trhway 7y ago
      using jet for trans ocean data delivery gives you several hours latency - acceptable for the logs - at the cost on the scale like $0.1-0.3/TB (really depends on the napkin used for the estimations)
      - derefr 7y ago
        Big capital costs in setting up the number of parallel writers required on one end and readers required on the other, though. And presumably human "IT teamster" labor, hooking and unhooking drives.
100k 7y ago
Naming is hard! Facebook used to have a _different_ Scribe (https://en.wikipedia.org/wiki/Scribe_(log_server) https://en.wikipedia.org/wiki/Scribe_(log_server)). We used it at a company I worked for, but it had long-since been deprecated, so I was confused when I saw this Scribe.
- johndoe345 7y ago
  This is the same scribe. Facebook closed-sourced it because it was too hard to maintain an open version and a version that addresses Facebook's needs.
  - naringas 7y ago
    I wonder why couldn't they just make the open source version address their needs...
    - javiermaestro 7y ago
      The version that was open-sourced kept evolving and integrating with other internal systems at Facebook. That's what made it hard to continue open-sourcing it (why the open-sourced version was discontinued) and why the current version is also hard to open-source. Maybe one day we'll have a version available. In any case, one of the larger parts of the system (LogDevice) is open source :) (disclaimer: I work in Scribe) LogDevice: https://engineering.fb.com/core-data/open-sourcing-logdevice-a-distributed-data-store-for-sequential-data/ https://engineering.fb.com/core-data/open-sourcing-logdevice...
- gnud 7y ago
  Scribe is also an enterprise integration/ETL tool. For _many_ years before FB existed.
  - javiermaestro 7y ago
    Interesting! Just found out about this, and I googled it out. TIBCO acquired Scribe Software, and Scribe Software was running as early as 1998-1999! :D https://web.archive.org/web/20120301000000*/http://www.scribesoft.com https://web.archive.org/web/20120301000000*/http://www.scrib... The initial stuff seems to not be so related, but the current description of what they do seems much much closer to what Facebook's Scribe does today :) Naming is hard! :D
dekhn 7y ago
braggy PR is misleading: the 25GB/s coming from CERN is after they filter the data down from 600TB/s because there are no commercial systems that can capture data at higher rates.
- gnufx 7y ago
  I'm not sure what that means, but the Cori filesystem is rated at 700GB/s and Summit's 2.5TB/s. See https://docs.nersc.gov/filesystems/cori-scratch/ https://docs.nersc.gov/filesystems/cori-scratch/ and https://www.olcf.ornl.gov/olcf-resources/compute-systems/summit/summit-faqs/ https://www.olcf.ornl.gov/olcf-resources/compute-systems/sum...
  - dekhn 7y ago
    it's pretty simple. the physical data acquisition devices (ATLAS is an example) collect data at rates in the 100s of terabytes/sec https://home.cern/science/computing/processing-what-record https://home.cern/science/computing/processing-what-record) No storage system can store that data (and most of it is not useful) so they have a series of hardware triggers and buffers that reduce the data down to roughly what modern (general purpose) hardware is capable of handling. They tune the thresholds to match what consumer hardware is capable of. With regard to supercomputer filesystems: nobody wants to use GPFS. CERN's EOS sustained (theoretical) 3.3TB/sec in Apr 2015, so it's not like they're uncompetitive with the largest supercomputer...
    - gnufx 7y ago
      I know how data collection works, but it sounded as if 25GB/s was regarded as high compared with filesystems you can buy. Obviously some people do want GPFS, if they can afford it, but Cori uses Lustre. I don't mean to claim that either is ideal for streaming high rate event data, of course.
      - adev_ 7y ago
        > Obviously some people do want GPFS, if they can afford it, but Cori uses Lustre Data model at CERN does not match the one of a supercomputer. CERN data are not processed locally but distributed and spread to ~100 of participating institute in the experiment. Moreover, "personal opinion", GPFS is crap. It's an old relic from the 90s that has so many quirk and problem of design that it would deserves an entire conference on it. Plus the fact it's proprietary and expensive. The only reason that make GPFS still alive is that for a long time, the only alternative was Lustre, and Lustre is even worst.
ninju 7y ago
How does this compare to a robust Splunk infrastructure?
- deleted 7y ago
  [deleted]
- packetslave 7y ago
  For what you'd pay for a Splunk license that can handle petabytes/hour of data, it would probably be cheaper to just buy Facebook and use Scribe. :-)
dividuum 7y ago
Is it normal for these internal system to not implement any kind of access control? From the post it seems every reader can access every stream?
- sjg007 7y ago
  Aka not a product requirement... Users are trusted etc... Works great until it doesn't..
- javiermaestro 7y ago
  That's actually not the case, there's access control :) The article just focuses on certain areas of the system and doesn't go into the security and privacy parts, that's all. (I work in Scribe)
bradhe 7y ago
So, if I'm reading this correctly, 2.5GB/s of log data being generated? If we assume (aggressively) that they have 5mil machines in their infrastructure, doesn't that mean that each machine would have to be generating 500kB/s of log data? Despite that, I find the claims to be underwhelming. So your system can process massive amounts of data by scaling massively horizontally...neat.
- javiermaestro 7y ago
  The number in the article is 2.5 TB/s, not GB/s :) (disclaimer: I work in Scribe)
  - bradhe 7y ago
    Right—sorry. But point still stands. Under what circumstances was that much data being generated from (what I’m assuming is) normal logging?
    - javiermaestro 7y ago
      I'm not following. I understood from your first comment that you think the amount of data is low ("underwhelming") and from your last comment that it's a lot ("that much data"). In any case, the data is "whatever needs to be logged". And it's not "server logs", which is what I'm interpreting from your comment. Scribe transports most data at Facebook to be processed by real-time systems (e.g. Puma, Scuba) and also "batch systems" (data warehouse). So, it's quite a lot, being "the ingestion pipe" for Facebook. Does this answer your question? :-? Puma: https://research.fb.com/publications/realtime-data-processing-at-facebook/ https://research.fb.com/publications/realtime-data-processin... Scuba: https://research.fb.com/publications/scuba-diving-into-data-at-facebook/ https://research.fb.com/publications/scuba-diving-into-data-...
      - bradhe 7y ago
        > So, it's quite a lot, being "the ingestion pipe" for Facebook. I see. I walked away from the article with the impression that it was meant to be a log aggregation service a la flume, splunk, or logstash. > the amount of data is low ("underwhelming") and from your last comment that it's a lot ("that much data"). I was remarking on the numbers in regard to generation, not consumption. Based on the article, my estimate is pointing out that generating 2.5TB/s of transactional logs and telemetry data using "millions" of machines would be technically possible but not reasonably practical...and thus likely not real ;). But, you corrected my understanding: That number isn't based on a different use case.
pvlak 7y ago
why can't Producer/Scribed write directly to LogDevice storage. Any reasons for routing through WriteService.
- thetrooperer 7y ago
  That's a good question. There are multiple reasons for this. I'll briefly mention two of them. One is the high fan in ratio - millions of machines are writing relatively small blobs of data, so the middle layer serves as an aggregator (which saves backend's IOPs, number of connections, etc). Another reason is the volume of metadata - it would be inefficient to keep all the LogDevice-level metadata on each of the producer hosts.
  - pvlak 7y ago
    Will the WriteService(Aggregator) make sense for environments having thousands of machines(not in millions) and they are all within the DataCenter. In our company, we are moving away from this design of having aggregators, to directly writing to Storage whereever possible, as it reduces the message loss. On the volume of metadata held by Producers, will there be any significant difference between holding WriteService & LogDevice meta.
    - thetrooperer 7y ago
      The devil is in the details probably, but if you have a single datacenter and all writes are coming from thousands of machines ("edge"), yes, it may make more sense to set up a single LogDevice / Kafka cluster and have all the edge hosts write to it directly.

124 points by gluegadget 7y ago

Keep in mind the network cost of petabytes per hour cross continent. Those of us who don't own our own cross-ocean fiber can't afford to design systems like this.

gnulinux 7y ago
Sounds like a startup idea.
- liveoneggs 7y ago
  Level 3 and Vodafone have a bit of a first mover advantage here
carapace 7y ago
Never underestimate the bandwidth of a cargo ship full of SSDs. (I'm paraphrasing an old, old joke.)
- noir_lord 7y ago
  Fairly high latency though I guess.
  - trhway 7y ago
    using jet for trans ocean data delivery gives you several hours latency - acceptable for the logs - at the cost on the scale like $0.1-0.3/TB (really depends on the napkin used for the estimations)
    - derefr 7y ago
      Big capital costs in setting up the number of parallel writers required on one end and readers required on the other, though. And presumably human "IT teamster" labor, hooking and unhooking drives.

100k 7y ago

Naming is hard! Facebook used to have a _different_ Scribe (https://en.wikipedia.org/wiki/Scribe_(log_server) https://en.wikipedia.org/wiki/Scribe_(log_server)). We used it at a company I worked for, but it had long-since been deprecated, so I was confused when I saw this Scribe.

johndoe345 7y ago
This is the same scribe. Facebook closed-sourced it because it was too hard to maintain an open version and a version that addresses Facebook's needs.
- naringas 7y ago
  I wonder why couldn't they just make the open source version address their needs...
  - javiermaestro 7y ago
    The version that was open-sourced kept evolving and integrating with other internal systems at Facebook. That's what made it hard to continue open-sourcing it (why the open-sourced version was discontinued) and why the current version is also hard to open-source. Maybe one day we'll have a version available. In any case, one of the larger parts of the system (LogDevice) is open source :) (disclaimer: I work in Scribe) LogDevice: https://engineering.fb.com/core-data/open-sourcing-logdevice-a-distributed-data-store-for-sequential-data/ https://engineering.fb.com/core-data/open-sourcing-logdevice...
gnud 7y ago
Scribe is also an enterprise integration/ETL tool. For _many_ years before FB existed.
- javiermaestro 7y ago
  Interesting! Just found out about this, and I googled it out. TIBCO acquired Scribe Software, and Scribe Software was running as early as 1998-1999! :D https://web.archive.org/web/20120301000000*/http://www.scribesoft.com https://web.archive.org/web/20120301000000*/http://www.scrib... The initial stuff seems to not be so related, but the current description of what they do seems much much closer to what Facebook's Scribe does today :) Naming is hard! :D

dekhn 7y ago

braggy PR is misleading: the 25GB/s coming from CERN is after they filter the data down from 600TB/s because there are no commercial systems that can capture data at higher rates.

gnufx 7y ago
I'm not sure what that means, but the Cori filesystem is rated at 700GB/s and Summit's 2.5TB/s. See https://docs.nersc.gov/filesystems/cori-scratch/ https://docs.nersc.gov/filesystems/cori-scratch/ and https://www.olcf.ornl.gov/olcf-resources/compute-systems/summit/summit-faqs/ https://www.olcf.ornl.gov/olcf-resources/compute-systems/sum...
- dekhn 7y ago
  it's pretty simple. the physical data acquisition devices (ATLAS is an example) collect data at rates in the 100s of terabytes/sec https://home.cern/science/computing/processing-what-record https://home.cern/science/computing/processing-what-record) No storage system can store that data (and most of it is not useful) so they have a series of hardware triggers and buffers that reduce the data down to roughly what modern (general purpose) hardware is capable of handling. They tune the thresholds to match what consumer hardware is capable of. With regard to supercomputer filesystems: nobody wants to use GPFS. CERN's EOS sustained (theoretical) 3.3TB/sec in Apr 2015, so it's not like they're uncompetitive with the largest supercomputer...
  - gnufx 7y ago
    I know how data collection works, but it sounded as if 25GB/s was regarded as high compared with filesystems you can buy. Obviously some people do want GPFS, if they can afford it, but Cori uses Lustre. I don't mean to claim that either is ideal for streaming high rate event data, of course.
    - adev_ 7y ago
      > Obviously some people do want GPFS, if they can afford it, but Cori uses Lustre Data model at CERN does not match the one of a supercomputer. CERN data are not processed locally but distributed and spread to ~100 of participating institute in the experiment. Moreover, "personal opinion", GPFS is crap. It's an old relic from the 90s that has so many quirk and problem of design that it would deserves an entire conference on it. Plus the fact it's proprietary and expensive. The only reason that make GPFS still alive is that for a long time, the only alternative was Lustre, and Lustre is even worst.

ninju 7y ago

How does this compare to a robust Splunk infrastructure?

deleted 7y ago
[deleted]
packetslave 7y ago
For what you'd pay for a Splunk license that can handle petabytes/hour of data, it would probably be cheaper to just buy Facebook and use Scribe. :-)

dividuum 7y ago

Is it normal for these internal system to not implement any kind of access control? From the post it seems every reader can access every stream?

sjg007 7y ago
Aka not a product requirement... Users are trusted etc... Works great until it doesn't..
javiermaestro 7y ago
That's actually not the case, there's access control :) The article just focuses on certain areas of the system and doesn't go into the security and privacy parts, that's all. (I work in Scribe)

bradhe 7y ago

So, if I'm reading this correctly, 2.5GB/s of log data being generated? If we assume (aggressively) that they have 5mil machines in their infrastructure, doesn't that mean that each machine would have to be generating 500kB/s of log data? Despite that, I find the claims to be underwhelming. So your system can process massive amounts of data by scaling massively horizontally...neat.

javiermaestro 7y ago
The number in the article is 2.5 TB/s, not GB/s :) (disclaimer: I work in Scribe)
- bradhe 7y ago
  Right—sorry. But point still stands. Under what circumstances was that much data being generated from (what I’m assuming is) normal logging?
  - javiermaestro 7y ago
    I'm not following. I understood from your first comment that you think the amount of data is low ("underwhelming") and from your last comment that it's a lot ("that much data"). In any case, the data is "whatever needs to be logged". And it's not "server logs", which is what I'm interpreting from your comment. Scribe transports most data at Facebook to be processed by real-time systems (e.g. Puma, Scuba) and also "batch systems" (data warehouse). So, it's quite a lot, being "the ingestion pipe" for Facebook. Does this answer your question? :-? Puma: https://research.fb.com/publications/realtime-data-processing-at-facebook/ https://research.fb.com/publications/realtime-data-processin... Scuba: https://research.fb.com/publications/scuba-diving-into-data-at-facebook/ https://research.fb.com/publications/scuba-diving-into-data-...
    - bradhe 7y ago
      > So, it's quite a lot, being "the ingestion pipe" for Facebook. I see. I walked away from the article with the impression that it was meant to be a log aggregation service a la flume, splunk, or logstash. > the amount of data is low ("underwhelming") and from your last comment that it's a lot ("that much data"). I was remarking on the numbers in regard to generation, not consumption. Based on the article, my estimate is pointing out that generating 2.5TB/s of transactional logs and telemetry data using "millions" of machines would be technically possible but not reasonably practical...and thus likely not real ;). But, you corrected my understanding: That number isn't based on a different use case.

pvlak 7y ago

why can't Producer/Scribed write directly to LogDevice storage. Any reasons for routing through WriteService.

thetrooperer 7y ago
That's a good question. There are multiple reasons for this. I'll briefly mention two of them. One is the high fan in ratio - millions of machines are writing relatively small blobs of data, so the middle layer serves as an aggregator (which saves backend's IOPs, number of connections, etc). Another reason is the volume of metadata - it would be inefficient to keep all the LogDevice-level metadata on each of the producer hosts.
- pvlak 7y ago
  Will the WriteService(Aggregator) make sense for environments having thousands of machines(not in millions) and they are all within the DataCenter. In our company, we are moving away from this design of having aggregators, to directly writing to Storage whereever possible, as it reduces the message loss. On the volume of metadata held by Producers, will there be any significant difference between holding WriteService & LogDevice meta.
  - thetrooperer 7y ago
    The devil is in the details probably, but if you have a single datacenter and all writes are coming from thousands of machines ("edge"), yes, it may make more sense to set up a single LogDevice / Kafka cluster and have all the edge hosts write to it directly.