squarecog

7 ms·

1.
S3 Is the New SFTP, or, how to get data to customers(materializedview.io)
7 points by squarecog 2y ago|0 comments
2.
Agents aren't all you need(parcha.com)
3 points by squarecog 2y ago|0 comments
3.
by squarecog 2y ago
Here are a couple of talks about orchestration and choreography, workflows vs sagas, etc. Both very technical, both with very little "sell" despite being given by a Camunda co-founder and a Temporal principal eng. https:/&#x
4.
by squarecog 3y ago
LLMs are Foundational Models, but not all Foundational Models are LLMs. If you are interested in this space, it does probably make sense to look up what Stable Diffusion (StabilityAI) and Claude (Anthropic) are. Not recognizing them is fine
5.
by squarecog 4y ago
LinkedDataHub, a "RDF-native notebook", is not to be confused with LinkedIn DataHub, which is a metadata store/crawler/ui for your data systems: https://datahubproject.io/ .
6.
by squarecog 4y ago
I should've added, there's an obvious example for the "SaaS control plane" separation, which is equivalent: "stop processing job X that is destabilizing the cluster" should be processed without needing to fight
7.
by squarecog 4y ago
These terms usually show up in the context of networking protocols. Cloudflare has a very quick explainer: https://www.cloudflare.com/learning/network-layer/what-is-th... . To make it even shorter: a control plane
8.
by squarecog 5y ago
Now? No idea. At the time I left in 2016, all Storm usage was replaced by Heron, that was the point of making it API-compatible.
9.
by squarecog 5y ago
I was in charge of the Twitter data platform team at the time we developed Heron and deprecated Storm. The Mesos component of your retelling is not quite right. Take a look at this comment I wrote around the time we started talking about He
10.
by squarecog 5y ago
Donated to Apache in 2018. https://blog.twitter.com/engineering/en_us/topics/open-sourc... (and open-sourced in 2016, having started development in 2014).
11.
Building a data team at a mid-stage startup(erikbern.com)
607 points by squarecog 5y ago|89 comments
12.
by squarecog 5y ago
Why is this odd? If he started two years later and there was not a trace of the Prize work at the company, that would be an indicator that the competition was not important. If he started and could still see knock-on effects from the compet
13.
How to Avoid Overengineering(evanm.website)
2 points by squarecog 6y ago|0 comments
14.
by squarecog 6y ago
> So basically all research projects have to redirect funds to cover part of the "open access fees" Yes, the cost of publishing results, as well as review, editing, etc, should be incorporated into the research budget (research
15.
by squarecog 6y ago
While HDFS is indeed used for exporting old data and storing some partition mapping metadata, it's clear from the blog post that MetricsDB is much more reliant on BlobStore as well as MetricsDB-specific services. > The servers check
16.
by squarecog 7y ago
I understand your architecture criticism, and think it has merit, but I'm not sure why Apache gets dragged into that. Apache Airflow is in Python. Apache Arrow is in C. CouchDB is Erlang. There's a ton of projects Apache Foundatio
17.
by squarecog 7y ago
Can you explain which part, in your opinion, is facepalm worthy? In your example conversation, the developer starts off by demonstrating that they are not good at thinking through how long things take them, by giving an impossibly optimisti
18.
by squarecog 9y ago
I helped build Twitter's data platform, 2010-2016. There isn't an "analysis server" and analyzing user activity is not done on a "user database backup" at Twitter's scale, though indeed that's a commo
19.
TigerGraph benchmarks: 4x-100x faster than Neo4J, Titan [pdf](doc.tigergraph.com)
3 points by squarecog 9y ago|0 comments
20.
by squarecog 9y ago
(ex-twitter engineer) I left before this project got started, and do not have any insider info on how they did it. Given what I know about the number of places tweet length assumptions were built into, it must have been a large, cross-team
21.
by squarecog 9y ago
hdf5, Feather, Arrow, protobufs, json, xml -- all solve the problem of binary representation of data on disk. They all leave the question of how to map said data to a specific problem domain up to the developer. Projects like ONNX define sa
22.
Houston by Turbine Labs: app routing and release infra(medium.com)
7 points by squarecog 10y ago|0 comments
23.
by squarecog 10y ago
Stopping when you hit 95% confidence is a classic failure mode. Yes, if you are doing classic t-test based A/B testing, you have to wait until a pre-determined threshold; otherwise, effectively, by looking at the p-value and stopping w
24.
by squarecog 10y ago
You may be interested in what Wes McKinney has to say on the topic: http://wesmckinney.com/blog/outlook-for-2017/ (you can read/write using pyarrow)
25.
by squarecog 10y ago
Yes: https://twitter.com/aneeshs/status/773547228694589440 "We use GraphJet for serving contextual, personalized content recommendations in real time, such as "You May Like", emails, notifications e
26.
by squarecog 10y ago
Hi, I manage the data science team at Zymergen, and I can tell you we are working on these topics (and hiring). Shoot me an email if you'd like to chat -- dmitriy at zymergen.com
27.
by squarecog 11y ago
Note how the creator of Apache Crunch is on the Dataflow committer list.
28.
by squarecog 11y ago
co-author of blog post here. We covered some of this, at a high level, here: https://blog.twitter.com/2015/the-what-and-why-of-product-ex... Certainly not everything needs to be A/B tested -- depending on the natu
29.
by squarecog 11y ago
Co-author of the Twitter post here. I don't necessarily disagree with you! Though I hope you are not calling Twitter a fly-by-night marketing startup :). As a company, we've found A/B testing very valuable, but it certainly h
30.
by squarecog 11y ago
Hi, author of the post here. A few brief points: 1. The post talks about most experiments not moving core metrics in a large way. This does not mean the experiments don't ship, it just means core metrics are very hard to move. So &quot

More ›