binarylogic

9 ms·

1.
by binarylogic 8mo ago
I agree with the framing. The goal isn't less data for its own sake. The goal is understanding your systems and being able to debug when things break. But here's the thing: most teams aren't drowning in data because they&#x27
2.
by binarylogic 8mo ago
Agree to an extent. There are absolutely unknown unknowns. But I think you'd be surprised how much data is obviously waste. Not the grey area, just pure garbage: health checks, debug logs left in production, redundant attributes. That&
3.
by binarylogic 8mo ago
Thank you! And you're right, it shouldn't cost that much. Financials are public for many of these vendors: 80%+ margins. The cost to value ratio has gotten way out of whack. But even if storage were free, there's still a sign
4.
by binarylogic 8mo ago
Thanks for the comment! Yes, I read that post. Great post. Feel free to reach out if you ever need help with Vector or have questions.
5.
by binarylogic 8mo ago
What you're describing is very real and it works to a degree. I've seen this same manual maintenance play out over and over for 10 years: cleaning dashboards, chasing engineers to align on schemas, running cost exercises. It never
6.
by binarylogic 8mo ago
You're right, it's not always binary. That's why we broke it down into categories: https://docs.usetero.com/data-quality/logs/malformed-data You'd be shocked how much obviously-safe waste (redu
7.
by binarylogic 8mo ago
The question is answered in the post: ~40% on average, sometimes higher. That's a real number from real customer data. But I'm an engineer at heart. I wanted this post to shed light on a real problem I've seen over a decade i
8.
by binarylogic 8mo ago
My apologies, I fixed the link. So much for restructuring the docs the night before posting this. You can read more here: https://docs.usetero.com/data-quality/overview To loosely describe our approach: it's inten
9.
by binarylogic 8mo ago
Yeah, it's funny, I never went down the regex rabbit hole until this, but I was blown away by Hyperscan/Vectorscan. It truly changes the game. Traditional wisdom tells you regex is slow. > I'm surprised it's only 40%.
10.
by binarylogic 8mo ago
100% accurate. It is very much political. I'd also add that the problem is perpetuated by a disconnection between engineers who produce the data and those who are responsible for paying for it. This is somewhat intentional and exploite
11.
by binarylogic 8mo ago
Thank you for the nice comment. I'm glad you enjoy Vector. I poured myself into that software for many years. I'm a bit bummed with its current trajectory, though. We hope to bring the next evolution with Tero. There were many pro
12.
by binarylogic 8mo ago
Hey Peter, I absolutely remember you! Thanks for the nice comment. And yes, data waste in this space is absurdly bad. I don't think people realize how bad it actually is. I estimate ~40% of the data (being conservative) is waste. But n
13.
How much of my observability data is waste?(usetero.com)
120 points by binarylogic 8mo ago|59 comments
14.
by binarylogic 8mo ago
I spent a decade in observability. Built Vector, spent three years at Datadog. This is what I think is broken with observability and why.
15.
Datadog, thank you for blocking us(deductive.ai)
77 points by binarylogic 8mo ago|30 comments
16.
by binarylogic 6y ago
Yeah, we're trying to solve for this over at https://vector.dev . We intend to decouple the pipeline from the method. As demonstrated by OT, this stuff changes, and you shouldn't have to rip out your plumbing every time
17.
Vector 0.9.0 Released – A lightweight and ultra-fast observability data router(vector.dev)
3 points by binarylogic 6y ago|0 comments
18.
by binarylogic 7y ago
Hey Lemmox, good questions: 1) Not currently, but it's certainly possible. We don't foresee any major fundamental changes. The most likely change is around transforming. Specifically, how you shape both log and metrics data struct
19.
by binarylogic 7y ago
I'm biased because my team and I created Vector [0], but I'd highly recommend investing in a vendor-agnostic data collector to start. You can use this to collect your data and send it wherever you please. This will afford you the
20.
by binarylogic 7y ago
Ah, I'm amazed that I did not know this. I just updated everything to use MiB. Thanks for pointing that out.
21.
by binarylogic 7y ago
They do, we actually put together a correctness test for this behavior: https://github.com/timberio/vector-test-harness/tree/master/... Logstash is not graceful. Our testing shows that they basically shu
22.
by binarylogic 7y ago
Absolutely, this is likely the next integration we'll be working on. There were a few features schema-wise that we needed support before we started, but we're _very_ close. We'd love beta testers to help us build it out. Feel
23.
by binarylogic 7y ago
Telegraf is nicely done. We spent a lot of time testing solutions in our test harness ( https://github.com/timberio/vector-test-harness ) and Telegraf was the most impressive of the tools we tested, so kudos to the Influ
24.
by binarylogic 7y ago
It's very likely we're doing something wrong with this test, but after many hours of trying we couldn't get our simple test to pass for Logstash, even though it passed for others: https://github.com/timberio&#
25.
by binarylogic 18y ago
This discussion is moot since Authlogic easily supports BCrypt: http://www.binarylogic.com/2008/11/22/storing-nuclear-launch...
26.
by binarylogic 18y ago
My comment about "anything not being used by the NSA" is not silly, it's true. Hell the NSA developed the Sha functions. I am not one to take security lightly, and salted Sha256 is not taking security lightly. I am not sure what kind of app
27.
by binarylogic 18y ago
Also, thanks to github, if a plugin is popular enough it never really gets abandoned. Look at the engines plugin. The author of that plugin rarely does anything to it, but people fork it and keep it up-to-date.
28.
by binarylogic 18y ago
Awesome, you shouldn't even consider using a plugin / gem unless it has a good name. The same applies with books. Unless you like title / cover you shoudn't read it.
29.
by binarylogic 18y ago
I am the creator of this plugin. Who cares about the name. I wanted something unique that would get people's attention and return unique content when searching. I also thought it was funny. People need to lighten up a little bit if the name