dsies

8 ms·

1.
by dsies 3y ago
Thanks for reading the post/article :) I did not see any point to listing out the cons as others have already done it a bunch of times. I was trying to focus on what moving to a monorepo solved for us (which was optics and poor ux).
2.
by dsies 3y ago
I was answering your request for a source. The linked article talks about identification numbers that can be used to link a person. I am not a lawyer but the article specifically refers to one person. By that logic, if the hash you genera
3.
by dsies 3y ago
https://gdpr-info.eu/art-4-gdpr/ paragraph 1: > ‘personal data’ means any information relating to an identified or identifiable natural person (‘data subject’); an identifiable natural person is one who can be iden
4.
by dsies 3y ago
Thanks Ivan - really appreciate the kind words and support!
5.
by dsies 3y ago
Re: docs - oops. We were frantically putting stuff together and linking before the docs were in that location - the link is supposed to be https://docs.streamdal.com/en/core-components/sdk/ . Fixed it in the
6.
by dsies 3y ago
Hmm good idea about the spark integration - integration is possible with basically anything that you’ve got code-level access to. I don’t know about messaging though - runtime data transformations for spark? I guess data folks would have no
7.
by dsies 3y ago
This is solid - thank you very much. We will do some more research but basically sounds like - go as low as possible, as long as the underlying libs support it. And re: kotlin - I last worked/played with it in 2016 and recall that it w
8.
by dsies 3y ago
I know you kid - but the _data_ in this context is the data that the app is processing at runtime. Ie. If the app is reading from a DB - that’s what we are tailing. And if that’s already clear - sorry :)
9.
by dsies 3y ago
I wonder if we screwed up by calling out “tail” - it is so much more then that - it executes wasm rules on the client that are pushed to it by the server AND because we have access to the data - we can expose a UI to see it flowing… like a
10.
by dsies 3y ago
Hey there - we have documented the tech stack here: https://docs.streamdal.com/en/resources-support/open-source/ Tldr: go, grpc, Protobuf, wasm, deno, reactflow, ts And yep, you’re right - we are using protob
11.
by dsies 3y ago
Good news then :) Everything stays on your network. Actually, in most situations - everything stays completely client-side. Because the rules that the client executes are Wasm modules, all data inspections and transformations occur in the c
12.
by dsies 3y ago
Hey peeps - Dan here - ready to answer any questions you've got!
13.
Show HN: Streamdal – an open-source tail -f for your data(github.com)
148 points by dsies 3y ago|37 comments
14.
by dsies 3y ago
Not being pedactic, I swear :) Can you elaborate on the Sun comparison? I am a huge fan of Sun and what they did for computing at large - designing hardware, creating specs, their contributions to evolving unix and so on. I'm not sure
15.
by dsies 3y ago
What a great idea. He was part of the hacker subculture - regardless of what your opinion of him or his abilities is. This made me think of the takedown.com telnet transcripts - what a blast from the past. Really sad day, RIP - will definit
16.
by dsies 4y ago
`protoc` is gnarly - `ffmpeg` unix graybeard vibes ;)
17.
by dsies 4y ago
If you have a gRPC service, you'd use `grpcurl`. This one is for RESTish HTTP 1 API's where the req/resp body is protobuf - something that `grpcurl` can't handle. In other words, you'd use this if your API uses trad
18.
by dsies 5y ago
The "reimplementing" part resonates with me 100%. I've also reimplemented this solution many times and every time it has been a pain in the ass. I built https://batch.sh specifically to address this - not having t
19.
by dsies 5y ago
^ 100% this. Do not delete events. They are your source of truth - you shouldn't really even modify them but stripping out PII is "alright". Re 24M+ records: create a batch runner that goes through "jobs" to perform
20.
by dsies 5y ago
Hi there, I started a company specifically centered around event sourcing - it's a saas platform for capturing all of your events, gaining the ability to granularly search them and then replay them to whatever destination. I'd be
21.
by dsies 5y ago
I'm late to this party but I'll chime in anyway - I love event sourcing. So much so that I actually built a company around it (and got into YC S20) - https://batch.sh Event sourcing is not easy but the benefits are hug
22.
by dsies 5y ago
Elixir! I've got such a mixed bag of feelings for that language. It's elegant but it's build-times are pretty terrible. It has the RPC stuff built-in but I've never seen anyone use it in production and instead fol
23.
by dsies 5y ago
This is cool - I love this space and it's indeed pretty complicated. And not that you asked for it but I'll try to add some additional insight. It depends on what you're trying to achieve - are you trying to increase reliabil
24.
by dsies 5y ago
Love this tool - use it to populate a bunch of "real-looking" data to test CDC functionality for Mongo. Pretty quick too - generated 100,000 records in <3 sec.
25.
by dsies 6y ago
Ahhh gotcha! Yes, that's actually one, decent sized pro for doing an event driven monolith haha With that said, in a monolith - there is still a chance of missed events - deploy, bugs, etc. and at that point, you'd still want to e
26.
by dsies 6y ago
Your sense is totally right and to boil it down further - you have to be OK with your system being eventually consistent.
27.
by dsies 6y ago
I might be missing something - what is the issue with being able to handle a replay event stream in a monolithic service? As the previous poster mentioned, if you are are able to handle events idempotently, you should be able to ingest even
28.
by dsies 6y ago
In my experience, lots of banking, some trading. In both cases, it is used for history, audit and precision. The idea is that for banking, it is not enough to just get the _current_ state - the more important thing is how someone _reached_
29.
by dsies 6y ago
Full disclosure: I co-founded a startup ( https://batch.sh ) which focuses specifically on solving problems in the event driven space - we do event archive, search and replay. So this space is super interesting to me :) You are
30.
by dsies 6y ago
Thank you very much!

More ›