edrenova

8 ms·

1.
by edrenova 2y ago
Just to jump in here -> We support RDS + more and you can self-host, Neosync. https://github.com/nucleuscloud/neosync (I'm one of the co-founders)
2.
by edrenova 2y ago
Thanks for the shout-out! Co-founder of Neosync here - love seeing more tools in this space and pushing the envelope further. Good luck!
3.
by edrenova 2y ago
Yup agreed. We built an orchestration engine into Neosync for that reason. Can handles all of the reading/writing from DBs for you. Also can generate data from scratch (using LLMs or not).
4.
by edrenova 2y ago
Nice write up, mock data generation with LLMs is pretty tough. We spent time trying to do it across multiple tables and it always had issues. Whether you look at classical ML models like GANs or even LLMs, they struggle with producing a lot
5.
by edrenova 2y ago
Thanks for the question! Faker is useful but doesn't have a lot of features. For example, referential integrity, data orchestration or the ability to read/write to a db. So faker can work for simple API schemas but if you need som
6.
by edrenova 2y ago
Thanks! Yeah we generally recommend not making your databases public and instead connecting to them using a bastion host. We support this at Neosync. Also, ideally, not connecting to a live DB and instead a snapshot or back up. A read repli
7.
Show HN: Open-Source Data Anonymization for Developers(docs.neosync.dev)
13 points by edrenova 2y ago|5 comments
8.
by edrenova 2y ago
cool to see this launch, actually came across this a few weeks ago and tried it out, really nice for local dev :)
9.
by edrenova 2y ago
The ideal experience is that you anonymize prod and sync it locally. Whether it's for testing or debugging, it's the only way to get representative data. When you write mock data, you almost always write "happy path" dat
10.
by edrenova 2y ago
Excited to announce a new partnership between Neon (open source serverless postgres) and Neosync (open source data anonymization) to give developers the easiest way to create data branches with anonymized production data for better testing,
11.
by edrenova 2y ago
hey! so sorry about this - it's fixed now! also - happy to chat further if you have any questions - evis@neosync.dev
12.
by edrenova 2y ago
Nice! appreciate you sharing it - would love to see the code at some point but looks like it's confidential. I spent a lot of time building tokenization solutions at a previous startup so we'll definitely support tokenization at s
13.
by edrenova 2y ago
Yup - totally hear you - hopefully we'll have a good solution for that in a few months :)
14.
by edrenova 2y ago
The ideal scenario is that you're able to augment your existing data with more data that looks just like it. The matter of statistical significance really depends on the use-case. For load testing, it's probably not as important a
15.
by edrenova 2y ago
Thanks for the comment and feedback! We're actually evaluating a clickhouse integration at the moment for a customer that we're working with so that might be coming in the future. Although today just PG and Mysql. To answer your q
16.
by edrenova 2y ago
yeah the referential integrity and constraints part is usually the most complicated part and everyone does things differently which adds another layer of complexity on it
17.
by edrenova 2y ago
yeah good question, if you're doing any sort of analytical work, then you'll care about the statistical distribution of your data. If you're running queries or sharing data with third parties, then you want to maintain the sa
18.
by edrenova 2y ago
we're actually working on this right, can see the PR here -> https://github.com/nucleuscloud/neosync/pull/1832/files it's a combination of creating a random number of records for foreign key
19.
by edrenova 2y ago
Thanks for the comment and hear you on the anonymization. What we see is that customers will go through and categorize what is PII and what is not and anonymize as needed. If not, they'll back fill with synthetic data. You can change
20.
Show HN: Neosync – Open-Source Data Anonymization for Postgres and MySQL(github.com)
246 points by edrenova 2y ago|44 comments
21.
by edrenova 2y ago
Hey HN - co-founder of Neosync here; We just launched our brand new AI Data Generation feature which allows you to use ayn LLM to generate synthetic data and insert that directly into a Postgres or Mysql database. Simply connect your databa
22.
by edrenova 2y ago
Looks pretty cool :) We can help with the anonymization piece if you're interested in checking us out - github.com/nucleuscloud/neosync
23.
by edrenova 3y ago
We solve this by managing referential integrity across relational databases when we subset. Effectively you have to build a graph of edges and nodes where edges are constraints and nodes are tables, then you have to follow these edges from
24.
by edrenova 3y ago
Logging someone out should be easy but it's surprisingly complex when you're working with different IDPs and auth providers. Here's what we learned building out our logout flow.
25.
by edrenova 3y ago
Right now its just based on a db schema (you connect your DB and we automatically read the schema) and we're adding support for a csv file schema in the next week or so. Our input system is pretty flexible, so it's not too hard to
26.
by edrenova 3y ago
just to jump in here - you can do this with Neosync, an OSS synthetic data generator and orchestrator for any schema and includes relational integrity, anonymization, + more (I'm one of the co-founders) If you're interested (githu
27.
by edrenova 3y ago
I hadn't seen this before but I did come across this recently (although it seems to be an older project) -> https://github.com/wangkuiyi/gotorch Agreed on the GC issue, going to be doing some perf testing on it
28.
The Future Is Synthetic Data Engineering(neosync.dev)
2 points by edrenova 3y ago|0 comments
29.
Show HN: GoTorch – A Go Implementation of PyTorch
9 points by edrenova 3y ago|3 comments
30.
GoTorch: A Go Implementation of PyTorch(github.com)
3 points by edrenova 3y ago|1 comments

More ›