tchiotludo

7 ms·

1.
Kestra raises another $8M for its open-source orchestration platform(techcrunch.com)
7 points by tchiotludo 2y ago|0 comments
2.
Tech Salaries Parsing from 2020 to 2023 with DuckDB and Orchestrated with Kestra(kestra.io)
3 points by tchiotludo 3y ago|0 comments
3.
Techniques You Should Know as a Kafka Streams Developer(kestra.io)
2 points by tchiotludo 4y ago|0 comments
4.
by tchiotludo 4y ago
All the issues described in this post lead me to create Kestra [0] . Airflow was a true revolution when it was open-source and we need thanks its innovation. But I totally agree that a large static dag is not appropriate in the actual dat
5.
Ask HN: Open-source with Kafka as dependencies, is this a instant turn off?
1 points by tchiotludo 4y ago|0 comments
6.
Debezium Change Data Capture Without Kafka Connect(medium.com)
7 points by tchiotludo 4y ago|0 comments
7.
by tchiotludo 4y ago
First of all, the article published is a retrospective, we are talking from decision in 2019, we can't talk from the past that leed us for a choice? Second, not all issues, some of them are fixed but there is still major issue, just di
8.
by tchiotludo 4y ago
There is no documentation for that for now but it's not complicated: - git clone the project - ./gradlew executableJar - cp build/executable/* docker/app/kestra - docker build -t myimage . In resume, you just n
9.
by tchiotludo 4y ago
Sorry, I didn't see your comment before. We provide 2 image tag, one with no plugins and one with all plugin, more information here ; https://kestra.io/docs/administrator-guide/deployment/docker... In re
10.
by tchiotludo 4y ago
Temporal.io is a really cool framework for building business process like managing microservice workflow (like paiement workflow: user pay, we call the shipping microservice, the billing microservice, ...) and good fit to handle individual
11.
by tchiotludo 4y ago
You can easily scale down for small project using a mono node using a simple docker compose setup: https://kestra.io/docs/getting-started/ Working well on a standard laptop easily
12.
by tchiotludo 4y ago
I know that that some issues are fixed in Airflow 2, they have made a large improvement with that release. But not all issues is resolved with this one. The performance issue is still here, just launch Airflow and submit thousand dagruns wi
13.
by tchiotludo 4y ago
The title was changed by moderators and I can't edit it anymore :'(
14.
by tchiotludo 4y ago
Airflow have design issue and performance issue, If you want to have some details, you can find some reason on this article: https://kestra.io/blogs/2022-02-22-leroy-merlin-usage-kestra... . For other workflow engine (d
15.
by tchiotludo 4y ago
The project start as a side project (yet another side project I do the night and weekend) but was quickly promoted and used in a French Big Retail Company. This one trust on the project and decide to go production with Kestra. So they decid
16.
by tchiotludo 4y ago
Do you have a screenshot please ? I didn't notice where. Thanks
17.
by tchiotludo 4y ago
Agree that both are expensive to scale on multiple node. But keep in mind, you can use it with a single node (like others do with a database like mysql). Just don't go multiple node if not needed by the project. But when you will need
18.
by tchiotludo 4y ago
Yes, of course! You have 3 solutions for that: - you can use this task using runner:DOCKER property and choose the image: https://kestra.io/plugins/core/tasks/scripts/io.kestra.core.... - you can also us
19.
by tchiotludo 4y ago
Ok got it, every task type are able to logged what they want, so basically, there is one task that is sending dirty output, I add a fix on the roadmap in order to clean this task, thanks for the report
20.
by tchiotludo 4y ago
Really thanks wpietri to have understand the hidden sense behind :+1:
21.
by tchiotludo 4y ago
I don't pitch as more complicated version of Airflow, rather, I think it's more simple than Airflow on the UX side: we use declarative flow with yaml and not python code that can be I agree with you that Kafka & ElasticSearch
22.
by tchiotludo 4y ago
Sorry I don't understand, we display with pretty print, see here : https://kestra.io/assets/img/05.8b5545ef.png It's not as json ? or I don't understand where you see that.
23.
by tchiotludo 4y ago
The support of windows server seems to be easy I think. Since it's java behind, most of the api is working on windows. Just need to create a custom task for windows, added in the backlog : https://github.com/kestra-io
24.
by tchiotludo 4y ago
- For performance mostly, Kestra rely a lot on Java thread to be able to handle a very large workload - Because the application is built on top of Kafka, and Kafka Streams that is only available on Java - Because the java ecosystem is very
25.
by tchiotludo 4y ago
To be honest, we don't have think about that for now. It's a complicated subject.
26.
by tchiotludo 4y ago
For now, we don't provide a SAAS for Kestra, it's definitely on the roadmap and our next project. In a meantime, we provide different installation : - Docker compose: https://kestra.io/docs/administrator-guid
27.
Show HN: Kestra - Open-Source Airflow Alternative(github.com)
142 points by tchiotludo 4y ago|69 comments
28.
Kestra: Infinitely scalable open source orchestration and scheduling platform(github.com)
7 points by tchiotludo 5y ago|2 comments
29.
by tchiotludo 6y ago
I confirm ! I need to rename KafkaHQ to AKHQ : https://twitter.com/akhq_io/status/1240037275565727744 Apache Committee ask to rename the project as CMAK and others ...
30.
by tchiotludo 7y ago
I confirm, this works ! Behind KafkaHQ is the official Kafka java client, so you can't connect to every cluster with any kind of security.

More ›