chenquan

14 ms·

1. 1.▲
   ArkFlow and Python: Easy Real-Time AI(arkflow-rs.com)

   2 points by chenquan 1y ago|1 comments
2. 2.▲
   by chenquan 1y ago

   Today, with great excitement, we proudly introduce a revolutionary update to the ArkFlow stream processing engine: full support for Python processors! This is not just a simple feature iteration; it marks a solid step towards more intellige
3. 3.▲
   by chenquan 1y ago

   In the recently merged PR, ArkFlow supports Python processors, which allows us to call Python scripts directly in ArkFlow. Using the Pyarrow library allows users to write data processing logic more freely. At the same time, relying on Pytho
4. 4.▲
   Show HN: Arkflow(stream processing engine) will soon support Python processors(github.com)

   2 points by chenquan 1y ago|1 comments
5. 5.▲
   by chenquan 1y ago

   Hello, I am the founder of this project and I am very happy that a friend has shared it. ArkFlow is positioned as a lightweight distributed stream processing engine that integrates streaming batches. With the help of datafusion's huge
6. 6.▲
   by chenquan 1y ago

   I'm writing a lightweight high-performance stream processing engine： https://github.com/arkflow-rs/arkflow docs: https://arkflow-rs.com/
7. 7.▲
   by chenquan 2y ago

   I’m grateful for your assistance.
8. 8.▲
   by chenquan 2y ago

   Hi, then I'm curious what do you think people care most about in data processing and database workloads?
9. 9.▲
   by chenquan 2y ago

   Yeah, I'm looking forward to having further conversations once I get to know tremor.
10. 10.▲
    by chenquan 2y ago

    Welcome to follow anytime.
11. 11.▲
    by chenquan 2y ago

    I don't know it exists.
12. 12.▲
    by chenquan 2y ago

    Hi, friend. How did you do it specifically?
13. 13.▲
    by chenquan 2y ago

    > Stability and safety are the least of the concerns in data processing and database workloads. I'm curious how you came to this conclusion?
14. 14.▲
    by chenquan 2y ago

    Hi! Brother, I think I will seriously consider what you said and I am honored to communicate with you.
15. 15.▲
    by chenquan 2y ago

    Good job, this is a rich reference.
16. 16.▲
    by chenquan 2y ago

    Hi,brother! I'm still thinking, but it's certainly not now.
17. 17.▲
    by chenquan 2y ago

    Why do you have to beat a native performance-oriented streaming engine written in C or C++? Currently, most of the mainstream stream processing engines are written in Java. Sorry, I may not add qualifiers to make you misunderstandings. Soft
18. 18.▲
    by chenquan 2y ago

    Yes, whether it is DataFusion, Arroyo, or Bentos, these open source products have made me profit a lot.
19. 19.▲
    by chenquan 2y ago

    Yes, RPCN (redpanda connect)/old benthos is very cool and can solve most of the scenes. Let me tell you quietly that I am using it too.
20. 20.▲
    by chenquan 2y ago

    Arroyo has been designed with more comprehensive consideration.
21. 21.▲
    by chenquan 2y ago

    I think stability, reliability and high performance are the foundation of infrastructure.
22. 22.▲
    by chenquan 2y ago

    Thank you for your valuable experience, I will seriously think about what you said.
23. 23.▲
    by chenquan 2y ago

    Sorry, please forgive me for not knowing Arroyo completely.
24. 24.▲
    by chenquan 2y ago

    Welcome to follow the latest news from ArkFlow at any time and even participate.
25. 25.▲
    by chenquan 2y ago

    Sorry, not yet, but this is the direction ArkFlow is working hard. Rust's own potential will also guide ArkFlow in this direction.
26. 26.▲
    by chenquan 2y ago

    Yes, Arroyo is entirely based on DataFusion, but ArkFlow is not exactly. In the future, ArkFlow will establish a plug-in ecosystem, allowing anyone to process data through plug-ins, not limited to DataFusion.
27. 27.▲
    by chenquan 2y ago

    Thanks for your recommendation.
28. 28.▲
    by chenquan 2y ago

    I think a stream processing engine written in rust will have better performance, lower latency, more stable services, lower memory footprint, and cost savings. At the same time, ArkFlow is based on DataFusion implementation, which will put
29. 29.▲
    by chenquan 2y ago

    Yes, they are similar. ArkFlow is mainly based on DataFusion. Bento actually comes from Benthos. Currently, the ArkFlow project is in the early stages and no performance comparison test has been conducted, but I believe that ArkFlow will ou
30. 30.▲
    ArkFlow – High-performance Rust stream processing engine(github.com)

    107 points by chenquan 2y ago|59 comments