jnewhouse

9 ms·

1.
by jnewhouse 3y ago
Yep, pretty much. Right now filesystem^ sources are finite, scanning the target path at operator startup time and processing all matching files. This processing is done by opening an asynchronous reader, courtesy of the object_store crate.
2.
by jnewhouse 3y ago
For the SQL interface, both sources and sinks are treated as tables. Sources you SELECT FROM, while sinks you INSERT INTO. Right now it is incumbent on the user to correctly specify the types of a source for deserialization. How getting thi
3.
The two semantics of streaming SQL(arroyo.dev)
4 points by jnewhouse 3y ago|0 comments
4.
Why Not Flink?(arroyo.dev)
3 points by jnewhouse 3y ago|0 comments
5.
by jnewhouse 3y ago
Hi there! We actually already have a built-in Nexmark source. It's pretty useful for developing new capabilities, and available as a source out of the box. Just read through the DBSP docs and it looks like it is working in a similar sp
6.
by jnewhouse 9y ago
Nope, Kudu https://kudu.apache.org/ . Although from Arrow's homepage it looks like it works with Kudu. "Apache Arrow is backed by key developers of 13 major open source projects, including Calcite, Cassandra, Drill
7.
by jnewhouse 9y ago
I don't think protobuf was around for public use when we came up with this format, which began around 2005. We use Protobuf internally, and some of our columns are actually byte[]'s containing protobuf data. We now support Parquet
8.
by jnewhouse 9y ago
It natively supports a limited set of Columns. Basically boxed primitives, java.util.Date, joda.time.DateTime, and arrays and double arrays of both boxed and unboxed versions of the preceding. The list of Columns being used is used to read
9.
by jnewhouse 9y ago
Those switches all occurred at the pipeline level, leaving the map-reduce platform untouched. Switching our base logs to something like Parquet, Thrift or Protobuf would be a much larger project. We do support writing and reading Parquet to
10.
by jnewhouse 9y ago
I think what I'd heard about was likely a poorly implemented use of Avro. I haven't actually worked with it.
11.
by jnewhouse 9y ago
We started developing rowfiles around 2005. Thrift wasn't open sourced until 2007. I couldn't find a date for protobuf's release, but I don't think it was standard outside of google at that time. We use protobufs interna
12.
by jnewhouse 9y ago
If you want more details, we were packing a Row class into a base64 encoded string using an ObjectOutputStream. This is a fine thing for small scale serialization but sucks at scale, because of the reasons mentioned in the post. Sorry we do
13.
by jnewhouse 9y ago
A standard database table isn't large enough to handle our large datasets. For example, the Hercules dataset was over 2 petabytes and even after optimization is almost 1 petabyte. Big data systems like Spark, Impala, Presto, etc. are d
14.
by jnewhouse 9y ago
We're a big data advertise and measure company based in San Francisco. We run online display ad campaigns for marketers across realtime bidding exchanges (RTB), such as those run by Google and AppNexus. We also provide a publisher prod
15.
by jnewhouse 9y ago
Author here, let me know if you have any questions/want more details.
16.
Saving Millions by Dumping Java Serialization(quantcast.com)
86 points by jnewhouse 9y ago|45 comments