willvarfar

8 ms·

1.
by willvarfar 5mo ago
Exciting! I have been on a similar odyssey making a 'zero copy' Java library that supports protobuf, parquet, thrift (compact) and (schema'd) json. It does allocate a long[] and break out the structure for O(1) access but do
2.
by willvarfar 5mo ago
With a long time in the industry and seeing how so many big software companies work, this really really chimed with me. Many/most teams and projects and busy work are not actually moving the bottom line, at massive opportunity cost!
3.
by willvarfar 8mo ago
Yeah it's pretty obscure, sorry. It's called cogroup in Spark and similar architectures. It does a group-by to convert data into the format (key_col_1, ... key_col_n) -> [(other_col_1, ... other_col_n), ...] This is useful and
4.
by willvarfar 8mo ago
This video is very liberal but does a good job of explaining which companies and industries pay for breaks and which don't. And uses soy bean farmers as a prominent example of a group who haven't been giving Trump bribes https:&
5.
by willvarfar 8mo ago
Can join cardinality can be tackled with cogroup and not expanding the rows until final write?
6.
by willvarfar 8mo ago
everyone is expecting everyone to actually go Gripen with Rolls Royce or MECA engines?
7.
by willvarfar 8mo ago
Having a pending order that can be cancelled is negotiation leverage?
8.
by willvarfar 8mo ago
And what about all those huge pending orders for F35 in ... Denmark and Canada? Etc.
9.
by willvarfar 8mo ago
The very last clip in the video says that it is kids in affluent families taking that direction.
10.
by willvarfar 8mo ago
(I work a lot with BigQuery's BigLake adaptor and it's basically caching the metadata of the iceberg manifest and parquet footers in Bigtable (this is Google) so query planning is super fast etc. Really helps)
11.
by willvarfar 8mo ago
Greenland and Denmark have always been encouraging minerals deals etc, they just haven't materialized.
12.
by willvarfar 8mo ago
Seriously, this is not what big data does today. Distributed query engines don't have the primitives to zip through two tables and treat them as column groups of the same wider logical table. There's a new kid on the block calle
13.
by willvarfar 8mo ago
specifically I've discovered how to 'trick' mainstream cloud storage and mainstream query engines using mainstream table formats how to read parallel arrays that are stored outside the table without using a classic join and t
14.
by willvarfar 8mo ago
crazy to think that soon not being able to successfully complete the captcha will be a signal that the user is human
15.
by willvarfar 8mo ago
I had a great euphoric epiphany feeling today. Doesn't come along too often, will celebrate with a nice glass of wine :) Am doing data engineering for some big data (yeah, big enough) and thinking about efficiency of data enrichment.
16.
by willvarfar 8mo ago
I agree that social media is a net negative, but want to also point out that before social media it was the mainstream press and TV have been shaping society for decades. Things like buying a used car from Nixon or fighting in Vietnam etc
17.
by willvarfar 8mo ago
I've always noticed and wondered, so I guess it's easy to overlook but it's there.
18.
by willvarfar 8mo ago
Presumably there will soon be banner ads in Claude Code then? </s>
19.
by willvarfar 8mo ago
I remember when syntax highlighting was introduced in Borland's Turbo Pascal editor (on DOS). It was a very major usability improvement and put TP's IDE at the forefront of getting things done. Fond memories :)
20.
by willvarfar 8mo ago
Kudos, neat digging and writeup that makes us think :) If you merge linear probed tables by iterating in sorted hash order then you are matching the storage order and can congest particular parts of the table and cause the linear probing wo
21.
by willvarfar 8mo ago
This is going to be an interesting take but I think it is plausible that we'll see a quiet growth in American tech companies having even bigger offshore campuses instead. Google Zurich or Google London could grow, Google does hardware
22.
by willvarfar 9mo ago
A really clear explanation! So if I were running a provider I would be caching popular prefixes for questions across all users. There must be so many questions that start 'what is' or 'who was' etc? Also, can subsequenc
23.
by willvarfar 9mo ago
absolutely, I almost missed it too. Took me a while to work out!
24.
by willvarfar 9mo ago
its a video? There was a preroll ad but you can also just click listen for the soundtrack.
25.
by willvarfar 9mo ago
I found this and was struck by how well formed and sharable this commentary was.
26.
'This is worse than the dot-com bubble' – Ed Zitron [video](youtube.com)
7 points by willvarfar 9mo ago|3 comments
27.
by willvarfar 9mo ago
It's also that Google brought Doubleclick and that completely flipped their incentives. Now they get money sending you to the content farms and what is good for their business isn't necessarily aligned with what is good for you..
28.
by willvarfar 9mo ago
I desperately want the kind of success we used to have searching on google back over a decade ago. Relevant results. A few annoying ads started appearing with paid placement, but we all ignored them. Nowadays google search results are so
29.
by willvarfar 9mo ago
I know I paid about 1000EUR for an air-air heat-pump with install in Sweden, but that was a decade ago and they cost 1500-2000EUR total these days. I also have a fancy big ground-source heat-pump bigger than most residential ones and that
30.
by willvarfar 9mo ago
The south of Sweden is expensive because Sweden did away with the previous single energy market and split into zones with sales abroad. Often Swedish producers sell to Germany at the same time Swedish consumers are forced to buy from Germa

More ›