entilzha

6 ms·

1.
by entilzha 1y ago
Great to see our paper here again! Since the paper release, we've also released model weights here for anyone interesting in building on top of it: https://huggingface.co/facebook/blt . We also added HF Hub code to
2.
by entilzha 2y ago
At least I wasn't aware of this work, but thanks for the refs! I'm always curious to read papers from 10-20+ years ago that have similarly inspired ideas. If it makes sense, we'll mention those in the next related work update
3.
by entilzha 2y ago
I don't believe so, or at least if someone tried it didn't work well enough that I remember :). Some of the motivation for the architecture changes in encoding patches stemmed from finding FLOP efficient ways to express relationsh
4.
by entilzha 2y ago
(Author here) If I understand your question right, this is one of the reasons BPE is nice and the parent liked it. For any character sequence, provided the characters are in the alphabet used to create the BPE vocab, there are no unknown wo
5.
by entilzha 2y ago
(Author Here) Good description! Maybe what parent got mixed up on is an alternate way to view this is trying to chunk bytes to have roughly similar information. EG we initially tried a bunch of patching schemes, EG, keep a running total of
6.
by entilzha 2y ago
(Author Here) Related thought, I think BPE is quite a good, cheap inductive bias to have in a model, which is part of what made it challenging to scale better against. I also suspect this is part of why with less training FLOPs BPE is bette
7.
by entilzha 2y ago
(Author Here) There is at least some work on character based modeling, but it hasn’t scaled well before. The challenge I think with something more adhoc for exceptional tokens is that it’s hard to see gains since they are by definition, inf
8.
by entilzha 2y ago
(Author Here) In editing we couldn’t find a good place for this so cut it in the current version, but at one point had discussed a parallel with information density of speech as described by one paper. Essentially the paper found that in la
9.
by entilzha 2y ago
(Author Here) Not sure what you mean by implicit? If you mean just treat bytes as tokens, one issue you run into is your sequence lengths get quite long, so compared to a regular token LLM, you can’t pack as many bytes in a batch, which mea
10.
by entilzha 2y ago
Author here :), I do think it’s a good direction to look into! That said, aside from it being a bit too much to do at once, you’d also have to be careful about how you distributed your FLOP budget across the hierarchy. With two levels, you
11.
by entilzha 3y ago
I tried a few a while back. What I really want is as close to 1-1 to obsidian UI as possible. I found with some of the plugins that it could be hit/miss on working correctly. If I were doing only markdown notes, then wouldn’t need obsi
12.
by entilzha 3y ago
While you’re here, a killer feature for me would be the ability to privately host obsidian sites (similar to publish). Even if it required subscribing to publish to download a tarball of the site (that isn’t public), it could still be worth
13.
by entilzha 7y ago
Any thoughts on how to access/modify on mobile without making it too cumbersome? I often think about todo on walk/train, but could see making it a computer only thing.
14.
by entilzha 7y ago
Totally agree on not trusting any one doctor. Nowadays, I basically assume doctors are narrow minded experts and do the broader thinking myself (by reading widely). It’s the old saying that when you have a hammer everything looks like a nai
15.
by entilzha 7y ago
Not entirely true. For good utilization you need both GPU/TPU ops to be fast (written in C), but that won’t get you far if your input pipeline (possibly written in python) is slow. I could imagine if all the TF calls work in PyPy, that
16.
by entilzha 7y ago
I’m a co-author and would be happy to answer questions about our work!
17.
Seeing How Computers Think Helps Humans Stump Machines and Reveal AI Weaknesses(cmns.umd.edu)
1 points by entilzha 7y ago|1 comments
18.
by entilzha 8y ago
Anyone know if puppy works with deep learning libraries like pytorch/tensorflow, or if there are plans to do so? Not looking for numerical speed ups, but for speed ups in preprocessing code
19.
by entilzha 8y ago
A similar issue got me from recommending android and google services to friends/family to actively discouraging them (and migrating off of every service I could feasibly do). The WiFi chip on my nexus 6 burnt out and neither Motorola o
20.
by entilzha 9y ago
You could actually simplify even more with the trick used in a comment farther down ( https://github.com/0101/pipetools ). That way you would implement __or__ on for example `pipe` and not have to wrap each function. As
21.
by entilzha 9y ago
True, but for better or worse python has support for spark/sklearn/pandas/numpy/plotting/deep learning/web which makes it a good jack of all trades language for data/research science. Its strength is that
22.
by entilzha 9y ago
Seq doesn't return a generator. At its core there is a concept of Lineage taken from Apache Spark. Essentially, when you do something like seq(data).map(func) it builds on a list of operations to execute and holds a copy of the base da
23.
by entilzha 9y ago
Data interchange between the two is relatively seeemless. seq(df) converts a pandas dataframe into a sequence of tuples/namedtuples. Calling .to_pandas(columns=cols) will convert a sequence of tuples/namedtuples to a dataframe wit
24.
by entilzha 9y ago
For operations where vectorization pandas will be much faster since its based on numpy. That being said, there are things which are very awkard to do in pandas which are very easy in pyfunctional
25.
by entilzha 9y ago
Thanks on name! Fair enough on docs being a bit scarce. The main intent is to being able to easily make a pandas dataframe a sequence of tuple/namedtuples, and convert a sequence of tuple/namedtuple to a pandas dataframes.
26.
by entilzha 9y ago
Not OP, but author. Line continuation is pretty annoying. Adding custom methods wouldn't be hard ( https://github.com/EntilZha/PyFunctional/issues/113 ), I just don't have time right now to add it. Wi
27.
by entilzha 9y ago
It would be great to have more native support for FP (flat map is a glaring example I run into a lot). It would help if there were 1) a less verbose anonymous closure syntax 2) support for a multi-line closure.
28.
by entilzha 9y ago
Author here. 1.0 has been out for a while now, and I'm very committed to API stability. 1.1 (or 1.0.1) should be out as soon as I have some time with mostly minor fixes here and there. I hadn't considered doing a comparison matrix
29.
by entilzha 10y ago
1password is a bit spendy, but I have found the convenience it provides to be valuable (browser integration, TouchId on iOS, and a few other things)
30.
by entilzha 10y ago
After Rice joined I actually completely stopped using Dropbox, transferred files, and deleted my account.

More ›