mlucy

5 ms·

1.
by mlucy 6y ago
This is really cool! I especially like that there's a premade colab notebook that lets you play with it: https://colab.research.google.com/github/openai/clip/blob/ma... . I'm a little surprise
2.
by mlucy 8y ago
I don't think you could get access to the actual models that are being used to run e.g. Google Translate, but if you just want a big pretrained model as a starting point, their research departments release things pretty frequently. For
3.
by mlucy 8y ago
> Is it always good enough to take the outputs of the next-to-last layer as features? It usually doesn't matter all that much whether you take the next-to-last or the third from last, it all performs pretty similarly. If you'r
4.
by mlucy 8y ago
There's actually been a lot of really good work recently around textual transfer learning. Google's BERT paper does sentence-level pretraining and transfer to get state of the art results on a bunch of problems: https:/&#x2
5.
by mlucy 8y ago
Yeah, I think this pattern is pretty common. (Basilica's main business is an API that does deep feature extraction as a service, so we end up talking to a lot of people with tasks like yours -- and there are a lot of them.) We'
6.
by mlucy 8y ago
I think if you have a small to medium sized dataset of images or text, deep feature extraction would be the first thing I'd try. I'm not sure what the most interesting problems with that property are. Maybe making specialized cla
7.
by mlucy 8y ago
I hadn't read it before! That's a fascinating result, actually. They emphasize interpretability in the paper, but I find it more interesting that you can do so well with only local information. My first thought is that it makes
8.
by mlucy 8y ago
I don't work with time series data much myself. I would imagine you can get at least some transfer learning, since there are patterns that show up across different domains. It looks like there's been a little bit of work done on
9.
by mlucy 8y ago
Definitely. There's been a lot of exciting work recently for text in particular, like https://arxiv.org/pdf/1810.04805.pdf .
10.
by mlucy 8y ago
Linear Algebra Done Right would be my recommendation.
11.
by mlucy 8y ago
Hi everyone! Author here. Let me know if you have any questions, this is one of my favorite subjects in the world to talk about.
12.
by mlucy 8y ago
> He's almost describing a future where we might buy/license pre-trained models from Google/Facebook/etc that are trained on huge datasets, and then extend that with more specific training from other sources of data i
13.
by mlucy 8y ago
Really cool idea. I hope you manage to get into a sustainable cycle of people you've helped with bankruptcy getting back on their feet and donating to help others in the same position.
14.
by mlucy 8y ago
I would second this; sentence embeddings outperform word embeddings on basically all tasks where you actually have sentences to work with. The only downside is that they're significantly more computationally intensive, especially for
15.
by mlucy 8y ago
Interesting. It took me a while to figure out what the main contribution here is, since doing dimensionality reduction on embeddings is fairly common. I think the main contribution is an empirical measure of A) how little is lost by reduc
16.
by mlucy 8y ago
A word embedding transforms a word into a series of numbers, with the property that similar words (e.g. "dog" and "canine") produce similar numbers. You can have embeddings for other things, such as pictures, where you w
17.
by mlucy 8y ago
It's really difficult to overstate how important embeddings are going to be for ML. Word embeddings have already transformed NLP. Most people I know, when they sit down to work on an NLP task, the first thing they do is use an off-the
18.
by mlucy 8y ago
Hi there :) Apologies for the super long response, but you had a lot of points. > Am I really missing something here or this thing is a complete nonsense with no actual use cases what's so ever in practice? Hopefully you're mis
19.
by mlucy 8y ago
You can definitely improve performance by choosing an embedding closely related to your task. In the future we're hoping to have more embeddings for specialized tasks. Kind of surprisingly, though, if you get your embedding by trainin
20.
by mlucy 8y ago
Thanks! No production use cases yet. This is the first usable release, and it's the bare minimum we felt we could build before showing it to people. > Are you a YC company? We have a YC interview on Friday, so hopefully in a few da
21.
by mlucy 8y ago
We aren't currently doing this. In the future I think we'll try to embed into a single space on a best-effort basis, assuming we can find the engineering resources. It will be really hard for some data types, but for the big ones
22.
by mlucy 8y ago
That's a really interesting idea. I can't really think of a barrier to this. Detecting the file format is straightforward, and generic image/text/etc. embeddings work surprisingly well. (In fact, you can actually get s
23.
by mlucy 8y ago
Hey! We're embedding images by feeding them through a deep neural net and using the activations of an intermediate layer as an embedding. You can read https://arxiv.org/abs/1403.6382 to learn more about this techn
24.
by mlucy 8y ago
"Word2vec for anything" is where we want to get to. Right now we only support images and text, but you can see the other data types on our roadmap at https://www.basilica.ai/available-embeddings/ .
25.
by mlucy 8y ago
Yeah, I agree. The number of new nouns per year is kind of ridiculous.
26.
by mlucy 8y ago
You're definitely right that you sometimes need to know the exact details of how an embedding is produced, especially if you're doing cutting-edge work. That's one of the things we really need to improve documentation-wise.
27.
by mlucy 8y ago
:P For what it's worth, people are doing job candidate clustering anyway right now. It's just that most people are doing it with keyword search or something. Doing it with embeddings instead would probably increase the quality
28.
by mlucy 8y ago
Apologies for the long answer, but this touches on a lot of interesting points: 1. Transfer learning / data volume. If you have a small image dataset, embedding it using an embedding trained on a much larger image dataset is really re
29.
by mlucy 8y ago
Definitely! In particular, semantically similar words are close to each other after embedding, so the space ends up with semantically meaningful clusters. Our embeddings have the same property. If you embed two similar images, they'll
30.
by mlucy 8y ago
We're feeding them through a deep neural network and using the activations of an intermediate layer as an embedding. You can read more about this technique in https://arxiv.org/abs/1403.6382 if you're interes

More ›