cubacaban

5 ms·

1. 1.▲
   by cubacaban 2y ago

   Rather superficial and obfuscating. The article keeps raising the question "why ignore the magnitude" and never answers it. "The important part of an embedding is its direction, not its length. If two embeddings are pointing
2. 2.▲
   by cubacaban 3y ago

   How big is the state space of the Markov chain corresponding to a LLM generating a sequence of tokens? Wouldn't it be (size of the vocabulary)^(size of the context window), i.e. ~ (100k)^(4k)? How useful is it to conceptualize LLMs as
3. 3.▲
   by cubacaban 4y ago

   Mapping corresponding text and image into the same vector space is exactly how CLIP and other contrastive learning setups work: Have text and computer vision networks embed input data, and teach the networks to embed related inputs closely