8 ms·
I haven't watched the video, but was wondering about the Tokenization part from the TL;DR: "|" "View" "ing" "Single" Just looking at the text being tokenized
by wolfhumble 2y ago
I haven't watched the video, but was wondering about the Tokenization part from the TL;DR:
"|" "View" "ing" "Single"
Just looking at the text being tokenized in the linked article, it looked like (to me) that the text was: "I View", but the "I" is actually a pipe "|".
From Step 3 in the link that @miletus posted in the Hacker News comment: https://x.com/0xmetaschool/status/1888873667624661455 https://x.com/0xmetaschool/status/1888873667624661455 the text that is being tokenized is:
|Viewing Single (Post From) . . .
The capitals used (View, Single) also makes more sense when seeing this part of the sentence.