jdspiral

5 ms·

1. 1.▲
   by jdspiral 1y ago

   I moved the app, it’s now tokenizer-machine.streamlit.app.
2. 2.▲
   by jdspiral 1y ago

   So I've taken the feedback and realized that I was misleading on the name and title. I'm updating the project accordingly. https://tokenizer-machine.streamlit.app/
3. 3.▲
   by jdspiral 1y ago

   Thanks! Yes — that’s on the roadmap, along with some other cool visualizations I’m working on. Machine translation is definitely something I want to work on: showing how models align meaning across languages using shared embeddings and atte
4. 4.▲
   by jdspiral 1y ago

   Yes, tokenization and embeddings are exactly how LLMs process input—they break text into tokens and map them to vectors. POS tags and SVOs aren't part of the model pipeline but help visualize structures the models learn implicitly.
5. 5.▲
   Meaning Machine – Visualize how LLMs break down and simulate meaning(meaning-machine.streamlit.app)

   114 points by jdspiral 1y ago|27 comments
6. 6.▲
   by jdspiral 1y ago

   I built a tool called Meaning Machine to let you see how language models "read" your words. It walks through the core stages — tokenization, POS tagging, dependency parsing, embeddings — and visualizes how meaning gets fragmented