parrt

5 ms·

1. 1.▲
   by parrt 2y ago

   That definitely makes it seem like it's noticing a great deal of its context window. impressive.
2. 2.▲
   by parrt 2y ago

   Wow. Cool. I have access to that model and have also seen some impressive context extraction. It also gave a really good summary of a large code base that I dumped in. I saw somebody analyze a huge log file, but we really need something l
3. 3.▲
   GPT-4o's Memory Breakthrough – Needle in a Needlestack(nian.llmonpy.ai)

   478 points by parrt 2y ago|239 comments
4. 4.▲
   by parrt 2y ago

   The article shows how much better GPT-4o is at paying attention across its input window compared to GPT-4 Turbo and Claude-3 Sonnet. We've needed an upgrade to needle in a haystack for a while and this "Needle In A Needlestack&quo
5. 5.▲
   by parrt 2y ago

   BTW, how did you manage all of the throughput to the models and navigate the various throttling strategies for all the models you mentioned?
6. 6.▲
   by parrt 2y ago

   Looks really cool and useful. Seems like GPT-4o it's a lot better than 4.
7. 7.▲
   by parrt 3y ago

   Sure thing! Very enjoyable to have people use our work.
8. 8.▲
   by parrt 3y ago

   :) Yeah, I use my own internal markdown to generate really nice html (with fast latex-derived images for equations) and then full-on latex. (tool is https://github.com/parrt/bookish ) I prefer reading on the web unl
9. 9.▲
   by parrt 3y ago

   Glad to be of assistance! Yeah, It really annoyed me that this critical information was not listed in any one particular spot.
10. 10.▲
    by parrt 5y ago

    Thanks! Took me a year to discover the key nut there. L1 vs L2 regularization is not well described I found so I went nuts trying to nail it down.
11. 11.▲
    by parrt 5y ago

    Also note we recently added 1D and 2D classifier decision boundary plots. See https://github.com/parrt/dtreeviz/blob/master/notebooks/clas...
12. 12.▲
    by parrt 5y ago

    Thanks. It's morphed over time as we add functionality so it's less clean than before.
13. 13.▲
    by parrt 5y ago

    That's a good idea. thanks!
14. 14.▲
    by parrt 5y ago

    Thanks! It took forever to bash my way to victory on that trees. The lib also supports the shallow trees in boosting machines.
15. 15.▲
    by parrt 6y ago

    Looks like my ANTLR 4 reference book is available, if anybody is interested.
16. 16.▲
    by parrt 6y ago

    True, but you have to do it one at a time to look at the variables. Also, the exception message doesn't tell you which operator among several is the issue. This just makes it easier. :)
17. 17.▲
    by parrt 6y ago

    Thanks, Jeremy. :) I didn't go into super huge detail in the article on the implementation part as most readers won't have interest in language nerd details like you and I do.
18. 18.▲
    by parrt 6y ago

    I wondered about that. It might work just calling my internal pyviz("some python code in string) function from the debugger. It'll execute in context (maybe?)
19. 19.▲
    by parrt 6y ago

    BTW, here's a link to the twitter thread. https://twitter.com/the_antlr_guy/status/1313569854918590465
20. 20.▲
    by parrt 6y ago

    One of the biggest challenges when writing code to implement deep learning networks is getting all of the tensor (matrix and vector) dimensions to line up properly, even when using predefined network layers. This article describes a new lib
21. 21.▲
    Clarifying exceptions and visualizing tensor operations in deep learning code(explained.ai)

    62 points by parrt 6y ago|12 comments
22. 22.▲
    by parrt 6y ago

    Thankfully pytorch does all of that calculus for us!
23. 23.▲
    by parrt 6y ago

    Glad to be of service. Yeah, I just couldn't see what RNNs were doing through all of the neural net stuff. Much easier to think of it as just gradually transforming a vector. I also can't see linear algebra w/o visualizing di
24. 24.▲
    by parrt 6y ago

    Vanilla recurrent neural networks (RNNs) form the basis of more sophisticated models, such as LSTMs and GRUs. There are lots of great articles, books, and videos that describe the functionality, mathematics, and behavior of RNNs so, don&#x2
25. 25.▲
    Explaining RNNs without neural networks(explained.ai)

    126 points by parrt 6y ago|7 comments
26. 26.▲
    by parrt 6y ago

    Here's the abstract: Practitioners use feature importance to rank and eliminate weak predictors during model development in an effort to simplify models and improve generality. Unfortunately, they also routinely conflate such feature i
27. 27.▲
    Can we get feature importance wo using predictions from a fitted model? Yep(arxiv.org)

    2 points by parrt 6y ago|1 comments
28. 28.▲
    by parrt 6y ago

    Indeed; lagrange multipliers are your friend! My problem initially was just the disconnect between the picture and the soft-constraint lagrange multiplier. Just couldn't figure out how that thresholded like that. :)
29. 29.▲
    by parrt 6y ago

    haha. :) Yeah, I got stuck for SOOooo long trying to reconcile the standard picture from ESL book with the math. Turns out they don't match! The picture is for a hard constraint whereas the math penalty term just makes bigger coeffici
30. 30.▲
    by parrt 6y ago

    The world certainly doesn't need yet another article on the mechanics of regularized linear models. What's lacking is a simple and intuitive explanation for what exactly is going on during regularization. The goal of this article