dmakian

6 ms·

1.
Show HN: Send AI-Generated facts about anything to your friends(getfacts.io)
2 points by dmakian 3y ago|0 comments
2.
by dmakian 3y ago
> A question, when GPT-4 contradicts in explanation, how much of them were in fact correct? It was mostly when a card is good in a vacuum but not as good in a specific set. WOE (which this was trained on) skewed pretty aggressive, so GPT
3.
by dmakian 3y ago
The longest running fine tuning job took about 8 hours, so ~$5. I think if you add up all of the learning and testing I did, probably closer to ~$50 total
4.
by dmakian 3y ago
> did you also try using weighted loss with Axolotl This is really smart, I didn't think about this! Will add it to my list of things to try, great idea! > Domain adaptation over subreddits/forums before finetuning may help
5.
by dmakian 3y ago
Still not clear maybe, I'm selecting players with a 62% lifetime win rate so mostly players who have been good over a larger number of drafts! Definitely not perfect data though, and agree that defining good in this context is hard --
6.
by dmakian 3y ago
There is not a lot of great content out there making this clear, but basically all that matters for basic fine tuning is how much VRAM you have -- since the 3090 / 4090 have 24GB VRAM they're both pretty decent fine tuning chips.
7.
by dmakian 3y ago
Only the statistics you see in the prompt (which are clearly limited). I have a lot of ideas about how you could improve that context (most likely letting the AI record and track notes throughout a draft), but this one was relatively simple
8.
by dmakian 3y ago
> Do you mean that you are looking at the draft picks from https://www.17lands.com/leaderboard and then sorting by Win Rate? Didn't you mean to choose Match Wins or Trophies? Otherwise, you're not measuring the
9.
by dmakian 3y ago
The two larger GPT-3.5 trials also got the card trivia examples, but like a bad scientist I don't have a great control group for those
10.
by dmakian 3y ago
> If I'm reading the author's writeup correctly, the prompt he's giving the agent at each pick contains only the names of the cards in its pool so far, and only gives the full text for the cards in the pack it's being
11.
by dmakian 3y ago
I've definitely thought about this problem and think it's in the range of 'feasible', but it would be pretty slow and expensive given how much context you need to provide a model for it to be able to reason about the gam
12.
by dmakian 3y ago
High level it's basically: 1. Generate a lot of text examples that look like this: https://gist.githubusercontent.com/davidhershey/f57d0b19563f... 2. The model is effectively trained to predict the next token base
13.
by dmakian 3y ago
> I was under the assumption that finetuneing LLMs was useful only when you need to change the model's tone (speak like a pirate, voldemort etc). A lot of why I tried this out was to test the limits of this belief, you see a lot of
14.
by dmakian 3y ago
Would love to compare notes, drop me a email at dshersh at umich dot edu if you'd be interested!
15.
by dmakian 3y ago
I hadn't seen this, this is awesome! You'd think given the volume of data available that this type of method would outperform an LLM, cool results. Still some fun things about LLM representations -- you can do fun things like give
16.
Fine-tuning Mistral 7B on Magic the Gathering Draft(generallyintelligent.substack.com)
356 points by dmakian 3y ago|100 comments
17.
How to Build a Feature Store(tecton.ai)
1 points by dmakian 5y ago|0 comments