4 ms·

Congratulation, great paper! It should have been put on HN earlier ;) I have a few questions: * you say (page 4): "We then perform standard instruction finetu

Congratulation, great paper! It should have been put on HN earlier ;) I have a few questions: * you say (page 4): "We then perform standard instruction finetuning on the base LLaMA-7B model" Could you perhaps provide a reference to the _exact_ finetuning approach you used? I'm afraid different groups of people have a different notion of "standart" (see for example pages 131-155 from https://arxiv.org/abs/2302.08575 https://arxiv.org/abs/2302.08575 for various fine-tuning approaches) and without knowing exactly how fine-tuning was carried out, it can be very difficult reproduce your research and results exactly. * the idea of using AST Sub-Tree Matching is nice. Could you please let me know which function in which file from your GitHub repository this is implemented in? Again, great job on publishing this paper! --- Best regards, friederrr.org

data_maan 3y ago
Seems @shishirpatil ran out of steam answering questions. Too bad.
- data_maan 3y ago
  (Or maybe the questions were too tricky and he wasn't able to answer, heh)
  - shishirpatil 3y ago
    Haha, was busy yesterday! Or was I? :P
shishirpatil 3y ago
Thanks @sfriedr We generate self-instruct data and then fine tune the base model with perplexity loss. The self-instruct data is https://github.com/ShishirPatil/gorilla/tree/main/data/apibench https://github.com/ShishirPatil/gorilla/tree/main/data/apibe... Thank you! Yes, the code can be found here: https://github.com/ShishirPatil/gorilla/tree/main/eval/eval-scripts https://github.com/ShishirPatil/gorilla/tree/main/eval/eval-... Hope this helps. Let me know if you have any follow-ups!
- data_maan 3y ago
  Awesome, thanks for letting me know! I'm still not sure though about some nitpicky things: - do you change all the weights, or just the ones from the last layer when fine-tuning? - do you just train on the _code_ field from the JSON file with the self-instruct data, or do you also use the other fields to train (or do you use the other fields just for downstream evaluation purposes)? I think it could be a major selling point of your paper if on Github (or in an appendix to your preprint, if you update it on arxiv), you had a section where you document the training process in detail
  - data_maan 3y ago
    (whoops, this comment/questions should have been to as an answer to your other comment @shishirpatil)

data_maan 3y ago

Seems @shishirpatil ran out of steam answering questions. Too bad.

data_maan 3y ago
(Or maybe the questions were too tricky and he wasn't able to answer, heh)
- shishirpatil 3y ago
  Haha, was busy yesterday! Or was I? :P

shishirpatil 3y ago

Thanks @sfriedr We generate self-instruct data and then fine tune the base model with perplexity loss. The self-instruct data is https://github.com/ShishirPatil/gorilla/tree/main/data/apibench https://github.com/ShishirPatil/gorilla/tree/main/data/apibe... Thank you! Yes, the code can be found here: https://github.com/ShishirPatil/gorilla/tree/main/eval/eval-scripts https://github.com/ShishirPatil/gorilla/tree/main/eval/eval-... Hope this helps. Let me know if you have any follow-ups!

data_maan 3y ago
Awesome, thanks for letting me know! I'm still not sure though about some nitpicky things: - do you change all the weights, or just the ones from the last layer when fine-tuning? - do you just train on the _code_ field from the JSON file with the self-instruct data, or do you also use the other fields to train (or do you use the other fields just for downstream evaluation purposes)? I think it could be a major selling point of your paper if on Github (or in an appendix to your preprint, if you update it on arxiv), you had a section where you document the training process in detail
- data_maan 3y ago
  (whoops, this comment/questions should have been to as an answer to your other comment @shishirpatil)