4 ms·

> We do the finetuning only on small semantic data were it helps a lot. This sounds interesting. Would you care to expand a bit on how you do this? What is thi

by runeks 24d ago

> We do the finetuning only on small semantic data were it helps a lot. This sounds interesting. Would you care to expand a bit on how you do this? What is this semantic data? > There are companies though which ahve this exact problem with programming languages you normally don't see. This applies to my company. We have a substantial code base in our own dialect of APL. You think post-training would help substantially here?

Zylokloto 24d ago
So we have free text paragraphs which describe machines. Our model detects if the machine in question is of the main category we are looking for, then our finetuned model extracts from the free form text semantic information about the machine. Like color, features, horsepower etc. This would normally take quite a long time to do manually but we already had the semantic version of these texts because the company was doing this for a while. We now use gemma or Qwen (we regularly re-finetune the newest models to just see if they get better and they actually do) and then use these finetuned models to save us a lot of time. ---- If I had a coding language which isn't available much online and coding LLMs are bad on it, I would definitly try to finetune this but its defintily a lot more work than just doing what I explained above. Depending on what your usage of this APL Dialect is, it might be easier to fine tune it to migrate from your APL to something a lot more common. If this is not an option at all: You need to start creating data for the finetuning. You need a few hundred up to a few thousand of them in different formats like Q & A pairs. Documentation, syntax, a lot of diverse small examples. The intersting thing about this data generation: You can either leverage, to a certain degree , what you already have, or you start collecting them through your team/work collegues or you do them by hand. You can put the text into RAG and experiment with context engineering until a LLM is 'good enough' in it to be able to help you generating examples for and with you. Like you give an LLM all the relevant context for it, then you let it generate pairs: { "instruction": "Write an expression to find the maximum along the rows of a 2D array.", "input": "Array matrix: A", "dialect_notes": "Custom dialect uses ⌆ (max-reduce) and ⌥ for axis specification instead of /[1].", "output": "⌆ ⌥2 A" } (I have no clue about APL this is just a random example I asked an LLM to generate). You might have luck and finding communities with the same issue you face.

by runeks 24d ago

Zylokloto 24d ago

So we have free text paragraphs which describe machines. Our model detects if the machine in question is of the main category we are looking for, then our finetuned model extracts from the free form text semantic information about the machine. Like color, features, horsepower etc. This would normally take quite a long time to do manually but we already had the semantic version of these texts because the company was doing this for a while. We now use gemma or Qwen (we regularly re-finetune the newest models to just see if they get better and they actually do) and then use these finetuned models to save us a lot of time. ---- If I had a coding language which isn't available much online and coding LLMs are bad on it, I would definitly try to finetune this but its defintily a lot more work than just doing what I explained above. Depending on what your usage of this APL Dialect is, it might be easier to fine tune it to migrate from your APL to something a lot more common. If this is not an option at all: You need to start creating data for the finetuning. You need a few hundred up to a few thousand of them in different formats like Q & A pairs. Documentation, syntax, a lot of diverse small examples. The intersting thing about this data generation: You can either leverage, to a certain degree , what you already have, or you start collecting them through your team/work collegues or you do them by hand. You can put the text into RAG and experiment with context engineering until a LLM is 'good enough' in it to be able to help you generating examples for and with you. Like you give an LLM all the relevant context for it, then you let it generate pairs: { "instruction": "Write an expression to find the maximum along the rows of a 2D array.", "input": "Array matrix: A", "dialect_notes": "Custom dialect uses ⌆ (max-reduce) and ⌥ for axis specification instead of /[1].", "output": "⌆ ⌥2 A" } (I have no clue about APL this is just a random example I asked an LLM to generate). You might have luck and finding communities with the same issue you face.