7 ms·
Why is GPT-3 15.77x more expensive for certain languages?
- rubywilde 3y agoActually, it is not true. Hilarious Author compares different encoders: for Facebook's NLLB and GPT2. Where did title came from? Another point is that OpenAI changed encoders for chat models. Link: https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb https://github.com/openai/openai-cookbook/blob/main/examples... Now English is less optimized for tokens usage and other languages are much more balanced. E.g. Ukrainian takes only twice as much tokens, before it had 6 times more tokens
- galaxytachyon 3y agoSo what I got from this is that GPT was trained on a dataset that biased in English contents. Is that right? I think even human has to spend extra energy to speak a language they were not born with, no matter how fluent they are in this language. I don't know about natural multilinguals.
- kevingadd 3y agoShould the cost really be 15x? Or even 5x? In this case, it's not even a question of whether the network is better at English, it's that the cost to communicate with it at all in other languages is higher. Once you pay that cost you now have to deal with the network potentially generating lower quality results for prompts in non-English languages too, which raises the actual cost of doing something with GPT beyond 15x since you probably will need more attempts.
- viscanti 3y agoBecause there's so much more English language for them to train on relative to most other languages, they're able to do some optimizations for English that they can't elsewhere. Should they not be able to implement optimizations for cases where they have the data volume to do so?
- kevingadd 3y ago"there's so much more English language for them to train on relative to most other languages" is an interesting assertion. There are billions of people on earth speaking languages other than English and they have access to the internet. Are you sure it's not just the case that we didn't scrape that data? Everyone has to choose what data to train on, you can't train against The Entire Internet, it's a limitless amount of data. But it becomes an intentional choice with consequences, like the 15.77x seen here.
- JCharante 3y agobut training against the entire Internet would still be biased towards English because English is the dominant language used on the Internet.
- MichaelZuo 3y ago> Everyone has to choose what data to train on, you can't train against The Entire Internet, it's a limitless amount of data. Isn't that exactly how OpenAI managed to 10x GPT 3.5 with GPT 4.0?
- famouswaffles 3y agoBoth of you are kind of misunderstanding a few things. Data used to train the tokenizer is entirely separate from data training the LLM. The tokenizer used to train GPT-3 was old, inefficient and targeted at tokenizing English. That's pretty much all there is to it. It's possible to train a tokenizer that is more efficient and more including of other languages. GPT-4's tokenizer is already far more efficient though still weighted to English.
- viscanti 3y ago> GPT-4's tokenizer is already far more efficient though still weighted to English. Right. It's a general question. Should they be allowed to take the kinds of optimizations they can with tokenization when it's a function of how much data they can use, even if that means some languages get more optimization than others? Or should users of those languages that could be optimized effectively pay a tax out of some sense of fairness?
- user_named 3y agoWhat makes you think there's a "should"?
- cool_dude85 3y agoThere's always a should. Society gets a say in what people and corporations can and can't do in (at the very least) the form of laws. There's your should right there.
- pixl97 3y agoWhich society? I mean OpanAI is a US company is unsurprisingly going to mostly communicate in English. Are we counting all societies, if so should our software cater to all their demands, language and/or culturally demanded?
- famouswaffles 3y agoBoth of you are kind of misunderstanding a few things. Data used to train the tokenizer is entirely separate from data training the LLM. The tokenizer used to train GPT-3 was old, inefficient and targeted at tokenizing English. That's pretty much all there is to it. It's possible to train a tokenizer that is more efficient and more including of other languages. GPT-4's tokenizer is already far more efficient though still weighted to English.
- terafo 3y agoNope, it's not about dataset. It's just bad tokenizer. Korean has couple of dozen of symbols in it's alphabet. Cyrillic languages have less than 50 symbols in total. Hiragana is 46 symbols. GPT-4 has 32k tokens IIRC. Including most significant alphabets would take less than a thousand.
- moelf 3y agobut the tokenizer is dataset-driven... it tokenize the most common pattern in your dataset to improve efficiency, so it's 100% about dataset?
- sillysaurusx 3y agoThere's dataset during training, and dataset for the tokenizer. The confusion here is that people are talking about the former, but you're correct that it's the latter. Remember, OpenAI's tokenizer was created in an era when 125MB was considered large for a language model. It's hard to fault them for making something that lasted four or five years.
- dragonwriter 3y ago> Remember, OpenAI’s tokenizer was created in an era when 125MB was considered large for a language model. GPT-2 and GPT-3 have different vocabularies and maximum token #s, which (even if the tokenizer architecture is the same) implies a different tokenizer model. GPT-3.5 might share the GPT-3 tokenizer, but even then I’d expect GPT-4 to have its own. But even if they are using the tokenizer from GPT-3, its not from “an era when 125MB was considered large for a language model”.
- sillysaurusx 3y agoActually, GPT-3's tokenizer is the same as GPT-2. https://datascience.stackexchange.com/a/109483 https://datascience.stackexchange.com/a/109483 You had me questioning myself for a minute. (The vocab size is still 50257. Even rounded up to a multiple of 128 for better sharding across the vocab embedding, only the first 50257 are used.) Believe it or not, 125M was large at the start of the GPT-2 era. No one knew LLMs could do anything interesting, let alone that they'd change the world.
- tlrobinson 3y agoI think yes, but more precisely the tokens were chosen to optimize training on a dataset that's biased to English content. I am curious how the token set affects quality of responses, ignoring the factors related to token count mentioned in the post (cost, prompt expressivity, latency, etc) Is it always better for the token set to be "native" to the majority of the training dataset and prompts/completions, or is it possible there's some "intermediate representation" (in compiler terms) that would be better?
- k8si 3y agoI don't know what you mean by compiler terms but basically, worse tokenizer = worse LM performance. This is because worse tokenizer means more tokens per sentence so it takes more FLOPs to train on each sentence, on average. So given a fixed training budget, English essentially gets more "learning per token" than other languages.
- GaggiX 3y agoThe article only clarifies that the dataset used to train the tokenizer is baised, not the entire dataset used by the GPT model.
- famouswaffles 3y agoData used to train the tokenizer is entirely separate from data training the LLM. The tokenizer used to train GPT-3 was old, inefficient and targeted at tokenizing English. That's pretty much all there is to it. It's possible to train a tokenizer that is more efficient and more including of other languages. GPT-4's tokenizer is already far more efficient though still weighted to English. You can test it here https://tiktokenizer.vercel.app/ https://tiktokenizer.vercel.app/
- seba_dos1 3y agotl;dr - because it operates on tokens, not words, and the set of tokens it uses is optimized for representing English text.
- hadlock 3y agoI would imagine they have far more english optimized compute instances running
- H8crilA 3y agoThere are no separate instances for different languages. ChatGPT is inherently multilingual.
- shagie 3y agoDo other languages have as nice a mapping to tokens? For example, if you were to go from French, you'd have 33 characters to work with rather than 26 (accents such). And you'd have chemisier and chemisière being two different genders of the same word that are used in different contexts. English tends to not have this difference. Likewise, French has more verb conjugation forms than English does. If you were to go to Japanese, you'd have the hiragana, katakana and kanji. While my Anglocentrism may be showing, I'm not sure there is another language that tokenizes as well when it comes to novel character combinations. Make up a new word. Use it in a setence. Give a definition for it. My new word is 'diflubble'. It is the feeling one gets when they are both excited and nervous in anticipation of an upcoming event. For example, I felt diflubble on the morning of my graduation ceremony. vs: Make up a new word in Japanese. Use it in a setence and give a translation for it. Give a definition for it. My new Japanese word is "keigarou", which means "being full of energy". 例えば、私は今日、keigarouな気持ちでいます。 Translation: For example, I am feeling keigarou today. The thing there is that you can't just make up new kanji. And it wouldn't be hiragana either.
- wolfium3 3y agoYou can use their online tool to see how it tokenizes words: https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer
- minimaxir 3y agoIt's worth noting that this only for GPT-3. If you're using ChatGPT or GPT-4, both use a different tokenizer that's more robust and uses/generates about 10% fewer tokens. (unclear how well it performs for non-English languages) You can test it offline using tiktoken: https://github.com/openai/tiktoken https://github.com/openai/tiktoken
- dchest 3y agoHere's online version: https://tiktokenizer.vercel.app/ https://tiktokenizer.vercel.app/
- sillysaurusx 3y ago10% smaller vocab size, or 10% fewer tokens on average? I assume the latter, but total vocab size is also an interesting metric. The tokenization speedups in that repo are very impressive. It was the most annoying part about processing 190,000 books. I think it took a few days on a server with 96 cores. Surprisingly hard to figure out the vocab size from that repo.
- minimaxir 3y ago10% fewer tokens on average. The vocab size itself is doubled. (~50k for GPT-2/3, ~100k for ChatGPT)
- sillysaurusx 3y agoWow. Does that help to double the vocab size? It certainly makes training more expensive. One clever trick to get some memory savings is to freeze the vocab embedding layer when fine tuning. It makes a noticeable improvement, both in speed and in mem required. Surprised they went the larger vocab route. LLaMA is only 30k. I wonder what the reason is... Thanks!
- 29athrowaway 3y agoIt is not that tokenization is optimized for English, but rather the other way around perhaps. Take "lampara" or "pantalones" in Spanish for example. English speakers were clever enough to shorten those words to "lamp" and "pants" respectively. And they have done this with many words. Translate text into Spanish and you will see text gets longer and there is more meaning encoded into words. "La mesa" refers to a female table, although tables are not lifeforms and have no sex. To me some languages impose a communication tax. It is taboo because people conflate language and culture and such.
- cool_dude85 3y agoOne wonders whether highly agglutinative languages, then, might have even better performance than English in the tokenizer since they can pack much more meaning into a single word. The linked article shows one such language, Malayalam, costing 15.7 times more. Try again.
- kevingadd 3y agoIf you familiarize yourself with ideographic/ideographic-adjacent languages like Japanese or Chinese you will probably notice that they are way more efficient than English. Yet those languages pay a tokenization tax too (thanks in no small part to the decisions of the largely western Unicode committees to favor western character sets - the UTF8 encoding favors ASCII tremendously)
- mcswell 3y agoI usually use the term "tokenization" to refer to breaking a text into "words" (tokens), although in the examples shown in the article, for the Latin script languages it seems to be doing tokenization into something like morphemes. This has nothing to do with the Unicode UTF-8 encoding system; Hindi would have the same number of tokens if you encode it with UTF-8 (where each character is 3 bytes) or ISCII (where each character is 1 byte). But when it comes to Chinese...something weird is going on.
- kevingadd 3y ago
- kouteiheika 3y agoSlightly offtopic, but: > One of the models listed above called NLLB (No Language Left Behind) has been open sourced by Facebook allowing for translation for 200 languages. It was not. The model's weights are under CC-BY-NC, which certainly motivates commercial entities to not leave those languages behind. /s
- adsfoiu1 3y agoIt was open sourced, just under a non-commercial license.
- teddyh 3y agoNon-commercial licenses are not Open Source: https://opensource.org/osd/ https://opensource.org/osd/
- voxic11 3y ago> The program must include source code, and must allow distribution in source code as well as compiled form. Its a model not a program. So that definition can't possibly apply to it no matter the license.
- kube-system 3y ago"open source" has a broader English meaning that predates the OSI for at least several decades. OSI does not have a trademark on "open source" because of this. This is the software licensing world's version of "a hotdog is not a sandwich"
- morelisp 3y ago> "open source" has a broader English meaning that predates the OSI for at least several decades. It is also not data collected by an intelligence agency or law enforcement from public sources.
- 3y ago
- FredPret 3y agoWhat an interesting aspect I haven't considered before. All the AIs will be trained on the available media - most of which is English. I sometimes wonder what it takes to unseat a lingua franca, but it looks like we won't see that soon. English is set to dominate for a long time.
- lucb1e 3y ago> All the AIs will be trained on the available media - most of which is English. Is it?
- generalizations 3y agoThe pile is an open dataset, and so is libgen. Should be pretty easy to confirm.
- kevingadd 3y agoThere's some nuance to this, I think. For one arbitrary example where this might not hold: NovelAI was trained on data from 'danbooru', an imageboard where people repost and tag art. All the tagging on that site is in English and they frequently also translate things like the author's description of the image and any in-image text. So if you were to use that site as a dataset, it would all be English. Or is it? The original source content was in a mix of languages - english, japanese, chinese, korean, etc. It only then got translated into english and tagged in english. So if you had trained on the original source content, you would have been training on a mix of languages, but that got erased for the convenience of the people training the network.
- unaindz 3y agoEven if the original images have a mix of languages I think the tagging is all done in english (I may be wrong). I would argue that the source material includes the tagging as it is necessary for the AI to get trained so the content is not really mixed but entirely english. But anyways the danbooru tags consist of things like: short hair, blue eyes, portrait. Things that are much more easier to translate (or "understand") in several languages than entire phrases like GPT does.
- karmoka 3y ago"Je voudrais une pizza" is better translated to "I would like a pizza" "I want a pizza" would be "je veux une pizza"
- idleproc 3y ago"J’aimerais..." is better translated to "I would like...", n'est-ce pas?
- ahoef 3y agoDepends. "Amerais" emphasises the liking of the pizza and "voudrais" emphasises getting the pizza.
- karmoka 3y agoBoth "je voudrais" and "j'aimerais" translate to "i would like", albeit with some nuances in the connotations. The later has more of a wishful quality, more open to rejection. In spoken form, they're mostly interchangeables.
- deleted 3y ago[deleted]
- Imnimo 3y agoSetting aside the specific choice of tokenizer for GPT models, I'm curious how much difference in performance is made by the features of the human language used to represent the training data. Like if you kept the exact same training corpus and could wave a magic wand and translate it into any language and could create a custom tokenization for each language, would some be more amenable than others to GPT-style language modeling?
- bob1029 3y agoIf you think about this from a "language is computation" perspective, it starts to get even more interesting. For example, what would the real-world performance of ChatGPT be if we had trained it predominantly on German or Korean text? Is English actually the best language/structure for this system?
- biztos 3y agoMaybe there’s a competitive advantage in training a new one on just German, say, and unleashing it on automotive engineering problems?
- DonHopkins 3y agoI'm afraid the Germans might weaponize it to come up with deadly funny jokes! https://www.youtube.com/watch?v=Qklvh5Cp_Bs https://www.youtube.com/watch?v=Qklvh5Cp_Bs
- wordpad25 3y agoHUGE SALE! Save 93% OFF on GPT API by translating prompt into English first!!!
- lukeschlather 3y agoI would want to see some data on tokenization for some real-world examples. "Je voudrais une pizza" actually translates more directly to "I would like a pizza" which is 5 tokens. But also I think there's some danger here in terms of this might be cherrypicking examples. Spanish is a lot more dense than English or French and might tokenize better. (I see "quiero pizza" is 4 tokens which seems like the right number of tokens to me - "quiero" actually contains "I want <present tense>") You could argue it's 2 or 3 tokens but 4 seems preferable. For diacratics in French or Spanish, diacratics are logically characters. I can't think of an example where it's actually useful to split the letter into a different token but I could see it happening and not being harmful. I do think it's possible French is just weird and just needs more tokens. When I think about how I process French, I probably do treat e.g. "Je l'ai aimé" as a pathological example as 3 tokens when I speak it out loud. But I can also see why you would tokenize it as 6 tokens, I'm not sure that's Anglocentrism so much as it's recognizing a complexity difference between French and English writing. But all this is contrast to how non-roman characters are tokenized at the byte level. That just seems bad and like it's definitely going to make it worse with non-roman languages. There's no point in having tokens that split characters.
- tough 3y agoI just found this tiktokenizer project in GH that might be of help to you https://tiktokenizer.vercel.app/ https://tiktokenizer.vercel.app/
- function_seven 3y ago> Spanish is a lot more dense than English or French and might tokenize better. I'm no linguist, so I apologize if I'm misinterpreting this statement. My impression has always been that Spanish is less dense than English, only because in almost all cases, the Spanish version of product instructions is wordier. Look at the back of a shampoo bottle[0] and notice that the Spanish version is either longer, or a smaller font, to fit it all. [0] https://i.postimg.cc/xd2X5WJN/Ghub-Fo-N11u8jz-Pjj-RDt-W-CGA9-Wr073qcu-QZx8-Sp-N-5k.jpg https://i.postimg.cc/xd2X5WJN/Ghub-Fo-N11u8jz-Pjj-RDt-W-CGA9...
- 3y ago
- mgaunard 3y agoSo for latin languages, they tokenize per word, and somehow for asian languages, it's tokenizing per radical. Of course you'd end up with a lot more tokens. Just tokenize by word regardless of language.
- crazygringo 3y agoWords aren't an equivalent count between languages either. English uses a lot of helper words, some other languages use multiple suffixes. Chinese characters don't even make it clear where "word" boundaries are -- there are no spaces.
- mgaunard 3y agoChinese does make it explicit where word boundaries are. The only language that doesn't is Thai, but there are still well-documented algorithms for it.
- crazygringo 3y agoHow does it make it explicit? You need a dictionary to figure it out, no? Same as e.g. Japanese?
- kccqzy 3y agoRight but such dictionaries are already built in to all major operating systems. The double-click-to-select-word interaction works well with Chinese and Japanese in all major operating systems. Without such dictionaries you can't even implement word selection.
- fomine3 3y agoIt works until it recognizes 外国人参政権 as foreign/carrot/regime
- biztos 3y agoReally, only Thai? Is there a reference for that? A quick search suggests it’s not the case, but I’m no expert. As a lowly beginner I find the lack of word boundaries in Thai frustrating but I think it’s just that I have not yet learned to think in syllables, I’m still always sounding them out in my head until I have a word I recognize, there’s no flow. This seems like something the LLMs should be very good at. Google Translate does OK-ish while Apple just throws up its hands in frustration and refuses to translate Thai texts.
- startupsfail 3y agoI’m finding it amazing that the model comes localized and supports obscure languages and is available. Compare this to traditional software. Or even to web software. Does Google come localized to all of these languages, for example? Yes, there is overhead from localization. So what, this overhead was always there for software.
- jinushaun 3y agoThe French example is strange and shows that the language model has an English bias. - “I want a pizza” = 4 tokens - “Je voudrais une pizza” = 7 tokens Why is “want” only 1 token in English, but “voudrais” 4 tokens? Following the French example, would “wants” and “wanted” map to 1 or two tokens?
- HDMI_Cable 3y agoI think it’s because the article itself is a bit wrong: ‘voudrais’ in French is more analogous to ‘I would like’ in English than ‘want’. Specifically, the ‘v-‘ indicates that this means ‘to want’, ‘-oud-‘ means that it is in the conditional or future, while ‘-ais’ would indicate its first person conditional. This being said, it makes sense ‘voudrais’ is more tokens than ‘want’, because it encodes more information.
- FrostKiwi 3y agoSo glad someone took the time to put up some data about it. Since day one, the subpar results for Asian languages has stuck out to me. It's especially true for LLama-derived models, where the output is just abysmal. It's my own pet theory, that bad tokenization is an important reason as to why they suck so much in the first place. It's not just broken grammar, it's a surprising lack of creativity, that English doesn't suffer from. ChatGPT English -> DeepL and fixing the auto-translation gives vastly improved results, than prompting ChatGPT to respond in an asian language.
- lee101 3y ago[dead]