7 ms·
A PhD student's perspective on research in NLP in the era of LLMs
- deleted 3y ago[deleted]
- dontupvoteme 3y agoAre papers becoming blogs?
- teruakohatu 3y agoIt has 10+ pages of references. More like a well researched lit review than a blog post with a few links.
- low_tech_love 3y agoYes, but putting a link to a Google Form on the abstract is a bit too much.
- whatyesaid 3y agoIt's arxiv, it's not a final version
- codethief 3y agoI'd say the culture of scientific publishing precedes blogs by "a few" decades. There have always been papers that were blog-like in nature.
- vpastore 3y agoOnly the one on arxiv. People shouldn’t take anything coming from that site seriously
- diffeomorphism 3y agoThat sounds highly misleading. It is the standard preprint site in many disciplines. So you should take it as seriously as any preprint and conversely if a published article is not on the arxiv, that would be suspicious.
- Frost1x 3y agoEh, there are plenty of actual pre-prints you can find on arXiv. Typical flow is you find a paper of interest published elsewhere then search arXiv for the same title, authors, etc. and can even find related work. A lot of publication isn't doing much in terms of genuine peer review but formatting, catching minor errors like grammar, typo, missing references etc. arXiv is great as a legitimate source for valid information tucked behind a paywall. It's not the only option but it's one of them (checking the primary author's personal site, researchgate, or associated research institutions repositories are also there, as well as contacting the author directly).
- 11101010001100 3y agoI extend the same courtesy to this site.
- maxov 3y agoWhat a strange recommendation. I do research in CS theory and machine learning and I try to find arXiv preprints when I can, they are usually more complete than conference versions of papers. If you stick to papers from authors you know or to those from well-known conferences, arXiv is often times the best source.
- infamouscow 3y agoSo the math, physics, and computer science departments of every major university should be ignored? What a brain dead comment.
- vpastore 3y agoSerious and trustworthy research is done on peer reviewed journals, not websites where everyone can post whatever he/she wants
- infamouscow 3y agoSerious and trustworthy according to whom, exactly? The term peer review was virtually non-existent prior to the 1960s. And despite that, nearly everything in modern society can ultimately be attributed to breakthroughs that happened prior to the advent of peer-review. https://books.google.com/ngrams/graph?content=peer+review&year_start=1800&year_end=2016&corpus=15&smoothing=3 https://books.google.com/ngrams/graph?content=peer+review&ye...
- vpastore 3y agoPrior to the 1960s there was no internet.
- infamouscow 3y agoI think you misunderstand how accessible the Internet was, and the history of open-access journals dating back to the late 40s. arXiv started in 1991. None of it explains why the peer review started surfacing in published books. It does correlate perfectly with when modern scholars point to when the institutions were captured.
- chaxor 3y agoThis is an extremely wrong viewpoint. "Many extremely important* papers have been published on arxiv alone - no other publisher. Just off the top of my head, without looking, ELMo from allenAI (the paper that started the contextualized representations revolution for BERT/attention) is only on arxiv.
- mschuetz 3y agoMany of the best papers appear an Arxiv first. In some fields, it is customary to put your preprint on Arxiv before/during the submission to the peer reviewed venue. Arxiv is vital for quickly developing research fields.
- SecretDreams 3y agoAlways have been.
- gumby 3y agoConsider the letters section of Phyicical Review, that turned into a weekly journal Physical Review Letters back in the 50s. It was a blog or forum back when papers were distributed on paper. Basically because of the slow pace of review and publication the letters column became a way to talk about recent results or problems, and then follow up letters (i.e. comments on the blog posts) became common. So the editors decided to hive it off and speed up its publication schedule.
- penguin_booze 3y agoYeah, wait for them to be twitter threads: 1/n - where (n >> 1).
- sdwr 3y agoThank **ing god! Hiding behind jargon and "the process" is an indicator of having nothing to say. I see this as a rolling up of the metaphorical sleeves, a sign that stuff is actually happening.
- rhdunn 3y agoI've been interested in NLP for tagging stories based on topics and themes (detectives, werewolves, murder mystery, etc.), so need accurate disambiguation of parts of speech and ways of detecting uses of metaphore, similies, etc. to describe those. I also want to be able to assess how much of the text is about a given topic, so that if I'm interested in reading a detective story from e.g. the Project Gutenberg collection, I don't want it to pick up a story where a detective is only mentioned in one paragraph. I've looked at several existing NLP frameworks (Open NLP, Stanford NLP) and none of them are accurate enough -- they fail on things like adjectives and old english second person pronouns. This makes them practically unusable for proper sense diambiguation, lemma and part of speech based rules, etc. The Open NLP tokenizer is also terrible at tokenizing title abbreviations ("Dr", etc.) and things like the use of "--" to delimit text, which is frequently found various Project Gutenberg texts. You can train the Open NLP tokenizer, but it works on what it has seen, so you need to give it every variation of "(Mr|Mrs|Miss|Ms|Rev|Dr|...). [A-Z]" for it to tokenize those titles; the same for other tokens.
- viksit 3y agoat this point the gpt api will do all of that for you! or you can use the embeddings api and build your own systems.
- rhdunn 3y agoThe problem with GPT and other LLMs is that they don't tokenize words at a word or morpheme level, it's just blocks of up to 4 characters, so you get tokens like `!"` instead of two separate tokens. -- That makes it harder to write custom tools on top of, unlike e.g. the output/model of things like the universaldependencies project.
- totorovirus 3y agowell I see topics like NLP in ethics, healthcare, etc, which I think is a sign they don't have much to do here.
- whatyesaid 3y agoYeah those aren't good especially stuff like healthcare a PhD student can't get the needed data. The other issue is, if you do focus on LLMs, it's too hyped your research would be too overlapping/competing especially as you've got a dissertation to write. It's a hard problem.
- deleted 3y ago[deleted]
- etamponi 3y agoIsn't the main problem with NLP research now that you'll need a ton of money to run your experiments? How can an "average" PhD researcher hope to validate their hypothesis if they need several thousand dollars per test?
- jasmer 3y agoMuch of the interesting pure research can be done at smaller scale, the larger models are arguably more product engineering than research. At least from a certain perspective.
- nl 3y agoNo. There are plenty of research directions that are outlined in this document that don't require huge compute budget.
- Randomizer42 3y agoThat is not a problem. There are literally thousands of ways to conduct any experiment at the fraction of the cost.
- LoganDark 3y agoThanks. Whenever I think I want to run something on a cluster of A100s, I'll just remember the thousands of ways instead, and train my LLMs that way. There's a reason why humans didn't need computers until just recently, after all. I must be so blind to not have seen this earlier.
- dcl 3y agoA PhD Student's Perspective... ~20 authors.
- SecretDreams 3y agoIt takes a whole village (of uncredited MASc students) to get a PhD!
- est31 3y agoThe particular headline pattern is used a lot and 99% of the time it indeed is at the front of opinion pieces written by individuals. But this time, it's different. In the syntax, in this specific form, the plural and the singular come out the same. E.g. in the sentence "I accompanied my friend to his parent's house", it can be either the house of his single mom/dad or the house of his two parents living together.
- bdsa 3y ago"I accompanied my friend to his parents' house"
- agalunar 3y agoLike @bdsa points out with their example, the singular and plural are actually spelled differently, viz. "parent's house" and "parents' house", despite being pronounced the same way. https://en.wikipedia.org/wiki/English_possessive#Nouns_and_noun_phrases https://en.wikipedia.org/wiki/English_possessive#Nouns_and_n...
- wahahah 3y agoI pronouce the latter as ~parent-siz; having them the same never sat right with me.
- est31 3y agoOh that's interesting, thanks! Not a native speaker but I dimly remember hearing about this in english class, thanks for reminding. I guess then the PhD student is indeed grammatically a singular then. It can still refer to a PhD student in general though instead of a particular one. https://www.rit.edu/ntid/sea/processes/articles/grammatical/types/specificgeneral https://www.rit.edu/ntid/sea/processes/articles/grammatical/...
- al__be__rt 3y agothis paper doesn't appear to have been edited by any reputable journal, so take its authenticity at face value...
- lgessler 3y agoI've got some bad news for you about how papers published at NLP conferences are reviewed.
- teleforce 3y agoAccording to the article, the original research on language model was kick started by Claude Shannon's early contributions in Markov chain model of English words. If you are in the field of Information and Communication Technology (ICT) there are hardly any area in the field which their fundamentals do not have Shannon's hands in it. Leonard Kleinrock once remarked that he has to focus on the exotic queuing theory field that later leads to the packet switching and then Internet because most of the fundamentals problems in electrical and computer engineering (older version of ICT) have already been solved by Shannon.