sacred_numbers

6 ms·

1.
by sacred_numbers 6d ago
The whole paradigm changes, though, when you can do daily cancer scans. You don't get a biopsy when the scan shows a lump. You get a biopsy after a couple weeks of daily scans showing the lump growing. Plus, having all the data from
2.
by sacred_numbers 3y ago
GPT-4 is not the same product. I know it seems like it due to the way they position 3.5 and 4 on the same page, but they are really quite separate things. When I signed up for ChatGPT plus I didn't even bother using 3.5 because I knew
3.
by sacred_numbers 3y ago
Based on my research, GPT-3.5 is likely significantly smaller than 70B parameters, so it would make sense that it's cheaper to run. My guess is that OpenAI significantly overtrained GPT-3.5 to get as small a model as possible to optim
4.
by sacred_numbers 3y ago
The reason cement is a major contributor to CO2 emissions is because of how much cement we produce. I don't know the lifetime or effectiveness of this catalyst, but typically you only need a tiny amount of catalyst to start a reaction
5.
by sacred_numbers 3y ago
Yes, with 4 bit quantization.
6.
by sacred_numbers 3y ago
We could not do that inadvertently. To block even 1% of light using Starlink sized satellites (~30 m^2 with solar panels deployed) would require tens of billions of satellites. We could do it on purpose with huge rotating solar reflectors,
7.
by sacred_numbers 3y ago
The quality difference is substantial. I don't care if it's wasteful to use something that has many uses for a supposedly narrow task (although I don't see translation as a particularly narrow task anymore than I see writing
8.
by sacred_numbers 3y ago
It's worse on English and a lot of other common languages (see Appendix C of the paper). It does better on less common languages like Latvian or Tajik, though.
9.
by sacred_numbers 3y ago
There will always be overhead, but that doesn't mean it will always be a huge amount of overhead. I believe the state of the art is 97% efficiency ( https://www.osti.gov/biblio/1495980 ) which is better than a lot o
10.
by sacred_numbers 3y ago
I did my own calculations based on plotting loss on benchmarks compared to models with known parameters and training data, as well as using a quote from Sam Altman that said that GPT-4 would not use very many more parameters than GPT-3. Ba
11.
by sacred_numbers 3y ago
I would bet money against that. Replicating GPT-4 pre-training with current hardware would cost about 40-50m in compute. Compute will continue to decrease in cost and algorithmic improvements may allow for more efficient training, but proba
12.
by sacred_numbers 3y ago
They do update the model in the background, although I'm not sure how often or how much they update it. To avoid issues with this practice they offer gpt-4-0314 which says this in the documentation: "Snapshot of gpt-4 from March
13.
by sacred_numbers 3y ago
It's unlikely that OSS LLMs will ever be able to compete with corporate LLMs. I can only think of a few scenarios where this could work: 1. Someone develops a procedure for training models with distributed computing resources, includi
14.
by sacred_numbers 3y ago
I can think of a few ways: 1. The ChatGPT web search plugin becomes standard protocol for every prompt. If you ask a factual question ChatGPT will first look up an answer with a search engine, then use the results to craft an answer. This
15.
by sacred_numbers 3y ago
If you bought an 8xA100 machine for $140k you would have to run it continuously for over 10,000 hours (about 14 months) to train the 7B model. By that time the value of the A100s you bought would have depreciated substantially; especially
16.
by sacred_numbers 3y ago
When I checked yesterday I believe the signature said OpenAI CEO Sam Altman, so it was definitely a joke signature, not a case of two people having the same name.
17.
by sacred_numbers 3y ago
The Reflexion paper ( https://arxiv.org/abs/2303.11366 ) that came out recently shows how this kind of mistake might be overcome. Asking the model to think about the answer after it's generated a first draft greatly
18.
by sacred_numbers 3y ago
Theoretically it should be way less energy intensive as well, since there won't be an animal expending energy to live for months before slaughter. Nor will there be a need to grow feathers, bones, or blood that end up as byproducts. Of
19.
by sacred_numbers 4y ago
When you are speaking to a person, they have inner thoughts and outer actions/words. If a person sees a chess board they will either consciously or unconsciously evaluate all the legal moves available to them and then choose one. An L
20.
by sacred_numbers 4y ago
Alternatively: 1. Quickly reduce costs by increasing model and computation efficiency. 2. Massively reduce prices while still maintaining some gross margin. 3. Massively increase market size and take the vast majority of market share. 4. En
21.
by sacred_numbers 4y ago
It could be even smaller than a Chinchilla optimal model. The Chinchilla paper was about training the most capable models with the least training compute. If you are optimizing for capability and inference compute you can "over-train
22.
by sacred_numbers 4y ago
We have, but it's not a single process. We can convert light to electricity quite cheaply and efficiently with solar PV panels and then use that electricity to electrolyze hydrogen from water and capture CO2 from air(or seawater). Then
23.
by sacred_numbers 4y ago
Vinyl chloride, when burned, can create poisonous byproducts such as phosgene and carbon monoxide. Vinyl chloride that leaks into the environment is a carcinogen that can cause damage decades into the future. It's a tradeoff, but proba
24.
by sacred_numbers 4y ago
I think the biggest reason for Tesla's gross margins is that millions of people want EVs for various reasons (gas prices, environmental concerns, fun, status) and Tesla is one of the only companies making them in large quantities. They
25.
by sacred_numbers 4y ago
Surprisingly it appears not to be too far off standard solar panel efficiencies. According to this source[0], five nines silicon (5N) is called Upgraded Mettalurgical-grade (UMG) silicon. According to this paper[1], efficiencies over 20% ha
26.
by sacred_numbers 4y ago
Unfortunately I think you're off by an order of magnitude. I think it would be 810 Kilojoules, which is approximately equivalent to a 1kg lithium-ion battery. Of course, you could move thousands of rocks up and down a big crater, rath
27.
by sacred_numbers 4y ago
Density in tilted installations is quite bad. If you want to capture morning and evening sun at an optimal angle you have to space the panels out a lot, like 5-10 panel heights. You can have them closer, but then you get shading, which defe
28.
by sacred_numbers 4y ago
Based on my calculations, at my latitude (40 degrees North), you would need about 16% more panels to generate an equivalent amount of energy per year. This isn't taking into account potential issues with snow buildup (which theoretical
29.
by sacred_numbers 4y ago
There is a small loss of efficiency (<2%) at extreme angles, but it is not too significant[1] [1]: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6611928/#:~:tex... .
30.
by sacred_numbers 4y ago
I think the reason hydrogen storage costs won't fall much is because the cheapest technology (metal tanks) have already benefited from economies of scale. The parts that make them suitable for hydrogen storage specifically will get che

More ›