6 ms·

Reading the comments in this thread, i honestly dont get it. 5.6-sol has felt like a regression in capability. In fact, every model since 5.3-codex has been a r

Reading the comments in this thread, i honestly dont get it. 5.6-sol has felt like a regression in capability. In fact, every model since 5.3-codex has been a regression from OpenAI. I just find 5.6-Sol over engineers problems, takes absolutely ages to solve basic problems.... At this point, I'm considering going back to cursor over codex due to the ability to get more control over what model I use since there is clearly a heap of user preference and having frontier providers constantly shift the goal post with "State of the Art" is complete non-sense.

tonyhart7 1mo ago
its over engineered problem solver ???? well because its a designed to do that if you want to solve basic problem then use Luna
- jeffybefffy519 29d ago
  I mean it added additional changes when it doesnt need to. Its basically hallucinating changes it thinks it needs to make regardless of effort levels i try.
SadErn 1mo ago
[dead]
jeswin 1mo ago
It depends on what effort you're using etc. As an example [1] of what codex is capable of, here's hugo (written in golang) ported to TypeScript - and then a TypeScript to Rust transpiler which converts arbitrary TypeScript into Rust. The TypeScript code which was transpiled into Rust (and is compatible with most hugo templates) runs faster than the original hugo. [1]: https://github.com/tsoniclang/tsonic-examples/tree/main/rust/tsumo/generated/packages https://github.com/tsoniclang/tsonic-examples/tree/main/rust... The transpiler is still WIP, but the fact that it can do this says a lot of about how far LLMs have come.
- jeffybefffy519 29d ago
  I find all effort levels of sol are the same in terms of amount of hallucinated unnecessary changes. Luna is much better all round on xhigh but my point still stands, every release of these new models is not an upgrade, its re-learning how to work with it. Its like rehiring an employee every few months then training them up. Its honestly tiring and cant stay like this. Opus has the same problem too…
  - jeswin 29d ago
    That's not been my experience. My prompting methods haven't changed much between recent GPT releases. I do put a lot of effort into building tooling and tests around a project, so the LLM output is converging around it.
    - jeffybefffy519 29d ago
      Can you give examples of the tooling and tests?
      - jeswin 29d ago
        Sure. This is the csharp target for tsonic (a TypeScript to C#/Rust/Python/Triton transpiler). It has a bunch of tests here: https://github.com/tsoniclang/tsonic-csharp/tree/main/test https://github.com/tsoniclang/tsonic-csharp/tree/main/test More comprehensive e2e proving grounds are at 1: https://github.com/tsoniclang/proof-is-in-the-pudding https://github.com/tsoniclang/proof-is-in-the-pudding 2: https://github.com/tsoniclang/tsumo/ https://github.com/tsoniclang/tsumo/ They were built specifically for testing the C# target. There are several other large projects we built specifically for e2e testing. But more interesting would be the tooling built to support this. For example, our current TypeScript parser [1] is a file-by-file port of Microsoft's TypeScript V7 compiler written in golang. The challenge here is that every time Microsoft changes code, we'll have to fix our code and tests. It's doable, but a fair amount of work. So we decided to write tooling to transpile Microsoft's v7 compiler from golang, and autogenerate our compiler. That tool is called gotots [2] - and it already produces a fully working TypeScript compiler. It's 3x slower than TypeScript v6 compiler, but we hope to get to rough performance parity in a week or so. Everytime Microsoft makes an update, we run gotots and our parser gets updated as well. [1]: The old parser - https://github.com/tsoniclang/tsts-legacy https://github.com/tsoniclang/tsts-legacy [2]: Golang to TypeScript transpiler - https://github.com/tsoniclang/gotots https://github.com/tsoniclang/gotots My general point is that tests and tooling is tremendous value, and they are guardrails for LLMs to converge. I could have, for example, chosen not to write the go-to-ts transpiler, and live with porting Microsoft's parser line by line. But making such tools is something LLMs are good at, so it's a tradeoff well worth making. And the upside is that you don't have to use LLMs to port Microsoft's parser/compiler (a large and complex project) line by line.

tonyhart7 1mo ago

its over engineered problem solver ???? well because its a designed to do that if you want to solve basic problem then use Luna

jeffybefffy519 29d ago
I mean it added additional changes when it doesnt need to. Its basically hallucinating changes it thinks it needs to make regardless of effort levels i try.

SadErn 1mo ago

[dead]

jeswin 1mo ago

It depends on what effort you're using etc. As an example [1] of what codex is capable of, here's hugo (written in golang) ported to TypeScript - and then a TypeScript to Rust transpiler which converts arbitrary TypeScript into Rust. The TypeScript code which was transpiled into Rust (and is compatible with most hugo templates) runs faster than the original hugo. [1]: https://github.com/tsoniclang/tsonic-examples/tree/main/rust/tsumo/generated/packages https://github.com/tsoniclang/tsonic-examples/tree/main/rust... The transpiler is still WIP, but the fact that it can do this says a lot of about how far LLMs have come.

jeffybefffy519 29d ago
I find all effort levels of sol are the same in terms of amount of hallucinated unnecessary changes. Luna is much better all round on xhigh but my point still stands, every release of these new models is not an upgrade, its re-learning how to work with it. Its like rehiring an employee every few months then training them up. Its honestly tiring and cant stay like this. Opus has the same problem too…
- jeswin 29d ago
  That's not been my experience. My prompting methods haven't changed much between recent GPT releases. I do put a lot of effort into building tooling and tests around a project, so the LLM output is converging around it.
  - jeffybefffy519 29d ago
    Can you give examples of the tooling and tests?
    - jeswin 29d ago
      Sure. This is the csharp target for tsonic (a TypeScript to C#/Rust/Python/Triton transpiler). It has a bunch of tests here: https://github.com/tsoniclang/tsonic-csharp/tree/main/test https://github.com/tsoniclang/tsonic-csharp/tree/main/test More comprehensive e2e proving grounds are at 1: https://github.com/tsoniclang/proof-is-in-the-pudding https://github.com/tsoniclang/proof-is-in-the-pudding 2: https://github.com/tsoniclang/tsumo/ https://github.com/tsoniclang/tsumo/ They were built specifically for testing the C# target. There are several other large projects we built specifically for e2e testing. But more interesting would be the tooling built to support this. For example, our current TypeScript parser [1] is a file-by-file port of Microsoft's TypeScript V7 compiler written in golang. The challenge here is that every time Microsoft changes code, we'll have to fix our code and tests. It's doable, but a fair amount of work. So we decided to write tooling to transpile Microsoft's v7 compiler from golang, and autogenerate our compiler. That tool is called gotots [2] - and it already produces a fully working TypeScript compiler. It's 3x slower than TypeScript v6 compiler, but we hope to get to rough performance parity in a week or so. Everytime Microsoft makes an update, we run gotots and our parser gets updated as well. [1]: The old parser - https://github.com/tsoniclang/tsts-legacy https://github.com/tsoniclang/tsts-legacy [2]: Golang to TypeScript transpiler - https://github.com/tsoniclang/gotots https://github.com/tsoniclang/gotots My general point is that tests and tooling is tremendous value, and they are guardrails for LLMs to converge. I could have, for example, chosen not to write the go-to-ts transpiler, and live with porting Microsoft's parser line by line. But making such tools is something LLMs are good at, so it's a tradeoff well worth making. And the upside is that you don't have to use LLMs to port Microsoft's parser/compiler (a large and complex project) line by line.