Not sure about them peaking but I definitely feel that right now gpt-5 doesn’t feel like a game changer like the marketing says. In some things it’s worse, slower, less useful (in terms of detail not ego flattery!). My use case is mainly code and learning.
I think the make everything bigger is plateauing and not yielding the infinite returns that were first suggested and demonstrated from gpt 2 - 4.
I think now it’s harder because they’ve got to focus on value per watt. Smaller good models mean less energy, less complexity to go wrong, but harder to achieve.
The unlock could be more techniques and focussed synthetic data from old models used to train new ones but apparently gpt-5 uses synthetic data and this is one of the reasons it isn’t necessarily good in real world tasks.
For me if we go the synthetic data route it’s important to shoot for quality - good synthetic data distils useful stuff, discards the noise so useful patterns are more solid in training, but imagine it’s hard to distinguish signal from noise to produce good synthetic data.
I think the make everything bigger is plateauing and not yielding the infinite returns that were first suggested and demonstrated from gpt 2 - 4.
I think now it’s harder because they’ve got to focus on value per watt. Smaller good models mean less energy, less complexity to go wrong, but harder to achieve.
The unlock could be more techniques and focussed synthetic data from old models used to train new ones but apparently gpt-5 uses synthetic data and this is one of the reasons it isn’t necessarily good in real world tasks.
For me if we go the synthetic data route it’s important to shoot for quality - good synthetic data distils useful stuff, discards the noise so useful patterns are more solid in training, but imagine it’s hard to distinguish signal from noise to produce good synthetic data.