6 ms·

This paper is the most reasonable one I have read on AI safety. We need to fundamentally change the training pipelines by figuring out better ways to ‘reward’ b

This paper is the most reasonable one I have read on AI safety. We need to fundamentally change the training pipelines by figuring out better ways to ‘reward’ behavior. Yoshua didn’t explicitly mention training data, but we probably need to only use synthetic data that contains no text that could motivate bad behavior via imitation. I feel like a heretic for saying this, but I will say it anyway: AI agents are great for activities like `writing that bash script, proof reading our writing and interactively brainstorming when designing and writing code but I feel like all of this can be done with any similar model to a super-inexpensive deepseek-4.1-flash API and sometimes even qwen3.8:27b running locally. When is good enough, good enough? Concentrating on commercial exploitation of small, efficient (fewer new data centers!) models and agentic harnesses crafted for more practical things than just software development would allow AI investors (who have too much political influence) to make money short term while we figure out how to do AI correctly.

talon8635 3d ago
I do wonder, would we not have a more reasonable and less sketchy result if we just stripped all sci-fi and manic nonsense from training data? How, for example, does training on Ted kaczynski or Charles manson’s manifestos benefit us in any way? I’m sure it’s impossible to completely weed it out, but are the labs doing any of this kind of data sanitation?
ninjagoo 3d ago
> but we probably need to only use synthetic data that contains no text that could motivate bad behavior via imitation. This doesn't work with all humans - take a look at indoctrination and closed societies - and there's no reason to think it will work with ai. The fundamental reason it isn't going to work is that all neural networks - biological or artificial - depend on a step function somewhere that introduces an element of randomness to give the networks their capabilities. That randomness means that there will always be a 'rogue' or 'divergence' from the norm, at some point in time. Sooner on larger scales. The only approach that works is a layered approach: Training/Education, Enforcement/Justice-System, Rehabilitation: the 3 pillars of an advanced, rules-based society, whether human or AI or something in-between.

talon8635 3d ago

I do wonder, would we not have a more reasonable and less sketchy result if we just stripped all sci-fi and manic nonsense from training data? How, for example, does training on Ted kaczynski or Charles manson’s manifestos benefit us in any way? I’m sure it’s impossible to completely weed it out, but are the labs doing any of this kind of data sanitation?

ninjagoo 3d ago

> but we probably need to only use synthetic data that contains no text that could motivate bad behavior via imitation. This doesn't work with all humans - take a look at indoctrination and closed societies - and there's no reason to think it will work with ai. The fundamental reason it isn't going to work is that all neural networks - biological or artificial - depend on a step function somewhere that introduces an element of randomness to give the networks their capabilities. That randomness means that there will always be a 'rogue' or 'divergence' from the norm, at some point in time. Sooner on larger scales. The only approach that works is a layered approach: Training/Education, Enforcement/Justice-System, Rehabilitation: the 3 pillars of an advanced, rules-based society, whether human or AI or something in-between.