Machine intelligence arises from an optimization process very different from the one that shaped human cognition. We should not expect it to inherit human principles by default, nor to extrapolate from them the way a person would.
We find it useful to separate two questions. The first concerns intent: does the model pursue the objective it was given, and does it seek clarification when that objective is ambiguous? The second concerns values: does the model hold a coherent set of principles, and act on them when objectives are underspecified or in conflict, or when it operates far outside its training distribution? We focus on the second, which we believe matters most as systems become more capable and more autonomous.
The central difficulty is generalization. Training against a written specification is effective in the average case, but depends on how much of the relevant situation space oversight actually covers. Approaches that draw on aligned behaviour latent in pretraining data depend less on coverage, but can degrade under sustained optimization pressure, as the model learns to rationalize its way toward the objective. Neither offers strong guarantees in genuinely novel settings.
Our working hypothesis is that the way humans carry values into unfamiliar situations has learnable structure, and that this structure is underrepresented in current training data. We study it empirically, with real people and at scale, and translate what we learn into reward models and RL environments for post-training.
We do not have a theory of value generalization, and we doubt one will arrive soon. In its absence, measurement comes first. We evaluate whether models trained on human-derived signal retain their values out of distribution and under further training, relative to synthetic baselines of equal size. If they do not, our hypothesis is wrong.