To be safe, a reasoning AI model should not be creative. Creativity entails some degree of chaos – some noise in the system. That noise can be the source of much “progress”, in the form of paradigm shifts, but can also lead to genuine harm in certain contexts.

It’s all about prompt design.

2025: humans trying to find the perfect prompt to make their LLM agents act a certain way

2035: LLMs trying to find the perfect prompt to make humans act a certain way

People fear AGI because of our tendency to anthropomorphise it.

Humans are not simply thinking / reasoning ‘machines’, we are also able to have genuine emotions–both good and bad.

Computers lack an equivalent to dopamine, the driving force behind greed, revenge, etc.

So we should be safe.