In 2011, Eliezer Yudkowsky gave an obscure but fascinating presentation on evolution and “outcome pumps”. It has garnered only 7.8K views on youtube over the last decade . These two processes are examples of optimization systems that can exhibit seemingly intelligent behaviors, yet these behaviors differ radically from human behaviors and preferences. The existence of such baffling processes raises the possibility that the first generally-intelligent AI systems could be similarly baffling for humans to reason about, and may act contrary to human wishes.
What sort of intelligent systems are possible? Can we assume that any intelligent non-human entities we create will necessarily have anything in common with us? Could it be that they automatically develop notions of morality similar to ours? Would their ways of thinking be at all similar to our ways of thinking? Many people currently working in AI alignment will answer “No” to the last three questions above: The Orthogonality Thesis , which is widely accepted in the AI community, argues that intelligence could theoretically come in many different varieties, and there is no good reason to assume that the property of “intelligence” is at all correlated with other properties of humans, such as our notions of morality.
At the January Chicago Rationality Meetup (Jan. 8 @ 2 PM), we will be discussing these questions, Eliezer’s presentation, and the Orthogonality thesis. Also, FYI, generally, topic meetups are the first Saturday of the month at 2 PM.  Eliezer’s presentation: https://www.youtube.com/watch?v=Uoda5BSj_6o The Orthogonality Thesis: https://www.youtube.com/watch?v=hEUO6pjwFOo