Copy of the slides "The Singularity Principles: Anticipating and Managing Cataclysmically Disruptive Technologies" presented by David Wood at the World Talent Economy Forum on 20th June 2022 A recording of the event is available at https://www.youtube.com/watch?v=d-X5Fxx-By8

1. @dw2 Page 1
The Singularity Principles
^
and managing
David W. Wood
@dw2

2. @dw2 Page 2
The Singularity Principles
^
and managing
What is the
Singularity?
The most important
concept in history?!
Subject to lots
of confusion
David W. Wood
@dw2

3. @dw2 Page 3
What is The Singularity?
1. A forthcoming unprecedented radical discontinuity in the
history of humanity,
2. Triggered by the emergence of AIs that are
comprehensively more intelligent than humans,
3. With the change, once it starts, occurring in a relatively
short period of time (lasting, at most, perhaps a decade),
4. With outcomes that are practically impossible to foresee.
No assumption
of conscious AIs
These outcomes can be influenced
by wise human actions
No
assumption
of smooth
exponential
progress
The Singularity
Principles

4. @dw2 Page 4
Time
Capability
Incumbent
technology
Disruptive
technology
OMG
phase
Human
intelligence
AI intelligence
AGI
determines
human
future
Humans are
primarily biological
AI capability
Humans are
primarily
technological
1/4
Time
Capability
2/4
Time
Capability
3/4
Human
intelligence
AI intelligence
Capability
Time
4/4
Unforeseeable
phase
Humans are
the 2nd most
intelligent
species
The outcome can
be influenced by
wise human
actions
“When humans
transcend biology”
Merger!
Eliezer Yudkowsky, Nick Bostrom Vernor Vinge

5. @dw2 Page 5
The Singularitarian Stance
AGI (Artificial General Intelligence) is possible
AGI could emerge within just a few decades from now
AGI would fundamentally change the nature of human existence
Once it starts, the emergence of AGI may be very hard to control
AGI could be deeply detrimental to human wellbeing
AGI could, in other scenarios, be profoundly positive for humanity
No metaphysical blocks
Supply
Demand
Multipliers
Not just one more new technology that we’ll learn to take in our stride
No easy solutions
AI Control Problem
AI Alignment Problem
Four catastrophic
error modes
Sustainable Superabundance
✓
✓
✓

6. @dw2 Page 6
Leo Gao, AI safety researcher
@nabla_theta
https://twitter.com/nabla_theta/status/
1502783399622111234
https://twitter.com/robbensinger/status/
1503220020175769602
Rob Bensinger, AI safety researcher
@RobBensinger
No easy solutions
Machine Intelligence Research Institute

7. @dw2 Page 7
Four catastrophic error modes
Defect in implementation: The superintelligence is by no means infallible:
it takes an action which is intended to progress a goal, but due to an error in
calculation (or an error in execution) a sudden disastrous outcome ensues
Defect in design: The superintelligence pursues goals originally designed into it by
humans, but pursues these goals in a way neither foreseen nor intended by humans,
resulting – again – in an outcome that is disastrous for human wellbeing
Design overridden: New goals or targets emerge, either within the superintelligence
itself, or within a larger system in which the superintelligence exists,
that no longer put a priority on human wellbeing (a bit like humans don’t
particularly prioritise supporting eight billion gorillas living on the planet)
Implementation overridden: The superintelligence is hacked, or reconfigured,
in ways that violate its original goals,
and its subsequent actions have a terrible impact on humanity
Defect in implementation
Defect in design
Design overridden
Implementation overridden

8. @dw2 Page 8
The Singularity Principles:
Short Form
As we develop and interact with increasingly powerful technologies,
we should be sure we understand:
1. The goals that we’re hoping to accomplish – rather than us merely drifting along
in some direction because it sounds nice, or has some alluring features, or it
seemed like a good idea the last time that we thought about strategic direction
2. What are the products & methods that are most likely to serve these goals well
– rather than us persisting with products or methods that happen to make us
feel comfortable, or which have given us some good results in the past
3. How we will manage any surprises arising en route to our goals
– rather than us being caught flat-footed as the victim of inertia or denial,
when unexpected signals start showing on our radars.

9. Question desirability
Clarify externalities
Require peer reviews
Involve multiple perspectives
Analyse the whole system
Anticipate fat tails
Reject opacity
Promote resilience
Promote verifiability
Promote auditability
Clarify risks to users
Clarify trade-offs
Insist on accountability
Penalise disinformation
Design for cooperation
Analyse via simulations
Maintain human oversight
Build consensus regarding principles
Provide incentives to address omissions
Halt development if principles not upheld
Consolidate progress via legal frameworks
The Singularity
Principles

10. @dw2 Page 10
https://transpolitica.org/projects/
the-singularity-principles/
Intended to counter today’s widespread
• Confusion
• Distraction
• Intimidation
By providing
• Clarity
• Credible grounds for hope
• An urgent call to action
Open access pre-publication preview
Feedback welcome!
Page 10