This document reports on a randomized controlled trial experiment testing the added value of an urban planning support system called Urban Strategy. The experiment involved master's students in urban planning completing a strategic planning task. Students were either given access to Urban Strategy or not. Evaluation results found that those using Urban Strategy perceived the planning process and outcomes as being higher quality. They also rated the usability characteristics of Urban Strategy positively. The document concludes that Urban Strategy shows potential for improving strategic planning processes and outcomes through its interactive interface and fast modeling capabilities.

1.
CESAR
WORKING
DOCUMENT
SERIES
Working
document
no.3
Support
for
planning
process
and
content
Urban
Strategy
Experiment
No2
M.
te
Brömmelstroet
13
April
2014
This
working
document
series
is
a
joint
initiative
of
the
University
of
Amsterdam,
Utrecht
University,
Wageningen
University
and
Research
centre
and
TNO
The
research
that
is
presented
in
this
series
is
financed
by
the
NWO
program
on
Sustainable
Accessibility
of
the
Randstad:
http://www.nwo.nl/nwohome.nsf/pages/nwoa_79vlym_eng

2. CESAR
Working
Document
Series
no.
3
Urban
Strategy
experiment
no.
2
Page
2
TABLE
OF
CONTENT
1.
INTRODUCTION
...............................................................................................................
3
2.
SETUP
OF
THE
EXPERIMENT
............................................................................................
4
2.1
Intervention:
Urban
Strategy
PSS
.........................................................................................
4
2.2
Mechanisms:
how
does
Urban
Strategy
bridge
the
PSS
implementation
gap?
...................
4
2.3
Setup
of
the
controlled
randomized
trial
.............................................................................
5
Treatment
.....................................................................................................................................
6
Evaluation
.....................................................................................................................................
7
3.
EVALUATION
RESULTS
.....................................................................................................
8
3.1
Perceived
quality
of
the
planning
process
............................................................................
8
3.2
Perceived
quality
of
the
planning
outcome
..........................................................................
8
3.3
Usability
characteristics
of
Urban
Strategy
..........................................................................
8
4.
CONCLUSIONS
AND
IMPLICATIONS
...............................................................................
10
4.1
Reflection
...........................................................................................................................
10
5.
CONCLUSIONS
AND
DISCUSSION
...................................................................................
11
5.1
General
conclusions
...........................................................................................................
11
REFERENCES
..........................................................................................................................
12
APPENDIX
I:
COMPETITION
SHEET
(DUTCH)
.........................................................................
13
Introductie
..................................................................................................................................
13
Herinrichting
Waalhaven
oostzijde
.............................................................................................
14
Opdrachtomschrijving:
...............................................................................................................
15
Uitgangspunten
en
randvoorwaarden
........................................................................................
15
APPENDIX
II:
EVALUATION
FORM
FOR
QUALITY
OF
THE
PROCES
........................................
16
APPENDIX
III:
EVALUATION
FORM
FOR
QUALITY
OF
THE
OUTCOME
...................................
18
APPENDIX
IV:
EVALUATION
FORM
FOR
USABILITY
OF
URBAN
STRATEGY
...........................
19

3. CESAR
Working
Document
Series
no.
3
Urban
Strategy
experiment
no.
2
Page
3
1. INTRODUCTION
In
this
working
document
we
report
on
a
second
randomized
controlled
trial
with
Urban
Strategy
to
test
the
added
value
of
this
Planning
Support
System
(PSS).
Urban
Strategy
is
a
software
package
developed
by
TNO
that
aims
to
improve
the
planning
process
and
planning
outcomes
of
strategic
planning.
It
does
so
by
offering
a
range
of
quick
models
that
show
the
effects
of
planning
interventions
in
an
easy
to
understand
visual
environment.
To
gain
more
insight
into
these
potential
improvements,
we
have
conducted
an
experiment
with
a
group
of
master
students
in
Urban
Planning
of
the
University
of
Amsterdam.
We
make
use
of
the
measuring
framework
as
introduced
and
discussed
in
CESAR
Working
Document
No.
1.
First,
we
describe
the
setup
of
the
experiment
(section
2).
Then,
the
findings
of
the
experiment
are
presented
(section
3).
In
the
fourth
section
we
will
briefly
discuss
the
implications
of
these
findings
and
further
research.

4. CESAR
Working
Document
Series
no.
3
Urban
Strategy
experiment
no.
2
Page
4
2. SETUP
OF
THE
EXPERIMENT
2.1 Intervention:
Urban
Strategy
PSS
TNO
started
around
2005
with
the
development
of
a
PSS
–
Urban
Strategy
(Borst
et
al.
2007;
2009a;
2009b)
–
specifically
aiming
to
bridge
the
existing
flexibility-­‐
and
communication
bottlenecks.
Urban
Strategy
aims
to
improve
complex
spatial
planning
processes
on
the
urban-­‐
and
regional
level.
To
do
this,
different
computer
models
are
linked
to
a
central
database
and
interface
to
provide
insights
in
a
wide
area
of
urban
indicators
and
maps.
The
effects
of
interventions
in
infrastructure,
land
use,
build
objects
and
their
functions
can
be
calculated
and
visualized.
Because
the
PSS
is
able
to
calculate
fast
and
present
the
results
in
an
attractive
1D,
2D
and
3D
visualisation
this
can
be
used
in
interactive
sessions
with
planning
actors.
Starting
point
for
Urban
Strategy
is
the
use
of
existing
state-­‐of-­‐the-­‐art
and
legally
accepted
models.
To
link
these
existing
models
a
number
of
new
elements
were
developed:
- a
database
with
an
uniform
datamodel;
- interfaces
that
show
a
3D
image
of
the
modeled
situation,
indicators
and
that
offer
functionality
to
add
interventions;
- a
framework
that
structures
the
communication
between
the
models
and
the
interfaces.
2.2 Mechanisms:
how
does
Urban
Strategy
bridge
the
PSS
implementation
gap?
The
goal
of
Urban
Strategy
is
to
enable
planning
actors
in
workshop
sessions
to
communicate
their
ideas
and
strategies
to
the
PSS
and
to
learn
from
the
effects
that
are
shown.
This
interactivity
calls
for
fast
calculations
of
all
the
model
and
fast
communication
between
all
elements.
For
this,
the
models
were
enabled
to
respond
on
events
(urban
interventions
from
the
participants
in
the
workshops.
A
new
software
architecture
was
developed
to
have
all
these
elements
communicate
(figure
2).
Through
this
increases
speed
and
the
wide
variety
of
models
that
are
linked
together,
the
PSS
aims
to
be
highly
flexible
in
offering
answers
to
a
large
number
of
questions
that
a
group
of
urban
planning
actors
can
have.
Figure
2
Schematic
overview
of
communication
architecture
of
Urban
Strategy.
The
3D
interface
generates,
based
on
objects
in
the
database,
a
3D
digital
maquette
of
the
urban
environment.
To
this,
different
information
layers
can
be
addes,
such
as
air
quality
contours,
noise
contours
and
groundwater
levels.
Also,
the
objects
can
be
colored
according
to
their
characteristics
(function,
energy
use,
CO2
emissions,
number
of
inhabitants,
etc).
The
2D
interface
can
be
used
by
the
end
user
(or
operator)
to
add
changes
to
the
database.
Objects
can
be
added
or
removed,
their

5. CESAR
Working
Document
Series
no.
3
Urban
Strategy
experiment
no.
2
Page
5
location
can
be
changed
and
the
characteristics
of
the
object
can
be
changed.
The
1D
interface
shows
indicators
that
are
calculated
by
all
the
models
that
are
included.
Examples
are
the
percentage
of
noise
hindrance,
group
risk
in
an
area
or
the
contribution
of
types
of
objects
to
CO2
emission.
Figure
3
The
three
interfaces
of
Urban
Strategy
2.3 Setup
of
the
controlled
randomized
trial
Together
with
TNO
we
organized
a
number
of
sessions
in
which
the
usability
of
Urban
Strategy
was
tested.
For
this
we
set
up
a
typical
urban
planning
challenge:
Optimize
existing
ideas
for
new
housing
and
working
functions
in
an
old
harbor
area
of
Rotterdam
(figure
4).
The
existing
ideas
face
problems
due
to
restrictions
on
noise,
external
safety
and
air
quality
(figure
5).
Figure
4
The
harbor
area
in
the
larger
Rotterdam
region

6. CESAR
Working
Document
Series
no.
3
Urban
Strategy
experiment
no.
2
Page
6
Figure
5
Current
strategy
for
the
area
(top
left),
air
quality
norms
(top
right),
noise
norms
(bottom
left)
and
external
safety
(bottom
right)
We
invited
24
students
from
the
course
Metropolitan
Transportation
Planning,
which
is
part
of
the
one-­‐year
Master
on
Urban
Planning
of
the
University
of
Amsterdam.
They
were
told
that
they
would
take
part
in
a
strategy
making
competition
and
were
not
aware
of
the
treatment.
They
represent
future
planning
practitioners
that
deal
with
planning
challenges
such
as
the
abovementioned.
The
students
were
randomly
split
into
six
groups
of
each
three
students.
Each
group
was
then
asked
to
perform
the
planning
task
as
a
group.
One
week
before
the
strategy
making
sessions,
they
were
introduced
to
the
planning
problem
and
criteria
on
which
the
developed
strategies
were
going
to
be
judged:
innovation,
workability,
relevance
and
specificity.
See
appendix
I
for
the
(Dutch)
form
with
information
that
each
participant
received.
The
six
groups
were,
again
randomly,
divided
into
control
and
treatment
groups.
Three
groups
performed
the
planning
task
without
support
of
Urban
Strategy,
while
three
other
groups
received
this
planning
support.
The
control
groups
worked
simultaneously,
while
the
treatment
groups
did
their
work
consecutively
(due
to
constraints
of
the
model
interaction).
All
groups
got
45
minutes
to
perform
the
task
and
were
asked
to
finalize
their
strategy
on
a
sheet
of
paper
and
an
empty
map
10
minutes
before
the
deadline.
The
control
groups
were
then
left
to
perform
their
task.
They
were
supported
by
the
maps
from
figure
5
and
empty
maps
to
draw
their
plans.
Also,
pens
and
markers
were
provided
(figure
6
left).
Treatment
The
groups
that
were
supported
with
Urban
Strategy
followed
a
fixed
protocol.
On
arrival
in
the
room,
Ralph
Klerkx
(TNO)
introduced
Urban
Strategy
to
the
group.
In
10
minutes,
he
showed
the

7. CESAR
Working
Document
Series
no.
3
Urban
Strategy
experiment
no.
2
Page
7
functionality
of
the
instrument
and
what
it
can
and
cannot
do.
After
this
introduction
Ralph
and
Sander
Schaminée
(TNO)
guided
the
groups
in
finding
solutions
for
the
planning
problem.
First,
they
were
asked
to
come
up
with
an
improved
design
of
the
location,
especially
the
place
and
orientation
of
the
housing
and
working
blocks.
With
this,
the
groups
started
to
interact
with
Urban
Strategy
to
see
the
effects
of
their
interventions.
During
this
interacting,
the
design
was
optimized
to
minimize
the
amount
of
housing
and
working
under
the
noise,
pollution
and
external
safety
norms.
Also,
strategies
were
developed
to
cope
with
other
negative
effects,
such
as
the
congestion
and
noise
of
the
road
adjacent
to
the
study
area.
During
these
design-­‐analysis
iterations,
the
groups
also
were
asked
to
write
down
the
strategies
and
their
reasoning.
The
strategy
making
took
place
in
a
classroom
in
which
the
group
had
a
table
to
discuss
and
draw
and
could
interact
with
the
different
interfaces
of
Urban
Strategy
(figure
6
right).
Sander
Schaminée
supported
this
by
drawing
the
proposed
interventions
into
the
model.
Figure
6
The
setting
of
the
control
group
(left)
and
the
treatment
group
(right)
Evaluation
Two
evaluation
instruments
were
used
to
test
the
effects
of
the
support
by
Urban
Strategy.
Using
a
7-­‐points
Likert
scale,
all
participants
were
asked
to
rate
a
number
of
statements
related
to
the
quality
of
the
strategy
making
process
(Appendix
II).
These
perceptions
were
aggregated
to
the
level
of
the
evaluation
dimensions
and
compared
between
the
control-­‐
and
treatment
groups.
Next
to
that,
the
resulting
strategies
were
collected
and
presented
to
two
external
raters.
Using
a
7-­‐
point
Likert
scale,
they
were
asked
to
rate
a
number
of
statements
related
to
the
quality
of
the
strategy
making
outcome
(Appendix
III).
The
strategies
were
presented
to
the
raters
without
the
hypotheses
or
indication
of
treatment/control.
These
perceptions
were
combined
and
aggregated
to
the
level
of
the
evaluation
dimensions
and
compared
between
treatment
and
control.
A
final
evaluation
form
was
provided
only
to
the
participants
of
the
treatment
groups.
They
were
asked
to
reflect
on
usability
characteristics
of
Urban
Strategy.
Again,
a
7-­‐point
Likert
scale
was
used
to
gather
their
perceptions.

8. CESAR
Working
Document
Series
no.
3
Urban
Strategy
experiment
no.
2
Page
8
3. EVALUATION
RESULTS
3.1 Perceived
quality
of
the
planning
process
Due
to
the
small
number
of
observations
we
cannot
assume
to
have
controlled
for
all
possible
other
influential
variables.
Also,
we
would
like
to
raise
awareness
for
the
fact
that
we
worked
with
students,
for
which
it
was
often
the
first
time
to
engage
in
such
a
strategy
making
process.
There
are
however
some
remarkable
outcomes.
On
the
overall
score
(at
right
of
table
1)
we
see
no
effect
of
the
support
by
Urban
Strategy.
On
the
level
of
the
dimensions,
we
see
a
meaningful
positive
effect
on
‘effectiveness’,
although
this
effect
is
not
significant
due
to
the
large
variation
within
the
control-­‐
and
treatment
group.
Apparently,
perceptions
of
this
vary
wildly.
A
very
negative
effect
is
found
on
‘consensus
on
goals’,
but
again
the
found
difference
is
not
significant.
Most
effects
on
the
other
dimensions
are
close
to
zero.
We
can
therefore
not
confirm
a
positive
effect
of
Urban
Strategy
on
the
strategy
making
process.
3.2 Perceived
quality
of
the
planning
outcome
The
overall
quality
of
the
planning
outcome
(at
right
in
table
2)
is
affected
negatively
by
the
support
of
Urban
Strategy.
This
effect
is
not
significant,
but
still
noteworthy.
Only
‘implementability’
is
positively
affected,
while
all
other
dimensions
show
a
negative
effect
of
Urban
Strategy.
This
is
most
remarkable
for
‘applicability’
and
the
related
grouped
dimension
of
‘relevance’
where
negative
scores
of
respectively
1,5
and
0,78
are
found.
Again,
we
can
conclude
from
these
scores
that
we
cannot
confirm
a
positive
effect
of
Urban
Strategy
on
the
strategy
making
outcome.
3.3 Usability
characteristics
of
Urban
Strategy
Although
a
positive
effect
of
the
instrument
could
not
be
confirmed,
the
participants
were
very
positive
about
its
usability.
The
scores
in
table
3
(on
a
7
point
scale
and
arranged
from
high
to
low)
indicate
that
all
usability
dimensions
score
over
4,6.
Most
positive
elements
are
the
‘support
for
evaluating
alternatives’,
‘communicative
value’
and
‘supporting
the
creation
of
ideas’.