The power transmission system is increasingly dependent on accurate time-stamping of digitally sampled values used for protection and control. In particular, real-time streaming of data from networked Phasor Measurement Units (PMUs) for wide area ‘closed loop’ automated control applications implies a critical dependence of accurate, available, and reliable microsecond-level timing. While microsecond accuracy is easily met by GNSS timing receivers, GNSS signals for open civilian use are weak and also lack effective authentication mechanisms. GNSS timing receivers are therefore vulnerable to interference from malicious or inadvertent radio noise (jamming) and susceptible to ‘spoofing’ with generated GNSS-signals containing misleading timing and navigation data. The overall goal of the COSECTIME project funded by Statnett is to demonstrate the applicability of state-of- the art fiber-optic time transfer techniques for traceable, secure and redundant synchronization of digital power transmission protection and control applications. In full deployment , the transmission system operator (TSO) will generate redundant autonomous UTC-traceable atomic timescales and distribute timing through redundant fiber optic networks also under TSO control. Here we present results from a pilot demonstration of timing distribution to the Statnett R&D project pilot IEC 61850 digital substation.

1. ฀ ฀ ฀ ฀ ฀ ฀ ฀
94
Summary
Thepowertransmissionsystemisincreasinglydependent
on accurate time-stamping of digitally sampled values
used for protection and control. In particular, real-time
streaming of data from networked Phasor Measurement
Units (PMUs) for wide area ‘closed loop’ automated
control applications implies a critical dependence of
accurate, available, and reliable microsecond-level
timing.
While microsecond accuracy is easily met by GNSS
timing receivers, GNSS signals for open civilian use are
weak and also lack effective authentication mechanisms.
GNSS timing receivers are therefore vulnerable to
interference from malicious or inadvertent radio noise
(jamming) and susceptible to ‘spoofing’ with generated
GNSS-signals containing misleading timing and
navigation data.
The overall goal of the COSECTIME project funded by
Statnett is to demonstrate the applicability of state-of-
the art fiber-optic time transfer techniques for traceable,
secure and redundant synchronization of digital power
transmission protection and control applications. In full
deployment , the transmission system operator (TSO)
will generate redundant autonomous UTC-traceable
atomic timescales and distribute timing through
redundant fiber optic networks also under TSO control.
Here we present results from a pilot demonstration of
timing distribution to the Statnett R&D project pilot IEC
61850 digital substation.
1. Introduction – timing
requirements in the power
system
Thepowertransmissionsystemisincreasinglydependent
on accurate time-stamping of digitally sampled values
used in protection and control. Power system uses
of timing and associated accuracy requirements are
summarizedinfigure1.Foracomprehensiveoverviewof
power sector timing issues, see references [NASPI2017]
and [GSA2018].
The IEC 61850 requirement of microsecond accuracy
with respect to UTC can be met by properly installed and
characterized GNSS timing receivers [EURAMET2016]
in combination with timing distribution using the IEEE
1588 PTP precision timing protocol on the substation
process bus. However, GNSS signals for open civilian
use are weak and also lack effective authentication
mechanisms. GNSS timing receivers may therefore be
vulnerable to interference from malicious or inadvertent
radio noise (jamming) and susceptible to ‘spoofing’ with
generatedGNSS-signalscontainingmisleadingtimingand
navigationdata[Shepard2012].Malicioustimingattacksor
simplyinadvertenttimingerrorsmayhaveadverseimpact
on monitoring and control applications [Almas2018].
Applications studied in [Almas2018] illustrate the role of
precision timing as a valuable cyber-asset in power sector
control systems. For critical applications timing accuracy
requirements need to be complemented by requirements
on availability and integrity.
Redundant secure timing sources and
timing distribution to digital power
protection and control applications
H. HAUGLIN¤
, T. DUNKER¤
, A. WALLIN+, O. TUNGLAND*, N. HURZUK*, R. LØKEN*
¤
JUSTERVESENET, +VTT MIKES,*STATNETT,
¤
* Norway, *Finland
KEYWORDS
UTC, traceability, secure timing, redundant timing, IEEE 1588 PTP, cesium clocks
* hha@justervesenet.no

2. ฀ ฀ ฀ ฀ ฀ ฀ ฀
95
Figure 2. Conceptual sketch of redundant traceable timescale generation and
distribution to digital power applications. UTC-traceability of TSO timescales
is maintained by fiber-optic time transfer to a national timing laboratory
participating in the realization of UTC [BIPM2013].
2. COSECTIME: Generation of
traceable atomic timing and
distribution through optical
networks
The COSECTIME (COordinated SECureTIME) project
is funded by the Norwegian TSO Statnett. The goal is
to demonstrate how synchronization requirements of
power system applications may be met independent of
GNSS. The project uses state-of-the-art fiber-optic time
transfer techniques for traceable, secure and redundant
synchronization. In full deployment (see figure 2), the
transmissionsystemoperator(TSO)generatesredundant
semi-autonomous UTC-traceable atomic timescales
and distributes timing through redundant fiber-optic
networks also under TSO control.
Figure 1: Power system uses of time-dependent data. Reproduced from [NASPI2017] Table 2.

3. ฀ ฀ ฀ ฀ ฀ ฀ ฀
96
3. Setup of pilot demonstration
timing distribution to an IEC
61580 digital substation
The COSECTIME project pilot demonstration is
integrated into another Statnett R&D project building
and evaluating a pilot IEC 61850 Process Bus compliant
digital substation. See [Hurzuk2019] for a description.
The pilot demonstration setup is outlined in figure 3.
Substation clocks
Substation clocks (Meinberg M3000) were chosen
because they handle multiple external reference inputs
(GNSS, PPS, IRIG-B, 10 MHz, PTP), support a number
of different timing distribution modules, and incorporate
a multi-channel measurement system. See figure 4 for
a sketch of the substation clock configuration. A multi-
channel measurement system monitors the timing offset
of available external references against the substation
clock timescale. The reference selection algorithm
chooses among external references based on availability
and configured priority, with an additional option of
excluding ‘unacceptable’ external sources with timing
offsets outside a selectable threshold. The local rubidium
oscillator (Rb) is steered to follow the substation clock
timescale and may be used for extended holdover when
no other acceptable external references are available. The
substation clock timescale is distributed to the various
output modules.All PTP-GM (Grandmaster) modules for
the substation buses get timing from the substation clock
timescale, but run separate CPUs and network stacks.
Redundant Statnett master timescales are generated
by industrial cesium clocks (Microsemi 5071A) at
separate locations. The accuracy and traceability of
Statnett clocks with respect to UTC is maintained
by time transfer to the timescale UTC(JV) generated
by the Norwegian national timing laboratory at
Justervesenet (JV). We have established continuously
running clock comparisons via dedicated high-accuracy
IEEE 1588-2019 PTP-WR (Precision Time Protocol;
‘White Rabbit’) fiber-optic links between Statnett and
Justervesenet. PTP-WR links have been demonstrated to
maintainhighstabilityovermorethan1000kmofoptical
transport networks [Dierikx2016], suitable for running
direct sensitive comparisons of stable atomic clocks at
separate geographic locations. Clock comparison data
are the basis for occasional steering of Statnett cesium
clocks. The cesium clocks have to be steered a few
times per year to stay within a target of 100 ns offset
from UTC. The required frequency of steering depends
partly on the stability of the clock environment at
Statnett (temperature, humidity and magnetic fields) and
has been evaluated in the COSECTIME project. Fully
deployed, Statnett will have autonomous control over
timing sources and timing distribution to critical power
system control and monitoring applications. The role of
the national timing laboratory is to provide traceability
of Statnett timescale(s) to UTC and a running indirect
verification and supervision of Statnett timing. See
the telecom sector.
Figure 3. Setup of pilot demonstration of optical network timing distribution to an IEC 61850 digital substation. Timescales are generated by cesium
clocksattwoseparatelocations.Onecesiumclockislocatedatclocksite1whereitisthetimingsourceforaPTPGrandmaster(MeinbergM500with
MRS input card).Timing links use IEEE 1588-2008 PTP(default profile) over a combination of CWDM/DWDM optical transport network.Another
cesium clock is located at the digital substation with its PPS output connected directly to the substation clocks. Substation clock#1 is configured to
use timing over PTPfrom Cs clock#1 as the preferred external reference.

4. ฀ ฀ ฀ ฀ ฀ ฀ ฀
97
well within a target 100 ns deviation over 100 days.
At the end of characterization at Justervesenet, both
clocks were nominally synchronized (frequency and
phase) to UTC using the ‘rapid UTC’ product from
BIPM [Petit2014]. Cesium clock#1 was deployed at
clock site 1 and placed in a server rack with forced air
(18 oC) convection cooling through the rack. Cesium
clock#2 was used as a portable clock for calibration of
link asymmetries and for on-site calibration of clock#1
at clock site 1. Cesium clock#2 was eventually placed at
the digital substation pilot to provide a redundant stable
time reference.
The influence of the clock environment was evident
for cesium clock#1. Through on-site calibration, we
found that the clock had gained a systematic rate change
of +4 ns/day compared to the calibration value in
Justervesenet’s laboratory. This change is mostly due to
a significantly lower operating temperature.
Cesium clock#2 deployed at the digital substation
was evaluated by comparison to the substation clock
GNSS timing (figure 6). Over 100 days, the average
clock rate error corresponds to 0.2 ns/day with respect
to the GNSS clock, and the long-term stability (> 10
days) is comparable to the stability measured in the lab
environment.
In conclusion: Cesium clocks are suitable for stable
timescale generation in an industrial environment, but
to maintain timing within 100 ns of UTC, clocks need
initial synchronization and sporadic steering based on
repeated on-site calibration or running verification.
Timing links
Timing links to the substation were established using
IEEE 1588-2008 PTP (default profile). The PTP link
to the digital substation is carried over a combination
of coarse (CWDM) and dense wavelength division
multiplexing (DWDM) optical transport networks.
Optical-to-electrical-to-optical transponder cards in the
DWDM system are used to convert between CWDM
and DWDM optical channels. See figure 3 for details.
Measurements and data archiving
Multi-reference source measurement data from both
substation clocks are logged every 12 s and automatically
archived daily at a central data repository for off-line
analysis.Additional measurements are performed using a
high-resolution time-interval counter (Keysight 53230A).
4. Results and operational
experiences from the pilot
demonstration
Suitability of cesium clocks for timescale generation
Clocks (Microsemi 5071A standard performance)
were initially calibrated at Justervesenet in a stable lab
environment, with strictly controlled temperature and
humidity. Clocks were also characterized in a climate
chamber to determine the influence of temperature
changes on the clock rate. Between 12 oC and 32
oC, cesium clock#2 showed a systematic rate change
correspondingto-0.5ns/day/K.Figure5leftpanelshows
the time deviation of cesium clock#1 and #2, indicating
that timing errors due to random clock noise should stay
Figure 4. Redundant digital substation clock configuration.

5. ฀ ฀ ฀ ฀ ฀ ฀ ฀
98
convert between CWDM and DWDM optical channels.
Recently, WR links have been established using alien
wavelength (i.e. without transponders) in the Statnett
DWDM lab network, but not yet implemented in the
production network.Asimilar alien wavelength link has
been described by [Dierikx2016].
Standard PTPlinks
Currently operating links are between clock site 1 and
the digital substation and between clock site 1 and
Justervesenet. The links use the default profile IEEE
1588-2008 PTP. Clock comparison data of cesium
clock#1andcesiumclock#2throughthePTPlink(figure
3) is shown in figure 7.With the exception of a particular
200 ns dip of unknown cause, the PTP link itself does
not contribute to the variation observed for , which is
dominated by random timing noise of the cesium clocks.
The pilot link (figure 3) was found to have delay
asymmetry resulting in a systematic offset of 340
ns for timing received at the substation clock. This
asymmetry was calibrated using cesium clock#2 as a
mobile time transfer reference. Such asymmetries may
be compensated in the substation clock multi-reference
source measurement system.
Figure 6. Timing offset between cesium clock#2 and substation clock GNSS
timing over 100 days (October ‘18 – January’19). Timing offsets have been
smoothed by a 1 day wide moving average filter.
PTP-WR links
High accuracy IEEE 1588 PTP-WR [IEEE2019] was
initially intended to be used for the network timing
links in the COSECTIME project. The inability of WR
equipment to achieve stable timing lock for the link
shown in figure 3 and a similar link set up between
Statnett and JV is likely due to excess jitter in the
optical-to-electrical-to-optical transponders used to
Figure 5. Cesium clock frequency stability (left panel) and timing stability (right panel) in climate-controlled lab environment measured against
UTCr. The time deviation (right panel) indicate projected timing errors due to random clock noise. Blue lines indicate factory specifications for
standard and high-performance clocks
Figure 7. Comparison of cesium clocks
#1 and #2 by timing distribution over
IEEE 1588 PTP. Clock measurement
data are taken from the multi-channel
measurement system of substation
clock#1. Short-term noise is dominated by
the limited resolution of the measurement
system. Long-term (>1 day) variation is
consistent with fundamental clock noise
of 5071ACs clocks. The cause of the ‘dip’
at day 58427 (also shown in the inset) is
unknown.

6. ฀ ฀ ฀ ฀ ฀ ฀ ฀
99
10 ns and will be evaluated during the COSECTIME
project. However, for any country-wide deployment of
network timing, using a mobile clock is labor-intensive
and logistically challenging. Calibration of network
delay asymmetries would benefit greatly if substations
were equipped with properly calibrated GNSS timing
reception chains (antenna/cable/lightning arrestor/
timing receiver) [EURAMET2016].
5. Discussion
The results presented above indicate that industrial
cesium clocks in combination with fiber-optic timing
distribution using IEEE 1588 PTP (standard and/
or high accuracy ‘White Rabbit’) is an alternative to
GNSS timing for power application sync requirements.
The clear benefit of this approach is that GNSS-timing
is only used as a secondary backup, thus eliminating
adverseimpactsofpotentialGNSSjammingorspoofing.
However, deploying redundant cesium clock timescale
systems and associated sync distribution networks
comes with additional challenges: (1) Maintaining
robust timescales, i.e. how to steer clocks and detect
failing individual clocks and still be able to provide an
accurate timescale; (2) Deploying a sync distribution
network with 100s of user sites and calibrating network
timing asymmetries; (3) Monitoring the overall integrity
of the timing distribution chains, including anomaly
detection. Such timing anomalies may be due to failing
equipment or changes in the sync distribution network
paths that affect timing asymmetries.
Disclaimer
Commercial products are identified for the sake of
completeness. No particular endorsements are implied.
Described apparent strengths or weaknesses may not be
characteristic of current equipment versions.
Bibliography
[Almas2018] Almas MS et al, Vulnerability of Synchrophasor-
based WAMPAC Applications to Time Synchronization Spoofing,
IEEE Transactions on Smart Grid 9(5) 2018. DOI: 10.1109/
TSG.2017.2665461
In conclusion: The established point-to-point links using
standard IEEE 1588-2008 PTPare sufficiently stable for
timing distribution to the digital substation. However, a
sensitive comparison of cesium clocks may benefit from
the high accuracy IEEE 1588 PTP-WR in combination
with high-resolution time interval measurements.
Cesium clock failure
Cesium clock #1 located at remote clock site 1 failed
after 482 days of continuous operation. While the clock
controller reported fatal loss of lock on the cesium
resonance, the clock continued operating on its internal
crystal oscillator. In normal operation, the crystal
oscillator is continuously disciplined to follow the long-
term accuracy and stability of the cesium resonance.
A system for detection of cesium clock alarms had
not been implemented, nor had a direct running link
between clock site 1 and Justervesenet been established.
However, PTP timing from cesium clock#1 was
distributed to substation clock#1 and used as the selected
synchronization source. Archived multi-reference data
show the effect of the clock failure on substation clock
timing (figure 8).
The clock failure could have been detected at several
layers of a timing chain as depicted in figure 2: (1) By
detecting ‘fatal error’alarms raised by the clock itself; (2)
By real time analysis of cesium clock comparison data,
either with another clock at the same site and/or through
high stability timing links to other sites with cesium
clocks; (3) At the substation clock, the PTP timing from
Cs#1 could be rapidly flagged ‘erroneous’and deselected
as the preferred sync source by a majority vote among
the available external synchronization sources (figure 8).
However, such a logic is not currently implemented in the
source selection algorithm of the substation clocks.
Calibration
Calibration of link delay asymmetries were carried out
using a cesium clock as a mobile timing reference. This
is convenient and accurate on short (< 1 day) calibration
campaigns. The calibration uncertainty is of magnitude
Figure 8. Failure of the remote cesium clock as measured by substation clock#1. Left panel: Time offset between PTP slave referenced to Cs#1 and PPS from Cs#2
(located at the substation). Middle panel: Timing offset between PTPslave referenced to Cs#1 and timing from the local Rb oscillator. Right panel: Multi-reference data
from all available external sources at the onset of failure.

7. ฀ ฀ ฀ ฀ ฀ ฀ ฀
100
Synchronization Protocol for Networked Measurement and Control
Systems. URL: https://standards.ieee.org/standard/1588-2019.html
[NASPI2017] Time Synchronization in the Electric Power System,
North American Synchrophasor Initiative 2017. URL: https://www.
naspi.org/node/608
[Petit2014] Petit G et al, UTCr: a rapid realization of UTC, Metrologia
51, 33-9 2014
[Shepard2012] D.P. Shepard, T.E. Humphreys, A.A. Fansler,
“Evaluation of the Vulnerability of Phasor Measurement Units to GPS
Spoofing Attacks,” International Journal of Critical Infrastructure
Protection, Vol. 5, 2012. Downloaded from http://radionavlab.
ae.utexas.edu
[Śliwczyński2019] Śliwczyński et al, Calibrated optical time transfer
of UTC(k) for supervision of telecom networks, Metrologia 56 015006
2019. DOI: https://doi.org/10.1088/1681-7575/aaef57
[BIPM2013] Practical realization of the unit of time, International
Bureau of Weights and Measures, Paris 2013. Available at: https://
www.bipm.org/utils/en/pdf/SIApp2_s_en.pdf
[Dierikx2016] Dierikx E et al, White rabbit precision time protocol
on long-distance fiber links, IEEE Trans. Ultrason. Ferroelectr Freq.
Control 63 945–52 2016
[EURAMET2016] Use of GPS disciplined Oscillators for Frequency or
Time Traceablility, Euramet technical guide no 3 2016. Downloaded
from www.euramet.org
[GSA2018] Report on Time & Synchronization User Needs and
Requirements, European Global Satellite Systems Agency 2018.
Downloaded from www.gsc-europa.eu
[Hurzuk2019] N Hurzuk et al, paper contribution to Cigre B5
Colloquium, Tromsø 2019
[IEEE2019] IEEE Approved Draft Standard for a Precision Clock