final report version ii on the total power system failure
Post on 05-Oct-2021
1 Views
Preview:
TRANSCRIPT
1
Final Report Version II on the
Total Power System Failure on
17th August 2020
Submitted to the Ministry of Power on Wednesday the 16th December 2020
NOTE:
THE AUTHOR OF THIS VERSION II OF THE FINAL REPORT HAS BEEN A MEMBER OF THE COMMITTEE
APPOINTED BY THE MINISTRY OF POWER IN ITS LETTER DATED 18.08.2020 (REF:
PE/DEV/01/81/2015). HE DOES NOT AGREE WITH SOME OF THE TECHNICAL INTERPRETATIONS
GIVEN BY SOME MEMBERS IN THE COMMITTEE AND THEREFORE PRESENTED THIS VERSION II AS A
SEPARATE DOCUMENT.
2
DISCLAIMER
The Conclusions and Recommendations stipulated herein have been framed based on the information
ascertained and discussions held with relevant officials of the Ceylon Electricity Board (CEB) through
number of visits made to Kerawalapitiya GSS, Lak Vijaya Power Station (LVPS), Protection Branch and
the System Control Center (SCC) of CEB. As such the professional judgment of the individual in making
the Conclusions and the Recommendations would depend on the accuracy of the said information
ascertained and those transpired from the said information and discussions. Further, the author would
not be responsible for the extrapolation of the Conclusions and the Recommendations stipulated
herein by any third party including the other members of the committee appointed for the same
purpose.
3
Contents
Introduction ………………………………………………………………………………………………………………………….…….4
1. Kerawalapitiya Grid Substation ..………………………………………………………………………………………..…5
1.1 Relevant observations and findings during the visit to Kerawalapitiya Grid Substation (GSS)
on the 19.08.2020
1.2 Analysis of the observations made during the visit to Kerawalapitiya GSS
2 Protection Branch / CEB .…………………………………………………………………………………………………6
2.1 Relevant observations and findings during the meeting with the Control and Protection
Branch Engineers on the 19.08.2020
2.2 Analysis of the observations made during the meeting Control and Protection Branch
Engineers
3. Lak Vijaya Power Station (LVPS) ……………………………………………………………………………………10
3.1 Relevant findings during the visit to the LVPS on the 20.08.2020
3.2 Analysis of the observations made during the visit to LVPS 3.2.1 Timing Data Analysis
3.2.2 Fault Current Analysis
3.2.3 System Frequency Analysis
3.2.4 Distributed Control System Data Analysis
3.2.5 LVPS Generator Current Analysis
3.2.6 Fast Cut Back (FCB) operation at the LVPS
3.2.7 Time sequence analysis of Over Speed Protection Control (OPC) at the LVPS
3.3 Observations on the LVPS behavior in this incident
3.4 Root Cause of the incident related to LVPS
4. System Control Centre / CEB ………………………………………………………………………………………..30
4.1 Relevant findings of the committee during the visit to the System Control Office (visited on
the 22nd August 2020)
4.2 Analysis of the observations made during the visit to System Control
4.3 Observations on the CCTV records at the Systems Control Center
5. Conclusions ..……………………………………………………………………………………………………………….34
6. Recommendations ..…………………………………………………………………………………………………….35
4
Introduction
This report has been prepared by a single author in the committee appointed by the Secretary to the
Ministry of Power in order to investigate the nationwide electricity supply interruption took place on
the 17th August 2020.
A nationwide electricity supply interruption had taken place around 12.30 pm on the 17th August 2020,
involving all Transmission lines, Power Stations and Grid Substations in the Power System in Sri Lanka,
thereby disconnecting all the electricity consumers of the country from the electricity power supply.
The said interruption is reported to have been initiated at Kerawalapitiya Grid Sub Station triggering
a total power system failure in the country. Consequent restoration of the transmission network, in
spite of all efforts initiated and managed by the System Control Center of the CEB with the nodal
station of the utilities had taken 6 hours and 16 minutes.
5
1. Kerawalapitiya Grid Substation
1.1 Relevant observations and findings during the visit to Kerawalapitiya Grid Substation (GSS) on the
19.08.2020
Routine maintenance work on the 220 kV isolators of the Bus Coupler Bay had been carried out on the
day of the incident by the Electrical Superintendent In Charge at Kerawalapitiya GSS, who apparently
has been attending routine maintenance work at the Kerawalapitiya GSS for the past 5 years. The
power in the Bus Bar 01 had been turned OFF for the maintenance, while the power of the Bus Bar 02
was ON. The Earth Switch 01 at Bus Bar 01 side had been OFF while the Earth Switch 02 at Bus Bar 02
side had been ON as shown in Fig. 1.2(a) at the time of incident. Under normal operations the Earth
Switch and the relevant isolator are interlocked, so that the isolator cannot be turned ON while the
Earth Switch is turned ON. However, during maintenance, this interlock had been bypassed, so that
isolator can be turned ON even with the Earth Switch is turned ON. At the end of the maintenance
work of the 220 kV Bus Coupler Bay, while the interlock is bypassed, the Isolator on the Bus Bar 02
side had been turned ON as shown in Fig. 1.2(b), creating a 3 Phase to Ground fault. The fault had
been isolated within 154 ms by tripping of all connected circuit breakers to the Bus Bar 2 by the
operation of Busbar differential protection.
It was revealed that the Electrical Superintendent In Charge at Kerawalapitiya GSS had been carrying
out the above maintenance work by himself with his team around. It was further revealed that this
maintenance work had been carried out without being supervised by any of his superiors, which is the
usual practice.
Bus bar 01: OFF
Bus bar 02: ON
Earth Switch 1
Earth Switch 2
Isolator 1
Isolator 2
Bus coupler
Fig. 1.2a
Bus bar 02: ON
Earth Switch 1
Earth Switch 2
Isolator 1
Isolator 2
Bus coupler
Bus bar 01: OFF
High fault current flow
Fig. 1.2b
1.2 Analysis of the observations made during the visit to Kerawalapitiya GSS
The total isolation time of 154 ms is too long on the busbar protection operation. As per the report of
Control & Protection branch also, the time to issue the trip command is 87 ms. Therefore, the fault
detection time of the busbar protection relay has to be checked. Turning ON the Isolator 2 could be a
human error (mistake), which should have been the Isolator 1 instead. However, if not for bypassing
the interlocking, a catastrophe by such a human error could have been avoided.
It is also noted that there is no robust maintenance protocol in place. The maintenance practices of
high risk areas are to be identified and best practiced protocols in maintenance to be incorporated in
particular in the said areas rather than having a general maintenance philosophy.
6
2. Protection Branch / CEB
2.1 Relevant observations and findings during the meeting with the Control and Protection Branch
Engineers on the 19.08.2020
During the meeting, the Protection Branch Engineers showed various data associated with the
incident using the data acquisition system in the CEB. Out of them, the bus voltage data shows
that the 3 Phase to Ground fault had been so severe that for 7.5 cycles, the 220 kV bus voltage
had been reduced to
1. 0V according to DDR record of Sub L bay of 220kV Kerawalapitiya GIS at 12:30:27 Hrs
2. 110 kV according to DDR record of Generator Transformer 2 bay of 220kV Lakvijaya GIS at
12:30:27 Hrs respectively. The corresponding voltages are reproduced from DDR in Fig. 2.1a
and 2.1b respectively.
Fig. 2.1a
220 kV reduced to 0 kV
7
Fig. 2.1b
Accordingly, in addition to the busbar voltage becoming zero at the fault location, reducing the
voltages to 50% at far away bus bars such Lakvijaya Power Station Norechcholei shows the severity
of the fault.
Further, the Disturbance Recorders (DDR) in the Control and Protection Branch / CEB shows
1. Busbar protection at Kerawalapitiya 220kV GIS operated and tripped all connected
Transmission Lines and Transformers by at 12:30:27.172 Hrs, due to a busbar fault.
2. 220kV Circuit breaker of Generator Transformer 3 at Lakvijaya Power Station tripped at
12:30:27.359 Hrs
3. 220kV Circuit breaker of Generator Transformer 2 at Lakvijaya Power Station tripped at
12.30.27.390 Hrs
4. 220kV Circuit breaker of Generator Transformer 1 at Lakvijaya Power station tripped at
12.30.27.429 Hrs
5. Phase B to Ground fault of approximately 8.5 kA occurred in 220kV Bus section 1/3 at
Lakvijaya Power Station 12:30:27.423 Hrs and all lines and Transformers connected to
Bus Section 1 and Bus section 3 tripped due to operation of busbar protection by
12:30:27.491 Hrs
6. As per the DDR of Kotmale PS, the system frequency dropped below 47 Hz within 1.9s
from the initial Busbar Protection fault at Kerawalapitiya Grid Substation.
2.2 Analysis of the observations made during the meeting Control and Protection Branch Engineers
Fig. 2.2a shows the transients of the bus bar voltages during and immediately after the fault isolation
in the nearby bus bars. Accordingly, Kerawalapitiya Bus bar 02 voltage drops from 220 kV to 0 kV
during the 3 phase to ground fault because the fault occurs through near zero resistance conductor
and does not recover because the busbar is isolated from the system by the operation of the busbar
220 kV reduced to ~110 kV
8
protection. The Kotugoda Circuit 01 (220 kV), which is the next node from Kerawalapitiya GSS, also
undergoes the same voltage drop. Though it recovers, it does not settle. Similarly, in the 132 kV
network also Kolonnawa GSS experience a drop from 132 kV to about 40 kV and many other places
experience similar drops. In far away places from the fault such as the Veyangoda GSS, the Generator
Transformer 2 bay of 220kV Lak Vijaya Power Station (Fig. 2.1 b), the 220 kV busbar voltage drops to
about 110 kV. The active power 𝑃 at a busbar is proportional to the voltage 𝑉𝐿𝐿 squared.
Fig. 2.2 a
The apparent power 𝑆 at a node can be expressed by
𝑆 = 𝑃 + 𝑗𝑄 =√3𝑉𝐿𝐿
2
𝑅 + 𝑗𝑋=
√3𝑉𝐿𝐿2
(𝑅2 + 𝑋2)2(𝑅 − 𝑗𝑋)
If the overall voltage drop is assumed to be 50% during the fault, the active power in the system
reduces to 25% of the pre fault value, which corresponds to a loss of load by 75%. The corresponding
dynamics can be modelled by
𝜏𝑔𝑒𝑛 − 𝜏𝑙𝑜𝑎𝑑 = 𝐽𝑑𝑓
𝑑𝑡+ 𝐵𝑓.
Where 𝜏𝑔𝑒𝑛represents generation, 𝜏𝑙𝑜𝑎𝑑 represents load, 𝐽 is the system inertia, 𝐵 system damping
and 𝑓 is system frequency, a drop in 𝜏𝑙𝑜𝑎𝑑 causes an increase in the difference in 𝜏𝑔𝑒𝑛 − 𝜏𝑙𝑜𝑎𝑑 which
gives rise to 𝑑𝑓
𝑑𝑡. Hence the rate of change of system frequency depends on the system inertia. If there
is adequate system inertia, the rate of change of system frequency can be kept within the accepted
limits.
0
50
100
150
200
250
12
:30
:26
.18
8
12
:30
:26
.28
8
12
:30
:26
.38
8
12
:30
:26
.48
8
12
:30
:26
.58
8
12
:30
:26
.68
8
12
:30
:26
.78
8
12
:30
:26
.88
8
12
:30
:26
.98
8
12
:30
:27
.08
8
12
:30
:27
.18
8
12
:30
:27
.28
8
12
:30
:27
.38
8
12
:30
:27
.48
8
12
:30
:27
.58
8
12
:30
:27
.68
8
12
:30
:27
.78
8
12
:30
:27
.88
8
12
:30
:27
.98
8
12
:30
:28
.08
8
12
:30
:28
.18
8
12
:30
:28
.28
8
12
:30
:28
.38
8
12
:30
:28
.48
8
12
:30
:28
.58
8
12
:30
:28
.68
8
12
:30
:28
.78
8
12
:30
:28
.88
8
12
:30
:28
.98
8
12
:30
:29
.08
8
12
:30
:29
.18
8
12
:30
:29
.28
8
12
:30
:29
.38
8
Bu
s b
ar v
olt
age
(kV
)
Time
Bus bar voltages in the Vicinity of Kerawalapitiya GSS during the fault and immediately after fault
Kearawalapitiya WCP 02 (220kV)
Kotugoda Circuit 01 (220kV)
Kolonnawa (132kV)
9
The above reduction in voltage reduces the power to 25% of the power flowing before the initiation
of the fault, meaning that the grid feels a loss of 75% of the load. The load at the time of incident had
been 1983 MW and loosing 75% of it gives the grid, a feeling similar to a single generator suddenly
loosing 75% of its load. In the single generator case, the generator frequency increases. Similarly, in
this case, the system frequency should increase. Once the fault is cleared, the frequency should stop
increasing and settle. However, the above scenario leads to a total system failure and therefore the
system frequency should start reducing. The system frequency waveform against time obtained by
the Control and Protection Branch / CEB at 12:30:27 hrs on 17.08.2020 at Kotmale Power Station,
reproduced in Fig. 2.2b confirms this hypothesis.
Fig.2.2b System frequency variation during the system failure
10
3. Lak Vijaya Power Station (LVPS)
3.1 Relevant findings during the visit to the LVPS on the 20.08.2020
The Power House, Switch Yard and the Control Centre of the LVPS were visited. During the visit,
the Switchyard layout, Voltage, Current, Power, Generator Frequency, System Frequency, Turbine
speed plots and the operation of the digital logic circuits of the relevant protection relays against
time, pre and post of the system power failure were inspected. Fig. 3.1a shows the Busbar
arrangement of the LVPS 220 kV GIS during the fault.
Fig. 3.1 a Busbar arrangement of the LVPS 220 kV GIS during the fault
Following elements have been reported in the report.
(1). Busbar protection at Kerawalapitiya 220kV GIS operated and tripped all connected Transmission
Lines and Transformers by at 12:30:27.172 Hrs, due to the three phase to earth fault in the Bus
coupler.
(2). 220kV Circuit breaker of Generator Transformer 3 at Lakvijaya Power station tripped at
12:30:27.359 Hrs
(3). 220kV Circuit breaker of Generator Transformer 2 at Lakvijaya Power station tripped at
12.30.27.390 Hrs
(4). 220kV Circuit breaker of Generator Transformer 1 at Lakvijaya Power station tripped at
12.30.27.429 Hrs
(5). Phase B to Ground fault of approximately 8.5 kA occurred in 220kV Bus section 1/3 at 12:30:27.423
Hrs and all lines and Transformers connected to Bus Section 1 and Bus section 3 tripped due to
operation of busbar protection by 12:30:27.491 Hrs.
(6). Lakvijaya-New Chilaw Line 01 and Lakvijaya-New Anuradhapura Line 02 have been tripped by
busbar protection at 12:30:27.478 hrs and 12:30:27.480 hrs respectively while Lakvijaya-New Chilaw
Line 02 and Lakvijaya-New Anuradhapura Line 01 were not tripped.
11
(6). As per the DDR of Kotmale PS, the system frequency dropped below 47 Hz within 1.9s from the
earth fault at Kerawalapitiya Grid. Frequency variation at the LVPS during the system failure,
reproduced from the report of CEB Protection and Control Branch, is shown in the Fig. 3.1b.
(7). Generator Transformers 3, 2 and 1 of Lakvijaya Power Station have tripped in 180ms, 210ms and
246ms respectively after the Kerawalapitiya three phase to earth fault has been cleared.
(8). Another Phase B - N fault has occurred in Bus Section 1/3 resulting in the operation of Busbar
protection in both Bus 1 and Bus 3. The Busbar section fault was cleared within 61ms. However, Bus
2 remained stable until the system was collapsed.
According to the above DDR records, it can be seen that Generator transformer CB of Unit 3 has been
tripped first and Unit 2 CB and Unit 1 CB have been tripped second and third following the earth fault
at Kerawalapitiya GSS.
Fig. 3.1b Frequency variation at the Lakvijaya Power Plant during the system failure.
12
3.2 Analysis of the observations made during the visit to LVPS
3.2.1 Timing Data Analysis
Table 3.2a shows the time records of relevant events at LVPS
TABLE 3.2a
Unit df/dt Generator Transformer Circuit Breaker Opening Time
Unit Shutdown Time
01 12:30:27:226.4 12:30:27:425.4 12:42
02 12:30:27:213.71 12:30:27:395.38 14:51
03 12:30:26:508.25 12:30:26:682 13:33
Since the system frequency is common to the entire grid, all three units should detect the system
frequency rise at the same time. However, as per the records in TABLE 3.2a, they are at different
instants in time, and hence the DFR clocks are out of synchronism. The difference between df/dt
record of Unit 01-Unit 03, which is (1000-508.25) +226.4 = 718.15 ms, which is approximately the time
synchronization error at the time of this recording.
Not having GPS synchronized clocks at all locations in the system has made the analysis some what
complicated.
3.2.2 Fault Current Analysis
According to the officials of the LVPP and the Transmission Protection branch, Kerawalapitiya three
phase fault has been picked up by the three generator units at LVPP as an external fault, when the
rate of change of frequency (df/dt) of the system increased beyond the currently set thresholds. This
df/dt threshold limit has been set to 2 Hz/s for all the three units at the LVPP. Consequently, the
generator df/dt protection scheme (ROCOF relay) has operated and tripped the generator transformer
circuit breakers. A phase to earth fault is also observed in the 1-3 bus section in the LVPS, after the
Kerawalapitiya three phase fault cleared. It is noted that, in the ROCOF relay, only df1/dt signal is
used to trip the Generator transformer circuit breakers while df2/dt, the other signal is currently not
in use. Apart from the indication about an advice provided by a Consultant to consider only df1/dt,
there is no proper evidence nor details of df/dt signal established at the commissioning stage were
made available. As per the LVPP officials, generator transformer Circuit Breaker (CB) tripping sequence
observed of the Units was Unit 03, Unit 02 and Unit 01 and the three units had been put on the Fast
Cut Back (FCB) operation and generator units had been put into the house load operation (Unit 01 -
12 minutes, Unit 02 – 2 hours and 20 minutes, Unit 03 – 1 hour), before the units shutdown. This
sequence of units going into FCB can be observed in the generator current waveforms in Fig. 3.2a,
3.2b and 3.2c whose data obtained from the BEN6000 recorder at LVPS which sampled at 200 µs.
13
Fig. 3.2a Current waveforms of Phase R of LVPS units during and after the Three Phase to Ground fault
at Kerawalapitiya GSS. Accordingly, Unit 03 current (Ash colour waveform) reduces first, Unit 02
current (Orange colour waveform) reduces next and the Unit 01 current (Blue colour waveform)
reduces last.
Fig. 3.2b Current waveforms of Phase Y of LVPS units during and after the Three Phase to Ground fault
at Kerawalapitiya GSS. Accordingly, Unit 03 current (Ash colour waveform) reduces first, Unit 02
current (Orange colour waveform) reduces next and the Unit 01 current (Blue colour waveform)
reduces last.
-60000
-50000
-40000
-30000
-20000
-10000
0
10000
20000
300001
96
19
1
28
6
38
1
47
6
57
1
66
6
76
1
85
6
95
1
10
46
11
41
12
36
13
31
14
26
15
21
16
16
17
11
18
06
19
01
19
96
20
91
21
86
22
81
23
76
24
71
25
66
26
61
27
56
28
51
29
46
30
41
31
36
32
31
33
26
34
21
Zoomed Generator Phase R Currents
Serial number : 20 GEN 1_IR A Serial number : 20 GEN 2_IR A Serial number : 20 GEN 3_IR A
-25000
-20000
-15000
-10000
-5000
0
5000
10000
15000
20000
25000
1
96
19
1
28
6
38
1
47
6
57
1
66
6
76
1
85
6
95
1
10
46
11
41
12
36
13
31
14
26
15
21
16
16
17
11
18
06
19
01
19
96
20
91
21
86
22
81
23
76
24
71
25
66
26
61
27
56
28
51
29
46
30
41
31
36
32
31
33
26
34
21
Zoomed Generator Phase Y Currents
20 GEN 1_IY A 20 GEN 2_IY A 20 GEN 3_IY A
14
Fig. 3.2c Current waveforms of Phase B of LVPS units during and after the Three Phase to Ground fault
at Kerawalapitiya GSS. Accordingly, Unit 03 current (Ash colour waveform) reduces first, Unit 02
current (Orange colour waveform) reduces next and the Unit 01 current (Blue colour waveform)
reduces last.
Hence with the synchronous sampled data, sampled at 200 µs by BEN6000 fault recorder at LVPS, it
is confirmed that the sequence of LVPS Generator Units going into FCB mode is Unit 03, Unit 02 and
Unit 01, irrespective of what can be derived using logic signals in DCS recorded at low time
resolutions in the range of 100 – 200 ms, which is three orders of magnitude lower in resolution.
The following parameters in TABLE 3.2b and 3.2c are used to calculate the fault currents of the LVPS
generators during the Three Phase to Ground fault at Kerawalapitiya GSS.
TABLE 3.2b
Line Section
No. of
Circuits
Cable type Distance
(km)
R (/km) X (/km)
Norochcholi - New Chillaw 2 AAAC 630 73.3 0.02378 0.2818
New Chillaw - Veyangoda 2 AAAC 400 44.2 0.03662 0.2961
Veyangoda – Kotugoda 2 Zebra 17.0 0.03807 0.306
Kotugoda – Kerawalapitiya 2 Zebra 18.0 0.03807 0.306
By using 100 MVA and 220 kV base, the impedance base is Z base = 2202/100 = 484 . The per unit
quantities of data in Table 3.2b are given in Table 3.2c.
-30000
-20000
-10000
0
10000
20000
30000
40000
50000
600001
96
19
12
86
38
14
76
57
16
66
76
18
56
95
11
04
61
14
11
23
61
33
11
42
61
52
11
61
61
71
11
80
61
90
11
99
62
09
12
18
62
28
12
37
62
47
12
56
62
66
12
75
62
85
1
29
46
30
41
31
36
32
31
33
26
34
21
Zoomed Generator Phase B Currents
Sampling Rate : 20 GEN 1_IB A Sampling Rate : 20 GEN 2_IB A Sampling Rate : 20 GEN 3_IB A
15
TABLE 3.2c
Line Section R () X () R (pu) X (pu)
Norochcholi - New Chillaw 1.74 20.66 0.0036 0.0427
New Chillaw - Veyangoda 1.62 13.09 0.0033 0.0271
Veyangoda – Kotugoda 0.64 5.20 0.0013 0.0107
Kotugoda – Kerawalapitiya 0.69 5.51 0.0014 0.0114
Generator transformer reactance = 0.14 pu on 360 MVA from the datasheet and it is 0.0389 on the
100 MVA base. As can be seen in Fig. 3.2d the RMS value of the fault current on 20 kV side is 10.094
kA.
Fig. 3.2d Circuit from LVPS to Kerawalapitiya GSS
Table 3.2d shows the RMS fault currents calculated from the measured currents in BEN6000 recorder.
TABLE 3.2d RMS fault currents calculated from measurements
Generator Unit 01
Generator Unit 02
Generator Unit 03
Phase R 10394 A 10499 A 10574 A
Phase Y 10600 A 10635 A 10722 A
Phase B 9415 A 9514 A 9613 A
Hence the calculated fault currents are in good agreement with the measurements despite possible
line parameter mismatches. It confirms that no incident has taken place at LVPS during the three
phase to ground fault at Kerawalapitiya GSS.
3.2.3 System Frequency Analysis
In case of an external fault, the Rate Of Change Of Frequency (ROCOF) is picked up by the Generator
df/dt protection incorporated in the ROCOF relay (81R in SIEMENS 7SJ6225). As the upper threshold
limit of df/dt has been set to 2Hz/s for all three units identically, in the case of an external fault, if the
df/dt limit exceeds the said threshold, the generator transformer CBs of all three units will trip
initiating the Fast Cut Back (FCB) operation and subsequently the generator units will be put into the
house load operation, usually for about 30 mins or so, expecting the grid power to put the units back
into operation.
16
According to the manual of ROCOF relay 81R of SIEMENS 7SJ6225, the measuring principle is (quoted
from the manual) “From the positive-sequence voltage, the frequency is determined once per cycle
over a measuring window of 3 cycles, and a mean of two consecutive frequency measurements is
formed. The frequency difference is then determined over a settable time interval (default setting
5 cycles). The ratio between frequency difference and time difference corresponds to the frequency
change; it can be positive or negative. The measurement is performed continuously (per cycle).
Monitoring functions such as undervoltage monitoring, check for phase angle jumps etc. help to
avoid malfunctioning.”
Since the only unknown here is the default setting which could be at least 2 cycles and assuming that
it does not go beyond 5 cycles due to long response times, it is varied from 2 cycles to 5 cycles and
df/dt has been calculated on the frequency data on the Kolonnawa 132 kV busbar sampled at 20 Hz
(sampling period of 50 ms) and is plotted in Fig. 3.2e. It shows the rate of change of the grid frequency
(df/dt) during the Three Phase to Ground fault at Kerawalapitiya GSS.
Fig. 3.2e Rate of change of frequency for different cycle times.
As can be observed, despite it is 20 Hz sampled dataset, for the 2 cycles average, 3 cycles average,
4 cycles average and 5 cycles average, the df/dt reaches above 2 Hz/s. Since the sampling itself acts
as a filter, a dataset sampled at a higher rate than 20 Hz (theoretically it can go up to 100 Hz, if zero
crossing method is used to find the frequency, but at the expense of accuracy) would result in even
higher figures for df/dt shown above. Hence it is concluded that the rate of system frequency has
reached 2 Hz/s which is sufficient to trigger the ROCOF threshold who is set to operate at 2 Hz/s.
-4
-3
-2
-1
0
1
2
3
4
1 3 5 7 9 111315171921232527293133353739414345474951535557596163656769717375777981838587
df/dt
2 cycle df/dt 3 cycle df/dt 4 cycle df/dt 5 cycle df/dt
17
3.2.4 Distributed Control System Data Analysis
In the High-Speed Recorders (HSR) of the respective Distributed Control Systems (DCS) in the Control
Centre of the LVPS, the Turbine Speed, the Generator Power, the Generator frequency and the rate
of change of frequency 𝑑𝑓
𝑑𝑡 recordings were studied. Since the sampling times used in the Unit 01, 02
and 03 DCS are as large as 105 ms, 200 ms and 105 ms respectively, the HSR records have only be used
to confirm if a certain event has taken place or not. Otherwise, they cannot be used to derive the time
sequence of events.
LVPS Unit 03 (Sampling period = 105 ms): According to Fig. 3.2f obtained from the Unit 03 DCS, the
FCB comes into operation following the activation of the 𝑑𝑓
𝑑𝑡, resulting in a drastic reduction in the
generated power, which causes an overshoot in the turbine speed.
Fig. 3.2f The activation of the 𝑑𝑓
𝑑𝑡 of Unit 03
LVPS Unit 02 (Sampling period = 200 ms): According to Fig. 3.7 obtained from the Unit 02 DCS, FCB
comes into operation, following the activation of the 𝑑𝑓
𝑑𝑡. This is observed in the drastic reduction in
the generated power and the overshoot in the turbine speed.
Fig. 3.2g The activation of the 𝑑𝑓
𝑑𝑡 of Unit 02
2800
2850
2900
2950
3000
3050
3100
3150
3200
0
50
100
150
200
250
300
350
1
22
43
64
85
10
6
12
7
14
8
16
9
19
0
21
1
23
2
25
3
27
4
29
5
31
6
33
7
35
8
37
9
40
0
42
1
44
2
46
3
48
4
50
5
52
6
54
7
56
8
58
9
Sample
Generator Unit 03 HSR Data
Load (MW) df/dt FCB Breaker Open Speed (rpm)
2800
2850
2900
2950
3000
3050
3100
3150
3200
0
50
100
150
200
250
300
350
1
21
41
61
81
10
1
12
1
14
1
16
1
18
1
20
1
22
1
24
1
26
1
28
1
30
1
32
1
34
1
36
1
38
1
40
1
42
1
44
1
46
1
48
1
50
1
52
1
54
1
56
1
58
1
Generator Unit 02 HSR Data
Load (MW) FCB df/dt Breaker Open Speed (rpm)
18
LVPS Unit 01 (Sampling period = 105 ms): According to Fig. 3.2h obtained from Unit 01 DCS, the 𝑑𝑓
𝑑𝑡 of
Unit 01 is activated which is confirmed by the overshoot in the Generator Frequency and the
overshoot in the Turbine Speed. This activation has reduced the power and tripped the 220kV Circuit
breaker of Generator Transformer 01 of the Unit 01.
Fig. 3.2h The activation of the 𝑑𝑓
𝑑𝑡 of Unit 01
3.2.5 LVPS Generator Current Analysis
The data in the BEN6000 at LVPS were synchronous sampled at a period of 200 µs and the first 2700
data points of the data records provided by the LVPS officials were used to extract the LVPS Generator
Unit current information shown in Figs. 3.2i, 3.2j and 3.2k. The GPS synchronized BEN6000 fault
recorder records data in a window from 100 ms before the fault to 2500 ms after triggering from the
fault. It is clear in them that there has not been any fault incident prior to the Three Phase to Ground
fault at Kerawalapitiya GSS. The fault at Kerawalapitiya GSS clears in nearly 7.5 cycles, i.e.,
approximately 150 ms, which is in agreement with the recorded fault clearance time of 154 ms
recorded and reported earlier. The screen shot in Fig. 3.2m confirms this recording window
information. Accordingly, for the Three Phase to Ground Fault at Kerawalapitiya GSS took place at
12:30:27.024 GPS, the BEN6000 at LVPS can only show data back to 12:30:26.924 GPS in relation to
the incident at Kerawalapitiya GSS.
0
500
1000
1500
2000
2500
3000
3500
0
50
100
150
200
250
300
350
Sam
ple 18
36
54
72
90
10
8
12
6
14
4
16
2
18
0
19
8
21
6
23
4
25
2
27
0
28
8
30
6
32
4
34
2
36
0
37
8
39
6
41
4
43
2
45
0
46
8
48
6
50
4
52
2
54
0
55
8
57
6
59
4
Generator Unit 01 HSR Data
Load (MW) Breaker Open df/dt FCB Frequency (Hz) Speed (rpm)
19
Fig. 3.2i BEN6000 record of currents of the three phases of Generator Unit 01 at LVPS: Sequentially
from left to right it shows (i) before the three phase to ground fault at Kerawalapitiya GSS, (ii) during
the fault at Kerawalapitiya GSS, (iii) after fault at Kerawalapitiya GSS, (iv) increase due to taking up
the load when Unit 03 leaving the grid and going to FCB, (v) increase due to taking up the load when
Unit 02 leaving the grid and going to FCB, (vi) Fault current in R and B phases, (vii) reduced current
due to entering FCB.
Fig. 3.2j BEN6000 record of currents of the three phases of Generator Unit 02 at LVPS: Sequentially
from left to right it shows (i) before the three phase to ground fault at Kerawalapitiya GSS, (ii) during
the fault at Kerawalapitiya GSS, (iii) after fault at Kerawalapitiya GSS, (iv) increase due to taking up
the load when Unit 03 leaving the grid and going to FCB, (v) reduced current due to entering FCB.
-50000
-40000
-30000
-20000
-10000
0
10000
20000
30000
40000
500001
74
14
7
22
02
93
36
6
43
9
51
2
58
5
65
8
73
1
80
4
87
7
95
0
10
23
10
96
11
69
12
42
13
15
13
88
14
61
15
34
16
07
16
80
17
53
18
26
18
99
19
72
20
45
21
18
21
91
22
64
23
37
24
10
24
83
25
56
26
29
Generator Unit 01 Currents
20 GEN 1_IR A 20 GEN 1_IY A 20 GEN 1_IB A 20 GEN 1_IN A
(ii)(iii) (iv)
(v)
(vi)
(vii)
-25000
-20000
-15000
-10000
-5000
0
5000
10000
15000
20000
25000
1
74
14
7
22
02
93
36
6
43
9
51
2
58
5
65
8
73
1
80
4
87
7
95
0
10
23
10
96
11
69
12
42
13
15
13
88
14
61
15
34
16
07
16
80
17
53
18
26
18
99
19
72
20
45
21
18
21
91
22
64
23
37
24
10
24
83
25
56
26
29
Generator Unit 02 Currents
20 GEN 2_IR A 20 GEN 2_IY A 20 GEN 2_IB A 20 GEN 2_IN A
(i)
(ii)
(iii)(iv)
(v)
(i)
20
Fig. 3.2k BEN6000 record of currents of the three phases of Generator Unit 03 at LVPS: Sequentially
from left to right it shows (i) before the three phase to ground fault at Kerawalapitiya GSS, (ii) during
the fault at Kerawalapitiya GSS, (iii) after fault at Kerawalapitiya GSS, (iv) reduced current due to the
Unit 03 entering FCB.
Towards the right end of Fig. 3.2i, it shows a huge current in the order of 50 kA in the R and B phases
of Generator Unit 01. Since the 20 kV winding side of the Generator Transformer is connected to the
Generator Stator windings is delta connected, it appears as a phase to phase fault current. However,
when the Generator Transformer 220 kV winding side, whose star point is solidly grounded, is
observed as shown in Fig. 3.2l, it is a Phase B to Ground (N) fault after the Generator Unit 01 going to
FCB.
Fig. 3.2l
-25000
-20000
-15000
-10000
-5000
0
5000
10000
15000
20000
250001
74
14
7
22
02
93
36
6
43
9
51
2
58
5
65
8
73
1
80
4
87
7
95
0
10
23
10
96
11
69
12
42
13
15
13
88
14
61
15
34
16
07
16
80
17
53
18
26
18
99
19
72
20
45
21
18
21
91
22
64
23
37
24
10
24
83
25
56
26
29
Generator Unit 03 Currents
20 GEN 3_IR A 20 GEN 3_IY A 20 GEN 3_IB A 20 GEN 3_IN A
(i)
(ii)
(iii)
(iv)
-8000
-3000
2000
7000
11
05
20
93
13
41
75
21
62
57
29
83
39
37
10
41
11
45
12
49
13
53
14
57
15
61
16
65
17
69
18
73
19
77
20
81
21
85
22
89
23
93
24
97
26
01
Generator Transformer 01 - 220 kV side Currents
Serial number : 220 GEN TR1_IR A 653 220 GEN TR1_IY A
Sampling Rate : 220 GEN TR1_IB A 5000 220 GEN TR1_IN A
21
This is the Bus Section 1-3 fault at LVPS occurring after Unit 03, Unit 02 and Unit 01 going to FCB mode.
According to records, this fault has got cleared within 61ms by operating Busbar protection. Hence it
is clear that Phase B - N fault in the Bus Section 1-3 occurred as the last major event in the series and
it has nothing to do with the Generator Units at LVPS going into to FCB mode. The results of the
investigations on finding the exact reason for the Phase B - N fault in the Bus Section 1-3 was not
available at the time of generating this report.
Even if the Phase B - N fault did not occur in Bus Section 1-3 of LVPS, it is impossible to avoid the
activation of 𝑑𝑓
𝑑𝑡 in any of the units because as shown in Fig. 3.2d, the df/dt exceeded the set value of
2.0 Hz/s threshold in the ROCOF relay, which took place during the three phase to ground fault at
Kerawalapitiya GSS which took 154 ms to clear.
However, Bus 2 remained stable until the system collapsed.
Fig. 3.2m
It was further noted that no auxiliary power supplying mechanism, which is as high as 10% of the
installed capacity, was available at the LVPS to accommodate the controlled shut down process of the
units. If adequate auxiliary power was available, the units could have subsequently been restarted
without waiting for 3, 4 days with the need to replace the mechanical protection devices which got
activated under uncontrolled shutdown. Further it could have avoided eliminated the huge financial
losses incurred due to unavailability of LVPS to grid for the said long period counted in days.
22
3.2.6 Fast Cut Back (FCB) operation at the LVPS
FCB operation is implemented in DCS control to put the generators into house load operation in case
generator transformer circuit breakers of Units have tripped due to external faults. In the house load
operation, generators will run at 3000 rpm powering auxiliaries in the plant. But the house load
operation is not recommended to maintain more than 30 minutes. The main objective of FCB
operation is to keep the generators running with auxiliaries until the grid is powered up. If the grid
power is restored in time, electricity generation could be resumed. The FCB operation will be
activated if one of the following three condition is met
1. df/dt relay activation
2. forward power protection activation
3. 35MW/s load drop of the unit is measured on a given condition i.e. the unit should be
operated more than 150MW and a rapid unit power drop to an operating point less than
80MW in 2s condition is monitored. (Subjected to operator enables the FCB function from
the DCS). This condition is enabled only for the Generator Unit 02.
3.2.7 Time sequence analysis of Over Speed Protection Control (OPC) at the LVPS
OPC comes into operation while the Generator Units are synchronized to the grid the generator
circuits breakers are ON. Once the over speed protection control is activated, high pressure regulating
valves and low pressure regulation valves will be closed, making a drastic reduction in the generated
power leaving the room for system to reduce the frequency. OPC will be reset to an assigned dead
band and currently it is set to 3000 rpm for all the units. The present OPC setting at the LVPP units is
as shown in the Table 3.2e.
Table 3.2e: Present OPC settings at the LVPS
Setting Unit 1 Unit 2 Unit 3
OPC 3100 rpm 3078 rpm 3110 rpm
RESET OPC 3000 rpm 3000 rpm 3000 rpm
DEAD BAND 100 rpm 78 rpm 110 rpm
OPC NO YES NO
The OPC triggering can be verified by plotting the Generator Current against time (sample number)
and frequency against time (sample number) in the same plot. It is understood that the frequency
cannot be calculated at a rate faster than 100 Hz by using the Voltage Zero Crossing method on a 50
Hz voltage or current signal. Since the frequency at LVPS is monitored using a 20 Hz sampled recorder
and since that is not sufficient for this kind of time sequence analysis of the OPC operation, the author
23
of the report has written a multilevel frequency evaluation algorithm and used on the voltage data
recorded at BEN6000 fault recorder at LVPS to find the Generator frequency of each Unit. The
corresponding findings are shown in Figs. 3.2n, 3.2o and 3.2p for the Generator Unit 01, Unit 02 and
Unit 03 respectively. The algorithm stabilizes within the first two cycles and frequency data is accurate
beyond that point.
Generator Unit 01: It is observed in Fig. 3.2n(a) that there are two peaks in the generator frequency
of Unit 01. The first generator frequency peak has a magnitude of 50.87 Hz (3052.2 rpm) and occurs
during the 3 Phase to Ground fault at Kerawalapitiya GSS. The second generator frequency peak has
a magnitude of 53.33 Hz (3199 rpm) and as can be seen in the Fig. 3.2n(a), several other incidents also
happen there. Therefore, the last 297 data points of Fig. 3.2n(a) is zoomed and shown on Fig. 3.2n(b).
Fig. 3.2n(a): Variation of Generator frequency and current over sample number
(derived using voltage measurements sampled at 200 µs) of Unit 01
As can be seen in Fig. 3.2 n(b), the incident (i) occurs first and it is the feeding of the fault current in
the Bus Section 1/3. Next is the incident (ii), where the generator frequency starts increasing. In this
context, such a speed increase can only happen when the generator is disconnected from the network.
Such a scenario is possible if the Unit 01 enters FCB. The third incident is hitting the OPC trigger point
at 3100 rpm (51.67 Hz). However, the Unit 01 is already in FCB and therefore OPC does not operate.
Therefore, the second generator frequency peak has no resultant effect to the system.
0 500 1000 1500 2000 2500 3000-100
-80
-60
-40
-20
0
20
40
60
Generator 1 frequency (Hz)
Generator 1 current (A/500)
24
Fig. 3.2 n(b): Zoomed Generator Unit 01 current and Generator Frequency during the last 2400 to
2697 datapoints.
Generator Unit 02: It is observed in Fig. 3.2o(a) that there are two peaks in the generator frequency
of Unit 02. The first peak in the Generator Unit 02 frequency has an amplitude of 50.92 Hz (3055.2
rpm) during the 3 Phase to Ground fault at Kerawalapitiya GSS, and the second peak with amplitude
51.3 Hz (3078 rpm) occurs when the unit is apparently in FCB. Since the sequence of occurrence of
the second peak in the Generator Unit 02 frequency is not clear, the last 2000 to 2697 data points are
zoomed as shown in Fig. 3.2o(b).
As can be seen in Fig. 3.2o(b), there is a rise in generator current at (i) which corresponds to taking the
load partially from the Generator Unit 03 going into FCB mode. Subsequently there is a drop in system
frequency which is seen in (ii). According to Fig. 3.2o(b), at (iii), the Generator Frequency starts rising
as opposed to system frequency fall. Therefore, this point must correspond to the Unit 02 going to
FCB. However, it is not visible until (iv), where it can clearly be seen that the Unit 02 is in FCB. It is
interesting to note that the Unit 02 OPC trigger level of 3078 rpm (51.3 Hz) is met only at point (v),
which is after FCB. Therefore, even though OPC is noted in the Table 3.2e in the data provided by the
LVPS, it happens after FCB.
25
Fig. 3.2o(a): Variation of Generator frequency and current over sample number
(derived using voltage measurements sampled at 200 µs) of Unit 02
Fig. 3.2 o(b): Zoomed Generator Unit 02 Current and Generator Frequency during the last 2000 to
2697 datapoints.
0 500 1000 1500 2000 2500 3000-60
-40
-20
0
20
40
60
Generator 2 frequency (Hz)
Generator 2 current (A/500)
26
It is observed in Fig. 3.2p (a) also that there are two peaks in the generator frequency of Unit 03 as
50.92 Hz (3055.2 rpm) during the 3 Phase to Ground fault at Kerawalapitiya GSS and 51.52 Hz (3091.2
rpm) when the unit is in FCB. Unit 03 in FCB can be verified by comparing the frequency curve with
the current curve and also the zoomed version of the same from the datapoint 2000 to 2697.
Fig.3.2p(a): Variation of Generator frequency and current over sample number
(derived using voltage measurements sampled at 200 µs) of Unit 03
In Fig. 3.2 p(b), it can be seen that the Generator Unit 03 enters FCB at (i). However, there is a delay
to start rising the Generator Frequency as seen in (ii).
0 500 1000 1500 2000 2500 3000-60
-40
-20
0
20
40
60
Generator 3 frequency (Hz)
Generator 3 current (A/500)
27
Fig. 3.2 p(b): Zoomed Generator Unit 03 Current and Generator Frequency during the last 2000 to
2697 datapoints.
Hence it is verified that even if the speed logic conditions are satisfied for the OPC of a particular
Generator Unit to trigger, since it happens after the Unit entering to FCB mode with the generator
circuit breakers open, OPC cannot come into effect.
28
3.3. Observations on the LVPS behavior in this incident
1. Generator Unit 03, Unit 02 and Unit 01 respectively went into Fast Cut Back (FCB) mode
in that sequential order after detecting the system frequency having df/dt above their
threshold limits of 2 Hz/s. This high df/dt has been an external event to LVPS.
2. The B-N Phase to Ground fault in Bus Section 1-3 had occurred even after the third
Generator (Generator Unit 01) going in to FCB mode. Hence it has no connection to the
Generator Units going into FCB.
3. Current df/dt protection scheme appears to be not the most appropriate scheme which
should have been there for the LVPP.
4. Had the df/dt thresholds of the Generator Units were set unequal, say 2 Hz/s, 3 Hz/s, 4
Hz/s a total failure could have been avoided.
5. The df/dt control logic seems to have been adopted without adequate justification on
the trade-off between the machine safety and the risk of collapsing the entire system.
6. Even after going to FCB, had there been a means of auxiliary power which is in the range
of 90 MW, a controlled shutdown could have been exercised, which could have avoided
3-4 days long, expensive startup procedures.
3.4 Root Cause of the incident related to LVPS
System frequency rise at a rate above the threshold setting of 2 Hz/s caused the Generator Units at
LVPS to go into FCB mode and disconnected from the network, which made a huge loss of generation
to the network, leading under frequency tripping of other generator units in the network, despite
some automatic load shedding.
NOTE:
If not for the current waveforms in Figs. 3.2a, 3.2b, 3.2c, 3.2i, 3.2j and 3.2k obtained using the
BEN6000 fault recorder at LVPS with sampling period 200 µs, the conclusions derived just by
observing the event plots in the respective Unit DCS reproduced in Figs 3.2f, 3.2g and 3.2g, which
are with sampling periods 105 ms, 200 ms and 105 ms respectively, could have lead to serious errors
misleading the reader.
29
4. System Control Centre / CEB.
4.1 Relevant findings of the committee during the visit to the System Control Office (visited on the
22nd August 2020)
The DGM, Chief Engineer and the Electrical Engineer at the System Control/CEB presented system
failure they observed at the System Control as shown in Fig. 4.1a, where it is clearly seen that the bus
bar voltages at the fault in Kerawalapitiya Grid Substation and the Kotugoda Grid Substation both
drops to 0V from 220 kV.
System Control Engineers presented the procedure followed to restore the system subsequent to the
system power failure. The same information titled “sequence of system restoration attempts” have
been submitted to the committee. Restoration has started in Mahaweli Complex, Laxapana Complex
and Samanalawewa Complex simultaneously. Out of them, the Samanalawewa Complex feeds mostly
the rural loads. After a couple of over frequency failures, the Samanalawewa Complex stabilizes at
15:37. Hence, in 3 hours and 7 minutes, part of the Sri Lankan power network becomes live since the
power failure at 12:30. After several attempts, at 18:08 Samanalawewa System & Kukule System
synchronized via Ambalangoda- New Galle 132kV circuit 01.
However, the Mahaweli Complex and the Laxapana Complex feed urban and industry loads. Industry
loads are typically large loads and therefore when energizing the network from a blackout, there is a
higher probability to mismatch the generation to the dispatch. As a result, there can be more over
frequency failures in stabilizing the Mahaweli Complex and the Laxapana Complex. At 18:46 Mahaweli
Complex is stabilized and syncrnoized to Samanalawewa Complex via New Laxapana- Polpitiya 132kV
circuit 1. Hence the transmission network has come back to operation fully 6 hours and 16 minutes
from the system failure.
Fig. 4.1a
0
10
20
30
40
50
60
0
50
100
150
200
250
12
:30
:25
.88
8
12
:30
:26
.13
8
12
:30
:26
.38
8
12
:30
:26
.63
8
12
:30
:26
.88
8
12
:30
:27
.13
8
12
:30
:27
.38
8
12
:30
:27
.63
8
12
:30
:27
.88
8
12
:30
:28
.13
8
12
:30
:28
.38
8
12
:30
:28
.63
8
12
:30
:28
.88
8
12
:30
:29
.13
8
12
:30
:29
.38
8
12
:30
:29
.63
8
12
:30
:29
.88
8
12
:30
:30
.13
8
12
:30
:30
.38
8
12
:30
:30
.63
8
12
:30
:30
.88
8
12
:30
:31
.13
8
12
:30
:31
.38
8
12
:30
:31
.63
8
12
:30
:31
.88
8
12
:30
:32
.13
8
12
:30
:32
.38
8
12
:30
:32
.63
8
12
:30
:32
.88
8
12
:30
:33
.13
8
12
:30
:33
.38
8
12
:30
:33
.63
8
12
:30
:33
.88
8
12
:30
:34
.13
8
12
:30
:34
.38
8
Kerawalap 220 kV Kotugoda 220 kV Kolonnawa 132 kV Frequency
53.729 Hz50.453 Hz
30
4.2 Analysis of the observations made during the visit to System Control Centre
The restoration attempts in the Mahaweli Complex are taken for analysis because it contains unusual
complications which were not seen in the other two complexes.
Restoration Attempt 01: The System Control had initiated the restoration with the Victoria Power
Station Unit 02, which was ready at that time (12:57 pm). They had restored up to Biyagama Grid
Substation in 7 minutes, and the Unit 02 had tripped due to excitation stage II failure, which is an
internal fault in the generator, after 6 minutes at 13:10. This could be due to load turning OFF in the
33 kV feeder taken for restoration.
Restoration Attempt 02: The second attempt had been with the Victoria Power Station Unit 01 at
13:21 which had been dispatching for 3 minutes and failed at 13:27 due to over frequency tripping,
which could be due to load tripping.
Restoration Attempt 03: The third attempt also with the Victoria Power Station Unit 01 at 13:33 which
also had failed at 13:36 due to excitation stage II failure, which is an internal fault in the generator, a
similar reason as in the first attempt, 3 minutes after restoring.
Restoration Attempt 04: Since there are lesser number of circuit breakers to Biyagama Grid
Substation, the fourth attempt had been from the Kothmale Power Station Unit 03 at 13:36, which
had energized Biyagama Grid Substation in 3 minutes at 13:41 and subsequently the Kotugoda,
Aniyakanda and Sapugaskanda Grid Substations respectively by 14:05. However, Kothmale Power
Station Unit 03 had tripped due to over frequency after delivering 30 MW for 12 minutes. At this point
a system frequency oscillating at a lower frequency than 50 Hz had been observed.
Restoration Attempt 05: The fifth attempt had been started from Kothmale Power Station Unit 01
and subsequently added Kothmale Power Station Unit 02 but again tripped due to over frequency
after about one hour of dispatching power while delivering 85 MW with the system frequency
oscillating at a lower variable frequency of approximately 8 Hz. The associated frequency variation
against time is shown in Fig. 4.2b.
Restoration Attempt 06: The sixth attempt had been from Kothmale Power Station Unit 03, tripped
the generator this time after 10 minutes delivering 23 MW due to over frequency.
Restoration Attempt 07: The seventh attempt had been successful, with the Kothmale Power Station
Units 01 and 02 with the Unit 02 generator governor on manual mode, i.e., with the frequency control
loop being open.
31
Fig. 4.2b
The failures in the restoration of the first and the third attempts were solely due to internal faults of
the units 01 and 02 of the Victoria Power Station.
The failures in the restoration of the second and the sixth attempts can be attributed to over frequency
caused by one or many of the large loads in the selected 33 kV feeders got tripped or a large solar PV
system coming into operation, which is reflected at the generator as an over frequency.
The failures of the fourth and the fifth restoration attempts could be attributed to over frequency due
to generator electrical frequency oscillating at a low frequency in the range of 10 Hz. This has
happened with two generators in the same busbar. Hence it is an electrically tight coupled situation
caused by reduced damping in the electrical governor of one or both of the generators. In this case,
it must be Kothmale Power Station Unit 02 generator governor, because in the 5th attempt, Units had
tripped when Unit 02 added and the 7th attempt was successful when the Unit 02 generator governor
was put on manual mode., i.e., frequency control loop being open. Improving the damping in the Unit
02 generator governor of the Kothmale Power Station may be needed.
It is further observed that the SCC has not followed the Restoration Manual in restoring the
transmission network. The explanations received was that the generators at the Victoria Power
Station were ready to connect while the others were not. From the surface, the long delay over six
hours to restore the transmission network can be attributed to the fact that SCC has not followed the
restoration manual. However, there is no guarantee that following the Restoration Manual would
ensure quicker restoration, because the restoration attempts of the Colombo City via the Mahaweli
Complex have failed in the 1st, 3rd, 4th and the 5th attempts due to internal faults in the stations, which
44
45
46
47
48
49
50
51
52
53
54
55
8/1
7/2
02
0 3
:32
:00
PM
8/1
7/2
02
0 3
:32
:04
PM
8/1
7/2
02
0 3
:32
:08
PM
8/1
7/2
02
0 3
:32
:12
PM
8/1
7/2
02
0 3
:32
:16
PM
8/1
7/2
02
0 3
:32
:20
PM
8/1
7/2
02
0 3
:32
:24
PM
8/1
7/2
02
0 3
:32
:28
PM
8/1
7/2
02
0 3
:32
:32
PM
8/1
7/2
02
0 3
:32
:36
PM
8/1
7/2
02
0 3
:32
:40
PM
8/1
7/2
02
0 3
:32
:44
PM
8/1
7/2
02
0 3
:32
:48
PM
8/1
7/2
02
0 3
:32
:52
PM
8/1
7/2
02
0 3
:32
:56
PM
8/1
7/2
02
0 3
:33
:00
PM
8/1
7/2
02
0 3
:33
:04
PM
8/1
7/2
02
0 3
:33
:08
PM
8/1
7/2
02
0 3
:33
:12
PM
8/1
7/2
02
0 3
:33
:16
PM
8/1
7/2
02
0 3
:33
:20
PM
8/1
7/2
02
0 3
:33
:24
PM
8/1
7/2
02
0 3
:33
:28
PM
8/1
7/2
02
0 3
:33
:32
PM
8/1
7/2
02
0 3
:33
:36
PM
8/1
7/2
02
0 3
:33
:40
PM
8/1
7/2
02
0 3
:33
:44
PM
8/1
7/2
02
0 3
:33
:48
PM
8/1
7/2
02
0 3
:33
:52
PM
Freq
uen
cy (
Hz)
Time
Frequency oscillations during Restoration Attempt 05
32
are beyond the control of the SCC and they are not visible until the units are connected. The
Restoration Manual also assumes that the machines are healthy and can be started and connected
reliably when a need arises. Probably it is due to this reason, the staff of the SCC has lack of confidence
Restoration Manual. In addition, it is a fact that the accuracy of the Restoration Manual can not be
tested practically. Therefore, it is vital that such a manual is prepared after rigorous dynamic studies
of the system using accurate dynamic models of the network components in the system, taking all
possible practical scenarios into consideration.
4.3 Observations on the CCTV records at the Systems Control Center
The CCTV records during the event on the 17th August 2020 at the CEB Systems Control Center show
the activities of the two engineers who were on duty during the time of fault. It is observed that both
the engineers were paying attention on laptop computers in front of them and it could be seen that
one of them browsing the Internet while the other one accessing the Facebook. It is questionable as
to whether the code of ethics in line with standard professional practicing would accommodate such
a conduct, related to a professionally challenging and extremely sensitive task where urgent attention
cannot be compromised. Even though they can’t avoid a blackout of this nature, there can be some
alarms/signals which could be attended and prevent any possible failures in the system if they keep
eye on the system.
33
5. Conclusions
1. The three phase to ground fault at Busbar No. 02 of the Kerawalapitya GSS on the 17th August
2020 at 12.30.27.018 hrs. has caused the island wide power failure. This is due to incorrect
operation of the isolator and earth switches of the bus coupler bay during a routine maintenance
by the Electrical Superintendent, in charge of the Kerawalapitiya GSS. It is also noticed that the
said maintenance work has been carried out by the electrical superintendent and his staff without
being supervised by his superiors. It is observed that has been no evidence of clear instructions
provided by Engineers of the Transmission Operation and Maintenance division who are
responsible for maintenance activities at Kerawalapitiya GSS to the ES in charge and his staff at
Kerawalapitiya GSS.
2. Three Phase to earth fault occurred during a routine maintenance/inspection task at the
Kerawalapitiya GSS has caused tripping of all transmission lines and generator transformer circuit
breakers connected to this GSS by the operation of busbar protection. By-passing the general
interlocking mechanism between the Bus Coupler Circuit Breaker and the Busbar 2 of the
Kerawalapitiya GSS without considering the potential risk of human error in carrying out the said
routine maintenance/inspection task is the root cause for this incident.
3. Due to the three phase to ground fault at Kerawalapitiya GSS, due to relatively longer time taken
to clear the said fault and due to lack of system inertia, the rate of change of system frequency
has increased beyond the current settings of the generator df/dt - Rate Of Change of Frequency
(ROCOF) protection relay of the LVPS. As a result, the generator transformer circuit breakers of all
three units of the LVPS have tripped in the chronological order Unit 03, Unit 02 and Unit 01,
making the LVPS unavailable to the national grid of Sri Lanka, which triggered cascaded under
frequency tripping of other generators in the network, despite automatic load shedding observed
in the post fault system frequency plot.
4. The SCC has not adhered to the restoration procedure stipulated in the Restoration Manual to
restore the power to the Colombo City. However, it cannot be concluded that the restoration got
delayed over six hours solely due to not following the Restoration Manual because out of 6
unsuccessful attempts to restore Colombo City via the Mahaweli Complex, 4 attempts failed
because of internal problems in the generator units, which are beyond the control of SCC and will
only popup once they are attempted.
5. The first option given in the Restoration Manual to restore Colombo City, i.e., using GT 7 machine
in Kelanithissa Power Station to energize the Kolonnawa GSS after getting auxiliary supply from
frame V Gas Turbines, perhaps the fastest option, has not been available due to a long overdue
repair. Because of its nonavailability without an alternative, has also contributed significantly to
the long delay in the restoration.
6. The generator units LVPS had been available as Unit 01 until 12:42, Unit 03 until 13:33 and Unit
02 until 14:51, where there was no energized transmission network to feed in. Had there been a
faster transmission network restoration, say via Gas Turbines at Kelanithissa Power Station, the
system could have been restored about 4 hours earlier.
7. The governors of the hydro-power machines, particularly in the Kotmale Power Station, in manual
mode during the system restoration had enabled a stable ramping than the automatic governor
control at the system restoration.
8. No load prediction tools have been used to predict the feeder loads at the time of loading the
feeders. This has led to mismatch between generation and loads, which caused over frequency
and under frequency tripping (ex: the 2nd and the 6th restoration attempts of via the Mahaweli
Complex).
34
6. Recommendations
1. Establishing a standard compliant, systematic, foolproof, safe procedures and maintenance
protocols during operation and maintenance in CEB is highly recommended. The
implementation of these procedures will have to be continuously monitored and supervised by
properly qualified, professionally trained, knowledgeable, experienced and skilled personnel. It
is further proposed a performance evaluating annual appraisal system which will help to
improve the above attributes of the CEB staff.
2. Establishing a risk management mechanism in order to determine the proper mix of preventive
measures, mitigation levels, shift or retention of risks and consequent level of robustness of
Operations & Maintenance protocols that would indicate the positive impact on the overall
system.
3. Establishing the Long Term Generation Expansion Plan in line with the current government
policy, considering the future requirements of the Sri Lankan power system in view of the
optimum generation mix based improved system stability, reliability, energy security and least
cost to the society.
4. Conducting a comprehensive review by CEB on the settings of the protection relays in the entire
network as it is evident that if the Kerawalapitiya Busbar protection 2 had operated quicker than
154 ms, the lack of inertia in the system would have passed unnoticed and the system could
have been prevented from going into a total blackout. It is further recommended to review the
existing protection strategy for frequency instability criteria used for df/dt in the LVPS.
5. Synchronizing all clocks in the national grid with the GPS clock is recommended so that the data
logged is consistent and easy to follow.
6. Reinstating Colombo City restoration Option 1 as per the current restoration manual, i.e., via Kalanithissa Power Station. Failing that, it is recommended to install and commission 70 to 100 MW gas turbines with frequency control, black start, ability to operate at leading and lagging power factors, stability at small loads, line charging for capacitive loads, features.
7. Studying and identifying the best mode of governor controls and settings to facilitate fast system restoration. It is evident that an improved damping in the frequency control loop in the generator unit 2 of the Kothmale Power Station could have saved a significant amount of time to restore the system.
8. Investigating better means of using past daily loading records of the feeders in the distribution network to predict more accurate load demands as this would drastically reduce the time to restore the complete transmission network.
9. Ensure the availability of an updated system restoration manual verified through accurate dynamic simulation studies of the system, so that the system can be restored reliably when it is used as a guide.
10. Staff involved in the control centres of the power plants and as well as those in the SCC, be receiving continuous professional training to become experts in their duties and to be disciplined to follow and handle emergency situations at the highest benchmarked skill levels.
11. Enforcing strict discipline on the staff of SCC while at work, recognizing the gravity of the correct execution of their duties.
35
Signed by
………………………………………
Dr. Lilantha Samaranayake Department of Electrical & Electronic Engineering Faculty of Engineering University of Peradeniya
top related