The "mystery" of what happened at the "Núñez de Balboa" photovoltaic power plant is, to this day, one of the most significant unresolved questions of the Iberian blackout. In this post I elaborate on this issue by using open-source intelligence, official reports and a bit of reverse engineering.
Introduction
In a recent official hearing of the Spanish Senate commission investigating the blackout, the president of REE (Spain’s TSO), Beatriz Corredor, stated the following.
"Let me tell you why we believe, why we know, that the whole process starting at 12:03 began in Extremadura, because we have physical evidence and therefore we can demonstrate that the extraordinary 0.6 Hz oscillation, that began at 12:03, was due to poor management and poor control of a high-power photovoltaic plant installed in the province of Badajoz[...] The same plant had had a similar failure, proven and documented, the previous year, and that the people in charge of that plant themselves said they had been conducting an experiment in terms of how to manage that plant."
That’s quite a statement, for which REE apparently has physical evidence and documentation that I assume will be made public at some point. This could happen in several ways: through one of the official reports still pending (CNMC or ENTSO-E final report) or directly in court during the highly likely legal proceedings where this matter will eventually be resolved. In fact, just two days ago, Iberdrola announced that they will take Beatriz Corredor to court over these statements.
This unnamed plant in Badajoz has been in the spotlight since the release of official reports from both the Government and REE. I have already covered this in previous editions (I and II). Long story short: the local 0.6 Hz oscillation (orange), which allegedly originated at this plant, is a strong candidate for the event that excited the subsequent inter-area 0.2 Hz oscillation (blue). The events resulting from this double-oscillation scenario, among others, ultimately led to the collapse of the power grid.
As a result, if proven, this plant would be a strong candidate for bearing a significant portion of the blame for the blackout, with all the monetary implications that entails. With so much at stake, none of the official reports publicly disclosed the name of this plant, although it was immediately identified by Spanish media outlets following the publication of the redacted government report: Iberdrola’s "Núñez de Balboa". It is one of Europe’s largest photovoltaic power plants, operated by a major player in the energy sector that has a 'complicated' relationship with both the Government and the TSO. All the ingredients for an explosive situation.
Buried among the dozens of figures in the ENTSO-E factual report, one immediately caught my attention: its description referred, once again, to an unnamed plant in Badajoz. It turns out that this figure fills a gap left by missing, or redacted, data in the other official reports.
The Figure 2-59
In the page 60 of the ENTSO-E factual report we can find the figure 2-59. Actually, there is a mistake in the report, as the paragraph introducing this figure refers instead to Figure 2‑58, which is unrelated and was already introduced in a previous paragraph.
"Figure 2-58 [it should be 2-59] shows fluctuations of active power with an amplitude of around 200MW and reactive power with an amplitude of around 180Mvar, occurring between 12:03 and 12:08."
The description of Figure 2‑59 reads as follows: “Active and reactive power generated by a power plant connected in the province of Badajoz". Before analyzing the figure, I recommend that the reader review the context in which it is introduced (see image below). I will return to this point later.
Analyzing the figure
Would ENTSO-E include a figure showing the P/Q time series of a random plant in Badajoz? Of course not. That figure appears in the factual report for a reason, and the plant is none other than "Nuñez De Balboa".
First of all, let's confirm the figure 2-59 actually corresponds to "Nuñez De Balboa". I'll use 4 different elements to demonstrate it, based on the anonymized descriptions found in previous official reports and just a bit of physics.
1. Constant Power Factor.
One of the issues in the Spanish power grid that played a significant role in the blackout was that the IBRs were not required to provide voltage support and therefore operated at a constant power factor (PF). The first thing we can observe in the graph is precisely that, the plant is operating at a constant power factor. We can easily calculate it using the formula cos(arctan(Q/P)). For example, given (per-unit) P = 0.5 and Q = -0.1 we have a PF = 0.98 (inductive) which corresponds to the expected PF for a photovoltaic power plant.
The reader will also notice the nearly instantaneous ramps in the power setpoints, which are characteristic of IBRs.
2. 10:30
According to REE (pag.16), the unnamed plant (the reports refers to this plant as "planta fotovoltaica A") began to oscillate at 10:30, though initially with low amplitude. We can see that exactly at 10:30 P began to oscillate in those terms.
3. 12:03
As the Government report notes (p. 76), at 12:03, the redacted plant's power output began oscillating within seconds, with an estimated peak-to-peak amplitude of 70%. In the figure we can observe this behavior.
4. Performance
According to REE (pag. 5), the plant increased its output from 250 MW to 350 MW and although active power (P) stabilized, reactive power (Q) continued to oscillate. Given that Nuñez de Balboa's grid-allowed capacity is 391 MW, it is clear that this scenario also aligns with the figure. There is not any other photovoltaic power plant in the province of Badajoz that matches this capacity.
Based on these 4 verifiable elements, we can now confirm the figure represents the P/Q time series of "Nuñez de Balboa".
Now let's get back to the ENTSO-E report.
Data (or the lack thereof)
Throughout the report, ENTSO-E highlights the challenge of accessing high-quality data from third-parties in Spain. In fact, some very specific data is missing.
"Overall, the Expert Panel was able to collect a lot of data from TSOs and parties connected to the TSO grids. However, some data remains missing, particularly related to some of the generation trips that occurred before the blackout. Several of the concerned parties (namely the owners of those facilities) informed the Expert Panel that they do not have this fault record data." (ENTSO-E report, pag.8)
I find it quite interesting that the very data crucial for assessing what happened, and for determining whether the disconnections were compliant or not, is missing.
On page 7, the ENTSO-E report explicitly states that "information on any identified malfunctions related to oscillations" was requested from "significant grid users (generators)". As shown in the image below, it is not entirely clear whether Iberdrola responded to this request with data for Nuñez de Balboa, even though, as seen in the analysis of Figure 2-59, the P/Q oscillations were evident.
In this regard, I find the tone used by ENTSO-E particularly suggestive in explaining why they have not yet been able to determine whether the 0.2 Hz oscillation is a forced oscillation, or whether it can be linked to a specific generator, despite the fact that the Nuñez de Balboa figure, although not explicitly named, was included in the factual report.
That is why I do not take any of ENTSO-E’s statements lightly, and there are numerous hints scattered throughout the report.
For me, there are two important unknowns:
- Which data Iberdrola provided to ENTSO-E, and which remained undisclosed or, allegedly, did not exist.
- What caused the malfunction at Nuñez de Balboa, assuming one occurred, as all the evidence seems to support this.
Obviously, I cannot answer these questions, but the idea is to provide verifiable technical details that may help everyone understand what seems reasonable and what does not, so we are better prepared for what may unfold in the coming months (ENTSO-E and CNMC final reports, legal proceedings...). On top of this, from the cyber perspective, it's crucial to fully understand the root causes of this blackout to see if a similar scenario could be replicated in a potential cyber-physical attack.
Inside the "Nuñez de Balboa" PV Power Plant
Introduction
The "Nuñez de Balboa" PV power plant is one of Europe's largest solar photovoltaic installations, with a rated capacity of 500 MW. It is connected to the Spanish transmission network through the 'Bienvenida' 400 kV interconnection facility (owned by REE). This node has a total available capacity of 541 MW, shared exclusively between "Calzadilla B" (150 MW) and "Nuñez de Balboa" (391 MW).
It is worth mentioning that REE’s report states that: “The other plant that evacuates power to the transmission grid through the same interconnection facility, as well as others connected at nearby substations, have been reviewed, and the only one exhibiting oscillations was the one indicated.”
Thus, no oscillations were detected at "Calzadilla B" but only at "Nuñez de Balboa".
Architecture
I created the diagram below to provide an approximate view and a clearer perspective for the next sections, however please note that it does not represent the exact architecture.
Nuñez de Balboa is comprised of 115 power blocks, each consisting of a medium-voltage (30 kV) skid and Power Electronics' HEC V1500 inverters. The plant has a 30/400 kV substation with two transformers and a line position toward REE's "Bienvenida" 400 kV substation. The SCADA system is implemented by DNV's GreenPowerMonitor.
Each of the inverter modules provides a fault-resistant, redundant architecture.
The modules are internally synchronized via the CAN bus. Externally, they support multiple remote communication interfaces for the Power Plant Controller and other plant components.
Each inverter module, which can be discovered by sending a UDP broadcast packet containing the string "pe_identify" (the manufacturer's name is PowerElectronics) to port 505, can be remotely controlled and monitored over ModbusTCP.
The analysis of the hundreds of parameters, including internal ones that can be configured via ModbusTCP, is useful for understanding an inverter's capabilities, both from a forensic and cyber (offensive) perspective.
From the monitoring (forensic) point of view, obviously, each module is able to report the regular real-time parameters such as I, Vdc, P, Q... But there are many others, such as dozens of different fault conditions, warnings and control parameters. In view of the P/Q oscillations, the correlation of these fundamental parameters with those collected from the Power Plant Controller could be used, at least, to determine whether inverter-level or plant-level controller issues were responsible for the oscillations observed in the figure 2-59.
Assuming Nuñez de Balboa's SCADA collected and persisted this data, I am not sure whether Iberdrola shared these datasets with ENTSO-E, but I guess they did not; otherwise, the factual report would likely have looked different.
Malfunction or not?
In their respective reports, ENTSO-E and REE suggest that the 0.6 Hz oscillation could be caused by an internal malfunction within the plant.
- ENTSO-E: "An inverter-based controller that malfunctions and sustains this malfunction could force this effect onto the grid."
- REE: "The grid conditions at the point of connection, short-circuit power, and voltage level have been analyzed, and both were correct. Therefore, it is likely that the oscillation was caused by a malfunction of an internal control or by an internal anomaly within the plant, which should be clarified by the plant’s owner."
The fact that both of them just hint at this possibility seems to confirm that neither received any internal data collected by Iberdrola from inside the plant. In the case of REE, this was somewhat expected, but I had been more optimistic about ENTSO-E. It might be possible that ENTSO-E received some data, but not sufficient to reach a definitive conclusion.
What about Iberdrola? They completely deny any malfunction. Iberdrola and Endesa commissioned a report from the Universidad de Comillas, which has recognized expertise in power systems, to investigate the blackout. The full report has not been publicly released, but a presentation and a summary are available, although with really limited information.
This is an excerpt of their summary:
"The 0.6 Hz oscillation (occurring from 12:03 hours) is an inter-area oscillation between generators of the Iberian system and generators of the rest of the continental European system, since the amplitude of the active power exchange oscillation (470 MW peak-to-peak) is comparable to the amplitude (1480 MW peak-to-peak) of the 0.2 Hz oscillation (occurring from 12:19 hours).
The statements by Red Eléctrica and ENTSO-E regarding the forced nature of this 0.6 Hz oscillation are not supported by evidence and/or analysis. Even if it were a forced oscillation caused by a specific installation, it should not be forgotten that the amplitude of oscillations caused by a forced oscillation depends on the proximity of the forced oscillation frequency to the frequency of one of the system's natural oscillations and on the damping of that natural oscillation."
It is worth mentioning that the authors announced that the report had been shared with ENTSO-E before its factual report was published. I don’t know whether, after studying the ENTSO-E factual report, the authors still stand by their assessment.
And the scientific literature? There are numerous documented cases that describe similar malfunctions of IBRs caused by either plant-level or inverter-level control issues. It wouldn’t be the first time something like this has happened. It should be noted that in most cases, these malfunctions are triggered by disturbances in the grid. On the day of the blackout, there were significant voltage fluctuations, particularly in the southwest nodes, exactly where Nuñez de Balboa is located.
Conclusions
After reading the ENTSO-E factual report, it is hard not to conclude that something occurred at the plant, within the context of the grid disturbances.
However, the grid-following control algorithms implemented in the HEC V1500 Inverters and the PPC are proprietary. There are dozens of parameters that could have been tuned to adjust the plant-level controller, as well as the inverter’s inner and outer control loops, specifically for the "Nuñez De Balboa" plant, both dynamically (during regular operation) and statically (when it was commissioned).
Therefore, it is extremely difficult to fully understand what happened inside the plant unless Iberdrola, or another entity, provides data and/or a coherent and transparent explanation for the P/Q oscillations observed there. The ENTSO-E final report may help clarify this issue, but the exact events may never be fully known.
Before presenting conclusions from a cyber perspective, I would like to clarify that these are generic conclusions intended for future scenarios: There is absolutely nothing in the physical behavior observed in the Spanish power grid on the day of the blackout that could realistically be attributed to threat actors rather than to power system dynamics.
That said, this does not imply that some of the key observed physical behaviors could not be replicated through cyber means. Power systems professionals will surely learn a lot from this incident, but all of us who work on the cyber side need to pay special attention.
There are two events in the Iberian blackout that should be carefully studied:
Distribution-side attacks
The distribution-side issues, with thousands of small-scale and residential (rooftop) solar inverters entering momentary cessation at the same time, had a significant impact on the transmission side. Replicating this by cyber means is totally feasible. It would not be easy or stealthy, but a determined actor could certainly attempt such an operation.
'Nuñez De Balboa'-style attacks.
In the conclusions of an analysis I published in May, a month after the blackout, I wrote the following:
"However, and this is purely speculative, which I generally avoid, there remain some highly sophisticated cyber-physical scenarios, possibly involving long-distance interactions, that could potentially align with what has been observed."
As we have seen, forced oscillations can also exacerbate natural inter-area oscillation modes. Therefore, the potential to generate forced oscillations by cyber means, impacting countries beyond the target plant is interesting, especially in Europe, due to an eastern neighbor whose behavior is not especially friendly.
This has been already explored [1][2] mostly from the academic side. This kind of attack is extremely sophisticated and requires a very precise approach, but technically feasible.
For instance, in this post we have seen how the inverters were remotely controllable via ModbusTCP. Among the controllable parameters, some could potentially be exploited to influence the inverter-level control algorithms, which in turn could induce oscillatory behaviors.
However, this attack vector is both less urgent and unlikely to occur compared with many others that also deserve consideration.