Following the March, 2022 disclosure of TLStorm—a set of critical vulnerabilities in APC Smart-UPS devices that allow an attacker to take control of Smart-UPS devices— Armis researchers have discovered five new vulnerabilities that share a common source. The root cause for these vulnerabilities was a misuse of NanoSSL, a popular TLS library by Mocana. By exploring similar faulty implementations of the Mocana NanoSSL library in network switches, Armis has discovered these new vulnerabilities in the implementation of TLS communications in multiple models of Aruba (acquired by HP) and Avaya (acquired by ExtremeNetworks) network switches.
Using the Armis knowledgebase—a database of over 2 billion assets—our researchers identified dozens of devices using the Mocana NanoSSL library. The findings include two popular network switch vendors that are affected by a similar implementation flaw of the library, leading to remote code execution (RCE) vulnerabilities that can be exploited over the network. While UPS devices and network switches differ in function and levels of trust within the network, the underlying TLS implementation issues allow for devastating consequences, if attackers are able to identify and exploit TLStorm vulnerabilities.
This new set of vulnerabilities, dubbed TLStorm 2.0, exposes vulnerabilities that could allow an attacker to take full control over these switches. The exploitation of these RCE vulnerabilities can lead to:
- Breaking of network segmentation, allowing lateral movement to additional devices by changing the behavior of the switch
- Data exfiltration of corporate network traffic or sensitive information from the internal network to the internet
- Captive portal escape
These research findings are significant as they highlight that the network infrastructure itself is at risk and exploitable by attackers, meaning that network segmentation can no longer be considered a sufficient security measure.
To learn more about the TLStorm 2.0 vulnerabilities and potential impact to organizations, see our detailed blog here.
TLStorm: Three critical vulnerabilities discovered in APC Smart-UPS devices can allow attackers to remotely manipulate the power of millions of enterprise devices.
Armis has discovered a set of three critical vulnerabilities in APC Smart-UPS devices that can allow remote attackers to take over Smart-UPS devices and carry out extreme attacks targeting both physical devices and IT assets. Uninterruptible power supply (UPS) devices provide emergency backup power for mission-critical assets and can be found in data centers, industrial facilities, hospitals and more.
APC is a subsidiary of Schneider Electric, and is one of the leading vendors of UPS devices with over 20 million devices sold worldwide. If exploited, these vulnerabilities, dubbed TLStorm, allow for complete remote take-over of Smart-UPS devices and the ability to carry out extreme cyber-physical attacks. According to Armis data, almost 8 out of 10 companies are exposed to TLStorm vulnerabilities. This blog post provides a high-level overview of this research and its implications.
Why is this research important?
Armis proactively researches various assets to help security leaders protect their organizations against new threats, including those targeting IT assets and cyber-physical systems (CPS). As part of this research, we’ve investigated APC Smart-UPS devices and the way they communicate with their respective remote management and monitoring services.
Attackers can remotely take over devices via the Internet.
The latest APC Smart-UPS models are controlled through a Cloud connection. Armis researchers found that an attacker exploiting the TLStorm vulnerabilities could remotely take over devices via the Internet without any user interaction or signs of attack. As a result, attackers can perform a remote-code execution (RCE) attack on a device, which in turn could be used to alter the operations of the UPS to physically damage the device itself or other assets connected to it.
Recent global events highlight the critical nature of TLStorm.
Attacks targeting the power grid and the apparatus of appliances within it have taken place in the past, with the most famous one being the Ukraine Power Grid attack that occurred in 2015 — in which UPS devices (as well as many other types of devices), have been remotely hacked which lead to wide-scale power outage. Recent events in the Russia-Ukraine conflict have raised concerns by US officials that the US power grid would be targeted by Russia via cyber attacks. The discovery of TLStorm vulnerabilities underlines the volatility of devices within enterprise environments responsible for power reliability, and stresses the need to act and protect such devices against malicious attacks.
Uninterruptible Power Supply (UPS)
Uninterruptible power supply (UPS) devices provide emergency backup power for mission-critical assets. In cases where a power disruption could cause injuries, business downtime, or data loss, UPS devices help ensure high availability of critical technology in:
- Server rooms
- Medical facilities
- OT/ICS environments
- Residences
What are the vulnerabilities?
The set of discovered vulnerabilities include two critical vulnerabilities in the TLS implementation used by Cloud-connected Smart-UPS devices, as well as a third critical vulnerability, a design flaw, in which firmware upgrades of all Smart-UPS devices are not properly signed and validated.
Two of the vulnerabilities involve the TLS connection between the UPS and the Schneider Electric cloud. Devices that support the SmartConnect feature automatically establish a TLS connection upon startup or whenever cloud connections are temporarily lost.
- CVE-2022-22806 – TLS authentication bypass: A state confusion in the TLS handshake leads to authentication bypass, leading to remote code execution (RCE) using a network firmware upgrade.
- CVE-2022-22805 – TLS buffer overflow: A memory corruption bug in packet reassembly (RCE).
These vulnerabilities can be triggered via unauthenticated network packets without any user interaction (ZeroClick attack).
The third vulnerability is a design flaw in which the firmware updates on affected devices are not cryptographically signed in a secure manner. This means an attacker could craft malicious firmware and install it using various paths, including the Internet, LAN, or a USB thumb drive. This can allow attackers to establish long-lasting persistence on such UPS devices that can be used as a stronghold within the network from which additional attacks can be carried.
- CVE-2022-0715 – Unsigned firmware upgrade that can be updated over the network (RCE).
Abusing flaws in firmware upgrade mechanisms is becoming a standard practice of APTs, as has been recently detailed in the analysis of the Cyclops Blink malware, and improper signing of firmwares of embedded devices is a recurring flaw in various embedded systems. A previous vulnerability discovered by Armis in Swisslog PTS systems (PwnedPiper, CVE-2021-37160) was a result of a similar type of flaw.
Armis disclosed these vulnerabilities to Schneider Electric on October 31, 2021. Since then, Armis has worked with Schneider Electric to create and test a patch, which is now generally available.
What are the risks?
The fact that UPS devices regulate high voltage power, combined with their Internet connectivity—makes them a high-value cyber-physical target. In the television series Mr. Robot, bad actors cause an explosion using an APC UPS device. However, this is no longer a fictional attack. By exploiting these vulnerabilities in the lab, Armis researchers were able to remotely ignite a Smart-UPS device and make it literally go up in smoke.
Since the TLS attack vector can originate from the Internet, these vulnerabilities can act as a gateway to the internal corporate network. Bad actors can use the TLS state confusion to identify themselves as the Schneider Electric cloud and collect information about the UPS behind the corporate firewall. They can then remotely update the UPS firmware and use the UPS as the entry point for a ransomware attack or any other type of malicious operation.
Attacks Targeting Cyber-Physical Systems Are On The Rise
Cyber-Physical Systems (CPS) are computerized systems that operate devices with real-world interactions, for example, automatic doors, PLCs, MRI machines and smart vehicles. The increasing adoption of IoT and CPS devices has created a wealth of new targets for bad actors.
The destructive implications of network-based attacks resulting in real-world damage are no longer theoretical. In 2014 there was an attack against a German steel mill where hackers infiltrated the mill’s network and tampered with a blast furnace shutdown mechanism. This attack resulted in a massive explosion, which luckily did not cause any casualties.
Illustrating the cyber-physical effect of the TLStorm attack, Armis researchers were able to damage a Smart-UPS over the network with no user interaction:
TLStorm: 3 vulnerabilities. Millions of devices at risk.
Affected devices
SmartConnect Family
| Product | Affected Versions | CVEs |
|---|---|---|
| SMT Series | SMT Series ID=1015: UPS 04.5 and prior | CVE-2022-22805 CVE-2022-22806 CVE-2022-0715 |
| SMC Series | SMC Series ID=1018: UPS 04.2 and prior | CVE-2022-22805 CVE-2022-22806 CVE-2022-0715 |
| SMTL Series | SMTL Series ID=1026: UPS 02.9 and prior | CVE-2022-22805 CVE-2022-22806 CVE-2022-0715 |
| SCL Series | SCL Series ID=1029: UPS 02.5 and prior SCL Series ID=1030: UPS 02.5 and prior SCL Series ID=1036: UPS 02.5 and prior SCL Series ID=1037: UPS 03.1 and prior | CVE-2022-22805 CVE-2022-22806 CVE-2022-0715 |
| SMX Series | SMX Series ID=1031: UPS 03.1 and prior | CVE-2022-22805 CVE-2022-22806 CVE-2022-0715 |
Smart-UPS Family
| Product | Affected Versions | CVEs |
|---|---|---|
| SMT Series | SMT Series ID=18: UPS 09.8 and prior SMT Series ID=1040: UPS 01.2 and prior SMT Series ID=1031: UPS 03.1 and prior | CVE-2022-0715 |
| SMC Series | SMC Series ID=1005: UPS 14.1 and prior SMC Series ID=1007: UPS 11.0 and prior SMC Series ID=1041: UPS 01.1 and prior | CVE-2022-0715 |
| SCL Series | SCL Series ID=1030: UPS 02.5 and prior SCL Series ID=1036: UPS 02.5 and prior | CVE-2022-0715 |
| SMX Series | SMX Series ID=20: UPS 10.2 and prior SMX Series ID=23: UPS 07.0 and prior | CVE-2022-0715 |
| SRT Series | SRT Series ID=1010/1019/1025: UPS 08.3 and prior SRT Series ID=1024: UPS 01.0 and prior SRT Series ID=1020: UPS 10.4 and prior SRT Series ID=1021: UPS 12.2 and prior SRT Series ID=1001/1013: UPS 05.1 and prior SRT Series ID=1002/1014: UPSa05.2 and prior | CVE-2022-0715 |
Technical Overview
As noted above, TLStorm is a set of three critical vulnerabilities. One is in the firmware signing of almost all APC Smart-UPS devices. The other two relate to the TLS implementation of the Smart-UPS devices with the “SmartConnect” feature, which automatically connects devices to the Schneider Electric management cloud.
Firmware upgrade vulnerability (CVE-2022-0715)
It is common practice to sign firmware files cryptographically and to check the signature during a firmware update. The APC Smart-UPS firmware is encrypted with symmetrical encryption, but is not cryptographically signed.
Our researchers were able to exploit the following key design flaws to fabricate a malicious firmware that was accepted by the Smart-UPS as official valid firmware:
- All firmware for Smart-UPS devices of the same model use the same encryption key.
- Symmetrical encryption: The same key is used for encryption and decryption, and the key can be extracted from a physical device.
- No signing mechanism exists.
The combination of these flaws allows an attacker to “upgrade” Smart-UPS devices over the network with customized and malicious firmware.
The firmware upgrade process depends on the specific model of the UPS device:
- The latest Smart-UPS devices featuring the SmartConnect cloud connection functionality can be upgraded from the cloud management console over the Internet.
- Older Smart-UPS devices which use the Network Management Card (NMC) can be updated over the local network.
- Most Smart-UPS devices can also be upgraded using a USB drive.
TLS vulnerabilities (CVE-2022-22805 and CVE-2022-22806)
Since software developers can’t reinvent the wheel every time they write new code, developers must rely on third-party code libraries for software development. In the case of Log4j2, almost every user of the library unknowingly inherited a remote code execution vulnerability (Log4Shell).
The root cause for both of the TLS vulnerabilities is improper error handling of TLS errors in the TLS connection from the Smart-UPS and the Schneider Electric cloud. APC uses Mocana nanoSSL as the library responsible for TLS communications. The library manual clearly states that library users should close the connection when there is a TLS error. In the APC usage of this library, however, some errors are ignored, leaving the connection open but in a state that the library was not designed to handle.
Smart Connect
The latest generation of Smart-UPS models implement a feature called SmartConnect, which is a dedicated Ethernet port through which the device will connect to the cloud service and allow remote management of the device.
TLS Authentication Bypass (CVE-2022-22806)
Ignoring the nanoSSL library errors causes the UPS to cache the TLS key in its uninitialized state. When an attacker uses the TLS resumption functionality, the uninitialized key (all zero) is fetched from the cache and the attacker can communicate with the device as if it was a genuine Schneider Electric server. As a seemingly verified server, the attacker can issue a firmware upgrade command and remotely execute code over the UPS device. The attack flow can be seen in the following diagram. On the left is a normal TLS session setup and resumption and on the right is the exploit handshake that is only possible due to the improper error handling:
TLS reassembly buffer overflow (CVE-2022-22805)
The same root cause – ignoring the nanoSSL library errors, leads to a memory vulnerability in the reassembly of TLS packets. The TLS reassembly feature allows TLS record fragmentation. The record is assembled chunk by chunk until the full record is received. An attacker can leverage the ignored nanoSSL library error and trigger a pre-authentication heap overflow condition that can lead to remote code execution.
How can you secure your UPS devices?
There are a few steps that you can take to minimize the risk of an attack. Armis recommends using all three mitigations and not just updating the device.
- Install the patches available on the Schneider Electric website.
- If you are using the NMC, change the default NMC password (“apc”) and install a publicly-signed SSL certificate so that an attacker on your network will not be able to intercept the new password. To further limit the attack surface of your NMC, refer to the Schneider Electric Security Handbook for NMC 2 and NMC 3.
- Deploy access control lists (ACLs) in which the UPS devices are only allowed to communicate with a small set of management devices and the Schneider Electric Cloud via encrypted communications.
How can Armis help?
Armis customers can leverage the Armis platform to:
- Quickly discover all Smart-UPS devices that need to be patched or protected from exploit attempts to plan and prioritize mitigation efforts. Armis not only detects the existence of these devices on the network, it can also provide valuable data about the device owner and physical location to expedite mitigation efforts.
- Detect exploit attempts in real-time and orchestrate the response through integrations with your IT and security tech stack.
- Continue to track the long tail of ‘still to be patched’ assets, and new assets that might be vulnerable. Ensure these assets aren’t targeted by exploit attempts at any time, and do not pose a threat to your network.
The Armis platform provides the required visibility to ensure all your assets, including cyber-physical assets that are not covered by traditional security solutions, are continuously protected against the latest threats.
Final notes
UPS devices, like many other digital infrastructure appliances, are often installed and forgotten. Since these devices are connected to the same internal networks as the core business systems, exploitation attempts can have severe implications.
It’s important for security professionals to have complete visibility of all assets, along with the ability to monitor their behavior, in order to identify anomalies and/or exploit attempts. However traditional security solutions do not cover these assets. As a result, they remain “unseen” and therefore expose the organization to significant risk.
Research Presentations
Armis experts will discuss the TLStorm research during the following virtual and in-person events:
- Webinar (Wednesday, March 30, 1:00 p.m. EST): Up in Smoke: 3 Critical Vulnerabilities in Smart-UPS Devices
- LinkedIn Live (Thursday, March 10, 2022, 12:00 p.m. EST): Lightning from the Cloud
- Nullcon Berlin 2022 (April 5-9, 2022) – Finding and Exploiting Critical Bugs in TLS Libraries used by “Smart” UPS Devices
- Black Hat Asia 2022 (May 10-13, 2022) – Like Lightning From the Cloud: Finding RCEs in an Embedded TLS Library and Toasting a Popular Cloud-connected UPS
Additional Resources
- Technical White Paper
- TLStorm 2.0 Technical White Paper
- Join the conversation with Armis on LinkedIn, Twitter, and YouTube
- Visit Armis.com to learn more about our agentless device security platform
Want to discuss this with one of our experts and/or schedule a demo? – Contact us here.