Enabling Secure Boot V2 on ESP32 Platforms in Production
This is the second of a series of two blog posts to talk about how to easily enable secure boot and sign firmware images for ESP32 series of chips.
In the first blog post, we talked about enabling secure boot V2 (SBV2) on ESP32 platforms in development, and released the developer tools to make firmware signing easy for secure boot enablement in development. In this post, we will talk about how to use the Thistle Web app to enable secure boot (V2) on ESP32 platforms in production, where the firmware signing keys are managed with the Google Cloud Key Management system (Cloud KMS).
Firmware Signing Key Management in Production
Recall from the previous blog post that for secure boot enablement, Thistle recommends to distinguish two scenarios: key management in development and in production.
If one would like to enable SBV2 on production devices, it's a good idea to perform firmware development and testing on SBV2-enabled development units. Ideally, production firmware should be a nearly faithful mirror of development firmware, where the two differ only in bits associated with public keys and signatures. Production and development firmware should be similar, so that after development testing is completed, a minimum level of additional testing would be needed on production firmware. Production and development firmware should be signed with different keys, because the security and operational requirements on them are different.
Development signing: Development firmware is typically flashed onto only a handful devices during earlier phases of hardware development. Signing key confidentiality in this case is usually not crucial, because development hardware is supposed to be physically more tightly controlled, and not not plugged into production systems. The development signing process should be easy to allow fast feedback loops.
Production signing: Production firmware runs on production devices, which usually ship in large volumes, and are used in the real world by end users. For this reason, production signing keys are typically highly confidential, and shall not be exposed to software or human operators. The production signing process in many cases should be easy to allow fast firmware releases.
The development tools Thistle released in the GitHub project thistletech/esp32-devenvs facilitate building and signing ESP32/ESP32-S2/ESP32-S3 images using file-based private keys, which satisfy development signing requirements. We learned from an earlier blog post that although storing a private key in a computer's filesystem is simple for development and testing, there are many reasons it may not meet production requirements. More specifically,
The private key file can be easily copied, and subsequently leaked, by an entity, e.g., careless or malicious personnel, a backup program, or malware running on the computer.
Even if the private key is password protected, the password can be brute-forced offline.
It isn't a straightforward task to securely back up a private key file. On the other hand, not backing up a private key file used for production signing can easily be a production continuity risk - imagine the consequences of a disk failure.
Leaking firmware signing keys can lead to severe impacts - remember the leaked MSI UEFI signing key incident not too long ago. It's not impossible, but usually difficult and costly to create and maintain a release process that can mitigate the above risks associated with private key management. Hence, a proper key management system is desired for secure boot signing.
ESP32 SBV2 Enablement with Thistle
Thistle provides a secure boot enablement Web service for production signing of ESP32* (we currently support ESP32 ECO3 onward, ESP32-S2 and ESP32-S3 series) firmware images, where the secure boot signing keys are created, used, and managed through Google Cloud KMS. This Thistle service has the following properties.
Signing key security. A customer's production signing key is created in the Cloud KMS, and is never exposed outside of the Cloud KMS. No entity, including Thistle employees, Thistle's users, and any Thistle software, has access to plaintext production signing keys, and thus cannot leak them, deliberately or by accident. In general, a signing key's lifecycle is managed in the Cloud KMS, allowing us to leverage Cloud KMS provided features, such as customer control, identity and access management (IAM), key access control and monitoring, and key backups.
Ease-of-use. Production signing key creation is transparent and simple. Production signing of firmware images is done in a few button clicks (or API calls).
Robustness. As long as locally signed firmware images (inputs to the Thistle service, produced using tools in thistletech/esp32-devenvs) work on development-fused ESP32* units, it's guaranteed that production signed firmware images (outputs of the Thistle service) will be readily flashable and work on production-fused units.
The following video demonstrates how to use the Thistle Web app to obtain production signed ESP32 firmware images from development images that were created using tools in thistletech/esp32-devenvs, and signed with non-security-critical, dummy keys.
Espressif DevCon 23 Secure Boot V2 Production Firmware Signing Demo by Thistle Technologies
How Thistle's Production Signing System Works
As mentioned earlier in this post, it's desirable that production firmware be a nearly faithful mirror of development firmware, where the two differ only in bits associated with public keys and signatures. Given that this is a practice adopted in the firmware development process, the idea behind Thistle's production signing system is simple: Starting from input firmware images signed with non-security sensitive keys (e.g., development keys or dummy keys), we parse the images, compute cryptographic information (including the signatures) associated with the KMS-backed production signing key, and patch the input images to insert the computed cryptographic information to the precise locations to obtain the production signed firmware images.
For production signing of ESP-IDF firmware images (bootloader and app images), the cryptographic information to compute and patch is the signature block. A valid signature block associated with a signing key can be generated using the rest of the firmware image (i.e., the image minus the signature block) and a signing key. When we generate the signature block using a production signing key managed in a proper key management system (e.g., Google Cloud KMS), and replace the old signature block with the generated one, we get a production signed image (illustrated below).
Signature block patching for production signed firmware image
Thistle has implemented this idea for ESP32 chips (including ESP32, ESP32-S2, ESP32-S3 families), for the ESP-IDF and MCUboot+ZephyrOS firmware bundle types. Technically, it's slightly more involved to create production signed images in the latter case, due to the need of additional public-key patching as well as checksum recomputation and patching. But the idea is the same. More importantly, the production signing keys never leave our Trusted Computing Base (TCB) - the Cloud KMS, and thus can't be exposed by humans or software.
Thistle's ESP32 secure boot signing system implements the following service architecture.
Production signing and patching system architecture
The system consists of several micro-services.
The IAM service takes care of user registration, authentication and authorization. It's also where secure boot signing parameters, including production signing keys and a per-Thistle-project signing policy, are configured. A signing policy allows firmware signing to happen only when certain (client/user-controlled) conditions are satisfied, e.g., a joint consent by two authorized firmware signers (dual-control).
The firmware patching service is responsible for fetching, parsing, and patching development or dummy-signed input firmware images, with production signatures obtained from the "code signing service". It pushes production signed firmware to an output storage location for the client to consume. This service does not perform any cryptographic operation, and thus minimizes the risk of cryptographic information leakage in it.
The code signing service is a lean service that interacts with the Cloud KMS to create digital signatures associated with KMS-backed firmware signing keys. While the cryptography largely happens inside the Cloud KMS, this service enforces the signing policy (e.g., dual-controlled signing), and allowlists the KMS key URIs, implementing a least privilege design. The code signing service can be deployed on a per-Thistle-project basis, maximizing signing key isolation and minimizing the impact of a service compromise (for whatever reason).
As seen in the earlier demo video, the Thistle Web app provides a straightforward way of using the system, and makes production signing of ESP32 firmware images a few button clicks.
Currently, all ESP32 SBV2 signing keys used by the system are managed in Thistle's GCP project (again, we still don't have access to the private keys). In the future, we will add support of Bring Your Own Key (BYOK). Stay tuned.
Sign Up with Thistle
Sign up with Thistle for free today to try out our ESP32 production signing feature!
You may use the following dummy signed IDF bootloader and application images (for ESP32 ECO3 and onward chips) as sample input files to upload in the UI:
