Live Linux kernel patching with progressive timestamped rollouts

The apt package manager is responsible for installing .deb packages on Ubuntu LTS (long-term support) and interim releases, including the .deb package for the Linux kernel. Updating the kernel package requires a system restart, leaving systems vulnerable between the moment the Linux kernel package is installed and when the machine is rebooted. In many cases, this exploit window is expanded by scheduled maintenance windows and delays associated with testing and validating security patches in staging environments.

Canonical Livepatch shrinks this exploit window by surgically modifying vulnerable kernel code in memory, and redirecting function calls to patched versions while the system continues operating. However, if the apt package manager has not also installed the security update from a newer .deb Linux kernel package, the in-memory security patches will be lost on reboot. This means that if the machine starts up in a vulnerable state, then Livepatch Client will have to reapply the Livepatch update. Ideally, system administrators will install security updates for the Linux kernel by upgrading the kernel .deb package, and rely on Canonical Livepatch service to secure the machine before the next reboot.

It is best practice to progressively roll out updates in test environments, before updating production environments. Until now, the only way to stagger Livepatch updates was to self-host a Livepatch Server, and control which machines received which Livepatch updates. Now it is even simpler to enable the Canonical Livepatch security patching automation with testing and validation in staging environments, before production. In internet connected environments, where Ubuntu instances can reach livepatch.canonical.com, Livepatch Client supports timestamp-based rollout configurations. Organizations can implement controlled and predictable update pipelines from staging to production environments, without the hassle of deploying a self-hosted Livepatch Server, and managing the distribution of Livepatch updates through Livepatch Server.

No Livepatch updates beyond this timestamp, please

The Livepatch cut-off date feature is enterprise focused, and is not available to users of the free Ubuntu Pro token. Configuring Livepatch Client with a specific timestamp in the past forces an Ubuntu machine to remain in a known, deterministic state. This can be achieved with 1 command, using “2024-10-01T12:00:00Z” as a hypothetical timestamp:

$ canonical-livepatch config cutoff-date="2024-10-01T12:00:00Z"

Even in tightly regulated production environments, system administrators can now move from a reactive patching to a proactive patching posture. Time-based control enables straightforward and rigorous testing workflows.

Progressing from testing, to staging, to production

1. In the development or testing environment, configure Livepatch without cut-off restrictions. This allows the latest patches to be applied immediately. If a Livepatch cut-off date has been set, setting it to a blank value will remove it:

   $ canonical-livepatch config cutoff-date=""


2. The staging environment should mirror production as closely as possible. Set a cut-off date that is ahead of the date set in production. This allows updates that have been withheld from production to arrive in the staging environment.
3. Once testing, development, and staging environments have received Livepatch updates, the updates can be promoted to production with a high degree of confidence. Match the cut-off date in the production environment with what has been applied in staging.

It is possible to identify which Livepatch updates have been applied by tracking the patched CVEs in the Livepatch Client status output:

$ canonical-livepatch status --verbose

Conclusion

The timestamp based rollout capability introduced in Livepatch Client provides a predictable and controlled pipeline of updates, without the complexity of managing your own Livepatch Server. Using graduated cut-off dates across environments enables the Livepatch security patching automation solution to conform with most enterprise security update protocols.

This powerful and now extremely convenient feature is not included in the Ubuntu Pro free tier. Virtual machines launched on major public cloud providers such as AWS, Azure, Google, or Oracle using an Ubuntu Pro image will have access to the cut-off date feature in Livepatch Client. Take control over your system reliability and operational confidence by enabling Canonical Livepatch on your Ubuntu instances today.