Open Source Market Segment LS
Open Source Market Segment RS
Wednesday, 14 May 2014 14:10

Patching a running Linux kernel: kGraft v kpatch Featured


Patching a Linux kernel without any downtime is likely to become a common practice over the next few years; two patches released earlier this year are likely to make what is a task requiring downtime something that can be done on the fly.

SUSE announced kGraft at the beginning of February while Red Hat made its pitch about kpatch towards the end of the same month. The patches have been submitted upstream for incorporation in the kernel – both companies have excellent records of sticking to the principles of open source.

To get an idea of the way the two processes work, iTWire asked both SUSE and Red Hat the following questions:

How does the process of patching differ between kpatch and kGraft?

What, according to SUSE/Red Hat, are the plus points of kGraft/kpatch?

Are there any features in kGraft/kpatch that you would like to incorporate into kpatch/kGraft?

Red Hat's Ron Pacheco, senior manager of technology product management, responded: "We can't speak to the nuances of kGraft, but the kpatch process works in the following manner:

"Analysis of the code change, which includes determining the patch candidates and an object-level comparison of the kernel pre-patch;

"Patch conversion, the point at which two extra sections are added to the object file to address the base function and the patched function; and

"Insertion of the patched entity.

"Ultimately, the kpatch process is heavily weighted towards step 1, analysing the code, in order to minimise and mitigate as much risk as possible during the patching process.

Regarding the plus points of kpatch vs kGraft, Pacheco said: "From a benefits standpoint, first and foremost, kpatch, like kGraft, is a completely open source solution and works with existing upstream infrastructures and functionalities.

"kpatch emphasises what we feel is the most important part of the patching process: the pre-patch analysis. kpatch provides the functionality to provide a full deep dive into the code change, picking out seemingly minor inconsistencies and issues that could have massive ramifications post-patch, as well as helping determine whether or not a patch function is even suitable for dynamic patching.

Pacheco added: "We want to ensure that IT teams understand the implications, limitations and ramifications of a suggested patch BEFORE they start the process, and kpatch delivers the tools to help answer these questions and eliminate human error."

In response to the first query, the director of SUSE Labs, Vojtech Pavlik, said: "As the goal is the same for both technologies, also the process is similar and at some points even identical. Both use ftrace for patching code, both use patches in a kernel module format, both use the kernel module linker functions to link patched code into the kernel. There are two points, where the process differs, though:

"1) Automated patch generation: kGraft starts with a list of changed functions and uses the DWARF debugging information of the old kernel package to generate a full list of functions that are in any way affected by changes in the patch. kGraft then compiles the post-patch kernel, same as kpatch.

"kpatch recompiles both the pre-patch and post-patch kernel sources with special compiler options (-ffunction-sections) and then compares the resulting object files, arriving at a list of differences.

"2) Patching process: kGraft patches at runtime and uses a RCU-like method of transitioning individual execution threads from pre-patch to post-patch functions that ensures consistency of function call format and semantics. It may take some time to transition all threads, as some processes may be idle and not reaching the transition point (kernel/userspace boundary) soon. Nevertheless, the kernel is protected from the patched bug already during the transition process as any new thread will start using post-patch functions automatically.

"kpatch stops all the processors in the system to make sure no thread is executing. It then checks whether any execution thread was stopped in a function that is to be patched or in any function that is called from a function to be patched by analysing stack return addresses. When that happens, kpatch has to resume all processors again, wait, and try later. If after trying several times it doesn't succeed, perhaps because an idle process is sleeping in a patched function, it fails.

Regarding what SUSE considered to be better in its solution, Matthias Eckermann, senior product manager for SUSE Linux Enterprise, said: "kGraft based patching does not require the kernel to be stopped, while all other methods require this for milliseconds up to seconds.

"kGraft is lightweight and builds on proven kernel infrastructure, namely ftrace'" he added. "For the use cases our customers are asking for, kGraft is just perfect. We are watching the other development paths (kpatch) carefully, yet for the moment our focus is on improving the quality of our mainline submission of the kGraft framework."

On the same question, Pavlik said: "Matthias already mentioned the major plus points - not having to stop the kernel; kGraft patches at runtime, while kpatch has to stop the kernel, potentially several times, for very short time periods, until it succeeds patching and never failing to patch.

"In a subset of scenarios, where kpatch would fail, kGraft may never finish the RCU transition process. This is not an issue, though, as the kernel is protected against the patched bug from the start of the patching process already.

"Less reliance on sane compiler behaviour: Recompiling the pre-patch kernel with additional options should not, but may affect compiler optimisation choices and thus render the object comparison invalid.

"To be fair, I'd like to also mention a plus point for kpatch: easier data format/semantics conversion.

"When the in-memory data format changes as a result of a patch, it is easier to do the surgery on the data while the kernel is standing still. Doing it on a running kernel requires more care to be applied by the programmer writing the conversion code. This code has to be created manually for both kpatch and kGraft."

Pavlik said that rather than incorporate features from kpatch, it was more beneficial to look at the code that was very similar and share it, "so that is what we're looking into now".

He said he had some thoughts about possible future synergies in places where the technologies differed, too.

"kpatch could continue with the kGraft method when it would otherwise fail," Pavlik said. "kGraft could resort to stopping the kernel for certain cases where in-memory data conversion has to be done and doing it at runtime isn't worth the additional developer effort."

He added: "kGraft could use the stack analysis part of the kpatch method to speed up the transition considerably and not be delayed by most idle sleeper processes."

The image used with this article is copyrighted to Larry Ewing, Simon Budig and Anja Gerwinski.

Read 15827 times

Please join our community here and become a VIP.

Subscribe to ITWIRE UPDATE Newsletter here
JOIN our iTWireTV our YouTube Community here


Thoughtworks presents XConf Australia, back in-person in three cities, bringing together people who care deeply about software and its impact on the world.

In its fifth year, XConf is our annual technology event created by technologists for technologists.

Participate in a robust agenda of talks as local thought leaders and Thoughtworks technologists share first-hand experiences and exchange new ways to empower teams, deliver quality software and drive innovation for responsible tech.

Explore how at Thoughtworks, we are making tech better, together.

Tickets are now available and all proceeds will be donated to Indigitek, a not-for-profit organisation that aims to create technology employment pathways for First Nations Peoples.

Click the button below to register and get your ticket for the Melbourne, Sydney or Brisbane event



It's all about Webinars.

Marketing budgets are now focused on Webinars combined with Lead Generation.

If you wish to promote a Webinar we recommend at least a 3 to 4 week campaign prior to your event.

The iTWire campaign will include extensive adverts on our News Site and prominent Newsletter promotion and Promotional News & Editorial. Plus a video interview of the key speaker on iTWire TV which will be used in Promotional Posts on the iTWire Home Page.

Now we are coming out of Lockdown iTWire will be focussed to assisting with your webinars and campaigns and assistance via part payments and extended terms, a Webinar Business Booster Pack and other supportive programs. We can also create your adverts and written content plus coordinate your video interview.

We look forward to discussing your campaign goals with you. Please click the button below.


Sam Varghese

Sam Varghese has been writing for iTWire since 2006, a year after the site came into existence. For nearly a decade thereafter, he wrote mostly about free and open source software, based on his own use of this genre of software. Since May 2016, he has been writing across many areas of technology. He has been a journalist for nearly 40 years in India (Indian Express and Deccan Herald), the UAE (Khaleej Times) and Australia (Daily Commercial News (now defunct) and The Age). His personal blog is titled Irregular Expression.

Share News tips for the iTWire Journalists? Your tip will be anonymous




Guest Opinion

Guest Interviews

Guest Reviews

Guest Research

Guest Research & Case Studies

Channel News