×

Message

Failed loading XML... Opening and ending tag mismatch: hr line 11 and body Opening and ending tag mismatch: body line 6 and html Premature end of data in tag html line 2
Wednesday, 14 May 2014 14:10

Patching a running Linux kernel: kGraft v kpatch Featured

By

Patching a Linux kernel without any downtime is likely to become a common practice over the next few years; two patches released earlier this year are likely to make what is a task requiring downtime something that can be done on the fly.

SUSE announced kGraft at the beginning of February while Red Hat made its pitch about kpatch towards the end of the same month. The patches have been submitted upstream for incorporation in the kernel – both companies have excellent records of sticking to the principles of open source.

To get an idea of the way the two processes work, iTWire asked both SUSE and Red Hat the following questions:

How does the process of patching differ between kpatch and kGraft?

What, according to SUSE/Red Hat, are the plus points of kGraft/kpatch?

Are there any features in kGraft/kpatch that you would like to incorporate into kpatch/kGraft?

Red Hat's Ron Pacheco, senior manager of technology product management, responded: "We can't speak to the nuances of kGraft, but the kpatch process works in the following manner:

"Analysis of the code change, which includes determining the patch candidates and an object-level comparison of the kernel pre-patch;

"Patch conversion, the point at which two extra sections are added to the object file to address the base function and the patched function; and

"Insertion of the patched entity.

"Ultimately, the kpatch process is heavily weighted towards step 1, analysing the code, in order to minimise and mitigate as much risk as possible during the patching process.

Regarding the plus points of kpatch vs kGraft, Pacheco said: "From a benefits standpoint, first and foremost, kpatch, like kGraft, is a completely open source solution and works with existing upstream infrastructures and functionalities.

"kpatch emphasises what we feel is the most important part of the patching process: the pre-patch analysis. kpatch provides the functionality to provide a full deep dive into the code change, picking out seemingly minor inconsistencies and issues that could have massive ramifications post-patch, as well as helping determine whether or not a patch function is even suitable for dynamic patching.

Pacheco added: "We want to ensure that IT teams understand the implications, limitations and ramifications of a suggested patch BEFORE they start the process, and kpatch delivers the tools to help answer these questions and eliminate human error."

In response to the first query, the director of SUSE Labs, Vojtech Pavlik, said: "As the goal is the same for both technologies, also the process is similar and at some points even identical. Both use ftrace for patching code, both use patches in a kernel module format, both use the kernel module linker functions to link patched code into the kernel. There are two points, where the process differs, though:

"1) Automated patch generation: kGraft starts with a list of changed functions and uses the DWARF debugging information of the old kernel package to generate a full list of functions that are in any way affected by changes in the patch. kGraft then compiles the post-patch kernel, same as kpatch.

"kpatch recompiles both the pre-patch and post-patch kernel sources with special compiler options (-ffunction-sections) and then compares the resulting object files, arriving at a list of differences.

"2) Patching process: kGraft patches at runtime and uses a RCU-like method of transitioning individual execution threads from pre-patch to post-patch functions that ensures consistency of function call format and semantics. It may take some time to transition all threads, as some processes may be idle and not reaching the transition point (kernel/userspace boundary) soon. Nevertheless, the kernel is protected from the patched bug already during the transition process as any new thread will start using post-patch functions automatically.

"kpatch stops all the processors in the system to make sure no thread is executing. It then checks whether any execution thread was stopped in a function that is to be patched or in any function that is called from a function to be patched by analysing stack return addresses. When that happens, kpatch has to resume all processors again, wait, and try later. If after trying several times it doesn't succeed, perhaps because an idle process is sleeping in a patched function, it fails.

Regarding what SUSE considered to be better in its solution, Matthias Eckermann, senior product manager for SUSE Linux Enterprise, said: "kGraft based patching does not require the kernel to be stopped, while all other methods require this for milliseconds up to seconds.

"kGraft is lightweight and builds on proven kernel infrastructure, namely ftrace'" he added. "For the use cases our customers are asking for, kGraft is just perfect. We are watching the other development paths (kpatch) carefully, yet for the moment our focus is on improving the quality of our mainline submission of the kGraft framework."

On the same question, Pavlik said: "Matthias already mentioned the major plus points - not having to stop the kernel; kGraft patches at runtime, while kpatch has to stop the kernel, potentially several times, for very short time periods, until it succeeds patching and never failing to patch.

"In a subset of scenarios, where kpatch would fail, kGraft may never finish the RCU transition process. This is not an issue, though, as the kernel is protected against the patched bug from the start of the patching process already.

"Less reliance on sane compiler behaviour: Recompiling the pre-patch kernel with additional options should not, but may affect compiler optimisation choices and thus render the object comparison invalid.

"To be fair, I'd like to also mention a plus point for kpatch: easier data format/semantics conversion.

"When the in-memory data format changes as a result of a patch, it is easier to do the surgery on the data while the kernel is standing still. Doing it on a running kernel requires more care to be applied by the programmer writing the conversion code. This code has to be created manually for both kpatch and kGraft."

Pavlik said that rather than incorporate features from kpatch, it was more beneficial to look at the code that was very similar and share it, "so that is what we're looking into now".

He said he had some thoughts about possible future synergies in places where the technologies differed, too.

"kpatch could continue with the kGraft method when it would otherwise fail," Pavlik said. "kGraft could resort to stopping the kernel for certain cases where in-memory data conversion has to be done and doing it at runtime isn't worth the additional developer effort."

He added: "kGraft could use the stack analysis part of the kpatch method to speed up the transition considerably and not be delayed by most idle sleeper processes."

The image used with this article is copyrighted to Larry Ewing, Simon Budig and Anja Gerwinski.


Subscribe to ITWIRE UPDATE Newsletter here

GRAND OPENING OF THE ITWIRE SHOP

The much awaited iTWire Shop is now open to our readers.

Visit the iTWire Shop, a leading destination for stylish accessories, gear & gadgets, lifestyle products and everyday portable office essentials, drones, zoom lenses for smartphones, software and online training.

PLUS Big Brands include: Apple, Lenovo, LG, Samsung, Sennheiser and many more.

Products available for any country.

We hope you enjoy and find value in the much anticipated iTWire Shop.

ENTER THE SHOP NOW!

INTRODUCING ITWIRE TV

iTWire TV offers a unique value to the Tech Sector by providing a range of video interviews, news, views and reviews, and also provides the opportunity for vendors to promote your company and your marketing messages.

We work with you to develop the message and conduct the interview or product review in a safe and collaborative way. Unlike other Tech YouTube channels, we create a story around your message and post that on the homepage of ITWire, linking to your message.

In addition, your interview post message can be displayed in up to 7 different post displays on our the iTWire.com site to drive traffic and readers to your video content and downloads. This can be a significant Lead Generation opportunity for your business.

We also provide 3 videos in one recording/sitting if you require so that you have a series of videos to promote to your customers. Your sales team can add your emails to sales collateral and to the footer of their sales and marketing emails.

See the latest in Tech News, Views, Interviews, Reviews, Product Promos and Events. Plus funny videos from our readers and customers.

SEE WHAT'S ON ITWIRE TV NOW!

BACK TO HOME PAGE
Sam Varghese

Web Analytics

Sam Varghese has been writing for iTWire since 2006, a year after the site came into existence. For nearly a decade thereafter, he wrote mostly about free and open source software, based on his own use of this genre of software. Since May 2016, he has been writing across many areas of technology. He has been a journalist for nearly 40 years in India (Indian Express and Deccan Herald), the UAE (Khaleej Times) and Australia (Daily Commercial News (now defunct) and The Age). His personal blog is titled Irregular Expression.

Share News tips for the iTWire Journalists? Your tip will be anonymous

WEBINARS ONLINE & ON-DEMAND

GUEST ARTICLES

VENDOR NEWS

Guest Opinion

Guest Interviews

Guest Reviews

Guest Research

Guest Research & Case Studies

Channel News

Comments