The company said 408 million units were sold in the quarter and marked the first year-on-year decline since it started keeping track of the market in 2004.
In 2017 as a whole, smartphone sales to end users topped 1.5 billion units, an increase of 2.7% from 2016. Huawei, ranked No. 3, raised its share in 2017, continuing to gain on Apple, while the combined market share of Chinese vendors in the top five increased by 4.2 percentage points. The market share of Samsung and Apple was unchanged.
"Two main factors led to the fall in the fourth quarter of 2017," said Anshul Gupta, research director at Gartner.
"Second, replacement smartphone users are choosing quality models and keeping them longer, lengthening the replacement cycle of smartphones.
"Moreover, while demand for high quality, 4G connectivity and better camera features remained strong, high expectations and few incremental benefits during replacement weakened smartphone sales."
Gartner said that while Samsung saw a year-on-year unit decline of 3.6% in the quarter, it still retained its spot as the top global smartphone, while Apple stayed second.
This was despite the beginning of a slowdown of Galaxy S8 and S8+ sales, Gartner said, noting that these two models had helped improve the overall selling price of its smartphones.
Apple's market share was steady in the fourth quarter of 2017 compared to the corresponding quarter in 2016, but iPhone sales fell 5%.
"Apple was in a different position this quarter than it was 12 months before," said Gupta. "It had three new smartphones — the iPhone 8, iPhone 8 Plus and iPhone X — yet its performance in the quarter was overshadowed by two factors.
"First, the later availability of the iPhone X led to slow upgrades to iPhone 8 and 8 Plus, as users waited to try the more-expensive model.
"Second, component shortages and manufacturing capacity constraints preceded a long delivery cycle for the iPhone X, which returned to normal by early December 2017. We expect good demand for the iPhone X to likely bring a delayed sales boost for Apple in the first quarter of 2018."
The Gartner report said Huawei and Xiaomi were the only vendors to achieve year-on-year unit growth (7.6% and 79%, respectively) and increased market share in the quarter. Huawei broadened the appeal of its smartphones with the Mate 10 Lite, Honor 6C Pro and Enjoy 7S.
"Xiaomi's competitive smartphone portfolio, consisting of its Mi and Redmi models, helped accelerate its growth in the emerging Asia/Pacific (APAC) market. It also helped Xiaomi win back lost share in China," the tech research firm said.
"Future growth opportunities for Huawei will reside in winning market share in emerging APAC and the US," said Gupta. "Xiaomi's biggest market outside China is India, where it will continue to see high growth. Increasing sales in Indonesia and other markets in emerging APAC will position Xiaomi as a strong global brand."