Hendricks made 102 off 89 deliveries to help South Africa post 363-7 and then bowl out the hosts for 285 in 45.2 overs. They took an unassailable 3-0 lead in the five-match series.
Dhananjaya de Silva tried to keep the chase alive with a career-best 84 but Sri Lanka's batting crumbled under the ever-increasing run-rate.
Fast bowler Lungi Ngidi claimed four wickets while fellow paceman Andile Phehlukwayo took three.
But it was Hendricks, a right-hand batsman who has played 12 Twenty20 internationals, who set up the win for his side that had been swept aside in the two Tests.
The 28-year-old became only the third South African batsman to register a hundred in his first ODI. Colin Ingram and Temba Bavuma are the other two.
Jean-Paul Duminy's 92 off 70 balls, and half-centuries by Hashim Amla and David Miller, also contributed to South Africa's imposing total after being put into bat first.
"Brilliant today. Once again, the batting was fantastic. We played aggressive," said skipper Faf du Plessis, who injured his shoulder while attempting a tough catch in the 10th over and stayed away for the rest of the game.
"And obviously, for a debutant to play that freely was great to see. Every game, there's one or two guys putting their hands up," du Plessis said of Hendricks.
Amla gave South Africa a brisk start with his run-a-ball 59 before being bowled by medium-pacer Thisara Perera, who returned impressive figures of 4-44.
Hendricks, who has been around since 2014 when he first played a T20 against Australia, built crucial partnerships including a 78-run fourth-wicket stand with Duminy.
Hendricks celebrated his century in just 88 balls with a boundary off paceman Lahiru Kumara, but was dismissed on the very next delivery.
- Duminy blitz -
Duminy, who got his second fifty of the series to lead the batting chart with 177 runs, maintained the momentum with a blazing knock laced with eight fours and four sixes.
Duminy got support from fellow left-hander Miller as the two put on 103 for the fifth wicket to pulverise the bowling.
Miller's quickfire 51 and Phehlukwayo's unbeaten 24 off 11 balls gave South Africa 98 off the last seven overs.
Sri Lanka were never in the hunt with South African bowlers getting wickets at regular intervals.
A 95-run, seventh-wicket partnership between de Silva and Akila Dananjaya, who scored 37, raised the hosts' hopes of making a go of their target.
But Phehlukwayo broke the dangerous-looking stand, dismissing Dananjaya and then having de Silva caught behind.
Sri Lankan captain Angelo Mathews saw several positives -- "Dhananjaya batted well, some bowlers bowled well."
"But credit to Reeza and Duminy. We need to think out of the box if we need to win away. I know we aren't doing well, but we need to play on good wickets to develop. We're playing for pride and we want to finish the series off well."
The fourth match is slated for Wednesday at Pallekele.
lungi ngidiReeza HendricksSouth Africasri lankaSri Lanka vs South AfricaSri Lanka vs South Africa 2018
First Published: August 5, 2018, 6:30 PM IST