Off-spinner Ali ended the match when, after having Dean Elgar and Kagiso Rabada caught at first slip by Ben Stokes with the last two balls of his 16th over, had Morne Morkel lbw with the next.
Resuming on 117 for four after being set a huge target of 492 for victory, South Africa were dismissed for 252 on the final day of what was the 100th Test at The Oval.
Opening batsman Elgar, who made a gritty 136, got good support from Temba Bavuma (32) in a stand of 108 after South Africa had collapsed to 52 for four on Sunday.
But Toby Roland-Jones's dream England debut continued when the Middlesex paceman struck twice in two balls before lunch on Monday to dismiss Bavuma and Vernon Philander.
And when Chris Morris fell to the last ball before lunch, edging Ali to slip, South Africa were 205 for seven.
This was a memorable match for all-rounder Stokes, who made 112 in the hosts' first innings 353 and then took two wickets in two balls late on Sunday.
Roland-Jones returned figures of five for 57 in South Africa's meagre first innings 175, while fellow debutant Tom Westley made 59 in England's second innings 313 for eight declared.
England, 1st Innings: 353 (B Stokes 112, A Cook 88; M Morkel 3-70, K Rabada 3-85)
South Africa, 1st Innings: 175 (T Bavuma 52; T Roland-Jones 5-57, J Anderson 3-25)
England, 2nd Innings: 313-8 dec (J Bairstow 63, T Westley 59, J Root 50; K Maharaj 3-50)
South Africa, 2nd Innings: 252 (D Elgar 136; M Ali 4-45, T Roland-Jones 3-72)
Ben StokesDean ElgarEngland hat-trickEngland vs South AfricaEngland vs South Africa 2017Faf du PlessisJoe RootMoeen AliMoeen Ali Hat-trickoval test
First Published: July 31, 2017, 7:22 PM IST