French forward Gouffran fired home a stunning angled volley midway through the first period at the iPro Stadium, with Yedlin putting the result beyond any doubt in stoppage time with his first goal since his move from Tottenham Hotspur last month.
County had the better of the opening exchanges, albeit without being able to muster a clear cut chance, but it was United who broke the deadlock in the 20th minute when Gouffran pulled the trigger, connecting perfectly with Jonjo Shelvey's deep corner.
Shelvey, Dwight Gayle and Mohamed Diame all threatened a second goal for the comfortable visitors, and it finally arrived with seconds remaining, as Yedlin beat Rams keeper Scott Carson to Ayoze Perez's knock-back.
The result was a fifth successive victory in all competitions - and a fourth clean sheet in a row - for Newcastle, who travel to Queens Park Rangers on Tuesday night.
Derby raced out of the traps and thought they had taken the lead when debutant Ikechi Anya raced in behind the visitors' defence and squared for fellow new arrival Matej Vydra to tap home. However, Anya had strayed offside and a raised flag curtailed County's joy.
At the other end, Gouffran saw a strike deflected off target before Gayle went even closer from 20 yards, having his low shot tipped wide by the fingertips of Rams keeper Scott Carson.
And it was from the resulting corner that United broke the deadlock - and in spectacular style.
Shelvey's delivery was deep towards Jamaal Lascelles, but as it dropped oer the skipper's head Gouffran met it on the volley and fired a thunderous effort into the far corner for his first goal since January 2015.
That knocked the stuffing out of the hosts and allowed Newcastle to control the remainder of the first half, letting their opponents have plenty of possession but refusing to allow them any chances in the final third.
The Rams made a bright start to the second period and full-back Marcus Olsson signalled their intent to get back on terms as quickly as possible with a curling strike from the edge of the area that drifted inches the wrong side of the upright.
United responded and Shelvey sent a 22-yard free-kick narrowly over the top after Gayle had been bundled over by Richard Keogh.
Suddenly the visitors had the bit between their teeth again. Ritchie twice sent crosses - one left foot, one right - agonisingly close to their intended targets, while Diame waltzed through the home defence but was unable to trouble Scott Carson with his finish.
But with only one goal separating the sides Derby always had hope and they almost equalised in the 68th minute when, first, Sels beat away Craig Bryson's long-range effort, before Anya sent an overhit cross onto the top of the bar and away to safety.
On 72 minutes, Yedlin was introduced from the substitutes' bench in place of goalscorer Gouffran, and it wasn't long before Perez entered the fray as well, replacing Gayle.
The latter's pace set up a late chance for Ritchie, as Yedlin raced past Olsson down the right flank as if he wasn't there and pulled a cross back for his teammate, whose shot was blocked.
Diame then had an effort saved by Carson, but the second goal finally arrived as the game moved into four minutes of added-on time.
It stemmed from another deep Shelvey corner but this one was knocked back across goal by Perez, and Yedlin showed brilliant determination to challenge Carson and bundle home from close range.
That secured all three points and sent more than 3,000 travelling supporters back to the North-East in great spirits.