Joe Willock, making his first competitive start for United since April, opened the scoring within 18 seconds at the City Ground, the club's fastest ever goal in the competition, before the midfielder was forced off in the early stages of the cup tie due to injury.
Newcastle found themselves on level terms shortly after the restart following Jota Silva's thumping strike before the contest was settled by a penalty shoot-out and, despite Joelinton being denied from the spot by Carlos Miguel, Ibrahim Sangaré and Taiwo Awoniyi both missed for the hosts before Sean Longstaff stroked home the winning spot-kick after successful conversions from Alexander Isak, Bruno Guimarães and Anthony Gordon.
Eddie Howe made six changes from the side which claimed a 1-1 draw against AFC Bournemouth on Sunday as Sandro Tonali, free to feature for United after serving his ten-month ban from competitive football, was handed his first appearance since October 2023 while Kieran Trippier, Lewis Hall, Willock, Harvey Barnes and Miguel Almirón were all handed starts.
United made the perfect start as Willock opened the scoring inside a mere 18 seconds. After Tonali and Almirón combined in the build-up, the latter threaded a through ball into Alexander Isak's path who saw his strike initially parried by Forest goalkeeper Carlos Miguel but Willock was on hand to tap the rebound into an empty net.
Tonali, who scored inside six minutes of his Premier League debut against Aston Villa last season, almost marked his highly-anticipated return to action with a goal as the Italian midfielder latched onto Isak's pass but was denied by Miguel, who rushed out quickly to block with his left leg.
Dan Burn prevented Forest from netting an immediate equaliser, hacking the ball away just metres from his own goalline after Magpies goalkeeper Nick Pope failed to collect Ramón Sosa's cross from the right flank.
Miguel Almirón went close to doubling the visitors' advantage as the Paraguayan was teed up by Harvey Barnes before his first-time shot inside the penalty area was turned out for a corner by Miguel.
Newcastle headed into the break with a one-goal advantage but were fortunate to avoid conceding on the stroke of half-time as Sosa's teasing free-kick flew dangerously towards Taiwo Awoniyi at the back post but the stretching Nigerian headed over the crossbar.
In the second-half, Forest equalised in the 50th minute through Jota. After the United defence failed to clear a long throw into the box, the loose ball fell kindly into the Portuguese forward's path, who lashed the ball into the net via the right-hand post.
Barnes was unable to restore United's one-goal cushion as the left winger raced onto Isak's pass but his drilled attempt inside the penalty area was blocked by Forest defender Àlex Moreno.
Newcastle continued their push for a late winner, with Dan Burn heading wide from Anthony Gordon's corner before Barnes' flashed a low effort into the side-netting minutes later.
With penalties looming, Lewis Hall went narrowly close to firing the visitors ahead in the closing stages as the left-back's bending effort from the edge of the box was clawed away by Miguel.
In the penalty shoot-out, Callum Hudson-Odoi and Alexander Isak converted the opening spot-kicks for their respective sides before Joelinton was denied by Miguel, who guessed the correct way to push away the Brazilan's effort from 12 yards.
Neco Williams netted Forest's third successful penalty in succession but, after Guimarães and Gordon netted Newcastle's third and fourth spot-kick, Sangaré's attempt struck the bar whilst Awoniyi wildly blazed his attempt over the crossbar.
Sean Longstaff was left to score the winning penalty in composed fashion, slotting the ball into the bottom right-hand corner to, ultimately, set up a third round clash away at League Two side AFC Wimbledon.
Newcastle United: Nick Pope, Kieran Trippier (c) (Tino Livramento 62), Emil Krafth, Dan Burn, Lewis Hall, Sandro Tonali (Sean Longstaff 62), Joelinton, Joe Willock (Bruno Guimarães 15), Miguel Almirón (Anthony Gordon 61), Harvey Barnes, Alexander Isak.
Subs not used: John Ruddy, Lloyd Kelly, Matt Targett, Jacob Murphy, Will Osula.
