The young Magpies netted twice in the space of three minutes, with Bailey, captain of the under-23s, opening the scoring with a fine header before Matthew Longstaff's first-time strike doubled the young Magpies' advantage before half-time.
Newcastle were unable to hold on for the win after 90 minutes, with Derby netting twice in the second half through Calum McDonald and Josh Shonibare.
In extra-time, Newcastle were awarded a penalty after Oliver Walters was fouled inside the box and, despite Bailey missing his attempt from 12 yards, Mo Sangare was quickly on hand to place the rebound into the net, to the skipper's delight.
Bailey told nufc.co.uk: "It's a really good result and all the lads showed that never-say-die attitude that we have in the team. We worked hard for the full 120 minutes and managed to get a great result that I think we've deserved in the end.
"We've made strong starts in recent matches. We went 2-0 up against Wolves in our last game and we were also ahead at half-time by the same scoreline against Derby. We are working to sustaining that over the full 90, or 120 like we have against Derby, so it can help the team going forward.
"It was a great ball by Matty (which led to Bailey's goal). He's always capable of putting a lot of dangerous crosses in and I probably should have had more goals. I didn't see it before I got my head to it but, thankfully, it managed to go into the net which got the team off a good start. The lads have been able to do the rest and take care of the game.
"Penalties are something that we have worked on since we have been in quite a lot of cup games. I was the penalty taker as I've scored the most in training and I scored in the Checkatrade Trophy. The goalkeeper read my spot-kick but, thankfully, Mo has helped me out by scoring and saved me a bit!"
United's youngsters are also chasing promotion to Division One of Premier League 2, with the young Magpies three points adrift of top spot in the second tier.
With a cup double in sight, Bailey, who led Newcastle to the HKFC Soccer Sevens title in Hong Kong last year, remains hopeful that his side can end an eye-catching season with silverware.
He added: "It shows the strong character that the lads have got. It's not the first time we have come from behind this season as we've come back from a goal down and showed a real resilience.
"In fairness, they are a good side. They are in the division above us and they made it difficult by working hard and scored two goals they probably deserved. It's not the best we have played but I think we can take plenty of positives from going ahead and then going on to win the game after conceding two goals.
"We have a good chance of winning the tournament. We still need to take it game by game and we will be focusing on our next league game against Stoke but it will be interesting to see who we will get in the semi-finals and I think we will fancy our chances, regardless of who we play.
"It's been a positive season so far but we need to make sure that it doesn't fizzle out into nothing. We need to make sure we can try to win at least one trophy and do our best to gain promotion at the end of the season."
