Mark Ruffalo once took Woody Harrelson out of a bar fight in New Orleans

Mark Ruffalo and Woody Harrelson’s memories of making the first Now you see me in New Orleans include a wild bar fight.

When Ruffalo, 57, appeared on Ted Danson and Harrelson’s SiriusXM show Where everyone knows your name for the Wednesday, Oct. 15, episode, the actor said he and Harrelson, 64, filmed scenes for the 2013 heist movie “completely on the streets” of New Orleans during Mardi Gras, which resulted in the men being “dragged into bars in the middle of a scene.”

“We were out one night and it was a crowded place. And a woman came up [Harrelson] and she said, ‘Oh my God, I love you so much,'” Ruffalo recalled. “And Woody put his hand on her arm and he said, ‘Oh, thank you. Thank you, darling. And this guy comes and he pushes her out of the way, and he pushes Woody.”

“Yeah, I remember that,” Harrelson added.

“Bad,” Ruffalo said of the situation. “Because Woody’s first response is not to push someone away, but to immediately punch them in the face. Which is the right thing to do, by the way. It’s absolutely the right thing to do. But then a whole melee broke out in this bar. He was in it – I was in the middle of it and it was going to be a big thing.”

As Ruffalo recalled, he pulled Harrelson out of the incident before the situation escalated, out of fear for their group’s safety. “I grabbed you and pulled you out, because I was like, this could be fun, but it could also turn into a disaster because maybe you and I could manage on our own, but the rest of the people we were with, I don’t think so,” he said.

Ruffalo and Harrelson starred with Jesse Eisenberg, Isla Fisher and Dave Franco in the original Now you see me film, which followed Ruffalo as an FBI agent and Mélanie Laurent as an Interpol agent who investigate a series of heists carried out by the group of magicians known as “The Four Horsemen.”

Never miss a story: Sign up for PEOPLE’s free daily newsletter to stay up to date with the best of what PEOPLE has to offer, from celebrity news to compelling human interest stories.

The film was a smash hit and spawned a sequel in 2016; Ruffalo and Harrelson return for the trio in the next Now you see me: now you don’t see me anymore. The OG cast members are joined by new costars Justice Smith, Dominic Sessa and Ariana Greenblatt in the new film, a synopsis of which shows the Four Horsemen “unite with a new generation of illusionists for their most global, high-stakes magical adventure yet.”

“Both generations of magicians must overcome their differences to try to defeat their cunning and dangerous adversary,” the synopsis adds. Rosamund Pike appears in the film as the antagonistic Diamond heiress, while Morgan Freeman also reprises her role from the previous two films as Thaddeus.

Now you see me: now you don’t see me anymore is in theaters November 14.