Between an audience and a performer there is the invisible fourth wall.
A difference between most performance and stand-up is that comedians “don’t have” the fourth wall. Or so it needs to seem.
Just my take, but I wouldn’t worry too much about “breaking” the wall.
There are all kinds of way successful comics out there, from the ultra who-are-you-what-do-you-do? types to the the tightly scripted one-liner merchants and there’s a place for them all.
People come to know you through what you’ve created. That’s intimate enough.
I reckon even the comedians that get audience members up on stage to dance/answer questions/pick a card haven’t broken the wall. It’s still a show, there’s still a wall, and those punters just became props.
I say pick your own relationship with the wall. For me it’s more of a serving hatch.
The diners can come into the kitchen if they like, but the stuff that’s relevant to them is coming through the hatch.