Inviting a friend to your game through Steam invites them to your party but not your game. As well as joining a game as the leader of a part doesn't bring the members of your party with you.

I agree, I've had times where I got disconnected and couldn't join my friends game back