Add a "recently played with" tab to the friends menu which would show the last 10 to 20 ish people who have shared a lobby with you.