Rejecting an invitation may not have the consequences people expect, study says