What is the origin of the idiom "backdoor man" and when to use it?

The phrase is thought to have originated in the early 1900s, when it was used to describe men who would sneak into the homes of married women through the back door. The phrase is now used in a more general sense to describe any man who is having an affair with a woman who is married or in a committed relationship.