Some people do have active polyamory relationships. I know a girl who is married but has a female partner, and her husband has another relationship.
Slightly different scenario than just having relations with someone but it does happen.
The thing is - to make a marriage work you have to go beyond a physical attraction. It's about existing with one person and working together as a team, being your complete self etc.
If you BOTH feel comfortable having sex with other people then the trust is still there - although this kind of thing can lead to mistrust and resentment.