Is my past haunting me?

I am in my early 30’s. I am single. But, there is a guy I have been dating for a while. I love him very much. However, I had sex with his older brother 10 years ago. His brother and