Am I in the wrong?

What makes you think life with him would be any different than it was for his wife of 11 years? I think you already know the answer to this...just afraid to face it. Marriage is tough enough...now add in an ex-wife and two kids plus your child and his father...could equal disaster.