Is there any hope should I just walk out? ?

When me and my bf met I was in a relationship but having problems so wanted to break up with the guy anyway but wasn't quite ready to take the leap. Told him I was not ready to get into another relationship but I enjoyed his company ( I know should have never talked to him in the first place if I still had bf but I was confused and he seemed so nice and charming and that he would treat me good) He said he didn't want to be just friends if I did not want to be with him he needed to move on.(I felt this kind of pushy but I like him So i panicked and got with him (broke up w/ ex but wanted to stay friends didn't hate the guy just couldn't be together he did drugs was not real emotionally available). New bf did not like me being friends with ex he said there is no reason for it unless u still have feelings. I felt torn. So I would stay friends with ex anyway. New bf found out and asked me over and over to stop (I felt like he was telling me what to do and felt kinda manipulated to be in relationship in first place). Well now he has like NO trust in me, its always look what I have done to him and tells me I brought this on myself and its up to me to give him reason to trust me again and I could have prevented this whole thing if I would have stopped being friends with ex in first place. Its like all my fault. I have tried over and over and over to get him to see I am not doing anything wrong I only want him but he still suspicioius and justifys it because of what I have done. So now its like an excuse to control me? I don't know? Its like he CAN'T let go of past. We are having a baby. I am thinking about walking away can't deal with this anymore. He always "you don't care about me" and this and that. He is a jerk and thats becasue "I have hurt him so much in past". He like is good at justifying everything. And I am always like well he does have a point so I deal with it but it taking its toll don't know what to do feel like I am going crazy. I mean yes I did wrong him or did I even wrong him does he just have me convinced I did? But even so I have tried and tried to get us past it and he like doesn't. He says I have never done anything to get him past it. Well thats cause I don't come over enough, do this or that or act like I care. Well how can I when I do try to be nice to him and he is rude or a jerk or makes some mean comment cause he is resentful at me. It makes me not want to be around so then he uses that as excuse like see you don't care you don't try to fix things. When I try to talk to him about that its "your the one that put us in this situation in the first place". Always justifys his actions. If I did wrong him yes I do feel I need to make right. But its like I can't win now. I have apologized said sorry told him i love him only want him admitted I should have stopped talking to ex bf but he still questions everything. I feel like I can't even go to friends house cause he thinks I am up to no good. When he questions me over and over and I get defensive he all "see there you go acting defensive u must be guilty". Point is WILL it ever end? Will he EVER get over it, like enough already ya know DROP it please dude its annoying the hell out of me! But its my fault ya know. How dare I act pissy over him questioning me. I am about ready to bail what do you guys say? Of course I will be the bad guy if I do "oh look she did this to me and now she just gonna leave" . HELP!