My ex is taking me to court over our son. Is there any way I can keep him from getting visitation?

My ex and I broke up when I was 2 months pregnant. He was absent the entire time I was pregnant. Did not offer to help out in anyway and was there for the birth or even the days in the hospital after the birth. Never once did I cheat on him but he claims I did. I have asked him for help here and there but he tells me that our son is not his and I should have kept my legs closed. Despite him denying our 3 month old son i am not taking him to court or even pushing for child support. He is denying him but he says he is taking me to court over him. I'm wondering why he would do that if he doesnt think he is his. Is there anyway I can keep him from getting rights when we go to court? He also has 2 domestic violence charges on him if that matters at all...