When my daughter was only a few weeks old, a neighbor moved in that made me feel exactly as you described. The jerk played music at such an unreasonable level that I finally had to take all of my shelves down because it kept rattling them off the walls. He also sold drugs right at his front door and I had all manner of people stumbling up to MY door in the middle of the night, thinking they were at his door.
I tried speaking to him and the landlord. I tried ignoring it. I tried calling the police to report the noise and the drug activities. I tried blasting Boléro or Il barbiere di Siviglia, but it just made him turn it up until cracked one of his own windows.
The constant stress and strain of having to deal with him and a newborn was nearly enough to give me a nervous breakdown. I was in tears every day and felt like I was going to throw up almost every morning. I thought for sure that he would be the death of me. I ended up practically camped out at my parents house while I searched for a new place to live. I had to take care of it all myself because my ex-husband was so absent from the home and our lives that he didn't care if we were being driving slowly insane or not.
Around this time, my dad died. Dad had been telling me that he was going to come down and take care of the jerk himself if things didn't improve. After Daddy died, I drove back home a few days later, dreading the hell that the neighbor would impose on me during my grief. When I got there, the neighbor's side of the duplex had been completely cleared out. It was as if he'd never been there. I like to think that maybe my Dad came back and haunted the you-know-what and scared him out of the place.
Still, the stress of the whole situation was so great, that I ended up moving anyway. It was like that horrible man had ruined the home for me forever. The rest of the time I was there (just a matter of a few months), I felt like I was walking on eggshells, waiting for the thumping to begin again.
In conclusion, my advice is MOVE. I know it sucks and its not fair, but you'll feel like the weight of the world has been lifted, I promise. It's not worth it to stay.