Gut - I agree with Gabriel. If they've just moved in, they are probably trying to get settled in and haven't even thought about the fence. If you know someone who does fence removal or that type of thing, it might be nice to pass along that contact info to the neighbors as they are new in town. Good luck! I hope you get your lovely garden back soon.