What would you do in this situation?

Well, there's a couple of options. You can sue your family for slander and defamation of character. Or you can move and start fresh.