Why is there controversy over vaccines?

I know that they have harsh ingredients but my grandma had TB and her brother has tons of small pox scars. What is worse, a world where polio, small pox, and other diseases are still widespread and common or the side effects of vaccines?