Sorry for being long, but should I let my brother join our sleepovers? sounds weird, but please read?

No, you should absolutely not let your brother join your sleepovers in future. It wasn't fair to him to put him in that embarring situation. To realize how wrong it was, just ask yourself what your parents would have thought if they had walked in on all that.