Perfect happiness is impossible if you have empathy. If you have perfect happiness, just for a second think of the suffering of just one unknown innocent person in some distant; or think of how many lonely desperate people there are behind closed doors in your own neighborhood. Only a psychopath could ignore that and remain perfectly happy.
In this world you can't make everyone happy. You listen to your conscience and see to the maximum extent that from your side you have no intention to hurt anybody. That is enough.
