We all have secrets. Things that, for cause of shame, embarrassment, or loss of friendship might be better off unsaid. Your internal cost/benefit calculus could tell you the pleasure would only cause more pain, or worse yet hurt innocent others.
But if keeping this secret comes between you and and what you want, what you really really want, than make sure to check your math. For a life of virtue is also not without regret.