Are you saying that everything that feels good should not be a sin?
If something feels good and does no harm to others, why should it be a sin?
If it feels good and does do harm to someone, then a moral calculus must determine whether the right to that pleasure outweighs the harm to another or vice versa--for example, Johnny taking Mary Lou to summer drive-in frolics might cause "harm" to Peggy Sue's feelings, but Peggy Sue has no right to deprive Mary Lou or her date of their pleasures.
Of course, if Johnny and Peggy Sue are married and share an expectation of fidelity, the harm is demonstrable and morally trumps Johnny's and Mary Lou's rights to pleasure: seems pretty straightforward.
Is there some element of moral arithmetic that I've missed in this equation?