The idea that the term came from non-religious people, or secularists, is just wrong.
And one I've never heard expressed, as well. It was historically (and maybe still?) far, far more important to members of minority religions, like Anabaptists or Catholics, depending on where they were, than to secular folks.