DeclinetoState,
Presumably, Abel received some unspecified reward from God (but it wasn't enough to keep him from being clobbered by his brother), while Cain either received no reward or was forced to suffer in some way.
Perhaps they knew the same way Aaron and his priestly sons knew in
Leviticus 9:24.
"Then the fire came forth from before Jehovah and consumed the burnt offering and the fat portions on the altar; and when all the people saw it, they gave a ringing shout and fell on their faces" (Lev. 9:24)
Is there any evidence, either in the Bible or elsewhere, of how God communicated his acceptance of the offering from Abel and his non-acceptance of Cain's offering? In other words, how did Cain know that God had not accepted his offering? Did God say, "Good job, Abel," and/or "Better luck next time, Cain"?
I don't think we are told this in the Bible. I offer Leviticus 9:24 as a possibility.