"The change in allele frequency over time" isn't a summary of evolution, and it isn't an explanation; it's a definition, in that it delimits what counts as evolution (as far as biology is concerned). Any process that involves the change in allele frequency over time is evolution, and any that doesn't isn't. Thus neutral genetic drift is considered part of evolution -- whether you think it's interesting or not is up to you, but it's still evolution. Phenotypic changes that aren't associated with changes in allele frequency (like the increasing height of humans in developed countries over the last 100 years) are not evolution.
This does not mean that all of evolution can be reduced to changes in allele frequency, and it also doesn't mean that the definition is perfect. Speciation is an important part of evolution, but it cannot be described solely in terms of allele frequency changes, and in fact need not involve any change in frequency at all.
When people object to evolution, of course, they're seldom objecting to allele frequency changes. What they're objecting to is common descent (something that involves allele frequency change but does not necessarily follow from it), and the explanation of change in terms solely of natural processes (mutation, natural selection, drift, etc).