In the oilfield business, we use flyash (= "poz" or pozzolan) all the time in with cement for cementing the pipe in wells into place. One important thing to note is that the flyash actually reacts with the cement over time, so that it pretty much becomes one entity with the rest of the sidewalk or cement sheath after a year or two. And its use can make better, more water-resistant concrete that just Portland cement can.
Second, yes, I agree that poz probably contains some dangerous stuff. After all, you burned some stuff you dug out of the ground to make it. Cement, on the other hand, is made of stuff that you dug out of the ground and burned. Different stuff, yes, but limestone and clay can have
toxic chemicals like barium in them, just like coal can. And given a choice between handling cement or flyash bare-handed or bare-nosed, I will take flyash. I've done both, and never been "scalded" by flyash like by cement. Cement is much more alkaline.
The difference, I think, is that cement is made with the express intent of it being the product for sale. Flyash is leftovers from making electricity, so it is "waste."
There is a product, generically called cenospheres, that has been in the oilwell market for 25 years or so that is also getting more common in construction. It is made by dumping flyash into water, and skimming off the portion that floats. (A decent percentage of those little sphere in Frako's post above are hollow, and rather astoundingly strong.) I'm not even real sure where I would look to determine whether cenospheres are classified as equally hazardous as the poz that sank - but my bet is that it's "safer" in one set of regulations somewhere, only because it is purpose-made like cement is.
(Why use cenospheres, you ask? So you can make strong cement that weighs 10 pounds per gallon instead of 14 or 15. That can be important in lots of applications.)