Depending on the field, impact may take a very long time. It takes forever to get an article or book published in many fields, reviewed and noticed, etc. When people talk of high-impact research, it is useful to give it some concreteness and a sense of what is required to increase impact--assuming that the world is not waiting desperately for your next piece of work.
And even then, superb work may not have a big impact over the medium run.
The idea to raise expectations and aspirations. Publishing is fine, good journals is fine, but in the end what matters in high-impact is whether others use the work or argue with it. Also, does it lead to "big money," in terms of grants now and in the future.
Your work may not be impactful, even if there are fans, and you have received some rewards and fellowships. Your colleagues may not much value your work, even if you are well valued outside your department. That is an indictment of your colleagues rather than a judgment of your contributions. It may even be age-related. But none of this is of matter.
In general, our job is to have people outside our university appreciate our contributions. If we are insufficiently appreciated, we might stay around for other reasons, or many receive offers from elsewhere and move on.