Honestly I believe that this is a mostly overblown belief, although some certainly did make efforts to destroy paganism in all writings and monuments I believe they were few and far between. Destruction through being forgotten is far more plausible, and also it must be noted that most likely destructions were committed by people who were formerly pagan themselves, and of course there are the famous cases which go against this like Boniface.