When exposed to high-energy radiations, the silicate aggregates used in concrete can exhibit some swelling. This, in turn, can result in some internal stress, which can lead to micro-cracks or de-bonding along the cement aggregate interface. However, there is presently some uncertainty regarding the longterm effect of such high-energy irradiation on aggregates and the associated risk of cracking and failure. Here, based on atomistic simulations, we investigate the effect of neutron irradiation on eight silicate minerals belonging to three different mineralogical families. We demonstrate that the irradiation-induced alterations do not depend on the silicate family but rather on the inherent structure and composition of each mineral. Interestingly, we show that vitrification can be used as a surrogate to assess the upper limit of irradiation-induced swelling. Further, we observe that this swelling potentially leads to high internal stresses, which can result in the initiation and propagation of cracks. These findings can guide the selection of optimal aggregate minerals for which the risk of concrete failure upon irradiation is minimized.