The micromechanical properties of shale are crucial for the modeling and prediction of its macromechanical properties. However, the elastic properties have not been comprehensively understood at nano- and microscales. In the present study, the mechanics-component mapping and logarithm filtering methods are proposed to overcome the defect of atomic force microscopy in identifying shale components to investigate the micromechanical properties of shale. Microscopically, the elasticity of shale is heterogeneous. Heterogeneous elastic characteristics of dolomite and quartz are caused by crystal structure anisotropy, isomorphism, and lattice defects. The anisotropy of crystal structure dominates the variation of Young’s modulus of dolomite and quartz. The Young’s modulus of framboidal pyrite depends greatly on its crystal evolution. The heterogeneous elasticity of organic matter is caused by the disordered molecular structure, the maturity variation, and the mixing of different organic macerals. Because of the difference in Young’s modulus among minerals, the metasomatism of feldspar by calcite, quartz, and clay minerals alters the micromechanical properties of shale. Our study reveals that mineral crystal structure characteristics, diagenesis, and organic petrological factors control the mechanical properties of shale microscopically.