Sparry calcite, which is extensively distributed in shale, is crucial for inverting burial evolution processes. The carbonate-rich (almost 50%) shale of the Jiyang depression has complex sparry calcite types. These sparry calcites serve as carbonate-rich shale storage spaces, in addition to performing as indicators of fluid–rock interactions. What effects the various forms of sparry calcite have on reservoirs and how they originate are still unclear. Sparry calcites in shale sampled from the Jiyang depression were analyzed and divided into two types: granular and columnar. Granular sparry calcite can be further classified into impurity-rich and impurity-poor granular sparry calcites based on the presence of minerals such as albite and quartz. Columnar sparry calcites can be further categorized into type I, which develops in organic-rich limestone, and type II, which develops in organic-rich mudstone or shale. The impurity-rich granular and type I sparry calcites, characterized by low FeO content, are derived from the dissolution of early carbonates and primarily governed by the mechanism of material dissolution–(migration)–precipitation. Impurity-poor granular and type II sparry calcite, characterized by high FeO content, mainly originate from the transformation of clay minerals, and the thermal evolution of organic matter in the wall rocks is primarily governed by the mechanism of material enrichment–precipitation. The formation mechanisms of these different types of sparry calcite provide important insights into the evolution process of calcite in shale and elucidate the relationship between the synergistic effect of multiple components and the formation of shale oil reservoirs.