The present study aims to analyze the thermal performance of the water-based nanofluids inside the wellbore heat exchangers to extract the geothermal energy from abandoned oil and gas wells (AOGW). Consequently, the study has considered Al2O3/water, Cu/water and CuO/water nanofluids as well as the Al2O3–Cu/water hybrid nanofluid with various particle volume concentrations ranging from 1% to 4% inside the coaxial wellbore heat exchanger (CWHE). For this purpose, a conjugate heat transfer module with a well-known k-epsilon (k-ε) turbulent model of COMSOL Multi-physics has been employed to simulate the fluid flow and heat transfer of the CWHE installed in AOGWs. Additionally, a sensitivity analysis has also been performed to determine the impact of individual operational parameters such as injection fluid mass flow rate and temperature on the system's thermal performance. It has been noted that the outlet fluid temperature enhanced by 25% with the addition of 4% of Cu nanoparticles to the water. Moreover, the maximum coefficient of performance of CWHE has been observed around 1.83 using Cu/water nanofluid with 4% volumetric fractions at an injection fluid temperature of 20 °C and a mass flow rate of 5 kg/s. Finally, the Cu/water nanofluid provided the best thermal performance followed by Al2O3-Cu /water, CuO/water and Al2O3/water nanofluids. It is expected that the findings of this study could motivate the researchers to work on the potential application of nanofluids/hybrid nanofluids to extract geothermal energy from AOGWs.