Northeast China (NEC) is one of the most important rice production areas in China, accounting for approximately 15% of its total national rice production, and is one of the regions most vulnerable to global climate change. Knowing crops’ potential yields is crucial to understanding comparisons of different cultivars, crops and environments as well as plausible future increases and limits to crop yields. To mitigate the impacts of climate change and enhance food security, investigating yield potential (YP), attainable yield (YPA), nitrogen-limited potential yield (YPN), and actual farmers’ yield (Ya) as well as rice yield gaps and their causes in NEC under the climate change background is necessary. In this study, the ORYZA (v3) crop model was calibrated and validated for rice phenology and yields. The validated model was then used to determine YP, YPA and YPN values as well as yield gaps caused by nitrogen management factors (YGAN) and other agronomic and socioeconomic factors (YGNa) for five agroclimatic zones in NEC from 1981 to 2010. The regional area-weighted mean YP levels estimated by the model were 12.9 t ha−1 for YP and 11.6 t ha−1 for YPA with decreasing trends and 8.7 t ha−1 for YPN with an increasing tendency. The Ya was 6.8 t ha−1, with a significant increase of 1.1 t ha− 1 per decade (p < 0.01), and farmers achieved 58.6% of the YP. The YPA accounted for 80.9–94.8% of the YP in NEC. The regional exploitable yield gap (YGE) was 4.8 t ha−1. The regional yield gaps were shrinking at 0.47 and 0.88 t ha−1 per decade for YGAN and YGNa, respectively. In conclusion, because of the persistently large yield gap between farmers and YPN, YPA provides the government and farmers an opportunity to significantly increase rice production by controlling socioeconomic factors and adopting high-yield agronomic management practices based on site-specific conditions, including optimized irrigation and fertilization practices and “super” rice cultivars.