Improving the energy efficiency of buildings is a key climate change mitigation strategy. The application of which will require substantial improvements in the pre-existing stock; a subset of which are buildings of historic importance. Retrofitting such buildings is controversial, as historic elements might be altered or covered up, thereby changing the character of the building. In this work, we introduce a novel socio-mathematical method to aid the resolution of this controversy. Firstly, we garner in a new way the views of 116 members of the public about the acceptability of 15 common retrofit measures. Secondly, the public's ranking of the acceptability of the measures with respect to heritage impact is compared to a ranking of the energy saving given by the measures when analysed using a dynamic thermal simulation of the building. No simple correlation is found; hence it is concluded that measures that present greater energy savings are not de facto more intrusive, and that there is the potential for a constructive dialogue between those inspired by a conservation agenda and those targeting carbon savings. Finally, by using a Pareto front approach, a new theory is developed of how to identify measures that are sensible in the eyes of both parties. This new three-stage process will be of use to those in Government attempting to resolve such conflicts or set national guidance.