There is an on-going debate in the seismological community as to whether stress drop is independent of earthquake size and this has important implications for earthquake physics. Here we investigate this question in a simple 2D cellular automaton that includes heterogeneity. We find that when the range of heterogeneity is low, the scaling approaches that of constant stress drop. However, clear deviations from the constant stress drop model are observed when the range of heterogeneity is large. Further, fractal distributions of strength show more significant departures from constant scaling than do random ones. Additionally, sub-sampling the data over limited magnitude ranges can give the appearance of constant stress drop even when the entire data set does not support this. Our results suggest that deviations from constant earthquake scaling are real and reflect the heterogeneity of natural fault zones, but may not provide much information about the physics of earthquakes.