AVHRR images do not have a geographical extent, per se. Instead, each scan line contains a set of “Earth location” points (GCPs) at various intervals. These points can be used to perform a polynomial transformation, or calibration, of the image, but since the points are determined systematically by the satellite, they may not be as accurate as ground truth.
Here are some suggestions that may improve your results with AVHRR georeferencing:
Perform panoramic distortion correction during import. This almost always reduces the RMSE of the polynomial transformation.
Extract the GCPs, rather than applying a calibration. This way, you have all of your geometric models (e.g., rubber sheeting) available to you, and you can examine the RMSE for various alternatives.
Look at the GCPs for any discontinuities. If your data is collected near the poles or the international dateline, you can get some very high RMSEs.
Consider converting your GCPs (using Coordinate Calculator) to your desired output projection before doing a geometric transformation. This is virtually a must if you have lat/lon discontinuities.