Terence Coelho comments on Why square errors?

[Terence Coelho]

In case it hasn’t crossed your mind, I personally think it’s helpful to start in the setting of estimating the true mean $\mu$ of a data stream. A very natural choice estimator for $\mu$ is the sample mean of the $x_i,$ which I’ll denote $\hat{\mu }$. This can equivalently be formulated as the minimizer of $\sum (x_i-\hat{\mu }{)}^2$.

Others have mentioned the normal distribution, but this feels secondary to me. Here’s why—let’s say $x_i\sim \sigma f\left(\frac{x-\mu }{\sigma }\right)$, where $f\left(x\right)$ is a known continuous probability distribution with mean 0 and variance 1, and $\mu ,\sigma$ are unknown. So the distribution of each $x_i$ has mean $\mu$ and variance ${\sigma }^2$ (and assume independence).

What must $f\left(x\right)$ be for the sample mean $\hat{\mu }$ to be the maximum likelihood estimator of $\mu$? Gauss proved that it must be $\frac{1}{2\sqrt{\pi }}{e}^{-x^2/2}$, and intuitively it’s not hard to see why it would have to be of the form $a{e}^{b{x}^2}$.

So from this perspective, MSE is a generalization of taking the sample mean, and asking the linear model to have gaussian errors is necessary to formally justify MSE through MLE.

Replace sample mean with sample median and you get the mean absolute error.