It is clear that the automorphism group of a generalized Reed-Muller code $\mathrm {GRM}(r,m)$ of degree $p^m$ over a prime field $\mathrm {F}_p$ contains the scalars $\mathrm {F}_p^*$ and the affine linear group $\mathrm {AGL}(m,p)$. It is shown that in general it is not bigger, i.e. $\mathrm {Aut}(\mathrm {GRM}(r,m))= \mathrm {F}_p^*\times \mathrm {AGL}(m,p)$ with the obvious exceptions for $r=0$, $m(p-1)$ and $m(p-1)-1$. This generalizes results known for $p=2$ to arbitrary primes. The proof depends on the ification of doubly transitive groups ; a few small cases are done by ad hoc methods.