Maximum likelihood estimation: derivations by hand
Ten problems on maximum likelihood estimation: the Bernoulli, categorical, Poisson and Gaussian MLEs by hand, the curvature of the log-likelihood, why the variance MLE divides by N and is biased, least squares and cross-entropy as negative log-likelihoods, MAP with a Gaussian or Laplace prior as ridge and lasso with the exact λ, and the uniform MLE that no derivative finds, with worked solutions and the mistakes that maximize the wrong thing.
Before you start
Almost every loss in machine learning is a negative log-likelihood in disguise. Least squares is the likelihood of Gaussian noise, binary and softmax cross-entropy are the likelihoods of Bernoulli and categorical labels, and weight decay is a Gaussian prior on the weights. These ten problems derive the standard maximum likelihood estimates by hand, check that each stationary point really is a maximum, work out what the estimates get wrong on average, and then turn the regression losses of the previous pages back into the probability models they came from, priors included. The five mistakes at the end are the ones that give a plausible number: the unbiased variance passed off as the maximum likelihood one, the exponential distribution's answer used for the Poisson, a constraint forgotten, a prior strength off by a factor of ,N, and a moment match used where the derivative has no zero.
The data x1,…,xN are independent draws from a distribution with density or probability mass ,p(x;θ), where θ is the parameter to estimate.
The likelihood is ,L(θ)=∏np(xn;θ), the probability of the data as a function of .θ. The log-likelihood is ,ℓ(θ)=logL(θ)=∑nlogp(xn;θ), with the natural log. Because log is strictly increasing, L and ℓ have the same maximizer, and the sum is easier to differentiate than the product.
The maximum likelihood estimate (MLE) is .θ^=argmaxθℓ(θ). A zero derivative only finds a stationary point; a negative second derivative (or a concave )ℓ) is what makes it a maximum.
xˉ=N1∑nxn is the sample mean.
N(μ,σ2) is the Gaussian with mean μ and variance ,σ2, density .(2πσ2)−1/2exp(−(x−μ)2/(2σ2)).
E is the expectation over a random sample. An estimator θ^ is unbiased if ,E[θ^]=θ, and its bias is .E[θ^]−θ. For independent variables the variance of a sum is the sum of the variances, and .Var(cX)=c2Var(X).
Regression uses the notation of the regression page: X∈RN×d has row n equal to ,x(n)⊤,y∈RN holds the targets, w∈Rd the weights, ∥v∥2=∑jvj2 and .∥v∥1=∑j∣vj∣.
Bayes' rule for the weights: ,p(w∣y)=p(y∣w)p(w)/p(y), where p(w) is the prior. The maximum a posteriori (MAP) estimate maximizes ;logp(y∣w)+logp(w);p(y) does not depend on .w.
A coin lands heads with probability .θ. In N independent flips, xn=1 for heads and 0 for tails, and k=∑nxn is the number of heads, with .0<k<N. Find the MLE ,θ^, show it is a maximum, and compute .ℓ′′(θ^).
··
A categorical variable takes one of K values, value i with probability ,πi, where πi>0 and .∑iπi=1. In N independent draws, value i appears ni>0 times. Find the MLE π^ with a Lagrange multiplier.
·
Counts xn∈{0,1,2,…} are independent Poisson with rate :λ>0:.P(x;λ)=λxe−λ/x!. Find the MLE ,λ^, assuming not every count is .0.
··
Let x1,…,xN be independent draws from ,N(μ,σ2), with N≥2 and not all xn equal. Find the joint MLE (μ^,σ^2) and show it is a maximum.
···
The xn are independent with mean μ and variance σ2 (no Gaussian assumption needed). Show that the variance MLE of Problem 4 has .E[σ^2]=NN−1σ2.
··
Linear regression with Gaussian noise: yn=x(n)⊤w+εn with the εn independent .N(0,σ2). Write the negative log-likelihood, and find the MLE of w and σ2 when the columns of X are independent.
·
Logistic regression models ,P(yn=1∣x(n))=σ(x(n)⊤w), with labels yn∈{0,1} independent given the inputs and .σ(t)=1/(1+e−t). Show that −ℓ(w)/N is the mean binary cross-entropy L(w) of the regression page.
···
Keep the model of Problem 6 with σ2 known, and give the weights independent Gaussian priors .wj∼N(0,τ2). Show that the MAP estimate minimizes the mean ridge loss ,N1∥Xw−y∥2+λ∥w∥2, find ,λ, and write .wMAP.
··
Same model, but with independent Laplace priors ,p(wj)=2b1e−∣wj∣/b,.b>0. Show that the MAP estimate minimizes N1∥Xw−y∥2+λ∥w∥1 and find .λ.
···
The xn are independent and uniform on ,[0,θ], with θ>0 unknown and p(x;θ)=1/θ for ,0≤x≤θ,0 otherwise. Find the MLE ,θ^, explain why setting a derivative to zero cannot find it, and compute .E[θ^].
Answers
,θ^=k/N, and ℓ′′(θ^)=−θ^(1−θ^)N
π^i=ni/N
λ^=xˉ
,μ^=xˉ,σ^2=N1∑n(xn−xˉ)2
E[σ^2]=NN−1σ2
w^=(X⊤X)−1X⊤y and σ^2=N1∥Xw^−y∥2
−ℓ(w)/N=−N1∑n[ynlogpn+(1−yn)log(1−pn)]=L(w)
λ=Nτ2σ2 and wMAP=(X⊤X+(σ2/τ2)I)−1X⊤y
:λ=Nb2σ2: the MAP estimate under a Laplace prior is the lasso
,θ^=maxnxn, with E[θ^]=N+1Nθ
Worked solutions
Problem 1
A coin lands heads with probability .θ. In N independent flips, xn=1 for heads and 0 for tails, and k=∑nxn is the number of heads, with .0<k<N. Find the MLE ,θ^, show it is a maximum, and compute .ℓ′′(θ^).
,p(x;θ)=θx(1−θ)1−x, so .ℓ(θ)=∑n[xnlogθ+(1−xn)log(1−θ)]=klogθ+(N−k)log(1−θ).One formula covers both outcomes: x=1 leaves θ and x=0 leaves .1−θ. The log of the product is the sum of the logs, and ,∑nxn=k,.∑n(1−xn)=N−k.
.ℓ′(θ)=θk−1−θN−k.logu differentiates to ;1/u; the chain rule on log(1−θ) supplies the minus sign.
.ℓ′(θ)=0⟺k(1−θ)=(N−k)θ⟺θ=k/N.Multiply by ,θ(1−θ)>0, which does not change where the expression is zero; the kθ terms cancel.
ℓ′′(θ)=−θ2k−(1−θ)2N−k<0 for every .θ∈(0,1).Both terms are negative because .0<k<N. So ℓ is strictly concave on (0,1) and its one stationary point is the global maximum.
At :θ^=k/N:θ^2k=θ^N and ,(1−θ^)2N−k=1−θ^N, whose sum is .θ^(1−θ^)N.Substitute k=Nθ^ and ;N−k=N(1−θ^); over the common denominator θ^(1−θ^) the numerator is .N(1−θ^)+Nθ^=N.
,θ^=k/N, and ℓ′′(θ^)=−θ^(1−θ^)NThe curvature grows in proportion to :N: more flips make a sharper peak. Its reciprocal with the sign flipped, ,θ^(1−θ^)/N, is the usual estimate of the variance of ,θ^, and here it is exactly Var(k/N)=θ(1−θ)/N with θ replaced by .θ^. If k=0 or ,k=N,ℓ is monotonic and the maximum sits at the boundary 0 or ,1, which is still k/N but is not a stationary point.
Problem 2
A categorical variable takes one of K values, value i with probability ,πi, where πi>0 and .∑iπi=1. In N independent draws, value i appears ni>0 times. Find the MLE π^ with a Lagrange multiplier.
.ℓ(π)=∑i=1Knilogπi.Each draw of value i contributes a factor πi to the likelihood, and there are ni of them.
,L(π,ν)=∑inilogπi+ν(1−∑iπi), with multiplier .ν.The πi are not free: on its own ℓ increases in every πi and has no stationary point (Mistake 3). The multiplier term enforces .∑iπi=1.
,∂L/∂πi=ni/πi−ν=0, so .πi=ni/ν.Only the i-th terms of the two sums contain .πi.
,∑ini/ν=1, so .ν=∑ini=N.Substitute step 3 into the constraint.
ℓ is a sum of concave functions ,nilogπi, and the set {π:∑iπi=1,πi>0} is convex.A stationary point of the Lagrangian of a concave function under a linear equality constraint is the constrained global maximum.
π^i=ni/NThe estimate is the observed frequency. At π^ every partial derivative ni/π^i equals ,N, so ∇ℓ is parallel to :(1,…,1): no move that keeps the sum at 1 can increase ℓ to first order. With K=2 this is Problem 1.
Problem 3
Counts xn∈{0,1,2,…} are independent Poisson with rate :λ>0:.P(x;λ)=λxe−λ/x!. Find the MLE ,λ^, assuming not every count is .0.
,logP(x;λ)=xlogλ−λ−logx!, so .ℓ(λ)=(∑nxn)logλ−Nλ−∑nlogxn!.The log turns the power into a product and the quotient into a difference; the last sum does not contain .λ.
.ℓ′(λ)=λ∑nxn−N=0⟺λ=N1∑nxn.The constant ∑nlogxn! differentiates to .0.
.ℓ′′(λ)=−λ2∑nxn<0.∑nxn>0 by assumption, so ℓ is strictly concave and the stationary point is the maximum.
λ^=xˉThe mean of a Poisson variable is ,λ, so the MLE is the sample mean. The exponential distribution, which models the waiting times between Poisson events, also has a parameter called a rate, but its mean is 1/λ and its MLE is 1/xˉ (Mistake 2).
Problem 4
Let x1,…,xN be independent draws from ,N(μ,σ2), with N≥2 and not all xn equal. Find the joint MLE (μ^,σ^2) and show it is a maximum.
.ℓ(μ,σ2)=−2Nlog(2πσ2)−2σ21∑n(xn−μ)2.The log of the Gaussian density, summed over the data.
,∂μ∂ℓ=σ21∑n(xn−μ)=0⟺μ=xˉ, whatever σ2 is.(xn−μ)2 differentiates to ,−2(xn−μ), and the −2 cancels the .−21. Since ,∑n(xn−μ)=Nxˉ−Nμ, the zero does not depend on .σ2.
With v=σ2 as the variable: .∂v∂ℓ=−2vN+2v21∑n(xn−μ)2=0⟺v=N1∑n(xn−μ)2.log(2πv) differentiates to 1/v and 1/v to .−1/v2. Multiply by 2v2>0 to solve.
Put μ=xˉ from step 2 into step 3.The joint stationary point has to satisfy both equations, and step 2 fixes μ independently of .v.
For each ,v,ℓ is a concave quadratic in μ with its peak at .xˉ. With ,S=∑n(xn−xˉ)2>0, the profile ℓ(xˉ,v)=−2Nlog(2πv)−S/(2v) has derivative ,(S−Nv)/(2v2), positive for v<S/N and negative beyond.So ℓ(μ,v)≤ℓ(xˉ,v)≤ℓ(xˉ,S/N) for every :(μ,v): the stationary point is the global maximum. S>0 because the xn are not all equal.
,μ^=xˉ,σ^2=N1∑n(xn−xˉ)2The variance estimate divides by ,N, not ;N−1; Problem 5 shows what that costs. Differentiating with respect to σ instead of σ2 gives the same answer, because the maximizer does not depend on how the parameter is written.
Problem 5
The xn are independent with mean μ and variance σ2 (no Gaussian assumption needed). Show that the variance MLE of Problem 4 has .E[σ^2]=NN−1σ2.
.∑n(xn−xˉ)2=∑n(xn−μ)2−N(xˉ−μ)2.Write xn−xˉ=(xn−μ)−(xˉ−μ) and expand. The cross term is ,−2(xˉ−μ)∑n(xn−μ)=−2N(xˉ−μ)2, because ;∑n(xn−μ)=N(xˉ−μ); adding the N(xˉ−μ)2 from the squares leaves .−N(xˉ−μ)2.
.E[∑n(xn−μ)2]=Nσ2.Each term has expectation ,Var(xn)=σ2, and expectation is linear.
.E[(xˉ−μ)2]=Var(xˉ)=σ2/N.,E[xˉ]=μ, so this is the variance of .xˉ. By independence ,Var(∑nxn)=Nσ2, and the factor 1/N divides that by .N2.
.E[∑n(xn−xˉ)2]=Nσ2−N⋅σ2/N=(N−1)σ2.Steps 1 to 3, by linearity of expectation.
E[σ^2]=NN−1σ2Divide step 4 by .N. The MLE is biased low: xˉ is fitted to the same data, and it minimizes ∑n(xn−c)2 over ,c, so the residuals about xˉ are smaller than those about the true .μ. Dividing by N−1 instead gives the unbiased sample variance; the two agree as .N→∞.
Problem 6
Linear regression with Gaussian noise: yn=x(n)⊤w+εn with the εn independent .N(0,σ2). Write the negative log-likelihood, and find the MLE of w and σ2 when the columns of X are independent.
.yn∼N(x(n)⊤w,σ2).yn is the constant x(n)⊤w plus Gaussian noise, and adding a constant shifts the mean without changing the variance.
.−ℓ(w,σ2)=2Nlog(2πσ2)+2σ21∥Xw−y∥2.Problem 4, step 1, with μ replaced by x(n)⊤w for example ;n;∑n(yn−x(n)⊤w)2=∥Xw−y∥2 because entry n of Xw is .x(n)⊤w.
For every fixed ,σ2,w enters only through ∥Xw−y∥2 with the positive coefficient ,1/(2σ2), so w^ minimizes :∥Xw−y∥2:.w^=(X⊤X)−1X⊤y.Minimizing a positive multiple of a function has the same minimizer; the least-squares solution is the regression page, Problem 2. The noise level does not affect .w^.
With ,w=w^, setting ∂(−ℓ)/∂σ2=0 gives .σ2=N1∥Xw^−y∥2.This is Problem 4, step 3, with the residuals yn−x(n)⊤w^ in place of .xn−μ.
w^=(X⊤X)−1X⊤y and σ^2=N1∥Xw^−y∥2Least squares is maximum likelihood under Gaussian noise, and the mean squared error at the optimum is the noise-variance MLE. Like Problem 5, it is biased low, now because d parameters were fitted: dividing by N−d removes the bias.
Problem 7
Logistic regression models ,P(yn=1∣x(n))=σ(x(n)⊤w), with labels yn∈{0,1} independent given the inputs and .σ(t)=1/(1+e−t). Show that −ℓ(w)/N is the mean binary cross-entropy L(w) of the regression page.
P(yn∣x(n);w)=pnyn(1−pn)1−yn with .pn=σ(x(n)⊤w).The Bernoulli of Problem 1, with its probability now different for each example.
.ℓ(w)=∑n[ynlogpn+(1−yn)log(1−pn)].The log of the product of the step 1 factors, as in Problem 1, step 1.
−ℓ(w)/N=−N1∑n[ynlogpn+(1−yn)log(1−pn)]=L(w)So minimizing the mean binary cross-entropy is maximum likelihood, and its gradient N1X⊤(σ(Xw)−y) (regression page, Problem 6) is the negative score divided by .N. The 1/N scales the loss without moving the minimizer. In the same way, softmax cross-entropy is the negative log-likelihood of Problem 2's categorical distribution with per-example probabilities.
Problem 8
Keep the model of Problem 6 with σ2 known, and give the weights independent Gaussian priors .wj∼N(0,τ2). Show that the MAP estimate minimizes the mean ridge loss ,N1∥Xw−y∥2+λ∥w∥2, find ,λ, and write .wMAP.
.logp(w∣y)=logp(y∣w)+logp(w)−logp(y).Bayes' rule; logp(y) does not depend on .w.
.logp(w)=∑j[−21log(2πτ2)−wj2/(2τ2)]=const−∥w∥2/(2τ2).The prior is a product over ,j, so its log is a sum of Gaussian log-densities with mean .0.
,−logp(w∣y)=2σ21∥Xw−y∥2+2τ21∥w∥2+c, with c free of .w.Problem 6, step 2, for ;−logp(y∣w); the log(2πσ2) and log(2πτ2) terms and logp(y) go into .c.
Multiplying by :2σ2/N>0:.N1∥Xw−y∥2+Nτ2σ2∥w∥2+c′.A positive multiple has the same minimizer, and this one makes the data term a mean.
The mean ridge loss with λ has minimizer ,(X⊤X+NλI)−1X⊤y, and .Nλ=σ2/τ2.Regression page, Problem 3, which exists for every .X.
λ=Nτ2σ2 and wMAP=(X⊤X+(σ2/τ2)I)−1X⊤yWeight decay is a Gaussian prior. A wide prior ()τ→∞) gives λ→0 and the MLE of Problem 6. For the mean loss, λ shrinks as N grows: the prior's pull is fixed while the evidence accumulates.
Problem 9
Same model, but with independent Laplace priors ,p(wj)=2b1e−∣wj∣/b,.b>0. Show that the MAP estimate minimizes N1∥Xw−y∥2+λ∥w∥1 and find .λ.
.logp(w)=−dlog(2b)−∥w∥1/b.The log of a product of d Laplace densities: each contributes .−log(2b)−∣wj∣/b.
.−logp(w∣y)=2σ21∥Xw−y∥2+b1∥w∥1+c.Problem 8, steps 1 and 3, with this prior in place of the Gaussian one; constants go into .c.
Multiplying by :2σ2/N:.N1∥Xw−y∥2+Nb2σ2∥w∥1+c′.The same positive rescaling as Problem 8, step 4.
:λ=Nb2σ2: the MAP estimate under a Laplace prior is the lassoThe factor is 2σ2 here and σ2 in Problem 8 because the Gaussian prior's exponent carries a 21 and the Laplace prior's does not. ∣wj∣ has a corner at ,0, so there is no closed form in general; the corner in the prior is also what lets the lasso set weights exactly to zero.
Problem 10
The xn are independent and uniform on ,[0,θ], with θ>0 unknown and p(x;θ)=1/θ for ,0≤x≤θ,0 otherwise. Find the MLE ,θ^, explain why setting a derivative to zero cannot find it, and compute .E[θ^].
L(θ)=∏np(xn;θ)=θ−N if ,θ≥m, and 0 if ,θ<m, where .m=maxnxn.One factor is 0 as soon as any xn lies above ,θ, and every xn lies in [0,θ] exactly when the largest one does.
L′(θ)=−Nθ−N−1<0 for every ,θ>m, and L=0 below .m.Above m the likelihood strictly decreases, so no θ has a zero derivative; below m it is zero, the smallest possible value.
.θ^=m.L jumps from 0 to m−N at m and decreases afterwards, so its maximum is the left end of the region where it is positive. The maximum is at a discontinuity, where no derivative exists: the calculus recipe assumes a maximum in the interior of a smooth region.
P(θ^≤t)=P(every xn≤t)=(t/θ)N for .0≤t≤θ.The maximum is at most t exactly when every point is, and by independence the N probabilities t/θ multiply.
.E[θ^]=∫0θt⋅θNNtN−1dt=θNN⋅N+1θN+1=N+1Nθ.Differentiating step 4 in t gives the density ;NtN−1/θN; then integrate .tN.
,θ^=maxnxn, with E[θ^]=N+1NθThe MLE can never exceed ,θ, so it is biased low, by ;θ/(N+1);NN+1maxnxn is unbiased. Problem 1 noted a boundary maximum at ;k=0; here the maximum is always at a boundary.
Where this goes wrong
1. Variance MLE divided by N − 1
pandas, spreadsheet VAR functions and most statistics courses report the sample variance with N−1 in the denominator, and that version is the one that sticks.
∂σ2∂ℓ=−2σ2N+2σ41∑n(xn−xˉ)2Right so far: Problem 4, step 3, at .μ=xˉ.
“The correct estimate of a variance divides by ,N−1, so the maximum likelihood one must too.”The shortcut that causes the mistake: “correct” there means unbiased (Problem 5), which is a different criterion from maximizing the likelihood.
σ^2=N−11∑n(xn−xˉ)2At that value the derivative in step 1 is ,−1/(2σ2), not ,0, so it is not the maximizer: the MLE divides by N (Problem 4). NumPy's np.var divides by N by default and pandas by ,N−1, so the same data give two answers in one notebook.
2. Poisson rate estimated as one over the mean
The exponential and the Poisson distributions both have a parameter called the rate ,λ, and the exponential's MLE is .1/xˉ.
ℓ(λ)=(∑nxn)logλ−Nλ+constRight so far: Problem 3, step 1.
“A rate is events per unit time, the reciprocal of the mean, so .λ^=1/xˉ.”The analogy that causes the mistake: an exponential variable is a waiting time with mean ;1/λ; a Poisson variable is a count with mean .λ.
λ^=1/xˉAt that value ,ℓ′(λ)=Nxˉ2−N, which is zero only when .xˉ=1. The MLE is xˉ (Problem 3): counts averaging 4 per hour give λ^=4 per hour, not .0.25.
3. Categorical log-likelihood made stationary without the constraint
Problem 1 had a single free parameter, and with K of them it is natural to set each partial derivative to zero in turn.
ℓ(π)=∑inilogπiRight so far: Problem 2, step 1.
“Set ∂ℓ/∂πi=0 for every ,i, as for any maximum of a function of several variables.”The shortcut that causes the mistake: the unconstrained condition applied to parameters that must sum to .1.
∂ℓ/∂πi=ni/πi=0This has no solution: ,ni/πi>0, so ℓ increases in every πi and, with nothing to stop it, every probability heads to .1. The constraint is what makes the classes compete for probability; with a multiplier the answer is π^i=ni/N (Problem 2).
4. Ridge strength set to σ²/τ² in a mean loss
For the summed loss the textbook correspondence is ,λ=σ2/τ2, and libraries document their loss as a mean.
−logp(w∣y)=2σ21∥Xw−y∥2+2τ21∥w∥2+cRight so far: Problem 8, step 3.
“Multiply by :2σ2: the penalty weight is .σ2/τ2.”The shortcut that causes the mistake: that is right for ,∥Xw−y∥2+λ∥w∥2, and the λ is then carried to a loss whose data term has a .1/N.
N1∥Xw−y∥2+τ2σ2∥w∥2 has the MAP estimate as its minimizerMultiplying step 1 by 2σ2/N gives λ=σ2/(Nτ2) (Problem 8). With σ2/τ2 the prior is N times too strong, and the more data there are, the further the estimate is from the posterior mode.
5. Uniform MLE taken as twice the mean
The derivative of θ−N never vanishes, and matching the mean θ/2 to xˉ looks like a reasonable way to get an answer anyway.
L(θ)=θ−N for ,θ≥maxnxn, and 0 otherwiseRight so far: Problem 10, step 1.
“There is no stationary point, so match the mean instead: .θ/2=xˉ.”The shortcut that causes the mistake: this is the method of moments, a different estimator with no claim to maximize .L.
θ^=2xˉFor the data ,0.2,0.3,1.9,,2xˉ=1.6, below the observation ,1.9, so :L(1.6)=0: the estimate says the data were impossible. The MLE is maxnxn=1.9 (Problem 10), and even when 2xˉ exceeds the maximum it has a smaller likelihood, because L decreases there.