From the order condition $$\sigma(w)=\frac{\rho(w)}{\ln w}+O(|w-1|^p)$$ I get $$\sigma(w)=1+\frac{3}{2}(w-1)+\frac{5}{12}(w-1)^2=-\frac{1}{12}+\frac{2}{3}w+\frac{5}{12}w^2$$

Those coefficients are for Adams-Moulton method $$y_{n+2}=y_{n+1}+h(\frac{5}{12}f(t_{n+2},y_{n+2})+\frac{2}{3}f(t_{n+1},y_{n+1})-\frac{1}{12}f(t_{n},y_{n}))$$

But how can I get the coefficients for Adams-Bashforth? $$y_{n+2}=y_{n+1}+h(\frac{3}{2}f(t_{n+1},y_{n+1})-\frac{1}{2}f(t_{n},y_{n}))$$

My text says

Thus, for quadratic $\sigma$ and order 3 we truncate, obtaining $$\sigma(w)=1+\frac{3}{2}(w-1)+\frac{5}{12}(w-1)^2=-\frac{1}{12}+\frac{2}{3}w+\frac{5}{12}w^2$$ whereas in the explicit case where $\sigma$ is linear we have $p=2$, and so recover, unsurprisingly, the Adams-Basforth scheme $$y_{n+2}=y_{n+1}+h(\frac{3}{2}f(t_{n+1},y_{n+1})-\frac{1}{2}f(t_{n},y_{n})).$$

The polynomial $\sigma(w)$ corresponds to a series expansion around $w=1$, not $w=0$. So when you truncate it to degree 1, you eliminate the term $\frac5{12}(w-1)^2$, not the term $\frac5{12}w^2$, so that it becomes $\sigma(w) = 1+\frac32(w-1)$.