When does linear system have linearly growing singular values?

Question

Suppose $W$ is a large matrix where $i$th smallest singular value grows as $O(i)$. What kind of matrix can $W$ be?

For instance, this appears to hold for random matrix with IID entries and for lower-bidiagonal matrix below, where else does this occur?

$$ \begin{array}{cc} \left( \begin{array}{cccc} 1 & 0 & 0 & 0 \\ 1 & 1 & 0 & 0 \\ 0 & 1 & 1 & 0 \\ 0 & 0 & 1 & 1 \\ \end{array} \right)& \left( \begin{array}{cccc} -0.5 & -0.4 & 1.3 & 1.3 \\ -0.7 & 0.7 & 0. & -0.7 \\ 0.3 & 0.6 & 1.2 & 1.2 \\ -1.5 & 1.6 & 0.8 & 0. \\ \end{array} \right) \end{array} $$

Notebook

Motivation: people noticed that trained neural network exhibits $1/x$ spectral decay. This is also the decay observed in the solution $W$ of $y=Wx$ when $x$ comes from a dataset with linearly growing singular values. Curious what other datasets exhibit this.

Answer 1

Pick a matrix that is diagonal with entries $1,2,3,\ldots$ on the diagonal. This matrix has the property you are asking for, but it's probably not one you would find very interesting.

The point I'm trying to make is that finding matrices with one specific property is not very difficult. It is substantially more difficult to find matrices among a specific class (like ones resulting from neural networks) that have that property, oftentimes because "specific class" is not defined based on mathematical properties but as arising from certain applications.