2
$\begingroup$

I simulated Oja's rule and BCM for a single postsynaptic neuron with two presynaptic neurons, and for 10000 inputs, where I randomly select one of $(0,1)$ or $(1/2,\sqrt{3}/2)$ as input. My learning rate was $1/100$ and I am using the sliding threshold for BCM. (I am following this tutorial: http://www.inf.ed.ac.uk/teaching/courses/nc/NClab8.pdf) However, both algorithms seem to converge to very different things, as pictured: Oja's rule

BCM rule

The axis are the weights from each presynaptic neuron, and the red stars are the two inputs I am feeding to the network. Blue means first steps of simulation and yellow last steps of simulation.

I know Oja's rule should extract the first principal component of the input data (which I think looks more like BCM did, so this is confusing me). But what truly puzzles me is that the weights converge to very different things! Is this supposed to happen? Why? How can I interpret the "final" weights in each case?

$\endgroup$

1 Answer 1

3
$\begingroup$

No, both look exactly like they should.

BCM is to be competitive between the input signals, thus finds that the vertical axis is a better separator for the two points than the horizontal axis. A very imprecise handling of the equilibrium conditions in their expectation form leads me to guess that the second weight oscillates around $4/\sqrt3=2.31$.

The Oja rule finds the dominant direction in an average sense, forced to lie on the unit circle or sphere. With two points of equal radius it is the bisector for the smaller angle between the lines through these points. That you get a segment on the circle is due to the learning rate, if that is continuously decreased, then the random oscillation also cover a narrower segment.

$\endgroup$
2
  • $\begingroup$ Thank you. I have been thinking about the first part of your answer, but I don't quite get it. What does it mean better separator? I am also not sure what you mean by "equilibrium conditions in their expectation form". Also, how do you get $4/\sqrt3$? The second paragraph is clear now (after a few more experiments), thank you! $\endgroup$
    – Lotte
    Dec 7, 2021 at 19:16
  • 1
    $\begingroup$ BCM is designed to highlight one input. The threshold makes it possible for the data to have two clusters, and such a clustering is the more stable situation. The rest was guessing, the average value of the vertical coordinate is $\sqrt3/4$. and the combination of $v-\theta$ and $v^2-\theta$ as factors on the right gives a tendency for $\theta$ to move towards $1$, which then makes the weight the reciprocal of the average. $\endgroup$ Dec 7, 2021 at 19:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.