I'm trying to make a neural network using MATLAB. But I'm finding that many problems do not get solved.

These are the formulas on which I based: $$ Layers\ in\ Neural\ Network = [0, ..., i, ..., out] $$ $$ W_{i,j} = weights\ of\ the\ layer_i\ to\ layer_j\ \in M_{m\ x \ n}\\ B_{i,j} = weights\ of\ the\ layer_i\ to\ layer_j \in \mathbb{R}^n \\ d_{i} = delta\ correction\ of\ the\ layer_i\ \in \mathbb{R}^n \\ out_{j} = output\ signal\ of\ the\ layer_j\ \in \mathbb{R}^n \\ $$

Update Weights Output layer $$ d_{output} = out_{output}·(1-out_{output})·(out_{output} - Y)\\ W_{j, output}^{+} = W_{j,output} - n·out_{j}^{T}·d_{output} \\ B_{j, output}^{+} = B_{j,output} + n·d_{output} \\ $$

Update Weights Hidden layers $$ d_{i} = out_{i}·(1-out_{i})·(d_{i+1}·W_{i+1, i+2}^T)\\ W_{i,j}^{+} = W_{i,j} - n·out_{i-1}^{T}·d_{i} \\ B_{i,j}^{+} = B_{i,j} + n·d_{i} \\ $$

Update Weights First Hidden layer $$ d_{1} = out_{1}·(1-out_{1})·(d_{2}·W_{2,3}^T)\\ W_{1,2}^{+} = W_{1,2} - n·X^T·d_{1} \\ B_{1,2}^{+} = B_{1,2} + n·d_{1} \\ $$

This is the code I generated:

classdef NeuralNetwork < handle
        % Wjk j inputs to k outputs (j inputs/neuron)
        % Bk (k bias)

        function obj = NeuralNetwork(num_inputs, num_outputs, hidden_layers, n_descensParameter)
            obj.num_inputs = num_inputs;
            obj.num_outputs = num_outputs;
            obj.n_descensParameter = n_descensParameter;

            size_ant = obj.num_inputs;
            for i=1:length(hidden_layers)
                obj.LW{i} = zeros(size_ant, hidden_layers(i));
                obj.LB{i} = zeros(1, hidden_layers(i));
                size_ant = hidden_layers(i);
            obj.LW{length(hidden_layers)+1} = zeros(hidden_layers(end), num_outputs);
            obj.LB{length(hidden_layers)+1} = zeros(1, num_outputs);

        function InitWeights(obj)
            b = sqrt(6)/sqrt(size(obj.LW{1}, 2)+obj.num_inputs);
            obj.LW{1} = -b + (2*b).*rand(size(obj.LW{1}));
            for i=2:length(obj.LW),
                obj.LW{i} = -b + (2*b).*rand(size(obj.LW{i}));
                b = sqrt(6)/sqrt(size(obj.LW{i}, 2)+size(obj.LW{i-1}, 2));

        function out = predict(obj, X)
            inputs = X;
            for i=1:length(obj.LW)
                inputs = obj.activationNeuronFunc(inputs*obj.LW{i} + obj.LB{i});
            out = obj.cOutput(inputs);
        function d = activationNeuronFunc(obj, x)
            d = (1./(1+exp(-x)));
        function d = cOutput(obj, o)
            d = round(o); % softmax
        function d = doutput(obj, o)
            d = o.*(1-o);
        function rval = cicle_backpropagation(obj, X, Y)
            o = {};
            inputs = X;
            for i=1:length(obj.LW)
                o{i} = obj.activationNeuronFunc(inputs*obj.LW{i} + obj.LB{i});
                inputs = o{i};

            LWplus = cell(size(obj.LW));
            d = cell(size(obj.LW));

            d{end} = obj.doutput(o{end}).*(o{end}-Y);
            LWplus{end} = obj.LW{end} - obj.n_descensParameter.*(transpose(o{end-1})*d{end});
            obj.LB{end} = obj.LB{end} + obj.n_descensParameter.*d{end};

            for i=length(obj.LW)-1:-1:2
                d{i} = obj.doutput(o{i}).*(d{i+1}*transpose(obj.LW{i+1}));
                LWplus{i} = obj.LW{i} - obj.n_descensParameter.*(transpose(o{i-1})*d{i});
                obj.LB{i} = obj.LB{i} + obj.n_descensParameter.*d{i};

            d{1} = obj.doutput(o{1}).*(d{2}*transpose(obj.LW{2}));
            LWplus{1} = obj.LW{1} - obj.n_descensParameter.*(transpose(X)*d{1});
            obj.LB{1} = obj.LB{1} + obj.n_descensParameter.*d{1};

            obj.LW = LWplus;

            rval = sum(abs(Y-o{end}));

And this is a small test which should create the XOR expression:

% Learn
a = NeuralNetwork(2,2,[2], 1);
X = [1, 1; 1, 0; 0, 1; 0, 0];
Y = [0, 1; 1, 0; 1, 0; 0, 1];
for i=1:1500,
    for j=1:4,
        a.cicle_backpropagation(c(j,:), b(j,:))
% Validate
a.predict([1,1]) % Must be 0, 1
a.predict([1,0]) % Must be 1, 0
a.predict([0,1]) % Must be 1, 0
a.predict([0,0]) % Must be 0, 1

The problem is that it does not learn correctly, I feel that there is an error in the code or formulas that have deduced. Could correct me? I'm pretty rookie and I want to learn to program it to understand fully.

  • $\begingroup$ What exactly is the problem you're trying to solve with machine learning? $\endgroup$
    – Paul
    Sep 25, 2015 at 14:14
  • $\begingroup$ None in particular, I'm programming a neural network to perform experiments. And I've been trying to learn the XOR operation. I wanted the neural network was N hidden layers (where you decide the structure to create it), but I can not successfully update the weights, and the network does not learn. $\endgroup$ Sep 25, 2015 at 17:09

1 Answer 1


One thing you should keep in mind, for future reference, is when you go to find the weights for an Artificial Neural Network using the Backpropogation algorithm with Gradient Descent (as you are), you're using a numerical method that will get trapped in a local minima if you're initial guess for the weights aren't good enough.

So if you are finding trouble with converging to the right solution, that should be one more thing you look into if you are certain your code follows the math correctly.

Now there's two things you might want to experiment with to help with this problem. First, you might want to add in a Momentum term to the Gradient Descent update formula. This momentum term can be as simple as just adding in a weighted version of the last update to the new one. You can get more info here.

You might also want to consider searching for the weights using a method that might help avoid being stuck at local optimums, like Particle Swarm Optimization, Genetic Algorithms, etc. You could even potentially use a hybrid approach, where you initially use PSO or GA and then use the Backpropogation algorithm to refine the result. Here are some links to help you with these topics; PSO Link, GA Link.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.