# Correct Weights Neural Network

I'm trying to make a neural network using MATLAB. But I'm finding that many problems do not get solved.

These are the formulas on which I based: $$Layers\ in\ Neural\ Network = [0, ..., i, ..., out]$$ $$W_{i,j} = weights\ of\ the\ layer_i\ to\ layer_j\ \in M_{m\ x \ n}\\ B_{i,j} = weights\ of\ the\ layer_i\ to\ layer_j \in \mathbb{R}^n \\ d_{i} = delta\ correction\ of\ the\ layer_i\ \in \mathbb{R}^n \\ out_{j} = output\ signal\ of\ the\ layer_j\ \in \mathbb{R}^n \\$$

Update Weights Output layer $$d_{output} = out_{output}·(1-out_{output})·(out_{output} - Y)\\ W_{j, output}^{+} = W_{j,output} - n·out_{j}^{T}·d_{output} \\ B_{j, output}^{+} = B_{j,output} + n·d_{output} \\$$

Update Weights Hidden layers $$d_{i} = out_{i}·(1-out_{i})·(d_{i+1}·W_{i+1, i+2}^T)\\ W_{i,j}^{+} = W_{i,j} - n·out_{i-1}^{T}·d_{i} \\ B_{i,j}^{+} = B_{i,j} + n·d_{i} \\$$

Update Weights First Hidden layer $$d_{1} = out_{1}·(1-out_{1})·(d_{2}·W_{2,3}^T)\\ W_{1,2}^{+} = W_{1,2} - n·X^T·d_{1} \\ B_{1,2}^{+} = B_{1,2} + n·d_{1} \\$$

This is the code I generated:

classdef NeuralNetwork < handle
properties
% Wjk j inputs to k outputs (j inputs/neuron)
LW;
% Bk (k bias)
LB;
num_inputs;
num_outputs;
n_descensParameter;
end

methods
function obj = NeuralNetwork(num_inputs, num_outputs, hidden_layers, n_descensParameter)
obj.num_inputs = num_inputs;
obj.num_outputs = num_outputs;
obj.n_descensParameter = n_descensParameter;

size_ant = obj.num_inputs;
for i=1:length(hidden_layers)
obj.LW{i} = zeros(size_ant, hidden_layers(i));
obj.LB{i} = zeros(1, hidden_layers(i));
size_ant = hidden_layers(i);
end
obj.LW{length(hidden_layers)+1} = zeros(hidden_layers(end), num_outputs);
obj.LB{length(hidden_layers)+1} = zeros(1, num_outputs);
end

function InitWeights(obj)
b = sqrt(6)/sqrt(size(obj.LW{1}, 2)+obj.num_inputs);
obj.LW{1} = -b + (2*b).*rand(size(obj.LW{1}));
for i=2:length(obj.LW),
obj.LW{i} = -b + (2*b).*rand(size(obj.LW{i}));
b = sqrt(6)/sqrt(size(obj.LW{i}, 2)+size(obj.LW{i-1}, 2));
end
end

function out = predict(obj, X)
inputs = X;
for i=1:length(obj.LW)
inputs = obj.activationNeuronFunc(inputs*obj.LW{i} + obj.LB{i});
end
out = obj.cOutput(inputs);
end
function d = activationNeuronFunc(obj, x)
d = (1./(1+exp(-x)));
end
function d = cOutput(obj, o)
d = round(o); % softmax
end
function d = doutput(obj, o)
d = o.*(1-o);
end
function rval = cicle_backpropagation(obj, X, Y)
o = {};
inputs = X;
for i=1:length(obj.LW)
o{i} = obj.activationNeuronFunc(inputs*obj.LW{i} + obj.LB{i});
inputs = o{i};
end

LWplus = cell(size(obj.LW));
d = cell(size(obj.LW));

d{end} = obj.doutput(o{end}).*(o{end}-Y);
LWplus{end} = obj.LW{end} - obj.n_descensParameter.*(transpose(o{end-1})*d{end});
obj.LB{end} = obj.LB{end} + obj.n_descensParameter.*d{end};

for i=length(obj.LW)-1:-1:2
d{i} = obj.doutput(o{i}).*(d{i+1}*transpose(obj.LW{i+1}));
LWplus{i} = obj.LW{i} - obj.n_descensParameter.*(transpose(o{i-1})*d{i});
obj.LB{i} = obj.LB{i} + obj.n_descensParameter.*d{i};
end

d{1} = obj.doutput(o{1}).*(d{2}*transpose(obj.LW{2}));
LWplus{1} = obj.LW{1} - obj.n_descensParameter.*(transpose(X)*d{1});
obj.LB{1} = obj.LB{1} + obj.n_descensParameter.*d{1};

obj.LW = LWplus;

rval = sum(abs(Y-o{end}));
end
end
end


And this is a small test which should create the XOR expression:

% Learn
a = NeuralNetwork(2,2,[2], 1);
a.InitWeights()
X = [1, 1; 1, 0; 0, 1; 0, 0];
Y = [0, 1; 1, 0; 1, 0; 0, 1];
for i=1:1500,
for j=1:4,
a.cicle_backpropagation(c(j,:), b(j,:))
end
end
% Validate
a.predict([1,1]) % Must be 0, 1
a.predict([1,0]) % Must be 1, 0
a.predict([0,1]) % Must be 1, 0
a.predict([0,0]) % Must be 0, 1


The problem is that it does not learn correctly, I feel that there is an error in the code or formulas that have deduced. Could correct me? I'm pretty rookie and I want to learn to program it to understand fully.

• What exactly is the problem you're trying to solve with machine learning?
– Paul
Sep 25, 2015 at 14:14
• None in particular, I'm programming a neural network to perform experiments. And I've been trying to learn the XOR operation. I wanted the neural network was N hidden layers (where you decide the structure to create it), but I can not successfully update the weights, and the network does not learn. Sep 25, 2015 at 17:09

One thing you should keep in mind, for future reference, is when you go to find the weights for an Artificial Neural Network using the Backpropogation algorithm with Gradient Descent (as you are), you're using a numerical method that will get trapped in a local minima if you're initial guess for the weights aren't good enough.

So if you are finding trouble with converging to the right solution, that should be one more thing you look into if you are certain your code follows the math correctly.

Now there's two things you might want to experiment with to help with this problem. First, you might want to add in a Momentum term to the Gradient Descent update formula. This momentum term can be as simple as just adding in a weighted version of the last update to the new one. You can get more info here.

You might also want to consider searching for the weights using a method that might help avoid being stuck at local optimums, like Particle Swarm Optimization, Genetic Algorithms, etc. You could even potentially use a hybrid approach, where you initially use PSO or GA and then use the Backpropogation algorithm to refine the result. Here are some links to help you with these topics; PSO Link, GA Link.