Odtnhj

Using the notations from Backpropagation calculus | Deep learning, chapter 4, I have this back-propagation code for a 4-layer (i.e. 2 hidden layers) neural network:

def sigmoid_prime(z): 
 return z * (1-z) # because σ'(x) = σ(x) (1 - σ(x))

def train(self, input_vector, target_vector):
 a = np.array(input_vector, ndmin=2).T
 y = np.array(target_vector, ndmin=2).T

 # forward
 A = [a] 
 for k in range(3):
 a = sigmoid(np.dot(self.weights[k], a)) # zero bias here just for simplicity
 A.append(a)

 # Now A has 4 elements: the input vector + the 3 outputs vectors

 # back-propagation
 delta = a - y
 for k in [2, 1, 0]:
 tmp = delta * sigmoid_prime(A[k+1])
 delta = np.dot(self.weights[k].T, tmp) # (1) <---- HERE
 self.weights[k] -= self.learning_rate * np.dot(tmp, A[k].T) 

It works, but:

• 
  

  the accuracy at the end (for my use case: MNIST digit recognition) is just ok, but not very good.
  It is much better (i.e. the convergence is much better) when the line (1) is replaced by:

  
  
  

  delta = np.dot(self.weights[k].T, delta) # (2)
  

  
  


• 
  

  the code from Machine Learning with Python: Training and Testing the Neural Network with MNIST data set also suggests:

  
  
  

  delta = np.dot(self.weights[k].T, delta)
  

  
  
  

  instead of:

  
  
  

  delta = np.dot(self.weights[k].T, tmp)
  

  
  
  

  (With the notations of this article, it is:

  
  
  

  output_errors = np.dot(self.weights_matrices[layer_index-1].T, output_errors)
  

  
  
  

  )

  
  

These 2 arguments seem to be concordant: code (2) is better than code (1).

However, the math seem to show the contrary (see video here; another detail: note that my loss function is multiplied by 1/2 whereas it's not on the video):

Question: which one is correct: the implementation (1) or (2)?

In LaTeX:

$$fracpartialCpartialw^L-1 = fracpartialz^L-1partialw^L-1 fracpartiala^L-1partialz^L-1 fracpartialCpartiala^L-1=a^L-2 sigma'(z^L-1) times w^L sigma'(z^L)(a^L-y) $$
$$fracpartialCpartialw^L = fracpartialz^Lpartialw^L fracpartiala^Lpartialz^L fracpartialCpartiala^L=a^L-1 sigma'(z^L)(a^L-y)$$
$$fracpartialCpartiala^L-1 = fracpartialz^Lpartiala^L-1 fracpartiala^Lpartialz^L fracpartialCpartiala^L=w^L sigma'(z^L)(a^L-y)$$

I spent two days to analyze this problem, I filled a few pages of notebook with partial derivative computations... and I can confirm:

• the maths written in LaTeX in the question are correct
  


• 
  

  the code (1) is the correct one, and it agrees with the math computations:

  
  
  

  delta = a - y
  for k in [2, 1, 0]:
   tmp = delta * sigmoid_prime(A[k+1])
   delta = np.dot(self.weights[k].T, tmp)
   self.weights[k] -= self.learning_rate * np.dot(tmp, A[k].T) 
  

  
  


• 
  

  code (2) is wrong:

  
  
  

  delta = a - y
  for k in [2, 1, 0]:
   tmp = delta * sigmoid_prime(A[k+1])
   delta = np.dot(self.weights[k].T, delta) # WRONG HERE
   self.weights[k] -= self.learning_rate * np.dot(tmp, A[k].T) 
  

  
  
  

  and there a slight mistake in Machine Learning with Python: Training and Testing the Neural Network with MNIST data set:

  
  
  

  output_errors = np.dot(self.weights_matrices[layer_index-1].T, output_errors)
  

  
  
  

  should be

  
  
  

  output_errors = np.dot(self.weights_matrices[layer_index-1].T, output_errors * out_vector * (1.0 - out_vector))
  

  
  

Now the difficult part that took me days to realize:

• Apparently the code (2) has a far better convergence than code (1), that's why I mislead into thinking code (2) was correct and code (1) was wrong




• ... But in fact that's just a coincidence because the learning_rate was set too low. Here is the reason: when using code (2), the parameter delta is growing much faster (print np.linalg.norm(delta) helps to see this) than with the code (1).




• Thus "incorrect code (2)" just compensated the "too slow learning rate" by having a bigger delta parameter, and it lead, in some cases, to an apparently faster convergence.

Now solved!