Odtnhj

I am trying to train a neural network in Python 3.7. For this, I am using Numpy to perform calculations and do my matrix multiplications. I find this error

RuntimeWarning: overflow encountered in multiply (when I am multiplying matrices)

This, in turn, results in nan values, which raises errors like

RuntimeWarning: invalid value encountered in multiply

RuntimeWarning: invalid value encountered in sign

Now, I have seen many answers related to this question, all explaining why this happens. But I want to know, "How do I solve this problem?". I have tried using the default math module, but that still doesn't work and raises errors like

TypeError: only size-1 arrays can be converted to Python scalars

I know I can use for loops to do the multiplications, but that is computationally very expensive, and also lengthens and complicates the code a lot. Is there any solution to this problem? Like doing something with Numpy (I am aware that there are ways to handle exceptions, but not solve them), and if not, then perhaps alternative to Numpy, which doesn't require me to change my code much?

I don't really mind if the precision of my data is compromised a bit. (If it helps the dtype for the matrices is float64)

EDIT:
Here is a dummy version of my code:

import numpy as np
network = np.array([np.ones(10), np.ones(5)])
for i in range(100000):
 for lindex, layer in enumerate(network):
 network[lindex] *= abs(np.random.random(len(layer)))*200000

I think the overflow error occurs when I am adding large values to the network.

This is a problem I too have faced with my neural network while using ReLu activators because of the infinite range on the positive side. There are two solutions to this problem:

A) Use another activation function: atan,tanh,sigmoid or any other one with limited range

However if you do not find those suitable:

B)Dampen the ReLu activations. This can be done by scaling down all values of the ReLu and ReLu prime function. Here's the difference in code:

##Normal Code
def ReLu(x,derivative=False):
 if derivative:
 return 0 if x<0 else 1
 return 0 if x<0 else x

##Adjusted Code
def ReLu(x,derivative=False):
 scaling_factor = 0.001
 if derivative:
 return 0 if x<0 else scaling_factor
 return 0 if x<0 else scaling_factor*x

Since you are willing to compromise on the precision this is a perfect solution for you! In the ending, you can multiply by the inverse of the scaling_factor to get the approximate solution- approximate because of rounding discrepancies.