Odtnhj

I've implemented a multi-layer perceptron neural network in Torch / lua and applied to an imbalanced dataset (33 features, around 300 elements, of which approximately 70% negative elements and 30% positive elements).

I tried several models and several settings. With the current model, I can get a quite high true negative rate (around 0.7) but a quite low true positive rate (around 0.3).
I really want to improve my true positive rate, but how?

I'm splitting the dataset into training set, validation set, and test set, and using the validation set to find the top number of hidden layers and hidden units. I'm currently using learning rate = 0.1 and iterations = 1000.
The activation function is a Sigmoid.
I tried several neural network tricks (momentum, dropout, Xavier initialization) but they did not lead to any improvement.

I worked with other imbalanced datasets in the past, and I was always able to get good rates even on the smaller class. But not this time...

The surprising thing is that my neural network does quite well during training, because I see the mean square error that drops very quickly. So I would expect good results on the test set as well, but it does not happen.

What can I do to improve my true positive rate?

Here's my model:

 perceptron = nn.Sequential() 

 perceptron:add(nn.Linear(this_input_number, this_hidden_units))
 perceptron:add(nn.Sigmoid())

 for w=1,this_hidden_layers do
 perceptron:add(nn.Linear(this_hidden_units, this_hidden_units))
 perceptron:add(nn.Sigmoid())
 end
 perceptron:add(nn.Linear(this_hidden_units, this_output_number))

And, for example, here's the MSE error during the second model training:

$$$ hidden_units = 25 hidden_layers = 1 $$$ 
completion: 10% (epoch=100)(element=194) loss = 0.04 error progress = 3.56623%
completion: 20% (epoch=200)(element=194) loss = 0.004 error progress = 2.2678%
completion: 30% (epoch=300)(element=194) loss = 0 error progress = 1.56329%
completion: 40% (epoch=400)(element=194) loss = 0.001 error progress = 1.2053%
completion: 50% (epoch=500)(element=194) loss = 0.003 error progress = 0.98677%
completion: 60% (epoch=600)(element=194) loss = 0.004 error progress = 0.84489%
completion: 70% (epoch=700)(element=194) loss = 0 error progress = 0.74345%
completion: 80% (epoch=800)(element=194) loss = 0 error progress = 0.67081%
completion: 90% (epoch=900)(element=194) loss = 0.021 error progress = 0.69629%
completion: 100% (epoch=1000)(element=194) loss = 0.001 error progress = 0.67999%

As you can see, the error quickly goes to zero.

You can find my complete working Torch code here, with the dataset included. You're more than welcome to test my code on your computer and try some additional variants of the method.

Do you have any suggestion on how to increase the true positive rate? Thanks!