Odtnhj

I am trying to use OpenMP to speed up my codes for Neural Network Computation. As I am using Visual Studio 2017, I need to enable the OpenMP Support in the Property sheets. However, after I have done that, some part of the code slows down by around 5 times even though I did not include any #pragma omp in the code.

I have isolated the sections and found out that this particular function is causing the problem:

void foo(Eigen::Matrix<float,3,Eigen::Dynamic> inputPts)

 std::vector<Eigen::MatrixXf> activation;
 activation.reserve(layerNo);
 activation.push_back(inputPts);

 int inputNo = inputPts.cols();

 for (int i = 0; i < layerNo - 2; i++)
 activation.push_back(((weights[i]*activation[i]).colwise()+bias[i]).array().tanh());

 activation.push_back(((weights[layerNo - 2]*activation[layerNo - 2]).colwise()+bias[layerNo - 2]));

 val = activation[layerNo - 1]/scalingFactor;

 std::vector<Eigen::MatrixXf> delta;
 delta.reserve(layerNo);

 Eigen::Matrix<float, 1, Eigen::Dynamic> seed;
 seed.setOnes(1, inputNo);
 delta.push_back(seed);

 for (int i = layerNo - 2; i >= 1; i--)
 
 Eigen::Matrix<float,Eigen::Dynamic,Eigen::Dynamic>
 d_temp = weights[i].transpose()*delta[layerNo - 2 - i],
 d_temp2 = 1 - activation[i].array().square(),
 deltaLayer = d_temp.cwiseProduct(d_temp2);

 delta.push_back(deltaLayer);
 

 grad = weights[0].transpose()*delta[layerNo - 2];

The two for-loops are the one that slow down significantly (from ~3ms to ~20ms). Strangely, although this function is called many times in the program, only some of them are affected.

I have included the header file <omp.h>. I am not sure whether it is due to the Eigen library, which is used everywhere. I tried defining EIGEN_DONT_PARALLELIZE and calling Eigen::initParallel() as suggested in the official site but it does not help.

The weird thing is that I did not even include any parallel pragma at all, there should not be any overhead to handle the OpenMP functions? Why is it still slowing down?

Eigen's matrix-matrix products are multi-threaded by default if OpenMP is enabled. The problem is likely the combination of:

1. Your CPU is hyper-threaded, e.g., you have 4 physical cores able to run 8 threads.


2. OpenMP does not allow to know the number of physical cores, and thus Eigen will launch 8 threads.


3. Eigen's matrix-matrix product kernel is fully optimized and exploits nearly 100% of the CPU capacity. Consequently, there is no room for running two such threads on a single core, and the performance drops significantly (cache pollution).

The solution is thus to limit the number of OpenMP threads to the number of physical cores, for instance by setting the OMP_NUM_THREADS environment variable. You can also disable Eigen's multithread by defining the macro EIGEN_DONT_PARALLELIZE at compilation time.

More info in the doc.

More details on how hyper-threading can decrease performance:
With hyper-threading you have two threads running in an interleaved fashion on a single core. They alternate every instruction. If your threads are not using less than half of the ressources of the CPU (in term of computation), then that's a win because you will exploit more computing units. But if a single thread is already using 100% of the computing units (as in the case of a well optimized matrix-matrix product), then you lose performance because of 1) the natural overhead of managing two threads and 2) because the L1 cache is now shared by two different tasks. Matrix-matrix kernels are designed with precise L1 capacity in mind. With two threads, your L1 cache becomes nearly ineffective. This means that instead of fetching the very fast L1 cache most of the time, you end up accessing the much slower L2 cache, and thus you get a huge performance drop. Unlike Linux and Windows, on OSX I don't observe such performance drop, most likely because the system is able to unschedule the second threads if the CPU is already too busy.