Deep learning is all about doing large searches and operations in the vector space. CUDA parallel programming is the main concept that allows you to divide any computation work over a large vectors per row, so you can distribute it more efficiently across different cores.
Nvidia has the dominance of Deep Learning because of the hegemony of CUDA, a proprietary programming framework. Because Nvidia was the first one to come up with a solution needed by data scientists to process large amounts of data.
NVIDIA dominates deep learning because Tensorflow and Pythorch both are implemented on top of cuDNN, also developed by NVIDIA and is built on top of CUDA, is a library of GPU-accelerated primitives for deep neural networks.
The main reason that you need an NVIDIA GPU is because of CUDA. CUDA is a proprietary programming framework developed by NVIDIA that facilitates massive parallelization of computing tasks using the cores in an NVIDIA GPU.
Actually, CUDA cores are not really the same as CPU cores. A CPU core is capable of fetching instructions, doing computations, replacing values in memory, releasing memory, writing to disk. A CUDA core on the other hand is only capable of processing computations in a single row of a vector. Much more limited than what a CPU core can do. But that doesn’t make it any less important to Deep Learning.
When picking the best CPU for Deep Learning, it is better to have a higher number of cores rather than having a high clock speed. The idea is that the higher number of parallel jobs you can start, the better it is for Deep Learning.