Loss landscape of deep neural networks

Finding the global minimum of a general non-convex function is in general an NP-complete problem. The optimization objective of deep neural networks (DNN) is not only non-convex but even non-smooth in case of ReLU activation function. Yet, practice suggests that even simple first-order methods such as gradient descent are able (in many cases) to find the global optimum of DNN at random initialization. Understanding this phenomenon is one of the core problems in theoretical machine learning.