Deep Learning Networks (DLNs) provide a versatile platform so that a lot of parameter may be learned to meet the demand of a broad spectrum of AI applications. We shall first address critical discrepancies between optimization and generalization, exempli-fied primarily by the four key gaps/issues: (1) data gap, (2) ca-pacity gap, (3) metric gap, and (4) algorithmic gap. The curse of depth on DLNs has widely been recognized as a cause of seri-ous concern for the execution of BP learning. To circumvent the depth problem, we resort to an Omni-present Supervision(OS) internal training strategy. This facilitates an internal OS learning strategy without invoking back-propagation. Two application sce-narios will be highlighted to showcase the merits of incorporat-ing internal OS learning into the external BP learning:

(a) OStrim: trim to derive cost-effective network in power, stor-ages, and FLOPS. (b) MIND-Net: designed to monotonically in-crease the network's discriminant capability.