Loss functions are among the most important parts of neural network design. A loss function helps us interact with a model, tell it what we want — this is why we classify them as “objective functions”. Let us look at the precise definition of a loss function.
In deep learning, an objective function is one whose output has to be minimized. Thus, the optimization algorithm needs to find a minima from the objective function. This is usually done by a backpropagation algorithm that calculates the gradients and then passes them over to the optimization algorithm. The optimization algorithm then changes the neural network parameters and weights so as to arrive at a lower objective function output.
