The historical prospect of solving optimization problems can be traced back to the era of Newton, Lagrange, and Cauchy. The development of differential calculus methods of optimization solely depends on the contributions of Newton and Leibnitz to calculus. The foundations of calculus of variations, which deals with the minimization or maximization of functions, depending on the nature of the objective function were incorporated by Bernoulli, Euler, Lagrange, and Weirstrass. Optimization is meant to maximize the efficiency of a system by intelligently controlling its control parameter(s), possibly subject to a set of stipulated constraints.