This short review presents a selected history of the mutual fertilization between physics and economics, from Isaac Newton and Adam Smith to the present. The fundamentally different perspectives embraced in theories developed in financial economics compared with physics are dissected with the examples of the volatility smile and of the excess volatility puzzle. The role of the Ising model of phase transitions to model social and financial systems is reviewed, with the concepts of random utilities and the logit model as the analog of the Boltzmann factor in statistic physics. Recent extensions in term of quantum decision theory are also covered. A wealth of models are discussed briefly that build on the Ising model and generalize it to account for the many stylized facts of financial markets. A summary of the relevance of the Ising model and its extensions is provided to account for financial bubbles and crashes. The review would be incomplete if it would not cover the dynamical field of agent based models (ABMs), also known as computational economic models, of which the Ising-type models are just special ABM implementations. We formulate the ``Emerging Market Intelligence hypothesis'' to reconcile the pervasive presence of ``noise traders'' with the near efficiency of financial markets. Finally, we note that evolutionary biology, more than physics, is now playing a growing role to inspire models of financial markets.