The emergence of large amounts of genomic, chemical, and pharmacological data provides new opportunities and challenges. Identifying gene-drug associations is not only crucial in providing a comprehensive understanding of the molecular mechanisms of drug action, but is also important in the development of effective treatments for patients. However, accurately determining the complex associations among pharmacogenomic data remains challenging. We propose a higher order graph matching with multiple network constraints (HOGMMNC) model to accurately identify gene-drug modules. The HOGMMNC model aims to capture the inherent structural relations within data drawn from multiple sources by hypergraph matching. The proposed technique seamlessly integrates prior constraints to enhance the accuracy and reliability of the identified relations. An effective numerical solution is combined with a novel sampling strategy to solve the problem efficiently.