19 September 2017
P2P botnet has become one of the major threats in network security for serving as the infrastructure that responsible for various of cyber-crimes. Though some existing approaches have claimed to detect P2P botnets with high detection rate and low false positive rate (FPR), none of them is solely capable of resolving all the problems, especially, when one wants to deal with: (a) botnet with encrypted command-and-control (C&C) channels, (b) stealthy botnet which are nearly impossible to observe any malicious activities in the network traffic, (c) botnet in waiting stage, and (d) botnet with randomized/dynamic communication patterns. In this work, we present PeerHunter, a community behavior based method, which is capable of detecting P2P botnet with all the challenges mentioned above. Specifically, PeerHunter (a) uses mutual contacts as the only natural feature to aggregate bots within the same botnet into one community through a community detection approach, (b) uses community behavior features to detect potential botnet communities and further identify bot candidates from each botnet community, and (c) combines a signature based method as an optional component to further categorize the bot candidates into specific or unknown botnets. Through extensive experiments with real network trace, PeerHunter can achieve both 100% detection rate and very low FPR.