In our previous research, a novel series of histone deacetylase (HDAC) inhibitors with l-phenylglycine scaffold were designed and synthesized, among which amides D3 and D7 and ureido D18 were far superior to the positive control (suberoylanilide hydroxamic acid [SAHA]) in HDAC inhibition, but were only comparable to SAHA in antiproliferation on tumor cell lines. Herein, further structural derivation of lead compounds D3, D7, and D18 was carried out to improve their cellular activities. Most of our newly synthesized compounds exhibited more potent HDAC inhibitory activities than the positive control SAHA, and several derivatives were even better than their parent compounds. However, compared with SAHA and our lead compounds, only secondary amine series compounds exhibited improved antiproliferative activities, likely due to their appropriate topological polar surface area values and cell permeabilities. In a human histiocytic lymphoma (U937) xenograft model, the most potent secondary amine 9d exhibited similar in vivo antitumor activity to that of SAHA.