In the 1950s the drug thalidomide administered as a sedative to pregnant women led to the birth of thousands of children with multiple defects. Despite its teratogenicity, thalidomide and its derivatives lenalidomide and pomalidomide (together known as Immuno modulatory Drugs: IMiDs) recently emerged as effective treatments for multiple myeloma and 5q-dysplasia. IMiDs target the CUL4-RBX1-DDB1-CRBN (CRL4 CRBN) E3 ubiquitin ligase and promote the ubiquitination of Ikaros/Aiolos transcription factors by CRL4 CRBN. Here we present the crystal structure of the DDB1-CRBN complex bound to thalidomide, lenalidomide and pomalidomide. The structure establishes CRBN as a CRL4 CRBN substrate receptor, which enantioselectively binds IMiDs. Through an unbiased screen we identify the homeobox transcription factor MEIS2 as an endogenous substrate of CRL4 CRBN. Our studies suggest that IMiDs block endogenous substrates (MEIS2) from binding to CRL4 CRBN when recruiting Ikaros/Aiolos for degradation. This dual activity implies that small molecules can principally modulate a ligase to up- or down-regulate the ubiquitination of proteins.