Targeting a universal host protein exploited by most viruses would be a game-changing strategy that offers broad-spectrum solution and rapid pandemic control including the current COVID-19. Here, we found a common YxxØ-motif of multiple viruses that exploits host AP2M1 for intracellular trafficking. A library chemical, N-(p-amylcinnamoyl)anthranilic acid (ACA), was identified to interrupt AP2M1-virus interaction and exhibit potent antiviral efficacy against a number of viruses in vitro and in vivo, including the influenza A viruses (IAVs), Zika virus (ZIKV), human immunodeficiency virus, and coronaviruses including MERS-CoV and SARS-CoV-2. YxxØ mutation, AP2M1 depletion, or disruption by ACA causes incorrect localization of viral proteins, which is exemplified by the failure of nuclear import of IAV nucleoprotein and diminished endoplasmic reticulum localization of ZIKV-NS3 and enterovirus-A71-2C proteins, thereby suppressing viral replication. Our study reveals an evolutionarily conserved mechanism of protein-protein interaction between host and virus that can serve as a broad-spectrum antiviral target.