Colloidal self-assembly -- the spontaneous organization of colloids into ordered structures -- has been considered key to producing next-generation materials. However, the present-day staggering variety of colloidal building blocks and the limitless number of thermodynamic conditions make a systematic exploration intractable. The true challenge in this field is to turn this logic around and to develop a robust, versatile algorithm to inverse design colloids that self-assemble into a target structure. Here, we introduce a generic inverse design method to efficiently reverse-engineer crystals, quasicrystals, and liquid crystals by targeting their diffraction patterns. Our algorithm relies on the synergetic use of an evolutionary strategy for parameter optimization, and a convolutional neural network as an order parameter, and provides a new way forward for the inverse design of experimentally feasible colloidal interactions, specifically optimized to stabilize the desired structure.