Animal eyes have evolved to process behaviorally important visual information, but how retinas deal with statistical asymmetries in visual space remains poorly understood. Using hyperspectral imaging in the field, in vivo 2-photon imaging of retinal neurons, and anatomy, here we show that larval zebrafish use a highly anisotropic retina to asymmetrically survey their natural visual world. First, different neurons dominate different parts of the eye and are linked to a systematic shift in inner retinal function: above the animal, there is little color in nature, and retinal circuits are largely achromatic. Conversely, the lower visual field and horizon are color rich and are predominately surveyed by chromatic and color-opponent circuits that are spectrally matched to the dominant chromatic axes in nature. Second, in the horizontal and lower visual field, bipolar cell terminals encoding achromatic and color-opponent visual features are systematically arranged into distinct layers of the inner retina. Third, above the frontal horizon, a high-gain UV system piggybacks onto retinal circuits, likely to support prey capture.