Pollinator-mediated selection towards larger and abundant flowers is common in naturally pollen-limited populations. However, floral antagonists may counteract this effect, maintaining smaller- and few-flowered individuals within populations. We quantified pollinator and antagonist visit rates and determined a multiplicative female fitness component from attacked and non-attacked flowers of the Brazilian hummingbird-pollinated shrub Collaea cipoensis to determine the selective effects of pollinators and floral antagonists on flower size and number. We predicted that floral antagonists reduce the female fitness component and thus exert negative selective pressures on flower size and number, counteracting the positive effects of pollinators. Pollinators, mainly hummingbirds, comprised 4% of total floral visitation, whereas antagonist ants and bees accounted for 90% of visitation. Nectar-robbers involved about 99% of floral antagonist visit rates, whereas florivores comprised the remaining 1%. Larger and abundant flowers increased both pollinator and antagonist visit rates and the female fitness component significantly decreased in flowers attacked by nectar-robbers and florivores in comparison to non-attacked flowers. We detected that pollinators favored larger- and many-flowered individuals, whereas floral antagonists exerted negative selection on flower size and number. This study confirms that floral antagonists reduce female plant fitness and this pattern directly exerts negative selective pressures on flower size and number, counteracting pollinator-mediated selection on floral attractiveness traits.