We report the direct observation of lattice phonons confined at LaAlO3/SrTiO3 (LAO/STO) interfaces and STO surfaces using the sum-frequency phonon spectroscopy. This interface-specific nonlinear optical technique unveiled phonon modes localized within a few monolayers at the interface, with inherent sensitivity to the coupling between lattice and charge degrees of freedom. Spectral evolution across the insulator-to-metal transition at LAO/STO interface revealed an electronic reconstruction at the sub-critical LAO thickness, as well as strong polaronic signatures upon formation of the two-dimensional electron gas. We further discovered a characteristic lattice mode from interfacial oxygen vacancies, enabling us to probe such important structural defects in situ. Our study provides a new perspective on many-body interactions at the correlated oxide interfaces.