The efficient coding hypothesis states that the front end properties of sensory systems, such as visual, can be understood from the statistics of natural stimuli. In particular, starting with the decorrelation, or whitening, assumption, Atick et al. and van Hateren attempted to derive the shape of spatio-temporal receptive fields (STRF) in the visual system. However, to obtain a unique STRF, they had to make additional assumptions. By extending the predictive coding framework (Srinivasan et al.), and without making additional assumptions, we derive optimal STRFs and find that they are consistent with STRFs in cat LGN and fruit fly second-order neurons (LMCs). In addition, we propose that the LGN STRFs are obtained in a stage-wise manner modeled by a lattice filter. Indeed, operation of the lattice filter is consistent with several physiological measurements in LGN neurons and fruit fly LMCs. Therefore, the lattice filter model is a useful abstraction that may help unravel visual system function.