The ability to forecast how different objects will be affected by the applied action (i.e. intuitive physics), is likely to be very useful for executing novel manipulation tasks. Similarly, the ability to forecast how humans will act in the future (i.e. intuitive behavior) can enable an agent to plan its interactions with humans. I will present results of some preliminary investigations on building intuitive models of physics (http://arxiv.org/abs/1511.07404 , https://arxiv.org/abs/1606.07419) and intuitive models of behavior (in context of water polo games) directly from visual inputs.