The waves are titanic tidal tsunamis created by the gravitational effect of the black hole Gargantua.
In Interstellar the waves are first seen when Brand spots some 'mountains' in the distance. The fact that they are waves is only discovered when Cooper says that: 'They're not mountains, they're waves!' From that point on, it is total chaos. Brand gets pinned down by some wreckage while the mountainous waves advance closer. Cooper and Doyle are forced to send CASE out to save Brand. CASE rescues Brand just in time as the waves curl over the robot and the astronaut. The two manage to get into the Ranger 1 as the waves thunder above them. However, Doyle waits until Brand gets in before he attempts to do so. He leaves it too late and he cries in pain as the waves wash over him. The Ranger 1 takes off without Doyle and leaves Miller's Planet. The last time the waves are seen is when the camera pans down over Doyle's still body. The waves rumble slowly in the distance.
1. The waves are as high as skyscrapers.