Everything you need to know is there in the music itself, the theory and written music are only the way we talk about it. So it's not just "by ear" players who should learn to play by listening, everybody should do it, and it should be a substantial element of learning.
All great "art" whether it is music, visual art, dance, or poetry can be experienced and as human beings we can be moved by those experiences. However to fully understand that art it needs to be broken down into its elements and undergo some type of analysis. The basic elements of music are Melody, Harmony, Rhythm, Dynamics, Timbre, Texture, and Form. To me it is that understanding that completes the experience of being moved by a piece of music or a particular performance. With that understanding I can communicate with others about my experience and I can look for those same elements in other pieces I hear for the first time.