I agree with most of what's already been said, but I would be careful about claiming the jazz greats didn't study theory. They may not have been studying chord/scale "theory" in the form it's evolved to today and is commonly taught, but they were definitely working on theory. Read Sonny Rollin's notebook that was published recently, look at Yusuf Lateef's work, John Coltrane, the list goes on.
I just started a 10 week in person course called "Theory of Jazz Improv". It's a dozen of a us in a room with our instruments. I will say from the outset I'm skeptical of how chord/scale theory is presented but I'd like to examine all of the concepts of improvisation with an open mind.
The educated early jazz players may have studied music theory, which, as you say is general and not the same as what is currently taught for improvisation. It's theory but only relates to how to use notated chords to build a solo. It's my understanding that this came about well after the fact via academic "reverse engineering". There was a demand for teaching students how to improvise... and that was the result. Unfortunately it resembles "paint by the numbers".
It's been my contention that those wishing to Creatively Improvise need to develop the ability to play by ear. This doesn't mean one shouldn't learn to read or study any form of music. Creative Improvisation is about hearing a line to play and being able to execute it in the same way one would sing. This is (for now) outside of what's easily taught.
If you can't hear a line to play, that's a problem. If you can hear what you would like to play and can't execute it that's also a problem. Those deficits won't be fixed by just learning to play finger memory riffs, arpeggios, or a host of other cut and paste lines. However after many hears you may be able to be familiar enough with your instrument to play what you would sing.
Developing a connection to your instrument so that you can play what you hear is the key. One exercise is to hear a line that you then immediately repeat without having to check your fingering, identify the note, the key you're playing in, etc. It's an ears to hands pathway instead of eyes to hands, or translating constructs to understand what you heard. It's also not the same a "ear training", which seems to be another academic type of "translating". It's direct and effective. Play what you hear.
The Other part of this is a matter of ones ability and experience. If you've got a good and broad mental musical library, then you have a lot to draw on. Creativity is however is pretty difficult to teach. Hearing in your head a line to play is not a lot different to playing what you just heard someone else play. The difficulty is in hearing a line that fits, communicates something more than technique, flows, and uses many of the general music theory constructs; melody, harmony, counterpoint, rhythm, etc.
If you go to most "jazz jams" you hear standards with players who may have studied jazz theory. The object in most cases is for the solo(s) to be a showcase for their technical level of playing. It can also (unfortunately) be a competition or "pecking order". How much creativity or musical appeal is there in that? Is there an audience for this or just others who are on the same tangent?
There is a world of music out there and improvisation can fit into much of it, however it's not always about solos, it's all of those other general theory aspects required for backing singers, playing harmonies, rhythmic lines, call and response, etc. Most popular music bands don't have written music. You bring to a group your ability to hear a line and be able to play that on the spot. The same would have been true for most of the jazz that isn't based on "standards" like Kind Of Blue and others. No fake book or continual repeats, they played what they heard in their heads in real time within a very simple structure. That's Creative Improvisation. IMHO most Jazz theory apes this without the spark of creativity. It gives the player something to play. Yet can they hear the notes they are playing before it comes out of their instrument? If you can't how is that you improvising?
A rather nasty quip by Truman Capote about another author's work is also apropos: That's not writing, that's typing.
Just because something is a “composition” doesn’t mean the material isn’t familiar to the composer. Even Bach had favorite licks….
The last line above is incorrect. Just listen to any of the many re-issue recordings that include alternate takes. The Cannonball record I referred to previously is one of them, with an alternate for Work Song, among others. In Cannonball’s solo on the alternate, there were two multibar phrases that were the same, though in one case he ended it differently. And a lot of the same licks. Nat’s solo had a whole chorus with the same general outline and rhythmic content. Bobby Timmon’s solo was very similar with the same structure throughout. I remember an Art Blakey record where Lee Morgan played the exact same intro on his solo on the alternate take, and there were clear examples of phrases and licks that were the same in both solos.
Bottom line, those Jazz Heroes of old worked on their improvisation all the time and developed strategies and licks and ideas which they then played night after night, show after show. Is this “real” improvisation? Of course it is. Does it mean they aren’t in the flow, they’re just phoning it in, they’re being lazy? No, it does not. Those guys studied their music and developed their own unique ways of playing it over many many hours, and they were deep into it. We should do the same.
Wade, I’ve listened to some of the music you have posted. You have ideas and concepts that are quite clear, and some are similar from piece to piece. Are you not improvising? Of course you are! But you have a vocabulary, and a thought process, and it’s recognizable from hearing you play.
I think a better way to think about this stuff is to think about having a conversation. You have the words you know, and the ones you use often. When you discuss a topic that you have discussed before, you often say the same kinds of things that you have before. Is this not a free conversation? Of course it is, it’s not a prewritten speech. But you have absolutely examined those ideas before, and maybe used phrases before that are similar or exactly alike to the ones you are saying now. Substitute “Tune” for “Topic”, and the process is the same.
@Mix O Lydian, I hope this conversation helps you. I’ve been playing a long time, longer than you have been alive. Playing seems natural to me. I keep trying to learn new things, and deepen my understanding of music in a multidimensional way. I keep looking for more things that sound good to me, and when I find one it’s like coming upon a gold nugget or a gem in the rough. So beautiful, so exciting! It’s a journey, and even after 68 years on it, I am amazed almost every day.
I greatly appreciate your post and agree with much of what you're saying. Yes I have musical quirks and phrases... vocabulary if you will. I don't think that's a detriment to me or anyone else. There's lots of jazz groups I've seen multiple times and it's certainly part of any performing group's playing to repeat tracks they may have recorded and play them with or without variations. It's expected! However those learning aren't in that category. My comments have always been about encouraging players to develop their ear and make the horn their voice. I've written this out once again (above).
Maybe my ability to communicate succinctly is lacking? However hopefully some will get the gist of this. Like you I'm old (79) and started with piano lessons at 6 years old (1952) and switched to woodwinds in the mid 1950s. I loved jazz at the time and still love players who are creative. However improvisation is not the sole provenance of jazz and can be, and is a part of many genres. I'd love to see that expand and hear more players develop their creativity. I don't hear that in most academic jazz players. I think that's a problem. Maybe you and others don't?
I'm not really here to debate and instead just want to throw out some ideas. Some seem to feel wounded or negatively challenged by those ideas. That's not a problem for me, and it shouldn't be a problem for them if they are playing and doing what pleases them.
We're old and have seen and heard a lot. We're in "legacy territory". I'm happy to try and inspire some with different information. Sharing information, ideas and choices that could benefit some is the goal.