This is very true...which is why characterizing the playing of Jazz (or any other genre) as just 'switching hats' isn't quite accurate. It is more than simply a matter of "ok so now I am playing THIS genre, so I gotta phrase like this and open up my Jazz bag of tricks".
This is why it isn't unusual when someone who is primarily a classical player sits in a jazz chair (say, for example, occupying a chair in a big band), many others in the band will quickly recognize "that guy/gal plays mostly legit , don't they ?"