First of all, thanks for all the replies so far. Appreciate the effort.
So maybe I'll just do an info dump here to let you all know what my attempts and progress has been, and maybe someone can identify the problem and suggest a course of action that I am capable of following.
So I've been trying a couple of things - basically throwing stuff at the wall to see what sticks.
... long description of various "methodologies" ... I think one of my main issues is that I am too rigid with patterns/formulas and not imaginative enough to be able to improvise. In all four methods that I have tried, I've been strictly following a pattern and unable to deviate out of the formula, which is probably why I get stuck after a while, with roughly the same line of thinking -- "what I'm doing now is cool, but how do I get from this to that other genre I'm interested in?"
I suspect you need to spend more time playing and listening to what you play, and less time trying strategies.
I am NOT saying that a knowledge of theory, functional harmony, chord-scale relationships, transcribing solos, all that, is not of value. But what I'm hearing here is what I've seen many times when people who've been trained in playing only what's written on the page try to improvise - they freeze up, they can only play what they've practiced, the exact notes I mean, but making up stuff out of their heads, they get stuck on little finger patterns learned from a book.
What you have to do is to get comfortable with playing stuff that's completely unwritten and with the bare minimum of planning; listening to what comes out; judging whether it sounds good or bad; doing more of what sounds good and less of what sounds bad.
Until you can let yourself go to do that (and accept that it's often going to sound bad, and that no catastrophe will follow), you'll be letting the strategizing and pattern memorization tail wag the improvisation dog. You need for all those books and patterns and transcriptions to be SUPPORTING the improvisation, not the other way round.
What you need to be doing is playing along with music on the radio or records. No music. No strategies. Pull stuff up that you don't even know the key. Or the tune. - Or, pull up stuff you've heard a thousand times, but never played it. Don't look it up for the key, the chord changes, the melody; just play the music and play along with it. Pick it out as you go along. If the melody's simple, play it some, if not, just go straight to improvising along with the record.
When you can get to the point where you can play stuff that more or less fits what you're hearing, using only those things on the sides of your head, THEN go back to the books and patterns and transcriptions - NOW you'll be ready to apply these things in SERVICE TO your playing, not thinking that those things ARE the playing.