First of all, thanks for all the replies so far. Appreciate the effort.
So maybe I'll just do an info dump here to let you all know what my attempts and progress has been, and maybe someone can identify the problem and suggest a course of action that I am capable of following.
So I've been trying a couple of things - basically throwing stuff at the wall to see what sticks.
The "start with single notes, then add a second note, then add a third, etc" as I have described in an earlier post above is one such attempt. I get to a point where I can play some random licks with the pentatonic scale... but I get stuck thinking what I'm doing is great for one genre - the blues - and I don't know how to apply any of that to any other genre. What if I want to play a jazzy ballad, or do a screaming rock/pop solo?
Another method I've been testing is to play chord tones along with the chord chart (or lead sheet) of a song. I start with the root note, then add the third, then add the fifth... and so far, I've gotten to being able to play arpeggios. Basically, playing patterns like 1-3-5-8va or 1-3-5-3 (I haven't gotten to 7ths yet), but then the gist of the problem again is, how do I go from arpeggios to actual solos/improv (and that are appropriate to the style/genre/song)?
One method that I feel I've gotten the most progress out of, is to embellish melodies. I've gotten to the point where I can be just sight-reading a song, but able to insert random embellishments like grace notes, turns and passing notes on-the-fly. But in the end, it still sounds like the original melody and I haven't been able to deviate far from it.
This video, for example, I'm amazed at how much he's able to embellish this song while I'm still stuck with simple grace notes, turns and short passing notes. That little embellishment at 0:26, for example, is something I'd love to be able to do to any song.
I've also tried to learn some short solos from songs/genres that I am interested in; in pop/rock songs, often that's learning the lead guitar part during the instrumental of a song. But again I hit the wall when I realize, the solos I learn are only appropriate for the song I'm playing along to. How do I take licks from them and apply them to other songs (and sounding appropriate to them)?
This led me to thinking maybe instead of learning the
hows, maybe I should try to tackle things from a different angle and study the
whys (hence, the books in question). That would, at least in theory, help with being able to learn solos from one song and try to learn the licks to apply them to another song.
But after a long, retrospective deep think, I think one of my main issues is that
I am too rigid with patterns/formulas and not imaginative enough to be able to improvise. In all four methods that I have tried, I've been strictly following a pattern and unable to deviate out of the formula, which is probably why I get stuck after a while, with roughly the same line of thinking -- "what I'm doing now is cool, but how do I get from this to that other genre I'm interested in?"