As Pete says, accidentals throw a monkey wrench into things. Here are some more rules for handling accidentals when reading bass clef on an Eb instrument:
...
johnggage.com
Keep the above page for reference for when you encounter that situation. Of course, your immediate problem is different, transposing on the fly from concert pitch to Bb, which is unrelated to the bass clef for Eb trick.
I think everybody has at least some difficulty keeping track of key signatures. There are so many other things to worry about while sight reading.
As far as remembering the order of sharps/flats, there are many mnemonics out there, so use whatever works for you. I personally just remember the word BEAD and the letters GCF forward and backward.
I also have a trick for handling keys with lots of sharps. Rather than remember what notes are sharp, I just remember what notes are NOT sharp using the flat key signature as a basis. For example, in the key of B, there are 5 sharps - F, C, G, D, A. Instead I think of B as Bb with every note sharped. What does that mean? If you sharp every note in Bb, then all the flats become naturals, and everything else is sharp. So thinking only of the Bb key signature - Bb and Eb, then changing both of those flats to naturals, and everything else sharp, that means I only have to remember that the key of B has only 2 natural notes - B and E, the exact same as the flatted notes in the key of Bb.
Sounds pretty complicated at first, but in the end, you only have to keep track of 2 things (the 2 natural notes in the key) rather than 5 (the 5 sharps in the key) as well as only remembering the order of flats, not sharps. Another example, the key of F# (6 sharps). So I think of the key of F which has only Bb. Sharping every note results in the only natural note being B. So all I have to remember is one thing, B natural. Everything else in the key of F# is sharp.
Conversely for keys with lots of flats, you can think about only the remaining naturals. For example the key of Db (5 flats) thinking of it as the key of D with every note flatted means the 2 sharps in D which are F and C become the only 2 naturals in the key of Db. So I only have to remember that all notes are flat except F and C.
I like to call this exception based key signatures. When I see huge numbers of flats or sharps (more than 3), I apply the rules above and keep track of just the naturals in those difficult keys. The end result is I never have to remember more than 3 notes in any key. So here's what I end up thinking for all keys:
C - all naturals
F - Bb
Bb - Bb Eb
Eb - Bb Eb Ab
Ab - all flats except F C G (think of the key of A but flatted)
Db - all flats except F C (think of the key of D but flatted)
[Gb - all flats except F (think of the key of G but flatted)]
G - F#
D - F# C#
A - F# C# G#
E - all sharps except B E A (think of the key of Eb but sharped)
B - all sharps except B E (think of the key of Bb but sharped)
[F# - all sharps except B (think of the key of F but sharped)]
This is not entirely my idea, but the concept came from the phenomenal bassist, Victor Wooten.