You're right that it is a problem. Now, I don't know how much of what you saw was related to caching. But, it would certainly be necessary to re-add all the role members to get the right data stored in the role-responsibility-action table. From what I remember, that flag is only used by the UI - to enforce how the data is established. Once in that table, the data continues to act per the flag's original setting (since the role member ID would be missing from the existing records.)
Would you say it's reasonable that we should make that field read-only on existing responsibilities, only allowing it to be set on new ones? If not, we could enhance the responsibility save to re-do the current action records upon a save. That has its own complications, though. Going from non-role-member to role-member assignment is easy, we simply apply the current setting to all members. However, going the other way could be tricky.
Given our current time-frame, I recommend going with the read-only option. If the other behavior is desirable (but not mandatory in your opinion), please add it as a JIRA for a future release.