I ride the bus to work some days. The majority of the riders are associated with the university--mostly students, but some faculty and staff (like me.) Other riders come from various urban demographics.
I often sit on the outside seat. If the rider beside me is getting off at the next stop, usually they will move to get up before the bus stops or even has started slowing down. I prefer to wait until the bus has stopped to move, so I'm not jolted back and forth. When that is the case, I'll tell the rider, "I'll move when the bus has stopped." The only drawback I can perceive is that it delays their exit of the bus by a couple seconds. Am I being rude, or just cautious?