This issue has been there for a long, long time. Just do the jump back like the rest of us and that’s it. If it was easy (or possible) to fix, I bet it would already have been done.
It seems to come and go between systems, and the issue is discussed here and there I’m afraid. I’ve had it quite rarely on the S922X box but much more frequently on the S905X4 one.
There’s the nice “Unpause Jumpback” addon which helps with the case when jerkiness starts after pausing and resuming.
There’s sometimes also a/v sync issues after pausing and they are resolved by jumping back so there’s probably something in the Amlogic side that is causing it and making things hard or impossible to fix.