Refactoring is not a Backlog Item.
Occasionally I see refactoring items on the backlog. This is very strange to me because it means you are externalizing development details to stakeholders. Stakeholders don't care about these things.
What then happens is the negotiating with stakeholders which is even weirder. Stakeholders are looking at you with no clue what you are talking about. They need to approve something that's gibberish to them.
Refactoring should be part of every Backlog Item. Just like every backlog item has analysis, implementation, and testing: it also has refactoring.
Refactoring should be your daily work. Don't ask permission to be a professional.