You don't control the truth in Production!
Now a days multiple teams/projects are working on the same Dataverse production environment.
So thinking you can manage the complete production truth in your ALM pipeline is strange.
Not every environment is đź’Ż equal to production. Accept this and things will be easier.
Don't treat production as an environment where your team controls the full truth, but as an environment that can be changed by other people or parties.
When you change this mindset, some things in ALM don't fit anymore.
When you accept change from outside, you will change your deployment. It will make your ALM simpler.
Example: Why move third-party solutions through your pipeline? Just install it manually.
When your ALM doesn't need to account for everything, it will be simpler.
When ALM is simpler, every team member can maintain it.
Do you think you control the truth in Production?