Indeed it is gradual because it took us time to get used to the quiet nights and just like in your case, the situation was never any better with our children visiting frequently. Whenever they left, we would be back to square one (again).
As much as I do not regret the course that this journey has taken, I do wish that I had easily figured out when my children needed some space to explore and get used to the newly-found 'freedom'. When they didn't (immediately) pick calls, I at times took it to mean that I was being ignored. Had I known better, it would have been an easier transition, without doubt.