First of all, it’s important to confirm that your secret phrase is backed up, either manually, digitally or both.
Then, open the xPortal app, click on “Profile” in the upper left corner and you will find the “Settings” button. Open the Privacy section, choose Phone Number and you will have the option to change or to unlink the phone number.