Under Settings, expand the option related to the phone number you would like to delete. Click on the settings dropdown-cog (top right), and then select the Delete Phone Number
option. This option will only be available for phone numbers not managed in Switchboard.
If you would like to delete a number currently managed in Switchboard, you will need to first stop managing the number in Switchboard and then the delete option will be available to you.
If you made a mistake in deleting a phone number, you will need to open up a support ticket with Twilio to see if it is possible to restore the number for you. This number will probably not be available to you after deletion so be sure before going down this path.