Yes, the contact will be able to complete the survey regardless of whether or not they are subscribed to a list.
If you deploy by link, the survey you create will be hosted publicly, and anyone that knows the URL to the survey will be able to complete it. The contacts to which you choose to distribute the URL are your choice, but anyone on the Internet could possibly come to your survey and fill it out. In that case, deleting a contact from your account will make no difference.
If you deploy a survey by email, we create an email for you that you will send to one or more of your contact lists. Anyone on those lists at the time the email is sent will be able to fill out the survey. If you send the email to one contact, then you delete that contact from your account, they will still be able to fill out your survey.
Each person that fills out your survey is required to enter their email address. If you need to disregard the answers of a particular person, you can remove their answers when you download the survey results.