Removing contacts

We have a situation where in multiple sheets we have columns that are assigned to contacts. When a contact leaves we remove them from the licensed / unlicensed users and also from the My Smartsheet Contacts lists.

However, in sheets where that contact has been assigned a task in the past, when you start to type a name their contact details come up as an option to choose.

In Column Properties, the Column Type is Contact List and there are no preferred contacts listed.

Does anyone know if there is a way to remove them from the sheets or what contact list the Column is referencing?