Revert "[hotword] fix race condition in destroy()"

This reverts commit 2c300979b59aa0acce8a815318598e874adc2245.

Reason for revert: b/312021415

Change-Id: I1f6aa0eba880131a8f9b165a55e770836d35389b
1 file changed