BJP Rajya Sabha MP Subramnaian Swamy on Thursday said the government cannot remove CBI Director Alok Verma on the basis of the Central Vigilance Commission (CVC) report without hearing him and felt if he is removed the problem would only "get worse, not better".