"I did feel a little bit embarrased but since it is not my fault... And the whole country knows... People who know my nature (know I am not at fault)," Singh said in an interview to PTI.
Singh said till now nobody has cast aspersions on his image but now some people had got an opportunity.
The saffron party leader said he had not backed Pragya Singh.
Rajnath Singh, who is under attack for reported remarks that Pragya is not a terrorist and that an Aarushi-like case was being built against her, denied he has not defended anyone in the wake of the investigations into the blasts.
"An organisation with established nationalist character with a record of exemplary social service can never take to the path of terror, whatever may be the provocation," Singh said.
Even Bajrang Dal leaders have not owned the people who were arrested on allegations of Malegaon blast conspiracy, he said.
If Congress was clear in its resolve to fight terror then it should have been the first party to condemn the irresponsible use of the word "Hindu terror" by some sections of politicians and the media.
"We believe firmly that terrorists do not have a religion or faith," he added.