Hilton said he would "continue to be a critic" and hoped that her fans would stop sending him death threats.
Gaga's fans continued to attack the blogger, which was made clear when he constantly retweeted the sometimes vulgar messages. One carried the hashtag #OperationKillThePig, others told him to kill himself and still more threatened to kidnap his young son.
"The suggestion that I pose a danger to Lady GaGa is preposterous," he said. "If her fans want to send a message of support to Lady GaGa, they should buy her music. Notwithstanding my feelings about our personal relationship, I do wish Lady GaGa happiness and continued success and I assure you my family house hunting has nothing to do with her."
Hilton returned to Twitter on Monday, reposting a link to his statement but also sharing his current state of mind.
"I can't sleep. I am literally sick to my stomach. Not playing any ... games here. I am afraid for my ... safety and that of my son," he wrote, adding, "I know some people are sick, and addiction will make you act up, but people need to take accountability for their actions!!!!"
"I know it's hard to see our idols/friends/people we looked up to as less than awesome. But sometimes those people are so hurtful!" he added.
Gaga dropped the video accompanying "Applause" on "Good Morning America" on Monday amid the controversy. And despite all the threats he had been receiving, celebrity chatterbox Hilton admitted that the video was "visually very compelling" but still thought it was "sadly much better than the song."
Once a critic, always a critic.
Follow Ministry of Gossip @LATcelebs.