A scathing US Senate report earlier said the CIA carried out "brutal" interrogations of al-Qaeda suspects.
The report also concluded the agency misled politicians and the public about the 2001-2007 programme for terror suspects.
The CIA has defended its actions in the years after the 9/11 attacks on the US, saying they saved lives.
'Beaten and drugged'
Hicks shouted at Mr Brandis after the attorney general finished speaking at a human rights awards gathering in Sydney on Wednesday.
"Hey, my name is David Hicks!" he said. "I was tortured for five-and-a-half years in Guantanamo Bay in the full knowledge of your party. What do you have to say?"
Mr Brandis, who was a member of PM John Howard's government when Hicks was at the US military prison in Cuba, walked off stage without answering.
"He's run away," Hicks said, describing the minister as a "coward".
Mr Brandis later described Hicks as a "terrorist".
"David Hicks, both in his book and also in his plea bargain document, confesses to acts which under Australian law... we define as terrorism offences", he said at a Senate committee hearing, according to Australia's ABC broadcaster.
He said he never felt his personal safety was threatened, although one senator raised concerns over security arrangements at the gathering.
Hicks - who is widely known as "the Australian Taliban" in Australia's media - was seized in Afghanistan in 2001 on suspicion of fighting alongside Taliban militants and sent to Guantanamo Bay the following year.
In 2007, he was transferred to a jail in Australia after being convicted of providing material support for terrorism.
Hicks later retracted his confession, claiming it was obtained under duress.
