Was this "retention" on the sale of the original property? Your post is a little unclear. Did they sell one and buy another?
The 3% (previously 5%) retention against possible CGT is only for non-residents. If your friends are genuinely residents, this should not have been withheld. Of course, it would have been up to them to prove their resident status at the time. If they were resident and sold their principle residence, and reinvested all the proceeds in another property, they should not be liable to any CGT anyway. If they were not legally resident, then they would be liable for CGT on any profit made on the first property. If the 5% amounted to more than their liabilty, they should be able to claim back the difference. They would also need to check the dates, because the law changed (not certain when), reducing the retention to 3% and the CGT from 35% to 18%, but took time to be implemented, so some non-residents paid more than they should have, and can now claim back the difference. (theoretically!)