Some 10,000 birds were released on Tuesday from Sha Tin Sports Ground to fly to their home provinces all over China, but several hundreds have refused to flutter further than a few blocks.
Officers from the Society for the Prevention of Cruelty to Animals have been spending the past two days collecting many of the saturated and hungry pigeons which have not left.
The organisation appealed last week for the release of the birds to be delayed until the bad weather had cleared.
But the Regional Services Department and the Hong Kong Racing Pigeon Association gave assurances that the birds would have no problems flying in bad weather.
Executive director Doreen Davies said the society had received dozens of phone calls from concerned residents of Wo Che Estate and nearby schools as hundreds of the birds lined their roofs in the wet.
"We asked them to delay until after the bad weather and now this is turning into the fiasco we thought it would," Ms Davies said.
"They have plenty of water up there sitting on those roofs but nothing to eat, so we are going to bring them back and get them fed and dried out."
The society had collected about 30 birds up until last night.
A department spokesman admitted it had received complaints from estate residents about the birds and officers were leaving food at the sports ground.
But she could not explain why the birds had refused to budge.
"Most of them have gone, but just like people these pigeons have their own characteristics and some don't want to go," she said.
"The association has assured us they will leave Hong Kong when better weather comes so we will keep feeding them at the stadium until then."