Four bodies were recovered during the night and early hours at the site in the southern city of George, city officials said.
Rescue teams have worked tirelessly since the five-storey apartment block, which was under construction, suddenly collapsed last Monday. Eighty-one people, mostly construction workers, were at the site at the time. Twenty-nine people have been rescued so far.
They include a man who was pulled out on Saturday after 116 hours under the rubble. "It is a miracle that we have all been hoping for," Western Cape provincial premier Alan Winde said on social networks.
Search teams have vowed to comb every cavity among the wrecked building, but the chances of survival are fading. Authorities said on Monday they were implementing a plan to "expedite the identification of the deceased".
Relatives of the victims have expressed frustration over the pace of the identification process. The reason for the collapse of the building, which was to include 42 apartments and had received planning permission, has not been established, and an inquiry is under way.