It is speculated that Gaaitjie was built around 1855 by the fishery firm, serving as a home for a local family. The old house was built right on the rocks on the Paternoster beachfront, boasting a foundation of solid stone. The foreman of the Paternoster fishery, Mr Sampson use to live in the historic building. Hence the road leading to Gaaitjie is named after him. The building housing Gaaitjie Restaurant is also a Cape Heritage building and still belongs to the fisheries
By 1902 a full-blown lobster industry was in operation, canning and exporting lobster to France in particular.