The production of ‘Premium’ olive oil depends in a large part on the quality of the fruit. Small producers see themselves confronted with vast investments and logistic snags when they intend to optimize the harvesting. Today, manual harvesting devices promise less damaged fruit when compared to the traditional methods with nets while the use of a cooling room on the farm is suggested as a solution when the harvesting needs to be stretched out over several days. The use of a manual inverted umbrella during the harvest, together with a storage up to 14 days at 5 °C at a family farm, was studied for three cultivars, ‘Arbequina’, ‘Picual’, and ‘Verdial’. Ten quality parameters of the produced oil were examined in two consecutive years together with an extended sensory analysis in the first year. The results underline the importance of the used harvesting and conservation method on the quality of the extracted oil. The strength of each factor varied in time and according to the cultivar. The ‘Arbequina’ c.v. showed a rapid increase in the importance of the conservation factor, while ‘Picual’ c.v. was the most resistant to deterioration, presenting a lower explanatory value of the conservation factor as compared to the harvesting one. The results indicate that small producers with financial and logistic restrictions can obtain a high-quality product. Either by combining both methods or by choosing the one that guarantees the best results given the cultivar and the specific storage time they need to consider.