African swine fever virus (ASFV) is a major threat for porcine production that has been slowly spreading in Eastern Europe since its first appearance in the Caucasus in 2007. ASFV enters the cell by endocytosis and gains access to the cytosol to start replication from late endosomes and multivesicular bodies. Cholesterol associated with low-density lipoproteins entering the cell by endocytosis also follows a trafficking pathway similar to that of ASFV. Here we show that cholesterol plays an essential role in the establishment of infection as the virus traffics through the endocytic pathway. In contrast to the case for other DNA viruses, such as vaccinia virus or adenovirus 5, cholesterol efflux from endosomes is required for ASFV release/entry to the cytosol. Accumulation of cholesterol in endosomes impairs fusion, resulting in retention of virions inside endosomes. ASFV also remodels intracellular cholesterol by increasing its cellular uptake and redistributes free cholesterol to viral replication sites. Our analysis reveals that ASFV manipulates cholesterol dynamics to ensure an appropriate lipid flux to establish productive infection.