Regulatory mechanisms associated with repeat-rich sequences and chromosomal conformations in mature neurons remain unexplored. Here, we map cell-type specific chromatin domain organization in adult mouse cerebral cortex and report strong enrichment of Endogenous Retrovirus 2 (ERV2) repeat sequences in the neuron-specific heterochromatic B2NeuN+ megabase-scaling subcompartment. Single molecule long-read sequencing and comparative Hi-C chromosomal contact mapping in wild-derived SPRET/EiJ (Mus spretus) and laboratory inbred C57BL/6J (Mus musculus) reveal neuronal reconfigurations tracking recent ERV2 expansions in the murine germline, with significantly higher B2NeuN+ contact frequencies at sites with ongoing insertions in Mus musculus. Neuronal ablation of the retrotransposon silencer Kmt1e/Setdb1 triggers B2NeuN+ disintegration and rewiring with open chromatin domains enriched for cellular stress response genes, along with severe neuroinflammation and proviral assembly with infiltration of dendrites. We conclude that neuronal megabase-scale chromosomal architectures include an evolutionarily adaptive heterochromatic organization which, upon perturbation, results in transcriptional dysregulation and unleashes ERV2 proviruses with strong neuronal tropism.