High intra-specific genetic diversity is associated with adaptive potential, which is key for resilience to global change. However, high variation may also support deleterious alleles through genetic load, thereby increasing the risk of inbreeding depression if population sizes decrease. Purging of deleterious variation has been demonstrated in some threatened species. However, less is known about the costs of declines and inbreeding in species with large population sizes and high genetic diversity even though this encompasses many species globally that are expected to undergo population declines. Caribou is a species of ecological and cultural significance in North America with a wide distribution supporting extensive phenotypic variation but with some populations undergoing significant declines resulting in their at-risk status in Canada. We assessed intra-specific genetic variation, adaptive divergence, inbreeding, and genetic load across populations with different demographic histories using an annotated chromosome-scale reference genome and 66 whole-genome sequences. We found high genetic diversity and nine phylogenomic lineages across the continent with adaptive diversification of genes, but also high genetic load among lineages. We found highly divergent levels of inbreeding across individuals, including the loss of alleles by drift but not increased purging in inbred individuals, which had more homozygous deleterious alleles. We also found comparable frequencies of homozygous deleterious alleles between lineages regardless of nucleotide diversity. Thus, further inbreeding may need to be mitigated through conservation efforts. Our results highlight the ‘‘double-edged sword’’ of genetic diversity that may be representative of other species at risk affected by anthropogenic activities.