From Muppet Wiki

The Castle of the Crystal was once the home of the UrSkeks, until their division into the Mystics and the Skeksis.

Once the Pure Crystal was cracked and became the Dark Crystal, the Mystics fled to their valley, and the Skeksis took rule of the Castle. Over time, the Castle became dark and foreboding, looking as evil as its rulers truly were. It is here that the Gelflings Jen and Kira healed the Dark Crystal and reunited the two species into one. When the Crystal was healed, the Castle shed its dark covering and revealed its true visage.