The Nightmare Realm is a seething, intergalactic foam between dimensions where Bill Cipher resides in after destroying his home universe. He sought to escape by entering the world of Gravity Falls, Oregon in an event known as Weirdmageddon.
At least 1 trillion years before the series, Bill Cipher "liberated" his home dimension by burning it and proceeded to take over the empty Nightmare Realm as his own. He described the Nightmare Realm as "[his] own decaying dimension" in which he had "been trapped for one trillion years" looking for a means to escape, as its lack of constant physics doomed it to self-destruct in the future. When Stanley Pines finally activated the Universe portal in an attempt to save his brother Stanford, it creates an Interdimensional rift. Bill later uses this to create a gateway between Dimension 46 (referring where Gravity Falls is) and the Nightmare Realm.
In "The Last Mabelcorn," Bill appears within Stanford's dreams to haunt him, and warn him of the upcoming inevitable events. While doing so, he gives Ford a glimpse of the Nightmare Realm by shattering the rift. Later, Bill, who is in the Realm, watches Ford and Dipper successfully establishing a shield around the Mystery Shack and starts to choose other people to resume his plan.
In "Dipper and Mabel vs. the Future," Bill has possessed the body of Blendin Blandin in order to retrieve the rift from Mabel Pines, which he uses to create the previously prophesied gateway to the Nightmare Realm.
In "Weirdmageddon Part 1," Bill uses the newly created portal in the sky to bring forth nightmare monsters and interdimensional criminals from the Nightmare Realm.
In "Weirdmageddon 2: Escape From Reality," the Nightmare Realm is still visible through the tear in the sky above Gravity Falls created by Bill. However, the interdimensional criminals can't leave Gravity Falls, Oregon; according to Ford Pines, natural law of weirdness magnetism of the city contains Bill's weirdness, and an equation can disable the barrier.
The Nightmare Realm, from what is shown, is a somewhat surreal unstable void filled with swirling shades of dark purple, red and yellow almost like a lava lamp. In the real-life cannonical Journal 3, it says that the Realm is littered with asteroids and has a "smell of burnt hair."
The Nightmare Realm has been used as a home for some of the worst criminals of the Multiverse, including Bill and his Henchmaniacs (such as Teeth and Keyhole) however, due to it connecting to many other universes, beings from more dimensions have found themselves stuck in the Nightmare Realm with no means of escape. These include a group of asteroid miners Ford encountered after first entering the Realm.