Recently, Trey of From the Sorcerer's Skull proposed a worldbuilding concept that interested me. How would it change the game to replace the species available to the player characters?
Trey first proposed imaging a fantasy setting without elves, dwarves, or hobbits, but featuring the species from TSR's Star Frontiers setting - the insect-like Vrusk, amoeba-like Dralasite, and flying-monkey-like Yazirians.
Later, he suggested a pulp scifi campaign using the same species, set entirely within the Solar System. (Trey has a bit of a cottage industry reimagining interstellar scifi as 1930s-style pulp confined to just our one star system. In addition to the Solar Frontiers I just mentioned, he's briefly written about Solar Wars, and he has a whole series of posts about Solar Trek.)
I want to build on his original suggestion though, and try some worldbuilding that stays relatively close to D&D's world, populated with Tolkien's heroes and monsters, just with, you know, different heroes, and different monsters.
We know that constraint encourages creativity, and I think that's especially true of worldbuilding. By limiting yourself to a small number of component parts, you can create a compact, thematic setting that's uniquely your own. Look at the 10 Monster Setting, or the 7 Class Setting, or the New New Crobuzon Challenge - they're all worldbuilding prompts that encourage you to select a small number of components and use them as the basis for a fantasy world.
If your handful of seedlings are particularly weird, so will be the garden that grows from them. The downside of that is that you can end up with a setting that's so weird that it doesn't really make sense to other people, and maybe not even to yourself. And obviously, actually running a campaign like this requires securing enough player buy-in beforehand, since their options for possible characters are radically restricted compared to a more open setting.
(Alternately, if you wanted, you could spend your Session Zero letting each player pick out their bespoke species and occupation combo, and then working together to imagine what sort of world it must be if those particular sorts of characters are a totally typical, run-of-the-mill adventuring team.)
A variation on this kind of prompt would be to try to make your own "French Vanilla" setting. By that, I mean a setting that mostly draws on the tropes of vanilla fantasy, but is specific enough to be uniquely your own, the kind of thing Trey writes about here. And as both Trey and Jack from Tales of the Grotesque & Dungeonesque have argued, vanilla fantasy is, almost by definition, something that is understood by almost all potential players, something that essentially generates its own buy-in because its expectations are so clear. Done well, french vanilla might give you the best of both worlds - enough familiarity to form a solid basis for a shared fantasy gameworld, but enough difference to make something that's personal and interesting.
(I had some success with this in a campaign where I offered my players a handful of classes - human fighters, thieves, and wizards, plus "venturers" who are basically adventuring capitalists, or like, white collar thieves; elven druids and "courtiers", another variety of legit thieves; and dwarven roboticists and tomb robbers, who are, you know, basically more thieves. My players seemed content with the assortment, and I felt like I had established a strong sense of what sort of people go adventuring in that setting.)
What I've called the "Non-Core" is one source for a french vanilla setting. The non-core is made up the kinds of ideas that don't usually make it into the core rules of most fantasy games, but do usually appear in the first batch of expanded content.
Imagine the sort of setting you end up with if you replace elves, dwarves, and hobbits with drow, duergar, and svirfneblin? Instantly you have a darker and spookier campaign setting, and one will make extensive use of the Underdark as a location.
Or what if you used only the species that got added when OD&D expanded to become AD&D? What if you had only half-elves, half-orcs, and gnomes? Unlike the "good fey" of a straight Tolkienian game, this settings imply that humans deal with the Unseelie Court as often as the Seelie, and that full-blooded elves are, in their own way, as monstrous as orcs, and both are ineligible as player characters.
Restrict yourself to just "new school" creature types like tieflings, dragonborn, and goblins, and while Old School players might grumble about the loss of elves, the younger generation of Critical Role and Adventure Zone fans might not even notice the restriction - or if they did, they might be more upset by the loss of aasimar and genasi than of hobbits and dwarves.
In a variation on Trey's initial idea to use the species from Star Frontiers, you could also borrow the playable species from TSR's Alternity game - the psychic Fraal, cybernetic Mechalus, bat-like Sesheyan, fringed lizard Tsa, and bestial Weren.
(One advantage of using totally alien species like these is that it lets you get away from an effect you sometimes see with the standard demi-humans, where each represents a different extreme, with humans like Goldilocks in the middle, defined by our flexibility and moderation. I suspect but I don't know, that Tolkien intended his elves to seem French and his dwarves German, one overly artistic and cultural and the other too industrial and militaristic compared to the "just right" modern middle-class British hobbits and medieval British humans. Though for all I know, old JRR could have been taking potshots at the Irish and Scottish, or maybe I'm imagining chauvinism where none exists. In other people's writing, I think I sometimes see elves, dwarves, and hobbits as representing feminine, masculine, and childlike qualities. This is fine if you want it, though it always contains some embedded assumptions about what humans are "supposed" to be like, which others might find objectionable. Scifi species potentially give you the chance to make humans just one species among several, rather than the center of Creation.)
But suppose for now that we want to stick fairly closely to the Tolkienian archetypes, but use alien species to create a kind of science fantasy french vanilla. Still unquestionably D&D, still built on familiar tropes, but with enough a difference to make this campaign feel special. For each species being replaced, I have a suggestion from regular Star Trek, from Babylon 5, and from the original Star Trek pilot episode "The Cage".
Elf Replacements - The alternate elves I've chosen here all reflect a vision of elves as "human, but better". I could write a whole post - and who knows, maybe sometime I will - about how fundamentally weird it seems to me that there's a widespread belief reflected in science fiction, everything from pre-Golden Age pulp stories to Star Wars and The Matrix suggesting that human perfection, whether it comes on a species level through evolution or individually through enlightenment, looks like an emotionless ascetic in a monochrome outfit, living in a harsh environment, spending all their time meditating for mental discipline and learning to fight real good.
Vulcans - The Vulcan homeworld is a harsh desert with habitation concentrated around the few oases. Both as individuals and a society, they used to be very passionate and emotionally-driven. At some point in their past, they adopted en masse a philosophy of logic, asceticism, and emotional self-control. The lawful Vulcans have a counterpart in the chaotic Romulans - who are the same biological species, but a different society, one that split off from the other Vulcans before the adopted their new philosophy (or perhaps as a rejection of it?) Vulcans famously have the the abilities to perform a Nerve Pinch that can stun most humanoids, and a Mind Meld that allows them to share thoughts and memories.
Minbari - Minbari society is divided into three castes - warriors, workers, and religious. They value honor and tradition, formal rituals and baroque decorations. Their society is the oldest of the current age, and they have the closest relationship with the Vorlons, the last remaining elder civilization of the previous age. They grow buildings out of crystal, and their ships incorporate biotechnology. Their warriors are skilled martial artists, while their religious caste can use crystal-based technology to produce almost magical effects.
Talosians - The remaining Talosians are the last survivors of a once great civilization. Their physical bodies are weak and somewhat frail, but their enlarged brains possess incredible psychic powers. They speak only telepathically, and routinely project illusions to disguise themselves and their allies. Talosians can read thoughts and project physical pain, but their powers can't penetrate really powerful emotions, such as rage.
I would represent Vulcans and Minbari with D&D's Monk class, which is a good match for their fighting style. I would probably pick the Cleric as a second option. Spells like Command, ESP, and Hold Person are pretty good representations of their abilities. Talosians should probably be Bards and Wizards, perhaps with an emphasis on illusions and enchantments to match their powers.
Dwarf Replacements - Although we might think of dwarves as being miners, or engineers, the replacements I've chosen all kind of tap into the "proud warrior race" archetype. Really, scifi has an embarrassment of riches for this particular trope, so you have plenty to choose from that aren't listed here. Your only limitation is probably your willingness to use a species that's been portrayed as an antagonist for your player characters, since these guys are rarely the heroes in fiction.
Klingons - The Klingons are a very proud society. They prioritize personal honor and the glory won in battle and they inflict harsh corporeal punishment for all crimes and moral transgressions, especially cowardice and other forms of perceived weakness. They consider a death during combat to be the only acceptable way to die. Klingons have distinctive ridged foreheads and fight with a two-handed, multi-pointed sword called the bat'leth.
Narn - The Narn have a reptilian appearance, hairless and mottled with spots. Although once a peaceful society, they were conquered by the Centauri and their planet occupied until they could drive the invaders out. They are now militaristic and intent on both protecting themselves and getting revenge on their enemies.
Rigelians - Okay, so technically these guys, the inhabitants of Rigel 7, are called Kalar, while it's the inhabitants of Rigel 5 who get the honor of being called Rigelians, but it's clearly the better name, and could be applied to anyone from that star system. They are notably taller than other humanoids, and prefer to fight with spear and shield. While Klingons and Narn routinely wear metal armor, Rigelian's wear thick layers of leather and fur.
All three of these species should probably receive a Fighter option, although they have different fighting styles - Klingons prefer two-handed weapons, Narn daggers, and Rigelians fight with one-handed weapons and shields, and are the only species here who wouldn't use guns. Although most Klingons vocally support honorable conflict, their ships use invisibility cloaks, they surgically alter spies to pass as human, and occasionally use poison and suicide bombs to achieve victory, even at the expense of honor. Their second option should probably be Assassin, or its equivalent. The Narn are shown to be pretty religious, so their second option should be Paladins or Clerics, even though in the show, it's a big deal that they're one of the only species to not have indigenous telepaths. Rigelians are clearly also Barbarians.
Hobbit replacements - The symbolic role of hobbits is a bit unclear to me. As I said above, I think Tolkien wrote them as being essentially like modern humans, contrasted against the old-fashioned humans of Rohan and Gondor. In most D&D settings, they don't seem to have a culture that's any different from human society, with the exception of Eberron, where they have their own little Dinotopia. For my replacements here, I've chosen species that are cast as sort of "dark mirrors" of humanity, reflecting our worst qualities back at us. Instead of modern middle-class homeowners, I've chosen capitalists, gangsters and slavers, and colonizers.
Ferengi - The Ferengi are mercantile capitalists. They don't particularly make anything, but they make money as middlemen, buying low and selling high. Their whole society is structured around commerce; their Rules of Acquisition are practically a sacred text, and their currency, gold-pressed latinum, is more-or-less the currency of universal interchange. Their leader is simply whichever Ferengi has the most money at any given time. The Ferengi have very prominent large ears, and they're immune to telepathy. Ferengi society classifies women as property, owned by their fathers and later by their husbands; by law and custom, their rights are severely restricted.
Centauri - The Centauri are a society past their prime. They once had an expansive empire, but most of their former colonies have since won independence. The hierarchy of their society is based on proximity to the hereditary emperor; status in the royal court is the primary determinant of social position. Centauri culture and fashion are still very much frozen as they were at the height of their power; anything new is almost automatically worse. Their taste is ornate and baroque. Centauri look very much like humans, although their women shave their heads bald, and their men grow fan-like crests of hair, the taller the better.
Orions - The Orion people have carved out a niche for themselves as dealers in illegal merchandise. While the Ferengi sometimes dabble in legal vice - such as gambling and holographic prostitution - Orions deal almost exclusively in contraband and outlawed services. Orion slavers kidnap members of their own species, and others, to sell into forced labor. Orion pirates raid ships and seize their goods. Orion gangsters make loans, collect debts, deal illegal drugs, and sell stolen goods they received from the pirates.
All three of these species should obviously receive Rogue as one of their class options. After that, I'm a little bit torn. Ferengi and Centauri would both probably alike another roguish archetype, but more along the lines of a Merchant or a Noble, though most rulesets don't treat those as playable classes. Bard might be a fair representation of their more flamboyant and charismatic members? As for the Orions, I think they could use a Fighter type to act as enforcers for some of their more violent crimes.