Here's a neat little essay by Desmond Reddick, entitled "The Unfilmable: What is Lovecraftian Cinema?"
Basically, the article takes stock of what made H. P. Lovecraft's literary work special and then examines which, if any, films have managed to catpure that spirit.
Reddick sums up the source of horror in Lovecraft's fiction as "cosmic dread," claiming that the author paints a nihilistic picture of the universe, wherein ignorance is bliss, where characters who learn the truth are driven to madness and despair.
That's not the kind of approach that would appeal to Hollywood, which more often than not wants to make feel-good movies. Consequently, most filmmakers have raided Lovecraft for titles, locations, and plot elements, while steering clear of the underlying philosophy.
Not surprisingly, Reddick finds most of the official adaptations, such as RE-ANIMATOR and THE HAUNTED PALACE (which was actually based on Lovecraft's "The Case of Charles Dexter Ward"), more entertaining as films than as adaptations.
Reddick reserves most of his praise for IN THE MOUTH OF MADNESS, directed by John Carpenter. Although the film is not based on a specific Lovecraft story, it uses many Lovecraftian elements to good effect. Unfortunately, Reddick also lumps in two other Carpenter films, PRINCE OF DARKNESS and THE THING, both of which are good movies but neither of which has much specifically in common with Lovecraft.
Nevertheless, this is an interesting essay, worth reading for any Lovecraft fan.
I will add just one more thing, regarding the pessimism of Lovecraft's work:
It is true that Lovecraft painted a dark picture of the universe in the horror stories he crafted during his most prodigious period. However, it is equally true that Lovecraft was not really a pessimist who thought that the universe was a dark and hopeless place. His later writing moves away from pure horror toward science-fiction, wherein the wonder of the universe replaces the fear of the unknonw. In fact, by the time of "Through the Gates of the Silver Key," Lovecraft was re-imagining his frightening "Elder Gods" and telling us that they only seemed like monsters to ignorant humans who didn't know any better. A life-long amateur astronomer, Lovecraft saw beauty in the cosmos, not despair and annihilation. Although he is best remembered for imagining unimaginable horrors, he also had a gift for creating awe and wonder.
And that, too, is something that Hollywood has never fully captured when adapting his work.