Harry's grandparents both maternal and paternal are dead. He has literally no other living relatives other than the Dursley's. If Lilys parents were alive that would be wonderful and Dumbledore would no doubt place Harry with them. If James's parens we're alive sorry Harry still has to go to the Dursley's.
However, since they were “old in Wizarding terms”, Rowling says they succumbed to a Wizarding illness which meant they were never part of Harry's life.
In an interview, Rowling confirmed that all four of Harry's Grandparents had died, which is why Harry couldn't have stayed with them after his own parents' deaths.
Fleamont's and Euphemia's deaths also meant that they did not meet their grandson, Harry James Potter, who would be born on 31 July 1980, nor did they realise tragedy would befall the Potter family soon thereafter.
Advancing in age, both Euphemia and her husband Fleamont succumbed to Dragon Pox shortly after the marriage of their son James to Lily Evans, in 1978–1979, dying within days of each other.
Since Harry inherited the Cloak indirectly from his father, many have wondered how James Potter came to possess it. James inherited the Cloak from his father, Fleamont Potter, who inherited it as a descendant of Iolanthe Peverell — a granddaughter of Ignotus Peverell.
As a single werewolf, it is understandable that Remus could not take care of Harry after his parents died. It's also possible he was not informed where Harry was placed and thus couldn't write to him in his early years at the Dursleys.
However, they all learned the truth in 1994, when Pettigrew was exposed by Sirius Black and Remus Lupin; "Scabbers" was, in fact, Peter Pettigrew, the wizard who betrayed James and Lily Potter to Lord Voldemort.
Several years later, however, Hermione was forced to alter her parents' memories and give them new identities as Wendell and Monica Wilkins, to protect them from the Death Eaters. After the Second Wizarding War ended, Hermione found Mr Granger and his wife in Australia and returned their memories.
Mr Evans and his wife died "normal Muggle death[s]" sometime before the murder of their daughter Lily and son-in-law James Potter, on the night of 31 October, 1981. This meant that their grandson, Harry Potter was left in the care of Petunia, his only surviving blood relation.
Theory 3 - The Baby Voldemort Was The Soul Fragment Left In The Dark Lord's Body. When Voldemort created his Horcruxes before the events of Harry Potter, he was left with a weak and vulnerable soul. He was the first in the wizarding world to make multiple Horcruxes, and he knew this was a risk.
Taking Harry in Wasn't Much of a Choice for the Dursleys
He informed Petunia and Vernon Dursley that the only thing keeping Harry safe was the magic of Lily's motherly love, and since Petunia was her only living blood relative, she was the baby's last hope to grow up safely.
This idea was incredibly embarrassing to him since Ron vehemently did not want to be cared for by his rich and famous best friend. This is precisely why Harry never tried to give the Weasleys money or give Ron lavish gifts - they didn't want pity.
Harry survived because Voldemort took his blood in GoF. His blood contains Lily's Sacrificial Protection. As some of it lived on, in Voldemort, Harry was tethered to life. It's like a light-side Horcrux, using someone else's love rather than your own soul.
Notably, Voldemort appeared to be entirely unaware of this split, as he later created a seventh Horcrux in his quest to have six, and continued to target Harry Potter despite him now holding part of Voldemort's own soul.
By the 1990s, most of the magical population were half-bloods. The children of Harry and Ginny Potter were considered half-bloods because although Ginny was pure-blood, Harry's mother was Muggle-born. The term was used as an insult from members of fanatical pure-blood families.
They were mean because they were a reminder to Vernon and Petunia of his powers and it reminded Aunt Petunia of the fact that a wizard killed her sister furthering her hatred for magic. Petunia hates Harry because she was jealous of her sister Lily's magical abilities and he reminded her of Lily.
Why didn't Remus ever get in touch with Harry before Prisoner of Azkaban?
Because he didn't exist before Prisoner of Azkaban. Perhaps because he didn't feel it would be appropaite to do so. A lot of people like to label Remus as Harry's uncle just because he was one of James's best friends but this does not automatically make one an uncle or aunt.
He believed that by marrying Tonks he had made her an outcast among society, because of his lycanthropy, as well as feared that he would pass on his affliction to their child.
The prolonged misery of being exposed to dementors kept the prisoners locked within their own minds, so Sirius lost the ability to transform. However, Sirius explained that the knowledge of his innocence kept him sane in a way most of Azkaban's prisoners were not.
As Ilovecats13 and DelphianDragon said, James' father Fleamont quadrupled the family's gold by inventing Sleekeazy's Hair Potion (the same that Hermione used before the Yule Ball to tame her mane). The family fortune was passed down to James, their only child, and then onto Harry. Rowling wrote about it on Pottermore.
In the Harry Potter and the Deathly Hallows movie, Snape showed how much he loved Harry, providing him with his memories and showing that he saw Lily in him, his last words echoing this: "You have your mother's eyes."
It is known that Harry is a descendant of Ignotus Peverell (The brother that received the Invisibility Cloak), and Voldemort is descended from Cadmus Peverell (The brother that received the Resurrection Stone).
