They live on a farm and Elmo and his sister Daisy visited them in the 1990 Sesame Street book Elmo Gets Homesick. The couple returned the favor 18 years later, visiting Sesame Street in the 2008 book What's the Word on the Street?
In 2014, Elmo's grandparents made their first on-screen appearance as Muppets. In episode 4417, Elmo's Grandpa and Grandma participated in a video-chat with Elmo and his friends on Sesame Street.
It is never stated if these are Elmo's paternal or maternal grandparents, however Elmo's Grandpa bears a strong resemblance to Louie, Elmo's dad, and both of their red fur imply that they may be the parents Elmo's dad (as opposed to being the parents of his mom). This would make them the parents of Louie and his brother Jack; and thus the grandparents of both Elmo and Daisy, and their cousin Jesse.
In the 1989 story "Is It Time for Bed Yet?", Elmo is excited for bedtime, because the next morning he has a fishing trip planned with his grandpa.