So how about this. Let's just agree that the two D's that we know of with the highest percent possibility of being Luffy's grandfather are Saul and Roger. Saul has the fact that he had an ordeal with Aokiji going for him, and the fact that he's a giant going against him. Gold Roger has the similar personality and looks going for him and the fact that we've nothing but circumstancial evidence going against him.
Other than those two, we have no names. So let's keep the speculation to them and simply state who you think it is and why. By doing this we can eliminate arguments.
For instance I think that it is Gold Roger, because I just don't see how a giant can produce a human grand child. But that's just me.