roger is his granpa not his father, luffys father is some normal guy with no speacial skill. he hasnt been shown an probobly never will.
luffy has no father nor a mother (note : he knows of)
just a brother who've been shown 4 or 5 times.
Roger isn't his grandpa, cuz he was scared as ajoki, tells he knew his grandpa. if roger was his grandpa luffy must be proud of this.