Holy Grandfathers

When Josef pray, he prays with the name of Azik & Ebereham who are his fathers and people follow him and no one complains but when I pray with my grandfathers names they antagonize me and say that I am worshiping idol. How & when has my grandfather become an idol?