People can't bully you for not being invested in the story if you don't tell people that you are not invested in the story. The people who have this problem are the people who go into chats where people like the story and talk about skipping or finding it boring. What do they expect? That itself doesn't usually get much response but when they continue pressing it they sometimes provoke a discussion that makes them unhappy.
When someone new talks about skipping, I do explain to them the importance of the story. It gives context to the fights and makes them better. It's not just a dragon, it's ______. It's not just a random npc, it's ______. It builds excitement for the next patch when you see _____ person in a trailer. But you wouldn't experience this excitement without knowing the story and I find many people who skip are not excited for the next patch the way people who followed the story are. The main purpose more than anything is to increase your excitement and enjoyment of the content you are doing.
I explain that it's a story-driven game where the story inspires the music, the mechanics, the bosses, the locations and the effects and there is no content without a story in this game, so it gets very boring just skipping constantly or trying to rush instead of soaking it in.
If, after explaining this, they still want to skip, that's fine, but it's important that they understand these things because there are people who skip who genuinely don't know that it's such a story-driven game, that the story impacts boss mechanics or that the story is so connected that even level 90 quests refer to lore from level 1-50 quests. I think that is why people try to explain it, because they don't want someone who skips to be unaware of what they are doing.
But again, if you do skip, the way to avoid feeling harassed is simply to not bring up the story.