You make a very important point. I would say, however, that our interpretation of what we read equally is important. People read the Bible and some like C.S. Lewis become Christians others become atheists or agnostics etc. The worldview we bring to our reading then perhaps is as or more important than what we read.
That being said the underlying thesis of what we read does matter. I have had to put a novel —” through the looking glass ”— down and stop reading because I found the underlying narcissism of an extreme sort and repulsive. When we put down a reading in such situations it is because we would rather feed our minds with something more noble, after all we do have a limited time for all of the reading we seek to accomplish in our lifetimes.