Google Scholar and Proprietary Science
I’ve used Google Scholar for quite a while, and most mathematicians I know use it too. Today, I discovered something rather interesting:


robots.txtTo be clear, I don’t necessarily blame Google for this, what with how LLM companies have scraped the internet. However, I feel as if I, and my community, have a duty to make my materials as widely available as possible. It should not be the case that my work is missed in literature searches because discoverability tools are blocked on one service vs. another. Indeed, I came to realize the above by using Claude for literature search and finding that it missed a fundamental author. I was quite surprised since, up till this point, I had found (or, perhaps, dreamed) that LLMs were an excellent tool for literature search.
What is the most discoverable way to host my research? That’s a dangerous question, let’s reframe it… How can I ensure my research remains easily accessible in today’s increasingly closed web? I have my publications hosted on my website, but is my website discoverable / indexed by all tools? Indeed, I doubt they’re able to use Google search either. ORCID? Semantic Scholar? Straight ArXiv? How are these supposed to avoid getting blasted by queries the same way, I assume, Google Scholar has?