AI Hype Analysis
Introduction
I visit Hacker News frequently enough that I just type in the letter "n" and press enter to navigate to it. There's no other site that Google Chrome would even consider suggesting instead. It's mostly just a time suck, but I get a little bit out of it.
One benefit to being a regular on Hacker News is that I'm informed about the latest trends. I'm almost never caught unaware about something new. In fact, usually I see stuff on Hacker News a few days before it starts appearing more generally. That got me thinking: the HN front page is probably a decent indicator of the current impact of a technology.
The subject I want to evaluate is whether "AI" was increasing or decreasing in popularity. To me, it felt like the popularity was dying down.
Data Collection
My approach was to scrape the thirty front page stories for every day since January 1, 2020. I think there are datasets available for Hacker News, but I just scraped it myself. It's easy to pull the front page for a specific day by using the ?day=<year>-<month>-<day> query parameter. My scraper and dataset are now on GitHub, and I set it up with a Github Action to keep it up-to-date automatically.
Analysis
To do the analysis, I did some finagling in Google Sheets. The approach is dead simple. I simply created a list of words that are related to "AI", and then counted how many articles contained them.
The data speaks for itself. "AI" is a more popular subject than ever. This result surprised me.
