MIT Webscraper

2026-08-15

When I reached out to MIT they told me their personal blog site had 7,270 blog posts. I know now that they have exactly 9,098, as well as every author, classifications of each article, and enough statistics for a blog post. (Hey, I should make one of those!)

This journey started about a month ago. I was researching MIT admissions on their blog site. MIT themselves constantly recommend their blog. This site contains some of the best admissions advice that I can find anywhere, but the vast majority of recent articles seem to not contain any advice at all.

The most recent article on the site is this:

towards the end of october, back in 2024, i was walking back to my dorm and i was about to make the turn from the asphalted footpath that went north from the music building (under construction then) onto vassar st., my thoughts still swimming above me in the elation of starry nights and tall roofs and vents to be crawled over and squeezed through, when i realized that i was halfway through the semester.

it was 3 am then, crickets humming in the grass, and i knew then that i was one-sixteenth of the way through my time at MIT.

but that was a good thing then, because one-sixteenth of my time at MIT had taken a long while. i knew then that i had fifteen more copies of what i had experienced so far,

fifteen times more psets and morning exams in walker memorial and evening square dance with warm bundt cake resting on the table and linguistics essays and midday bluebike errands to chinatown and auroras and books read on the grass at boston commons and leaves along vassar st. turning to the color of the sunset.

i knew that i had fifteen times more of all this.

but now? now i am halfway through. and already i do not have many of those fifteen left.

Where is the admissions advice? For a site that is about MIT admissions this is a pretty poor article.

This is common as well. Looking through the first pages of the site, you can easily see that most of the articles on this site are not about admissions at all. They are blog posts for students already in MIT.

No matter how much I claim that all the blog posts are not admissions focused, I cannot prove it. According to MIT there are more than 7,000 articles. I cannot look at all of them!

MIT's Response

Email response from MIT from 7/30/26:

There have been over 7,270 blog entries posted to MITAdmissions.org. They're all worth reading, but there are a few you really need to read. These are the blog entries that cut deep into our admissions process. Not only what we're looking for, but what we want from you, and how much we care about the work that we do. These blogs will help you "get" what the MIT admissions process is all about.

They include, in no particular order:

It's More Than A Job by Ben Jones
Applying Sideways by Chris Peterson
Bleary-Eyed by Mikey Yang
The Room(s) Where It Happens by Jessica Ch'ng
There Is No Formula by Ben Jones

(BTW, those articles listed are some of the only articles that I have found, and people have told me about, that talk about admissions.)

This is also a pretty old statistic as I now know there are exactly 9,098 articles on that site.

Building the Scraper

When I was in 7th grade I made a "search engine." To be honest, it kind of sucked and did not work well, but it did teach me how to make a link scraper and scrape websites in Python.

A few weeks beforehand I also made my first thing using AI in Python with Ollama. This gave me an idea: what if I find all the blog article posts, and then use AI to classify them?

That is why I am now writing this blog post here and now.

Starting this project I knew that the challenge would not be to actually do the scraping and analyzing. That is not difficult to make. The challenge would be speed.

Since I am writing this in Python, it was going to be a challenge to make it work quickly and efficiently. Writing this in a language like Go would have been a good idea, but I had no idea how to make a link scraper in it!

Scraping an Article

To start this project out I made a simple program that would go to a blog post, get all of the text, title, date released, etc.

I was worried that getting this data would be difficult, but unironically, they literally have a single div for all of the text in the article and a single div for the header that contains the article name, author and date released.

IT'S LIKE THEY WANTED ME TO SCRAPE THE SITE.

The names of these divs were article__body and page-topper_tittle respectively. THIS MADE THINGS EASY BEYOND BELIEF.

I started out with the article Applying Sideways , which is easily the most referenced article about MIT admissions. It is a good article, but I do not 100% believe it. (That is a story for another day.)

Once I scraped the first article, it should have been obvious: it was a big article. I do not actually need the entire article to classify it.

The beginning of the article sets the scene, and the ending is the conclusion, just as your 5th grade teacher told you. The subject of an article is often in the first or last sentence.

Because of this I decided to cut out the middle 40% of the article to save tokens. I chose this because I did not want to cut articles too short if they were short articles, but also wanted to cut out a decent chunk.

Once I had gotten all the data from an article, I would save it to a JSON file for further analysis.

Link Scraping

The biggest problem with this current system is that I only had one article to analyze, and I had thousands more articles that I did not have the links to!

Link scraper time!

The structure of this scraper is pretty simple. The idea is that I have a list of links that I need to visit. As soon as I visit a link, it is removed from that list.

I check if that link contains the prefix: https://mitadmissions.org/blogs/

This was to make sure that I only scraped the MIT website, not any other sites. As soon as I let the bot escape from this site I would be scraping billions of sites. My poor laptop cannot handle that.

If the link contained that prefix I would check all the links on that site using Beautiful Soup. Great library, by the way. Every time I use it I have zero complaints. It works first try and makes scrapers really easy to make.

Then I check each of those links to make sure they are actually full links.

When doing these projects I will often find links like /applying-sideways, #body, and other "fragment" links.

These links work fine as long as you are already on a webpage and are really convenient if you are trying to navigate a page, but for bots those are your worst nightmare because they will just waste time and not give you any more information.

I used a handy library called URL parse for this.

If a link does not fail any of the checks it is saved to the list of URLs to visit. Once a site is finished I save it to a set of URLs. I continue this loop until there are no more URLs to visit in my list.

I then trim all of the URLs. Even though all my sites are in that subdirectory, I only want blog posts, and luckily for me all blog posts start with this:

https://mitadmissions.org/blogs/entry/

I swear to god that MIT is begging for people to scrape them. I mean, it is MIT, so it might be on purpose.

AI Classification

Once I have these links I send them to the scraper, and it gives me a JSON file of all the articles, dates released and authors.

By the way, the length of the JSON file in lines was 45,492. I am so glad I coded this in Neovim. VS Code would have died by line 500!

I wanted these so that I could do some interesting statistics, like what articles before a certain date are like.

Then the interesting thing happens: it is shipped off to an Ollama model.

This is not a particularly interesting piece of code. All I do is give it some categories to organize into:

Student Admissions
Student Blog Posts
Student Stories
Memes and Shitposts
Other

Then I send it off to the AI and save the results in a JSON file while it is running.

This is done so that if for some reason the program stops, it can start up again and not lose several hours of inference.

Then, when it is all done, I put it in Postgres.

Now for the Results!

Student Admissions

2,089 (23.0%)

Student Blog Posts

6,510 (71.6%)

Student Stories

9 (0.1%)

Memes and Shitposts

81 (0.9%)

Other

409 (4.5%)

Total

9,098

Wow... I was really, really wrong about the Student Admissions.

One thing that I was worried about is the accuracy of the AI. I checked the first 10 articles that the AI classified, and I give it an 8/10.

Ed. Note: I realize that this was not the most thorough analyzation of the data. I am thinking about revisiting this, but I was NOT in the mood for looking through random articles for an hour.

At face value, this looks pretty poor. The two articles that failed were:

One about what a student did that year and what they learned, classified as Student Admissions: Learning by Doing .
One about the process to apply to be a blogger that year, classified as a Student Blog Post: Be a Blogger 2026 .

Fun fact: the person who wrote the second article was the same guy who wrote Applying Sideways, one of the best articles on that site ever made!

The first one is just plain wrong, but it is not ridiculously wrong. I would not classify it that way. That said, it has some reasonable interpretation to classify it that way.

The second article did talk about the man's personal life, and the majority of the part that was talking about being a blogger was in the middle 40%, which makes sense why it was classified wrong.

It would probably be a good idea to rerun it with modifications, but I really do not want to wait another 10 hours for it to finish.

Author Statistics

The biggest shock I got when looking through the data was this:

Author	Articles	Percentage
Matt McGann '00	695	7.64%
Chris Peterson SM '13	436	4.79%
Waly N. '24	244	2.68%
Sam M. '07	239	2.63%
Snively '11	233	2.56%

The first digit is how many articles those people have written, and the percentage is the percentage of all articles written they have done.

The top five authors have written over 20% of all the articles on the site.

There are only 182 authors, almost one for each country! With 9,098 articles we should expect around 50 articles per person.

13.74% of all authors on the site have written less than 5 articles, and 12.09% of authors have written more than 100 articles. The median amount of articles written is 25. The trimmed mean after removing the top and bottom 15% is 32.2151 articles.

Over a quarter of all the authors are at the extreme ends of the spectrum, either writing very little or a lot of articles.

These statistics are weird, but not out of the norm. Chris Peterson is an admissions officer, and Snively is very active online, even being a moderator on a subreddit called r/MITadmissions.

Are the Top Authors Actually Writing About Admissions?

The top two authors are admissions officers. The rest being students, you would think that the majority of their posts would be about admissions, right?

Nope.

Classification	Articles	Percentage
Student Blog Posts	1,892	65.99%
Student Admissions	708	24.69%
Other	239	8.34%
Memes and Shitposts	27	0.94%
Student Stories	1	0.03%

The top ten authors wrote 24.69% of their articles about admissions instead of the overall 23%.

OK, maybe having students in the top was bringing the average down.

Top 2 authors' percentage about admissions: 35.09%.

THAT LOW?!?!?

From actual admissions officers, only one-third of their articles are about admissions.

I checked articles from one author and I can verify those stats are actually good.

Has the Blog Changed Over Time?

OK, OK. Maybe the articles just got less and less admissions focused as time went on. So maybe in the beginning of the blog posts there were a lot more articles about admissions, right?

start_year | end_year | admissions_articles | total_articles | admissions_percentage
------------+----------+---------------------+----------------+-----------------------
2005 | 2007 | 470 | 2020 | 23.27

Why does this happen?

For a site that is cited as being about applying and admissions into MIT, does it look like the majority of the articles are not about that at all?

Before I began this project, a person told me that this is on purpose. Part of the reason why this happens is to show what the culture at MIT is like.

They want to show what students at MIT are like and who they are. By having a place where all the students can make posts is an effective way to do this.

The only problem is that the forms are heavily moderated. To just make a post you have to be accepted as a blogger and fill out an application, and you then, as far as I know, go through an editor before it is posted.

With a rough total of 259,952 students at 11,900 students a year, only 182 of them have become authors on these blog posts. Slightly less, as some people on the blogs are not MIT students.

0.070013% of students have become a blogger. Which means that roughly 8 students a year become bloggers. The culture of MIT is propagated by these few.

Conclusion

I think that this blog post has been shifting since it started. The amount of admissions articles was slightly higher in the past, and so it makes sense for a blog post that is 22 years old for it to have shifted values.

I am a little frustrated about this, but I am not infuriated about it.

As I have been writing this I have read more and more articles and I have gotten an appreciation for these blog posts a lot more. Even if they are not all focused about admissions, I still enjoy reading the articles.

A lot of them, while not directly about admissions, are good reads and would be valuable to any future applicant.

Thank you so much for reading this article!

It took me a while to write it, so if you got to it from Hacker News please leave a comment :)

And for those who found it elsewhere, I promise a comment system on the blog site itself is in the works!