How it works
There are two parts to the project: collecting the raw GTFS realtime files, and creating the CSV journals.The raw data
At a high level the idea is very simple: just download the data from the MTA every 5 seconds and every calendar day collect the downloaded files into an archive file.
The problem is that this is hard to do reliably. Any computer hosting the download-and-archive program is liable to have failures or expected restarts (e.g. to update the operating system). Also if the download-and-archive program is itself modified and updated, there will be downtime while the old version stops and the new version starts up. This is a very general problem in software engineering: "run thing all the time" is not very feasible.
One solution is to have multiple replicas of the data collection system running on different machines. For subwaydata.nyc this system is called Hoard and is independent of the subway data project: it can be used to reliably collect any data with a refresh time of ~seconds. There is a three replica deployment of Hoard collecting NYC subway data. The deployment is diverse on a few dimensions (cloud provider; geographic location) to mitigate the risk of correlated failures.
Hoard creates archive files on a per-feed and per-hour basis. Every morning NYC time, the subwaydata.nyc ETL pipeline runs and aggregates all the data for the previous calendar day into a single archive file and makes this available to download on the website.
The CSV data
The CSV data is generated by a GTFS journaling program. Given a list of GTFS realtime feeds this program creates a "journal" of all the vehicles in the feed.
The GTFS realtime data contains information about the state of the transit system at the current time whereas the journal is a record of what happened in the past. The algorithm to go from current to past is pretty simple. For every vehicle, the GTFS realtime updates are read in order. These updates contain predictions for the station arrival times for that vehicle for every future stop. When a station disappears from the feed, it means the stop is now in the past. The journaling software uses the last predicted time in the feed as the estimated time of arrival.
This is the happy path, but in a system as complicated as the NYC subway there are a bunch of edge cases (for example, a train may be rerouted and now have a different schedule). The journaling software tries to handle these gracefully. But in addition the CSV data does contain columns that mark the vehicle as having a complicated history. It is possible to filter the dataset to only the simplest vehicle journeys, which cover the vast majority of the dataset and whose arrival times are accurate.