This is an important question because the cost (be it financial, environmental, etc) of building, operating, and maintaining a transportation facility is generally determined by the maximum capacity it is expected to provide. To avoid ruining any surprises, all calculations here are derived from the publicly available 2007 Hub Bound Report, and implemented in this spreadsheet. The "hub" here is Manhattan below 60th Street -- New York City's official CBD.

Just to get warmed up, chew on this -- from 8:00AM to 8:59 AM on an average Fall day in 2007 the NYC Subway carried 388,802 passengers into the CBD on 370 trains over 22 tracks. In other words, a train carrying 1,050 people crossed into the CBD every 6 seconds. Breathtaking if you ask me.

Over this same period, the average number of passengers in a vehicle crossing any of the East River crossings was 1.20. This means that, lacking the subway, we would need to move 324,000 additional vehicles into the CBD (never mind where they would all park).

What does it take to move that many additional vehicles? Well, it depends. Different auto facilities in the city appear to have different capacities (as expressed in vehicles per hour per lane):

At best, it would take 167 inbound lanes, or 84 copies of the Queens Midtown Tunnel, to carry what the NYC Subway carries over 22 inbound tracks through 12 tunnels and 2 (partial) bridges. At worst, 200 new copies of 5th Avenue. Somewhere in the middle would be 67 West Side Highways or 76 Brooklyn Bridges. And this neglects the Long Island Railroad, Metro North, NJ Transit, and PATH systems entirely.

Of course, at 325 square feet per parking space, all these cars would need over 3.8 square miles of space to park, about 3 times the size of Central Park. At that point, who would want to go to Manhattan anyway?

Without the NYC Subway, I'm pretty sure this is what it would look like if we provided for to everyone commute by car:

Update: people seem to be taking this map quite seriously, whereas I would call it ... conceptual. The blocks in Manhattan below 57th Street are the theoretical parking lots, whereas everything else is the additional roads, bridges, and tunnels we would need to move the cars.

The result, after much whacking, is, I think, compelling, but you'll have to see for yourself. The general idea it that the history of subway ridership tells a story about the history of a neighborhood that is much richer than the overall trend. An example, below, shows the wild comeback of inner Williamsburg, and how the growth decays at each successive stop away from Manhattan on the L train:

This is somewhat in contrast to the South Bronx, which is yet to see the resurgence in ridership, other than at Yankee Stadium and the Grand Concourse:

The stations around Wall Street tell a totally different story, in which the ups and downs of each dep/recession have more immediate but temporary effects:

My first stab at visualizing this data was a traditional cartographic approach, showing the overall growth from 1977 to 2006 at each station. This told an approximate story at the level of the whole the city, but did not leave much room for detailed exploration. Thanks to geoserver's awesome new(ish) dynamic symbolizers functionality, it was trivial to plot the station-by-station time series sparklines (generated in R of course) onto the interactive online map. (Originally I the plots were produced in Perl and placed onto the map with a Javascript WFS layer, but that is so 2005.)

For all this, I never really felt like this little experiment was ready for an audience. That all changed when OpenGeo put up its Open StreetMap base layer for the web, giving fancy overlays like this one the context they need.

http://transit.frumin.net/subway/spark.php

UPDATES:

At least 2 people have taken the data I put out there and used it to make some zippier interactive flash apps:

• The first is very polished, but I think the designer is quite misled in his desire to not plot dots on a map, and thus to plot what looks like a network flow diagram but with totally bogus data

• The second is a little rougher around the edges, but I'd say is much more honest, and thus useful

Not sure if anyone knows, but I also have GIS files for the subway here: http://transit.frumin.net/trx/Alignment

Not that I ever really thought it would be. What I did think about was the pictures I would get back. And they are money (perhaps even more so than the nap I took in the machine). Here's the best of the crop (click for enlargements, or go here):

Images | AVI Video | Flash Video	Images | AVI Video | Flash Video
Images | AVI Video | Flash Video	Press play on this video: AVI Video | Flash Video

I don't think I should have expected to recognize myself from a cross-section of my brain, eyeballs, and sinuses. In fact, I think I look damn silly.

I do recognize my crooked nose in this one, but I didn't expect the crookedness to continue inside the skull. That huge white ball inside my right cheek -- is that mucous? Is there a doctor in the house?

I can't resist this one: peep the brain stem.

I have been thinking about mass transit trip planning software for the web and for mobile devices. Between the individual efforts of agencies around the world, and Google's efforts towards open sharing of structured transit system data, we seem to be on the right track, institutionally speaking. As a user, however, I am perpetually frustrated by the focus that every transit trip planner I have ever used puts on the supposed schedule, even for services that are high frequency and/or less-than-perfectly reliable.

This general feeling, combined with two recent and exciting meetings I have had, leave me with a few nagging questions:

• In providing transit users with such software, how useful is the schedule by which the transit provider has planned their operations?

• When are expected waiting and travel times more useful than precise trip-by-trip itineraries?

• What effect do randomness and unreliability have on those expectations?

• Should the passenger plan her trip differently if she has to be on time than if her schedule is flexible?

• Finally, does real-time information obviate the need for any or all of these other inputs?

The answer: it depends. The actual schedule (R trains leave Union St at 8:13, 8:25, 8:37 arriving at Union Square at 8:39, 8:51, 9:03, etc) is only relevant to the degree to which operations follow the plan. And even in the face of near-perfect operations, I only care about the schedule of departures when I have something to lose by ignoring it (i.e. when there's not always another train or bus in tolerably few minutes).

Expectations implied by the schedule (I should wait 6 minutes on average, but never longer than 12, and the ride is expected to take 26 minutes) are meaningful even when the precise schedule isn't, but only if those expectations are reasonable. For example, a simple model shows that as the service becomes even slightly variable, expectation of waiting time increases, as does the maximum. Of course, many things that cause some passengers to wait longer are experienced by other passengers as delays along the way.

Let's now think specifically about trip planning software for relatively high frequency urban transit services with normal amounts of variability. I don't want to be bothered with exact but fairly useless times of scheduled departures and arrivals. I just want to know how long I can realistically expect to have to wait, and how long the trip is likely to take. And when I have a hard timeline, like getting to a meeting or a catching an airplane, I want to know the (approximately) worst case scenario.

Current levels of unreliability in our transit systems are not something we should have to live with. More funding, saner public policy, and better management can go a long way towards fixing some problems. I am not focusing here on the sources of unreliability, but suffice it to say they are many, some debatably the provider's responsibility (eg missing drivers, faulty equipment) and some debatably not (eg on-street traffic, passenger behavior). But given that they are here today, would you rather think a trip will be fast and have it end up being slow, or would you prefer to have the best information possible when making your own decisions?

The copious amounts of real service data collected by transit providers from bus GPS and rail signaling systems are of great value here. They allow us to fairly easily and cheaply describe distributions of waiting and travel times, and thus estimate expectations and approximate maximums for use in trip planning software.

Often, those systems were in fact installed to provide real time data, with historical performance analysis a secondary or accidental purpose. The notion of an expected waiting time changes radically when real-time "next-vehicle" information is provided, assuming the real-time predictions are in fact accurate. However, even perfect real-time data doesn't protect from problems from occurring down the line or reduce the variability inherently introduced by successive transfers.

In the next generation of (open source?) web and mobile transit trip planning, please:

• Give me the option to use the schedule or to use expected values, but try to be smart about the default.

• When not using the schedule, please allow me to plan depending on how flexible my own schedule is.

• Use real performance data to generate realistic expected and worst case scenarios.

• When possible, especially when the trip is imminent, use real time data to reduce uncertainty in my trip plan, but make use of realistic expectations for forecasting the balance of the trip.
  .

To implement such a trip planner, a number of open questions remain:

• Even for a perfectly reliable system, where exactly is that threshold between using the schedule and using expectations?

• How does this threshold change as a function of normal or excessive variability in operations?

• What is the best way to integrate real-time data (of varying predicative quality) with realistic expectations for trip planning on-the-go?

If you're still awake, and have comments or questions, let's talk. The fact that this post found its way onto your computer makes it highly likely you already know how to get in touch.

Rather than try to explain the work, I've just cut and pasted the executive summary from the report and included some of my favorite figures (with no explanation). It's not a terrible paraphrasing, but if there is a lot of really good meat in the document if you are bored and hungry. Snooze on...

The London Overground is a pre-existing rail service in London whose operating responsibility and revenue risk were recently granted to Transport for London (TfL). Here we discuss the prospect of using data from the Oyster smartcard ticketing system to evaluate the performance of the London Overground explicitly from a passenger’s perspective.

The core idea behind our approach is to directly measure end-to-end individual journey times by taking the difference between entry and exit transactions stored by the Oyster system. The focus of this study is Excess Journey Time (EJT), calculated on a trip-by-trip basis as the difference between the observed journey time and some standard. In this case, the standard is determined for each trip with reference to published timetables, indicating how long the trip should have taken under right-time operations. A positive EJT indicates that the journey took longer than was expected.

Excess Journey Time is interpreted as the delay experienced by passengers as a result of services not running precisely to schedule. The distribution of EJT indicates reliability. We validate these interpretations using a detailed graphical analysis, and then aggregate them to the line and network level over a variety of time periods. Our analysis is conducted on large samples of Oyster data covering several months and millions of Overground trips in 2007.

At the aggregate level, relative values of Excess Journey Time are largely in line with expectations. The North London Line has the highest average Excess Journey Time of all lines on the London Overground, around 3 minutes, and the widest distributions (i.e. least passenger reliability). On all lines, there is significant day-to-day variability of Excess Journey Time. For the whole London Overground, and for the North London Line in particular, Excess Journey Time is worst in the AM and PM Peak timebands.

The current performance regime for the London Overground is the Public Performance Measure (PPM), which measures the fraction of scheduled vehicle trips arriving at their destinations fewer than five minutes late. Over time, EJT shows a strong correlation to PPM. There is clear additional variation in EJT, indicating that it captures certain information about passenger experiences that PPM does not. This variation tends to increase as PPM decreases, particularly in the AM and PM peak timebands, which suggests that the effectiveness of PPM as a measure of the passenger experience decreases as service deteriorates.

Another quantity of interest derivable from Oyster data is the time between passenger arrival at the station and the scheduled departure of the following train. The spread of this distribution of this quantity indicates the degree to which passengers arrive randomly (i.e. "turn up and go") rather than time their arrivals according to schedules. We have found that on the North London Line, especially during the AM, interpeak, and PM peak periods, passengers tend to arrive randomly. This is apparently in contrast to conventional wisdom for National Rail services, and has distinct implications for crowding levels and timetabling practice. In an appendix to this report we look at this in detail, and recommend that even headways be prioritized in timetabling the North London Line.

The Overground is, by design, part of a larger integrated multimodal network. Oyster data, by nature, is somewhat ambiguous in representing passenger trips on such a network that involve transfers or multiple routing options. This poses certain problems to our methodology, but also presents the opportunity to quantify and understand the experience of passengers across the entire network. We discuss these problems, potential solutions, and opportunities at length, as well as other applications for this methodology, and future research directions.

We have concluded that Oyster-based metrics are effective for monitoring and identifying problems as experienced by passengers on the London Overground. They may be even more effective for use across the whole of London's public transport network, particularly as Oyster is in the process of being rolled out to all National Rail services in the Greater London Area.

While the caption of image points out that "some riders use the $81 passes for 40 or fewer rides," it fails to point out that anyone making 46* or fewer trips is losing money on their pass. The calculus of "losing" or "making" money on a monthly pass is of course fraught with nuance (e.g. how much is it worth to me to not have to think about paying on each trip? are any of these passes subsidized?) but the article doesn't touch on it at all.

It's no secret that I love NYC Transit and transit in general, but that doesn't mean people should be buying passes when it's far from beneficial. Don't even ask about the London case...

* 46 trips is the breakeven point when the monthly pass cost $81 and individual trips cost $1.74 (after the bonus)

What the chart below shows to me is that the notion of "climbing the charts" is basically defunct. Albums hit their peak Billboard positions immediately or very shortly after their release dates, and then fade out like like the ego and stature of your high school football team's quarterback.

(click for a full size image)

There are a number of ways I have thought that this particular graphic could be improved -- make it interactive, somehow link albums by the same artist, etc -- but, really, who cares?!?

I definitely never thought I'd actually get to say that. Fortunately, the guy I know told me to bring a camera. The battery was dying so I didn't get to snap that many pics, but what I got follows. I think these pictures capture, to a degree, how messy the prospect of assembling a TBM way under ground really is.

The overriding feeling I had throughout was of being inside, around, and on top of a Sandworm in the novel Dune. What really impressed me was the number of people necessary just to put one of these things together, and the enthusiasm of all the people working on the project (sweating it out under 200 feet of bedrock).

The entrance to the work site, in Sunnyside, Queens

The 30+ years old but never used LIRR tunnel under the East River

The beast itself:

So, point being, East Side Access is really happening! This visit was in late August, just before the machine was brought on line. Perhaps I'll get to visit again to see it in action.

Specifically, in the UK tradition, all of the cameras relay a full video feed to some central processing location. Not only is this absurdly costly (think fibre!!) but it allows for plenty of privacy invasion by anyone who has access to the cameras' feeds. The proposition for New York is very different. The proposition is much cheaper and seems to all but eliminate the possibility of using the cameras for anything but looking at license plates. That's because the cameras would be equipped with enough smarts to know when to snap a photo, and only that still image would be sent to be processed. If you don't believe me, read this excerpt from IBM's recently released proposal:

A worst case analysis shows that for a very busy lane, with one thousand vehicles passing the detection equipment every hour and forced to send two 100kB images for each vehicle, the bandwidth requirement is a mere 57kB/s. This is within the capacity of wireless networks today, but is not the optimal solution approach.

A more realistic case, in which 50% of vehicles are equipped with an E-ZPass tag, 90% of the remaining license plates are read with a sufficient confidence at roadside and 80% of charges are paid in a timely manner, leads to a bandwidth requirement of 8kB/s. A very busy, six-lane detection point would thus be well within the capacity of NYCWiN, even without local reinforcement of the wireless network.

We estimate that, with our proposed solution approach to vehicle detection at the edge of the network and given the estimated amount of traffic in the city, the average local bandwidth requirement across the system will be on the order of less than 1kB/s per lane, and the overall load on the backbone of the wireless network will be small.

More generally speaking, I think it suffices to say that Congestion Pricing uses technology, and as we know technology only gets better and cheaper over time, so we can be sure that NYC's Congestion Pricing technology will be much better and cheaper than London's.

Now, if only somebody would only explain this to all the privacy freaks and civil libertarians that are making this process so painful...

What a platform:

The boroughs, in order of time I've spent in them:

Some lovely embellishments on the Hudson River:

And my favorite little twist:

Not a bad running mate:

But not a good day either (they came in 4th):

For more info on the campaign itself, check out a recent NYTimes podcast and an interview on WNYC with Jimmy Breslin, Mailer's running mate and another icon of New York realness.

For example, a recent piece (of what I don't know) by Stephen Fleming of the Georgia Public Policy Foundation, entitled In Transportation and in Technology, Packets Beat Circuits, starts off "Why are so many mass transit policies doomed to failure? Because packets beat circuits. Let's explore an analogy."

You can imagine where it goes from there ("cars are like packets, mass transit is like circuits, so cars are better"). The guy claims to have worked in digital communications for 10 years; I wonder if he's just bitter because he was on the wrong side of the packets vs circuits debate.

For the sake of all 6 people likely to read this, I hereby debunk this terrible analogy:

The fundamental aspect of a circuit-switched network, as stated in the first sentence of the Wikipedia article on Circuit Switching is that it "establishes a dedicated circuit (or channel) between nodes and terminals." That is, the bandwidth for the flow is reserved end-to-end for the life of the circuit. Traditionally, when I make a phone call, part of the 'space' on a bunch of copper wires connecting where I'm calling from to where I'm calling to is reserved even if no one is saying anything over the line.

Clearly a road network is not circuit-switched -- when you start out from your house you don't have a dedicated lane all the way to your destination (if you did, I might just own a car). In a circuit-switched transit network, not only would I have a seat on one R train from Union Street to Union Square, I would have a seat on every R train over the same route for the duration of my trip. Realizing this, the analogy breaks down completely.

Fleming's main argument as to why Transit is like Circuits is that the bandwidth hierarchy, in traveling by train, then bus, then feet, is like the digital transmission hierarchy of telephone (i.e. circuit-switched) networks. Perhaps he lives and works directly on top of an interstate and has never driven by highway, then arterial, then local street in his car. Or never noticed that the bandwidth on his home broadband connection is orders of magnitude smaller than the trans-Atlantic fiber lines that connected me in London to his web server in Georgia.

In a circuit-switched network, I only use as much bandwidth as I need at that moment, and only on the single link I'm currently traversing. When I get to the end of that link, I am put in a queue until the next link on my path is ready to receive me. It's true that it's a bit more obvious to see a road/auto network as analogous to a packet-switched network, but only because of the apparent simplicity of the rules of the system. Transit networks are of the same nature, it's just that the way packets (i.e. people!) are queued and switched where links connect is more complicated and constrained than on roads.

Speaking in data-network terms that we are all familiar with, transit mops up auto when it comes to bandwidth (total bits, or people, per unit time). The problem with transit is, in some circumstances, higher end-to-end latency (the time it takes for the first bit, or person, to get where they're going). But once you get the flow started, we all know that one track of even light rail service can carry the same number of passengers per hour (or was it bits per second) as 7 lanes of freeway or 17 lanes of street (see here).

Unfortunately, people actually read and believe this kind of proof-by-bad-analogy thinking. In a recent newsletter from the Reason Foundation, Robert Poole, Reason's Director of Transportation Studies, claims to have had his "Aha!" moment when reading Fleming's piece. Too bad for him the "Aha!" wasn't a realization to be much more careful with his analogies.

The first strategy for adding additional housing outlined in Mayor Bloomberg's 2030 plan is to pursue transit oriented development. In simple terms, dense development around transit stations and hubs. However, the plan also describes capacity issues that our transit system faces today and will face in the future. So, we should build all our new housing clustered around a transit system already reaching capacity? Hrmmm...

Looking at subway ridership in a long term context, it's pretty clear however that while some lines may be quite crowded today, the system as a whole is substantially below the highest usage levels it has supported historically. On the whole, we have recovered from a nadir of 915 million trips in 1977 to 1.5 billion trips in 2006 -- about the same as in 1952.

We all know that commuters from Williamsburg and the Upper East Side are suffering, but the question remains -- are there parts of the city where subway usage is substantially below levels that have been supported in the past? Comparing each station's 2006 annual ridership to levels in 1952 yields the following map, where red indicates a net decrease over the last 54 years (and thus, theoretically, excess capacity):

This analysis of course doesn't account for the fact that bringing the South Bronx back to historical levels would make the problems on the Lexington line even worse than they are today, but it at least gives a sense of what areas could accept housing growth around subway stations if the most pressing line-level capacity issues were resolved.

What if we want to look at each station's or line segment's pattern over time? As they say here in London -- "watch this space" (and think sparklines).

[Technology Shoutout: most of the work for making the above map was done by the ever-more-brilliant open source PostGIS and GeoServer packages.]

While usually calling something a zero-sum game is a bad thing, in this case I mean it positively. The $500M for a congestion pricing pilot that New York has lost may still lose would go to somewhere else like Dallas, San Diego, Atlanta, San Francisco, Denver, Miami, Seattle or Minneapolis. I am a New Yorker born and bred, and am as disappointed as anyone about this, but I think in this case we all may be suffering from a slight case of New-York-is-the-center-of-the-world-itis.

None of these other cities have nearly the mass transit system that New York has. Probably combined their mass transit systems don’t carry half the passengers ours does, and most people living in those places think of cars as an absolute necessity for practically everything they do. In a broader sense, it could very well be an overall net positive that the congestion pricing pilots happen in other cities. Perhaps those places will make incremental progress in shifting people out of cars and more importantly, changing peoples' value systems when it comes to cars vs. other modes.

If we are really lucky, this could be the first wave in a national shift towards more rational thinking about transportation, which would definitely benefit NYC in the long term. In truth, I'm a lot less worried about NYC than I am about other cities and the country/world as a whole, so I wonder if it doesn't hurt to at least imagine the possibility of some greater good coming from Albany's (seeming) ineptitude.

Working with the singular Jeff Zupan and his former sidekick Alexis Perrotta, I helped to develop a possible alignment for the Triboro RX, and a crude estimate of what levels of initial commuter ridership one could expect to see if it were built. The fruits of this labor can be seen on the web at http://transit.frumin.net/trx/TriboroRX (including sections on the alignment, our data sources, the demand model, and detailed results). There I describe in detail how the line and its stations are laid out and how we made our estimates. At the end of the day, we can comfortably say that at least 76,000 New Yorkers (including 32,000 diverting from other modes of transportation) would use the Triboro RX to get to and from their jobs every day. This number that is quite competitive with many existing lines, and without ever touching the island of Manhattan.

At the heart of our ability to make this estimate is the Journey-to-Work data published by the census -- counts of commuters between every census tract and every other census tract in the city. Given these flow data, the shape of the subway network with and without the Triboro RX, and a rough model of how people make travel decisions on public transportation, it's not so hard to guess which subway riders would use a new transit line if it were built. Estimating new transit riders is more nuanced, but we did our best with limited resources.

This study of the Triboro RX has, for me, been much more than a semi-traditional transportation modeling exercise. I took it as an opportunity to get intimately familiar with the state of the art in Open Source mapping and GIS software, including PostGIS, GeoServer, and OpenLayers. These pieces represent a full network-enabled stack for, respectively, storing and manipulating, mapping and presenting, and client-side interfacing of spatial data. I don't think they are quite yet usable by the non-hacker, but I wouldn't be doing this work if I didn't think that my computing skills brought something special to the table. That said, I encourage you to check out the following:

• Interactive map of New York City, its subway system, and the Triboro RX (don't forget to try changing the background map and various overlays)
• Subway and Triboro RX in Google Earth
• The whole enchilada in Google Earth

Now, it wouldn't be a perfect project to do, for free, when I should be saving money for school, if it didn't also involve getting my hands dirtier than they already do from all the crumbs in my keyboard. It seems absurd to talk about planning a transit line without actually having visited the areas it would connect. Having synced the clocks on my GPS device and digital camera, I twice explored the Triboro RX right-of-way and its environs from Flatbush to Bay Ridge, in Brooklyn. The results are viewable either in Google Earth or directly on the web. What really struck me was the diversity of neighborhoods -- Flatbush, Ocean Parkway, Borough Park, Sunset Park, Bay Ridge -- traversed by the Triboro RX in less than a third of its length. Continuing on, it runs through East New York, Brownsville, Cypress Hills, Middle Village, Jackson Heights, Astoria, and Mott Haven. You could eat your heart out while getting from Brooklyn to the Bronx, skipping "the city" entirely.

Finally, no contemporary New York transportation project is complete if it doesn't some how tie into Congestion Pricing. In terms of providing mass transit to unserved communities in the outer boroughs, methinks this graphic speaks for itself:

PS Please forgive me, I know these maps need legends for the quantitative parts. It's all a big hack, trust me!

PPS The interactive web maps work much better in FireFox than Internet Explorer. Save your soul and get a real browser.

Instead of relying purely on viral marketing or mass media marketing alone, big-seed marketing combines the two approaches so that a large initial audience spreads the marketing message to a secondary audience, yielding more overall interest than either approach would have by itself, even if the message isn't that contagious. "Because big-seed marketing harnesses the power of large numbers of ordinary people, its success does not depend on influentials or on any other special individuals; thus, managers can dispense with the probably fruitless exercise of predicting how, or through whom, contagious ideas will spread."

While I was not listed as an author in the HBR paper, I am listed on the as yet unpublished version on Dr. Watts' web site (abstract).