all about D3 time scales

all about D3 time scales

The Problem

At Tidepool, we have need to plot data that comes out of diabetes devices that are completely naïve with respect to timezone information. All we have in our data is the timestamp reflecting the time displayed on the user's device at the time of the event in question. We call this deviceTime, and we store it in ISO 8601 format without a timezone offset - i.e., YYYY-MM-DDTHH:MM:SS. Because they are completely timezone-naïve, diabetes devices do not automatically make the switch from standard to daylight time and vice versa (where such transformations are the norm). Furthermore, because (at least in the United States) the change from daylight to standard time (or, again, vice versa) occurs in the middle of the night - specifically, at 2 a.m. - we can almost assume that very few or no users of diabetes devices change the time on their devices at the precise moment when this transformation is called for. Thus, we can assume that most - if not all - users will generate some datapoints that are technically undefined on a timezone-aware scale. For example, 2:30 a.m. on March 9th, 2014 is undefined (for American locales) because the switch from standard to daylight savings time happens at the turnover from 1:59 a.m. directly to 3:00 a.m. in the next minute.

Until we have timezone-aware data coming out of diabetes devices, we want to present a user's data to them as transparently as possible. Plotting diabetes device data on a scale that automatically accounts for the switches to and from daylight savings time is not transparent: two datapoints - at 1:30 a.m. and 2:30 a.m. on March 9th, 2014, for example - will be plotted in the exact same place even though the user saw (or would have seen, had they not been asleep) two different timestamps on their device with an hour of time passing between glances at the device. On the other hand, if we plot the data we retrieve from diabetes devices on a scale that knows nothing of daylight savings time, then when the user eventually changes the time on their device to reflect the switch to or from daylight time, they will see an overlap or a gap in their data that directly corresponds with their manipulation of the device's settings and thus remains perfectly transparent. (The same applies when a user travels across timezones and adjusts their device accordingly.)

The Solution

D3 provides two types of time scales: UTC (d3.time.scale.utc()) and non-UTC (d3.time.scale()). For the purposes of this document, the most important feature of UTC time is that it does not undergo any transformation to or from daylight savings time. This feature makes D3's UTC scale the only appropriate scale for us to use when plotting timezone-naïve diabetes device data, at least if we are to maintain the transparency described above.

In the vast majority of cases, we are not, in fact, plotting UTC data; we are merely exploiting - in a somewhat kludgey fashion - the fact that UTC does not undergo any scheduled transformations but rather retains the same timezone offset (namely, 0) all year round. (Because pretending we are plotting UTC data when we really are not is a somewhat ugly solution, we choose to keep the code that applies the transformation to "UTC" - really just appending Z, the ISO 8601 shorthand for the UTC timezone offset - segregated from the rest of our code.)

As will be discussed further below, there is a second reason for employing D3's UTC scales to plot our data, and that is to ensure consistent results across different browsers and when viewing data generated in one timezone (e.g., Pacific) in another (e.g., Eastern).

Examples

This gist contains three examples of D3 time scales that - at least in theory - share a common domain and range.

The domain for the scale is March 8th, 2014 at noon to March 10th, 2014 at midnight. We chose these dates to highlight the change from standard to daylight time that occurred on March 9th, 2014 at 2 a.m.

The range for each scale is 0 to the width of the SVG.

tideline's current implementation

Tideline's current implementation of time scales first applies the transformation of appending Z to the timezone-naïve ISO 8601 (without a timezone offset) deviceTime that is present in the data. We use d3.time.scale.utc() and set the domain like so:

.domain([new Date('2014-03-08T12:00:00.000Z'), new Date('2014-03-10T00:00:00.000Z')])

using JavaScript's built-in Date constructor.

without UTC

So, what happens if we don't apply the append Z transformation and don't use a UTC scale? The second example displays the result of using d3.time.scale() and setting the domain like so:

.domain([new Date('2014-03-08T12:00:00'), new Date('2014-03-10T00:00:00')])

still using JavaScript's Date constructor.

Several things to note here:

• The user-facing domain (i.e., what the tick labels on the drawn axis say) changed! (At least if you're viewing in Chrome or Safari.) Why did this happen? The issue here is that the Chrome and Safari implementations of JavaScript Date assume that a timezone-naïve date string of the sort we've used to set the domain here is UTC time, but d3.time.scale() generates an axis according to your browser's local time, so the domain you see should reflect the current timezone offset between you and UTC. The disconnect here is that the domain hasn't changed - the axis still runs from noon on March 8th UTC to midnight on March 10th UTC but the labels you can see are showing you what those UTC times are translated into your local timezone. Pretty confusing, and there are more problems.

• Firefox has implemented JavaScript Date slightly differently such that it assumes that a timezone-naïve date string is your local time. This removes the disconnect between what you pass as the domain and what you see, but obviously the cross-browser inconsistency is a problem, and not using UTC means that the time period between midnight and 3 a.m. on March 9th is foreshortened to two hours instead of three because of the switch to DST. See above in 'The Problem' for why we don't want this behavior.

• Finally, using d3.time.scale() with its coercion to the browser's local time means that (unless you're using Firefox) when you view the data in a different timezone, the domain that is presented to you will change again. Compare these two screenshots (taken at an earlier version of this gist):

  • taken in Pacific timezone, #2 is offset eight hours earlier than UTC (4 a.m. instead of noon) at the beginning of the domain (seven at the end, after switching to DST)

  • taken in Central timezone, #3 is offset six hours earlier than UTC (6 a.m. instead of noon) at the beginning of the domain (five at the end, after switching to DST)

D3's suggested solution for the JavaScript Date constructor problem

D3 is aware of the JavaScript Date constructor implementation difference between browsers and provides an alternative to using this constructor in order to ensure consistent results across browsers. (Unfortunately, this advice is given neither in D3's time scales documentation nor in the documentation for time formatting, so I only discovered it recently, via this discussion on GitHub.)

This example was produced using a D3 time-formatting function:

var timeFormat = d3.time.format('%Y-%m-%dT%H:%M:%S');

which will return a cross-browser-consistent Date object when used to set the domain of the scale:

.domain([timeFormat.parse('2014-03-08T12:00:00'), timeFormat.parse('2014-03-10T00:00:00')])

The append Z trick to produce "UTC" date strings is no longer necessary here, but the same undesirable (if correct) foreshortening of the time span between midnight and 3 a.m. on March 9th occurs and results in overlapping data at the switch to DST:

In addition, although the axis itself is identical across all three browsers under consideration here, we don't actually achieve full cross-browser consistency:

• Firefox doesn't properly deal with the "undefined" timestamp of 2:30 a.m. on March 9th - it's plotted at 3:30 a.m. on the axis:

• Safari plots both the "undefined" 2:30 a.m. on March 9th and the 2:00/3:00 a.m. moment of the switch to DST an hour too late:

achieving #1 with D3's alternative to the Date constructor

In fact, if we want to ensure cross-browser consistency and remove the foreshortening of the timespan between midnight and 3 a.m. on March 9th, the only way to do it seems to be a D3 notational variant of the current tideline implementation using d3.time.scale.utc() again. Namely, using a time format specification for parsing date strings that assumes the appending of a UTC timezone offset (here +0000, since D3 doesn't appear to include the shorthand Z as a proper offset when parsing timezone offsets with the %Z format specification option):

var timeFormat2 = d3.time.format('%Y-%m-%dT%H:%M:%S%Z');

And setting the domain like so:

.domain([timeFormat2.parse('2014-03-08T12:00:00+0000'), timeFormat2.parse('2014-03-10T00:00:00+0000')])