On graphics, math and data visualization. Assembled by creator of digital goods, Riley Davis.

On graphics, math and data visualization. Assembled by creator of digital goods, Riley Davis.

On graphics, math and data visualization. Assembled by creator of digital goods, Riley Davis.

Weather Comparison Chart

March 17, 2015

weather chart thumbnail

All my friends scattered to the wind after college and left Kansas City for other places in the country. Now, a few years after this diaspora, and a couple of children later, we talk about getting back together. There are arguments about what city has the best beer and the best weather, but still offers lots to do, and is relatively close to our families. No one city has the best of everything. A couple of our friends are moving to Portland and while it sounds pretty great, my wife gets a bit of the SAD if it's too cloudy.

I thought I would find out for myself just how cloudy it really is. Wikipedia lists days of sunshine, but that seemed incomplete. Is it cloudy all day? Or is it cloudy in the morning like in San Francisco? This visualization is my attempt to quantify weather conditions and such. It's probably not going to end any debates, but it's a good start if weather is high on your list of must-haves before you move across the country.

Getting the data

The 2014 history data was scraped from the wunderground history api. You have to create an account, but it's free up to 500 calls a day (per email, wink). Each response is a json blob of the daily summary and ~24 hourly observations with cloud cover, temperature, humidity, precipitation, etc. Here is a glossary of the terms in the response. I just scraped for the year of 2014, which corresponds to the black bars in the chart.

I found historical weather data from NOAA but it's quite raw. Luckily, the University of Dayton has already done some of the heavy lifting for me (thanks, Ohio!). The Dayton data is broken down by a daily average temperature from 1995-2014 for many domestic and international cities, like so:

 1             1             1995         55.0
 1             2             1995         57.1
 1             3             1995         44.4
 1             4             1995         47.6
 1             5             1995         44.1
 1             6             1995         45.2
 1             7             1995         49.5
 1             8             1995         42.5
 1             9             1995         43.6
 1             10            1995         41.0

Preparing the data

Parsing the data was fairly straightforward. JavaScript isn't a good language for scripting, but the functional programming features like map and reduce are really nice. We can eliminate some of the callback mess by using the someFuncSync() version of io methods, since we're not creating a client-facing app. (please do not do this on a server).

Basically our goal is to create a list of averages per day. We'll end up with

var fs = require('fs');

var files = {
    'CODENVER': 'denver',
    'NYNEWYOR': 'nyc',
    'CALOSANG': 'la',
    'ORPORTLA': 'portland',
    'ILCHICAG': 'chicago',
    'MOKANCTY': 'kc',
    'IEDUBLIN': 'dublin',
    'TUISTNBL': 'istanbul'

var finalData = {};

try {
} catch (e) {

Object.keys(files).forEach(function (file) {

    var temps = [];
    // initialize array
    for (var i = 0; i < 12; i++) { temps[i] = []; }

    fs.readFileSync('weather_data/' + file + '.txt')
        .forEach(function (line, i) {
            var observation = line.trim().split(/\s+/);

            var date = new Date(observation[2], observation[0] - 1, observation[1]);
            var temp = observation[3] === '-99' ? null : parseFloat(observation[3], 10);

            // this might be confusing since we're representing dates as 0-indexed instead of the js convention of 1-indexed
            if (!temps[date.getMonth()][date.getDate() - 1]) temps[date.getMonth()][date.getDate() - 1] = [];
            temps[date.getMonth()][date.getDate() - 1].push(temp);

    // clear out null values
    temps.forEach(function (month, monthIndex) {
        month.forEach(function (day, dayIndex) {
            var thisDay = day.filter(function (temp) {
                return temp;

            temps[monthIndex][dayIndex] = {
                date: new Date(2014, monthIndex, dayIndex + 1),
                avgs: thisDay.sort(function (a, b) {
                    return a - b; // sort ascending
                max: Math.max.apply(Math, thisDay),
                min: Math.min.apply(Math, thisDay)

    // transform temps back into a flat array of dates
    var finalTemps = temps.reduce(function (memo, monthly) {
        return memo.concat(monthly);

    // splice out leap day
    finalTemps.splice(59, 1);

    finalData[files[file]] = finalTemps;

    console.log('created finalData[' + files[file] + ']');


fs.writeFileSync('averages.json', JSON.stringify(finalData));

Many of the architecture (using that word loosely) decisions for this project are guided by the fact that while d3 can do many awesome stats things, you have to consider file download size. I definitely could have piped all these files directly to the browser and probably written less lines of code overall, but the 2014 data for ONE city is 27mb. That being said, this could be pretty intresting if set up as a service with some of the node.js streaming tools parsing the JSON and an upgraded account from wunderground.

Making the chart

If you haven't tried d3 yet, it's time to take the plunge. Basically, this is a glorified bar chart where the bars don't end at 0, but go from a high to a low temperature. Here are some of the intresting bits:

// setting x and y scales.
// the x domain is hard-coded to 2014, but it doesn't really matter, I just picked a year with no leap year.
var x = d3.time.scale()
    .domain([new Date(2014, 0, 1), new Date(2014, 11, 31)])
    .range([0, width]);
// start at 0° and end at 100°F
var y = d3.scale.linear().domain([0, 100]).range([height, 0]);

d3 has some nice stats stuff built in, just make sure to check first before you try to write your own methods. I wanted to show the interquartile range for the historical average temperatures, and d3 has quantile methods.

// draw inner quartile
        .attr('class', 'quart')
        .attr('x', function (d) { return x(; })
        .attr('y', function (d) { return y(d3.quantile(d.avgs, 0.75)); })
        .attr('width', barWidth)
        .attr('height', function (d) {
            return height - y(d3.quantile(d.avgs, 0.75) - d3.quantile(d.avgs, 0.25));
        .attr('fill', centerColor);

I had trouble getting the months to show up as ticks on the x-axis, but here's what I ended up with:

var xAxis = d3.svg.axis()

// later...
svg.append('g') // x axis
    .attr('class', 'x axis') // this is so you can style the svg elements with css
    .attr('transform', 'translate(0,' + y(0) + ')') // cross @ 0
    .selectAll('.tick text')
        .style('text-anchor', 'start')
        .attr('transform', 'rotate(45)') // rotate so labels fit on mobile
        .attr('x', 6)
        .attr('y', 6);

Refining the chart

I wanted to compare 2014's temperatures to the running average, but overlaying all the data made the chart too busy. I tried different colors and opacities, and in the end I settled on a mouse-driven interaction, where you can see 20 days at a time. This still gives you a sense of what the year was like without overwhelming the rest of the data.

Showing the sunshine as a percentage was pretty fun. The measurements are from 2014, so I have the hourly condition measurements. Conditions can be something like "cloudy", "snow", or "brimstone", etc. I have the daily list of 24 observations, and made the yellow bar more translucent the more cloudy it was. I counted the conditions "clear", "sunny", "scattered clouds", "partly cloudy" (I'm a glass-half-full sort of guy) and "mostly sunny" as clear-ish. So if it was snowing for half the day and clear for the other half, the yellow bar would be at 0.5 opacity.


It's the middle of the winter here in nyc, and LA looks pretty great after building this chart. And to answer the original question, yes, it really is that cloudy in Portland.


Fermat's Spiral in JavaScript

March 7, 2015

dots final


var scaleFactor = 2;
for (var i = 0; i < num; i++) {
    var theta = 2.39998131 * i;
    var radius = scaleFactor * Math.sqrt(theta);
    var x = Math.cos(theta) * radius;
    var y = Math.sin(theta) * radius;

I needed to visualize some survey data where I could see proportions of categorical data as well as the actual number of respondents. While almost all data could be bar charts, this isn't very exciting or pretty. The number of respondents was a manageable number (in the hundreds) so I could represent each one atomically. The next obvious thing would be to make a grid of squares, which is neat, but we can do better.

data grid codepen

It would be fun to make a force layout in d3, which would get us individual nodes showing proportions, but all that movement amounts to chartjunk. I love me some verlet simulations, but less is more when trying to show data.

So basically what we want is a circle packing algorithm. All the nodes are the same size, since they represent a person. This will save us having to write a more complex algorithm to place each circle. I thought a great way lay this out would be something like a sunflower. So how would we accomplish this?

You might remember the Fibonacci sequence from elementary school. The ratio generated by one number in the sequence to the next approaches 1.6180339887...:1 or φ the Golden Section.

So how does this relate to our sunflower? Basically, the placement around the circle is a function of the Golden Angle. Basically, if we divide the circumference () of the circle into two sections, the ratio of the length of the smaller to the larger is the golden section (φ). The arc length of the smaller section comes out to 136.508° or 2.39998131 radians. The radius at whatever point is the square root of the angle times an arbitrary scaling factor.

So if I wanted to place the dots for my visualization, I would iterate through a range of numbers, moving 136.508° around the spiral each step, gradually increasing my radius. I end up with something like this:

dots codepen

Changing the scale factor will make the dots more spread out.


JavaScript Date object and Arrays

February 7, 2015

Short post. Putting this here for myself and future googlers.

Did you know that you can easily create an array of all the days in a year? You don't have to explicitly set the month parameter in new Date().

first we'll make a range function, unless you're using the underscore/loash method _.range

function range(num) {
    return Array.apply(null, Array(num)).map(function (_, i) {return i;});

which will give us an array of numbers from 0-364. All we have to do to transform that into an array of Date objects is map them. The day parameter will automagically do a modulo into the month parameter:

var dateArr = range(365).map(function (i) {
    // must add 1 to the date
    // January 1, 2014 would be new Date(2014, 0, 1)
    return new Date(2014, 0, i + 1);

Presto! You don't ever have to remember the number of days in each month again!