dygraphs 2011 year in review

It's been a big year for dygraphs!

Looking back at the last 365 days in the commit log, there were 356 commits. A big "thank you" to everyone who contributed:

• danvk (207 commits)
• kberg (107 commits)
• nealie (13)
• jeremybrewer (9)
• antrob (8)
• flooey (6)
• paul felix (3)
• bombela (1)
• ben winslow (1)
• Lee (1)

One thing I learned this year is how to better manage the commit history on the master branch. So those counts may understate the contributions of people like Paul who contributed later in the year.

So what changed? The flashy stuff first! Here were the new features:

• Range selector (Paul Felix)
• Log Scale (kberg)
• Chart labels & "legend: always"
• Animated Zooms
• CSS-based sizing & automatic resizing
• Better float format handling (jeremybrewer)

It's also been a year of greatly-improved documentation. Some of the biggest wins here have been:

• The options reference page, and logic which ensures that it stays up-to-date
• JSDoc for our public APIs.
• A comprehensive guide to dygraphs and IE
• This dygraphs blog (kberg)

Finally, some of the biggest changes happened behind the scenes. kberg and I both learned a lot about JavaScript this year and used that knowledge to improve the dygraphs development process. The biggest changes in this category are:

auto_tests (kberg gets a dygraphs gold star for this one). Automated tests help us prevent unexpected side-effects from changes, as well as regressions. We had no automated tests at the start of the year, but now we have 83. This is good news for users as well as developers, who can make changes with a lot less fear than they did a year ago. Another gold star goes to Cory Smith for developing JSTD.

We also learned about the new "script mode" in ECMAScript 5, which tells the browser to complain when you use certain JavaScript features like global variables. All of dygraphs' JS files now run in strict mode and pass through jshint, a JavaScript linter. This will also help keep us from shooting ourselves in the foot.

There have been many other internal changes — we fixed a bajillion bugs, and antrob made some contributions which resulted in significant speedups for large charts.

So where is dygraphs going next? Here are some areas I'd like to work on next year:

• Improved auto_test coverage
• More intuitive/discoverable interactions
• Nicer-looking charts
• Better warnings when you misconfigure a Dygraph
• Customized interactions for touch-screens
• Speed improvements (maybe by using WebGL and reducing internal copying)
• Improving dygraphs.com

All-in-all, 2011 has been a great year for dygraphs. Here's hoping that 2012 will be even better!

Dygraphs Coordinate Systems, 3/3

This is the last of three posts about the coordinate systems used throughout Dygraphs.

• Part 1: Introduction and the Dygraph Data Coordinate System
• Part 2: DOM Coordinate System
• Part 3: Percentage Coordinate System

Dygraph Percentage Coordinate System

Update: Yes, it's really the fraction of the drawable area (from zero to one) rather than a percentage (0 to 100.) If we rename the methods, I'll update this post.

There’s an additional coordinate system which represents percentages from the edges of the drawing area, relative to upper-left corner. These aren’t often required for public API, but can be useful when having to work strictly in the viewable area of the canvas. This is different than the DOM coordinate system which has its origin off the viewable area of the canvas. In this case, this coordinate system is all about the viewable area of the canvas.

These are accessible by the methods toPercentXCoord and toPercentYCoord.

toPercentXCoord takes a data X data value and converts it to a percentage from the left edge of the canvas. This diagram gives you an idea:
And to hit the point home, here’s a snippet of code which demonstrates values created by this function:

for (x = 1; x <= 9; x++) {
 console.log(x, '=>', g.toPercentXCoord(x));
}

1 "=>" 0

2 "=>" 0.125

3 "=>" 0.25

4 "=>" 0.375

5 "=>" 0.5

6 "=>" 0.625

7 "=>" 0.75

8 "=>" 0.875

9 "=>" 1

toPercentYCoord does the same thing along the Y axis, but goes from top to bottom.
The drawing area along the y-axis goes from 0-131, so let’s look at toPercentYCoord.

g.toPercentYCoord(0)
1


g.toPercentYCoord(131)
0


g.toPercentYCoord(120)
0.08396946564885496


g.toPercentYCoord(65.5)
0.5

Converting percentages back to data coordinates can be performed with this function:


function(pct) {
  var range = g.xAxisRange();
  return ((range[1] - range[0]) * pct) + range[0];
}

and a similar one for y-axis conversion. Validation is left up to the reader.

Summary

Each coordinate system has its own origin:

• The Data coordinate system has its initial origin in the lower left corner, with increasing values moving up and to the right. This doesn’t mean the lower-right corner is (0,0), since the graph can be moved or resized.
• The DOM coordinate system has its origin in the upper left corner of the unclipped canvas, with increasing values moving down and to the right. This is not the same corner as the percentage coordinate system. However, you are limited to drawing inside the graph area.
• The percentage coordinate system has its origin in the upper left corner of the clipped canvas, with increasing values moving down and to the right.

API that can be used for translating between these coordinate systems are:

• toDomXCoord and toDomYCoord convert data coordinate values to DOM coordinate values
• toDataXCoord and toDataYCoord convert DOM coordinate values to data coordinate values
• toPercentXCoord and toPercentYCoord convert data coordinate values to canvas percentages.

Dygraphs Coordinate Systems, 2/3

Update: 8:30AM Oct 4: to clarify several items.

This is the second of three posts about the coordinate systems used throughout Dygraphs.

• Part 1: Introduction and the Dygraph Data Coordinate System
• Part 2: DOM Coordinate System
• Part 3: Percentage Coordinate System

DOM Coordinate System

Another coordinate system is the DOM coordinate system, which maps to coordinates of the underlying canvas.

(DOM coordinates are useful for mouse event handling. drawing on the underlay canvas, to name two things.)

DOM coordinates typically have (0,0) in the upper-left corner. While you might expect the corner to be in the drawing area, you’ll be surprised where that corner precisely is.

The Dygraph canvas coordinates don’t begin and end in what you would think of as the drawing area. No, in fact the canvas is widened to take up the space of the entire parent div element.

(People not familiar with Dygraphs may not realize, but the labels themselves are not canvas-drawn. They're part of the HTML DOM. As someone who had to understand these coordinate systems the DOM coordinate origin was a surprise. Other graphing systems draw their labels right on the canvas, e.g. http://code.google.com/apis/chart/, Dygraphs' does not.)

Note: A demo of a Dygraph without axis, using the full parent div element for its drawing area can be found at http://dygraphs.com/tests/unboxed-spark.html.

Clipping and other hackery

Here's an example that will help bring home home the difference between the full canvas area and the area exposed to the user, by way of a fun example, by showing how the image above was drawn:


g.updateOptions({
 underlayCallback: function(ctx) {
   ctx.canvas.width = ctx.canvas.width; // 1
   for (var y = 0; y <= 350; y += 50) {
     ctx.moveTo(0, y);
     ctx.lineTo(500, y);
     ctx.stroke();
   }
   for (var x = 0; x <= 500; x += 50) {
     ctx.moveTo(x, 0);
     ctx.lineTo(x, 350);
     ctx.stroke();
   }
}});

The line marked // 1, above is important. When the underlayCallback is called, it hands the caller a pre-initialized canvas context such that anything outside the graph area is clipped. (Having a clipping area is useful for graphing lines that exceed the boundaries of the clipped viewport.) A statement like // 1 reads like a no-op, but it has a side-effect of resetting canvas context, clearing the viewport, making that drawing possible. This is what the graph looks like without line // 1:

I hope you see now that (0, 0) doesn't represent the corner of the drawing area, but some spot which is fundamentally inaccessible to the user.

Note: Please don’t use this trick in graphs and applications you care about. This is only a hack for demonstrative purposes.

Conversion methods

The Dygraphs API comes with methods for converting between Data coordinates and Canvas coordinates

toDomXCoord converts from data X coordinates to DOM X coordinates

for (x = 1; x <= 9; x++) {
 console.log(x, '=>', g.toDomXCoord(x));
}

1 "=>" 56

2 "=>" 110.875

3 "=>" 165.75

4 "=>" 220.625

5 "=>" 275.5

6 "=>" 330.375

7 "=>" 385.25

8 "=>" 440.125

9 "=>" 495

toDomYCoord converts from data Y coordinats to DOM Y coordinates

for (y = 0; y <= 120; y+=20) {
 console.log(y, '=>', g.toDomYCoord(y));
}

0 "=>" 330

20 "=>" 279.618320610687

40 "=>" 229.23664122137407

60 "=>" 178.85496183206106

80 "=>" 128.4732824427481

100 "=>" 78.09160305343512

120 "=>" 27.709923664122137

and toDataXCoord and toDataYCoord convert X and Y values from the DOM coordinate system to the Data coordinate system.

g.toDataXCoord(56)

1

g.toDataYCoord(330)

0

Dygraphs Coordinate Systems, 1/3

This is the first of three posts about the coordinate systems used throughout Dygraphs.

• Part 1: Introduction and the Dygraph Data Coordinate System
• Part 2: DOM Coordinate System
• Part 3: Percentage Coordinate System

Introduction

This series of articles covers the three primary coordinate systems used in Dygraphs. They are: data coordinates, DOM coordinates, and percentage coordinates.

Each has its own distinct rules about origins and directions.

The entire series is based on this graph:

which is generated by this snippet of code:

new Dygraph(document.getElementById("graph"),

    [

      [1, [10,  10, 100]],

      [2, [15,  20, 110]],

      [3, [10,  30, 100]],

      [4, [15,  40, 110]],

      [5, [10, 120, 100]],

      [6, [15,  50, 110]],

      [7, [10,  70, 100]],

      [8, [15,  90, 110]],

      [9, [10,  50, 100]]

     ], {

       width: 500,

       height: 350,

       customBars: true,

       errorBars: true

     });

Dygraph Data coordinate system

The most obvious coordinate system to dygraphs is its inherent data coordinate system. The best way to demonstrate that coordinate system is through an example:

This highlighted point can be represented by the data coordinate system (4, 40). We’re even aided by the display on top showing the currently selected point.

Not to put too fine a point on it, but the x-value is 4, and the y-value is 40.

Graph data is always specified in the data coordinate system, and dygraphs does the job of translating those into points on the canvas. This should be no surprise to you, reading the code above shows the middle value for x=4 is 40.

This doesn’t always mean that the origin is in the lower-left corner of the graph area, since a graph can be resized and moved, but values almost always increase up and to the right.#

Before moving on to the next coordinate system, let’s play with the graph and its API. You can skip this section if you’re already comfortable with pulling values out of the graph.

Sidebar: Extremes and Visual Ranges

Extremes

Dygraphs provides a function that describes the extremes, or minimum and maximum data values, of the x-axis with xAxisExtremes:

g.xAxisExtremes()

[1, 9]

There is currently no yAxisExtremes function but Dygraphs an open source project; contributions are always welcome.

Visible Ranges

xAxisRange shows the visible data range along the x axis:

g.xAxisRange()

[1, 9]

Similarly yAxisRange shows the visible data range along the specified y axis:

g.yAxisRange()

[0, 131]

You’re wondering why the y axis range ends at 131 and not 120. This is because Dygraphs automatically adds a buffer to the top of most graphs’ y ranges for improved readability.
If you want the range to be precise, you can use the valueRange option with g.updateOptions({valueRange: [0, 120]}):

(Note there is currently a rendering bug where setting the value to 120 does something funny. This graph was rendered with a value range of 0-120.1.)

Remember these are visual ranges and not extremes of the data. So panning a graph along the x axis would result in a change in the x-axis range, but not x-axis extremes.

Dygraphs and IE9

(This is a retread of a mailing list post from earlier this year Dan wrote, but it's so interesting I wanted to mildly edit it and repost it here. In fact, this very post was the motivation for finally starting this blog. -Rob)

---

This is a somewhat lengthy update on the current state-of-the-art when it comes to dygraphs and Internet Explorer. If you just want something to copy and paste to make dygraphs work in all versions of IE, here it is:

<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="X-UA-Compatible" content="IE=EmulateIE7; IE=EmulateIE9">
    <!--[if IE]>
    <script type="text/javascript" src="path/to/excanvas.js"></script>
    <![endif]-->

The dygraphs home page and all tests now use this prologue. Here's the sequence that led to it:

1. IE9's native <canvas> support is great.
2. IE9 requires an HTML5 doctype to enable its native <canvas> support.
3. An HTML5 doctype puts IE8 in standards mode, which breaks excanvas.
4. excanvas can be fixed, but it requires a hack and doesn't work as well as in quirks mode.
5. Using a very magical incantation, we can get the right combination of IE document modes.
6. As a result of these, it might be time to try out FlashCanvas for dygraphs on IE8.

Longer story...On March 14, Microsoft released IE9. The focus of IE9 is on HTML5 support, including a native implementation of the <canvas> tag. This was an exciting development because it meant that, for the first time ever, dygraphs could run natively on the most-recent versions of all the major browsers.

Because many sites rely on quirks in older versions of IE, IE9 will, by default, render all pages in "Quirks mode". This is basically IE5.5 and it most certainly does not feature a native canvas tag.

To tell IE9 that you're quirk-free and would like all the new HTML5 features, you need to preface your page with an HTML5 doctype:

<!DOCTYPE html>
<html>
...
</html>

All of the dygraphs documentation and test pages use this doctype.

This doctype is also recognized by previous versions of Internet Explorer and puts them into Standards mode, rather than Quirks mode. Unfortunately, IE8 standards mode changed some of the behavior of VML, which is what excanvas used to emulate the <canvas> tag.

At this point, there are three different directions to go:

1. Get IE to display in standards mode in IE9 but not in IE8.
2. Fix excanvas.js to work in IE8 standards mode.
3. Switch to a different <canvas> emulation system.

Let's look at each of these:

1. Get IE to display in standards mode in IE9 but not in IE8.

This would normally be done via a doctype, however, dygraphs is very much cutting against the grain by wanting IE9 standards mode but specifically wanting to avoid IE8 standards mode. There are a variety of sources of information about how to set the document mode from a web page, for instance:

• http://ieblog.members.winisp.net/misc/How%20IE9%20Determines%20Document%20Mode.svg
• http://ieblog.members.winisp.net/images/MarcSil_IE8_Document_Mode_2.png
• http://msdn.microsoft.com/en-us/library/ms537512(v=VS.85).aspx
• http://blogs.msdn.com/b/ie/archive/2010/03/02/how-ie8-determines-document-mode.aspx
• http://msdn.microsoft.com/en-us/library/cc288325(VS.85).aspx
• http://hsivonen.iki.fi/doctype/

Many of these resources are either incorrect, incomplete or outdated. I tried many different combinations of doctypes, conditional comments and <meta> tags before arriving at the magical incantation at the start of this email. I'm not entirely sure why it works, but it does seem to. It's fragile — if you reverse the order of the "IE=" clauses, it will break. MS typically goes to great lengths to avoid breaking existing behaviors, so if it works now, I believe that it will continue to work.

Just to be clear
- The <!DOCTYPE html> before the HTML suffices to get the native <canvas> tag in IE9.
- If you want to support IE8, you'll need the <meta> tag and the conditional excanvas include.
- If you only need to support IE7 and earlier, the <meta> tag is optional.

The snippet at the top of this email will always work, so you may as well just include it if you're going to support IE<9. The one quirk is that it will cause IE8 to render your page in IE7 standards mode. This isn't too big a deal, however, as relatively little changed between these two.

2. Fix excanvas.js to work in IE8 standards mode.

Microsoft changed quite a few things about VML in IE8, much to the consternation of VML users:
http://ajaxian.com/archives/the-vml-changes-in-ie-8

The update appears to have broken excanvas under some circumstances when used in IE8 standards mode. Dygraphs is one of these circumstances.

Reading around, I noticed this:
http://stackoverflow.com/questions/941170/does-anyone-could-make-excanvas-work-in-ie-8-with-jquery/1740754#1740754

As Ronald B. suggests, commenting out one line in excanvas.js un-breaks it in IE8 standards mode:
// el.style.overflow = 'hidden';  // fix IE8

However, as the blogger complains, VML in IE8 standards mode doesn't work as well as VML in IE7 standards mode or Quirks mode. You're better off dropping into IE7 mode.

3. Switch to a different <canvas> emulation system.

FlashCanvas (http://flashcanvas.net/) is a drop-in replacement for excanvas. Instead of relying on VML to work in older versions of IE, it uses Flash. This has a couple of consequences:
- it's faster
- it's less subject to the whims of IE
- flash and the DOM can get out of sync, leading to odd animations when panning.

It's also being actively maintained, unlike excanvas (which hasn't been updated in over a year). My copy of IE8 is a little bit janky, so it's hard for me to tell exactly how well it works. The dygraphs.com home page doesn't load for me when using it, for example. That's why I'd appreciate some feedback from the community about the relative merits of excanvas and FlashCanvas.

Dygraphs has an Annotated Range Selector!

Dygraphs now has an annotated range selector, thanks to an external contribution by Paul Felix. Inspired by Google's Annotated Time Line and other time series chart tools, you can now have easy zooming and panning of the data along the X axis. The widget is displayed at the bottom of the graph when you set the option showRangeSelector to true. Here is a simple example:

  new Dygraph(
    document.getElementById("graphdiv"),
    data,
    {showRangeSelector: true}
  );

Some other notes:

• Initially, the entire X range is shown.
• Zooming is performed by dragging one of the two zoom handles.
• Once zoomed in, panning is performed by dragging the section between the zoom handles.
• A mini plot of the data is displayed in the range selector.
• For multiple series, the mini plot is an average of the series values.

Additional range selector options are described in the Interactive Elements section of the dygraphs options reference.

Hello, World!

What's the smallest HTML/Javascript snippet required to make a Dygraph? First, you need a div element to host your graph:

<div id="hello-world"></div>

And then you create the graph. A dygraph requires two parameters:

• The document element created for the graph
• A description of the data to be graphed

Additional options may be specified in a third parameter (though it's not required for the demo.)

There are several documented data formats; for this demo we're going to use multi-line text:

<script>
new Dygraph(
    document.getElementById("hello-world"),
    "Date,Hellos,Worlds\n" +
    "2011/10/01,250,280\n" +
    "2011/10/05,260,295\n" +
    "2011/10/09,400,240\n" +
    "2011/10/13,225,325");
</script>

Et voilà!