Tuesday, September 9, 2014

PyDyGraphs, a dygraphs plotting module for IPython Notebooks

(Guest post by Zach Dischner)

PyDyGraphs on Github (mostly written by Kevin Dinkel)

The IPython Notebook is an excellent tool for interacting with Python and displaying data in real time. While tools like matplotlib can generate static plots in an IPython Notebook, we often find their plots to be slow to render and the minimal interactivity limits their usefulness in practice. Being a long time fan of dygraphs, we wrote a module to render the JavaScript plots in the Notebook. This module can be readily included in an IPython Notebook and then used to generate beautiful, fast, and interactive time series plots with dygraphs, all stored locally with no need for remote hosting.

Screen Shot 2014-09-08 at 11.06.55 AM.png

PyDyGraphs is far from mature, but what proved to be a simple effort of integrating the DyGraphs library into the Notebook environment has already been enormously useful and fun to use. The whole thing only took a day or two to write, and has become my new go-to plotting tool for time-series data.

Purpose


Originally, PyDyGraphs was meant as a Pandas Dataframe plotter. The Dataframe is a powerful data structure, and the lack of an adequate visualization tool lead to the creation of the PyDyGraphs plotting module. The Dataframe is already a relational data structure, and is easily provided to Dygraphs for visualization. Support for typical time series arrays has been added, but those simply get translated to DataFrames along the way.

I am one of the lead software developers and main instrument operator on the BRRISON/BOPPS high altitude balloon mission. With flight times on balloon missions potentially being limited to hours, immediate and intuitive interpretation of flight telemetry is absolutely essential in order to make the fast decisions needed to maximize achievement of science goals. We found that matplotlib figures were simply too slow and too static for our needs. Hence PyDyGraphs! We created the module to follow the same basic behavior as matplotlib, but give you much of the awesome functionality of dygraphs.

Enough words, here are some examples.

2x2 subplot




Similarly 3x3 Plot


How it Works


At its core, this module creates a JavaScript string for use within an IPython Notebook. It takes advantage of the Notebook’s built in “Kernel” and “Callback” JavaScript interactivity tools to get data between Python and JavaScript. We based it off of this SO question (thanks danvk!)




Conclusion

PyDyGraphs is a simple, easy to use wrapper of the epic dygraphs plotting library. We are using it for personal and professional projects, and are just plain stoked about being able to mix Python/dygraphs.

For more examples, check out our github: https://github.com/DayStarEngineering/PyDyGraphs

There are plenty of todos and ideas, so feel free to fork and contribute.

Monday, June 23, 2014

dygraphs as an NPM package

dygraphs is now available as an NPM package.
NPM is the NodeJS package manager. It works well enough that it's used for all sorts of packaging, even when NodeJS isn't involved.

To install dygraphs via NPM, simply run:
npm install dygraphs
This will install dygraphs under a node_modules directory, for example:
ls -lh node_modules/dygraphs/dygraph-combined.js 
Once you've got it installed, you can source it from that location in your web page, web server, Python program, whatever.

If you create a package.json file, you can specify a specific version of dygraphs that you'd like to depend on. That way, you can keep getting a compatible version in the 1.x series, even after we move on to 2.x.

Wednesday, August 14, 2013

Announcing dygraphs 1.0.0!

Today is a big day for dygraphs. With a controlled set of scripts and procedures, we are moving dygraphs to controlled releases, and with that, we’re proud to announce dygraphs version 1.0.0!

While this means that you’ll still have access to the very latest source, there will always be permanent, stable resources available for 1.0.0:
That means the next bug fix release will be version 1.0.1, the next feature release will be 1.1.0, and who knows what 2.0.0 will look like?

The master branch will continue to have the latest development code.

Brief History of dygraphs

For its first official "release", let's take a quick trip down dygraphs memory lane.

Dan originally created dygraphs in 2006 to serve his team's internal dashboard at Google. It was based on PlotKit, by Alastair Tse. (Curiously, Dan and Alastair would share an office in 2012 before discovering this connection.) Even now, dygraphs is often used for internal dashboards, a reflection of this original use case. Back in 2006, Firefox 1.5 and Safari were the only browsers which supported <canvas>. Given their market shares, Dan didn't see much value in releasing dygraphs publicly.

In 2009, Dan found that dygraphs was useful to his new group at Google for something other than an internal dashboard. So he decided to open source it! The initial commit was to Google Code, but the project moved to github shortly thereafter. dygraphs gained some initial attention at this time via a Hacker News post.

In 2010, Robert started working on dygraphs, contributing several major features and greatly improving the community. You can thank him for the dygraphs tests suite, the dygraphs gallery and this blog!

In 2012, Klaus began contributing to dygraphs and encouraged us to move towards a plugin model. As we did this, it became increasingly clear that we needed a way to deprecate certain parts of the API. And that's how we've arrived at version 1.0.0!

Why Do Release Branches

Here are the benefits of creating release branches:
  • It will allow us to add powerful new features: We have lots of features we want to add, but some of them are hard to do with some of our archaic features, or will break existing APIs, which leads to the next benefit.
  • It will allow us to break backward-compatibility: there are some old options, API and behaviors that we would like to remove for the sake of cleanliness. We would also like to change how charts appear in some ways that might frustrate users who rely on the existing rendering. We also want to drop support for excanvas and Internet Explorer earlier than IE9 (which added native support for <canvas>). 
  • It will allow you to have reliable, consistently tested, releases: Someone who relies on a specific version doesn’t want to have to know a specific git commit SHA, nor should a user always have to suffer when new patches are released that may break backward compatibility.
  • It will help turn plug-ins to a first-class feature. Plug-ins can be a powerful feature, again, if we break backward compatibility.
Here’s what won’t change: 
  • Easy configuration: The Hello, World! of dygraphs is pretty darn simple, and that’s how we want it to stay.
  • Rendering large data sets quickly: This has been an extremely powerful part of dygraphs, and it has to stay that way. In fact, we have a change pending that will improve performance by 100%. There’s plenty of things to work out: deployment strategy, testing strategy, git branching strategy, refactoring for the plug-ins architecture, and so on.
  • Quick support on the mailing list: We enjoy making sure dygraphs works for you, and really enjoy the people in the community growing around it.

Waaaait a minute. Wasn't dygraphs at version 1.2 for, like, forever?

Yeah, we feel sheepishly guilty about that. But we're moving to an active versioning numbering scheme with three number identifiers, so while it may seem like we'll release another 1.2 one day, in fact it will be 1.2.0, and before too long we'll have 1.2.1, 1.3.0, and on into the future, and this whole "1.2" thing will be a distant memory.

Want to help craft the future of dygraphs?

Here’s what we need from you: are you interested in helping us refactor dygraphs? Do you like writing tests? Is there a feature you're dying to implement? Do you have opinions about the plug-in architecture? Then we want to speak with you. Drop us a note at dygraphs-users@googlegroups.com

Thank you

We wouldn’t have even considered building a new release if we felt it wouldn’t be a great benefit to all of you. Your support and feedback has been valuable and encouraging, so please keep it up.


Here’s Dan, hard at work releasing 1.0.0 for you!

Monday, December 31, 2012

Changes to the Experimental Palette

Hey Dygraphers,

I've been working on some bugs and features, and while at it, been pushing some nice new features into the experimental palette that makes it eminently more useful. Let's take a tour. You can play along at http://www.dygraphs.com/experimental/palette

1. Option Set drop-down
Using the Option Set dropdown on the palette, you can configure any axis or series that you like.

Go ahead and select line (series). You'll see a subset of the options that apply to individual series.



2. Move the line series to the second y-axis.

In the axis text box, type "y2" and press "enter" (or click Redraw, if you prefer.) The second axis will appear.

3. Configure the second y-axis.

The line is on the second y-axis, but it's not obvious. Let's make all series on the second y-axis stand out. Select y2 axis from the Option Set drop-down, and set drawPoints to true, and and pointSize to 5. Click Redraw. Presto!


4. View the options as a hash.
On the top-right of the palette is a link "to hash". Click it, and the options text box appears.


5. Change the options as a text object.

Sometimes tweaking via the object hash is the way to go. In the text box, change axisLineColor from white to green. Then click OK.

OK I didn't pick the best example because it's hard to see, but this axis is green.


Let's zoom in:
There you go! (hmm... why is there a little red in the axis?) Probably to help celebrate the christmas spirit. Happy New Year!

Tuesday, December 25, 2012

Dygraphs 2012 Year in Review


2012 was a busy year for Dygraphs.

We had almost 400 commits over the year (with a month to go, no less.)  And something you'll notice is that this year we had a _huge_ number of external contributors. So first and foremost, thank you to everybody who contributed to Dygraphs.
Robert Konigsberg: 175
Dan Vanderkam: 137
Klaus Weidner: 30
Adam Vartanian: 5
Uemit Seren: 5
Josep LlodrĂ : 4
David Moena: 3
Joshua: 3
Paul Felix: 3
Beda Kosata: 2
Jason Hollingsworth: 2
Wim Bruynooghe: 2
wimme: 2
David M Sibley: 1
David Watrous: 1
Helder: 1
James H Thompson: 1
Matt Miller: 1
Richard Klemm: 1
Russell Valentine: 1
bluthen: 1
mbghsource: 1
shole: 1
timeu: 1

Here's an overview of the major new features:
What a great year! See you in 2013.

Saturday, December 1, 2012

The New and Better Way to Specify Series and Axis Options

TL;DR: Put series inside the series option. If you want to put a series on the second y-axis, then inside that series specify axis : 'y2'. You can say that a series belongs on the first y-axis with axis : 'y', but that's also the default. This only applies inside the series option. Also, per-series options can also be specified in an axis, so it applies to every series on that axis.

In a recent post, I described the bizarre behavior required to make multiple axes work. Fortunately, because of the experience writing the post, I submitted a series of patches that rewrote the internals for handling options processing, and also improve the API.

Specifying a Series

Traditionally, per-series options were specified on the top-level of the options dictionary, like so:

{
  pointSize : 5,
  legend : 'always',
  T : {
    pointSize : 6
  }
}

This behavior is now discouraged. Instead you should put your per-series options in the top-level series option:

{
  pointSize : 5,
  legend : 'always',
  series : {
    T : {
      pointSize : 6
    }
  }
}

Now, if you want to deal with multiple axes, pay close attention, because things have changed.

It used to be that if you wanted to specify that a series was on the second y-axis, you did this:

T : {
  axis : { }
}

And then if you wanted to put another series on the second y-axis, you would reference the first series, like so:

S : {
  axis : 'T'
}

This behavior still works, until you put it inside the series option.

So let's say you had 

{
  pointSize : 5,
  legend : 'always',
  T : {
    axis : { }
  },
  S : {
    axis : 'T'
  }
}

and then you moved it into the series option:

{
  pointSize : 5,
  legend : 'always',
  series : {
    T : {
      axis : { }
    },
    S : {
      axis : 'T'
    }
  }
}
... the graph would fail to render, with the exception
"Using objects for axis specification is not supported inside the 'series' option."
That's because we did per-axis specifications right this time. All you have to do is directly specify the axis you want a series to appear on, like so:

{
  pointSize : 5,
  legend : 'always',
  series : {
    T : {
     axis : 'y2'
    },
    S : {
      axis : 'y2'
    }
  }
}


No more weird {}, and no more referencing other series.


Remember, behavior inside the series option is different from outside. The outside-series options have been effectively deprecated. However, all the old behavior has been preserved. so your old graphs will still work. It's different from the old way, but it's easier to use.

Specifying Per-series options on an axis

As I mentioned at the top, options parsing has been rewritten. This allowed for a nice small change, which is the rule in which options are found:
When searching for a per-series option, first see if it's specified in the series. If it's not there, see if there's an option for it on the axis. If it's not there look for a user-specified option, and then finally rely on a default value.
This implies the new rule for finding per-axis options:
When searching for a per-axis option, first see if it's specified in the axis. If it's not there, look for a user-specified option, and then finally, rely on a default value.
Make sense? Let's look at a demo:

{
  drawPoints : true,
  pointSize : 5,
  legend : 'always',
  series : {
    T : {
      axis : 'y2',
      pointSize : 10
    },
    S : {
      axis : 'y2'
    }
  },
  axes : {
    'y2' : {
      drawPointCallback: Dygraph.Circles.PENTAGON
    }
  }
});

In this case,  notice that series T and S are both on the y2 axis. The point size is overridden for T by being specified directly in T. However, both S and T have pentagons for points because of the drawPointCallback specification in axes.

Friday, November 23, 2012

Using the second y-axis

Short summary: Consider all the series you want on the right-side axis. Choose one as (for lack of a better term) the master series. For that series, add the option axis : {}. For the others, use axis : 'T', replacing T with the master series' name. Yes, we know this is confusing, and we have some ideas for fixing it.

Update: The behavior for specifying a second y-axis has been simplified. See this recent post.

One of the nice new features Dan added to Dygraphs was the ability to use a second y-axis. Unfortunately, we all agree the mechanism to specify using the second y-axis is a little confusing, so this post ought to make things easier to understand.

Take note, though, that we're working out a proposal to make using the second y-axis easier to understand, so please feel free to read and comment as you see fit. (Once we simplify the API for using the second y-axis, this post will be obsolete, but hopefully it'll be replaced by another one. (Remind me to add references to this post when that time comes.))

Demonstration: Here's a graph with three series (named R, S and T), all on the left-side y-axis.

To move one of the series to the right-side y-axis, you need to add an option for the series. For instance, let's move series T to the right-side:
T : {
  axis : { }
}

The short explanation of what this means is that it instructs Dygraphs to create a new axis, and assign T to it.

Which looks like this:

You might think that if you wanted to add S to the right axis you would add
S : {
  axis : { }
}
but if you did, you would get this error:

dygraphs: Only two y-axes are supported at this time. (Trying to use 3) (axes.js:65:7) 

Yeah, confusing. Instead, what you do is:
S : {
  axis : 'T'
}
This says that whatever axis T is on, put S there too.

Here's what it looks like: