The benefits of running specs against nightly releases of dependencies.

Spend some time and resources to set up your Continuous Integration infrastructure to run your spec suites against nightly releases of your dependencies. The benefits are larger than the costs.

Context

To further explain the point I will use an example from today.

We run our specs daily against the latest nightly release of BABYLON.js. On Friday one spec failed. I reported in the forum (not even a github issue). A few hours later there was a fix and PR merged with the main branch of BABYLON.js. We would have the new nightly in a day or two.

Our specs pass with version 4.2.0 of BABYLON.js, but they fail with BABYLON 5.0.0-alpha.6. A few of the hundred of extensions running in the Instructions Steps (IS) Framework are using BABYLON.js. The IS framework is powering the 3D building instructions at FLLCasts and BuildIn3D.

BABYLON.js provides two releases of their library.

  1. Stable – available on https://cdn.babylonjs.com/babylon.js
  2. Preview – available on https://preview.babylonjs.com/babylon.js

How do we run the specs against the preview (nightly) release of BABYLON.js?

We’ve configured Jenkins to do two builds. One is against the official release of BABYLON.js that we are using on production. The second run is against the preview release.

When there is a problem in our code both builds will fail. When there is an issue with the new version of BABYLON.js only the second build fails.

What is the benefit?

I think of the benefit as “being in the context’. Babylon team is working on a new feature or they are changing something. If we find an issue with this change six months later it would be much more difficult for them to switch context and resolve it. Probably there are already other changes. But when we as developers are in the “context”, when we are working on something and have made a change today and there is an issue with this change it is much easier to see where the problem is. You are in the same context.

The other large benefit is that when 5.0.0 is released we will know from day one that we support it and we can switch production to the new version. There are exactly 0 days for us to “migrate” to the new version.

How much does it cost us?

Basically – zero. The specs are run in under 60 seconds and the build is configured with a param.

What if there are API changes?

Yes, we can’t just run the same code if there are API changes in BABYLON.js. That’s why we have the branch. If there are API changes we can change our code in the babylon-5.0 branch and keep it up to date with changes in dev, which is most of the time resolved with a simple merge.

But BABYLON.js is a stable library. There are not many API changes that are happening. At least not in the API that we are using.

For fun

As you are here, here is one instruction

Large Spaceball from Geosmart Spaceball set in 3D