Gain function of a filter

In version 1.2.1 of ssci.js I added a gain and phase shift function to the ssci.smooth.filter() function.

In this post I’ll go through what a gain function is and give an example of it. According to the ABS:

Gain functions can be used to look at the effect of a linear filter at a given frequency on the amplitude of a cycle in a given series. In other words, it shows what happens to the amplitude of a given input cycle on the application of a moving average.

The function I have defined takes a single value, the period, and returns the factor being applied by the filter to that period.

Therefore if we have a filter defined as:
var orp = ssci.smooth.filter()
.filter(ssci.smooth.henderson(13));
orp();

Then we can use the following to get the gain function:
var gout=[];
for(var i=1; i<20; i=i+0.1){
gout.push([i, orp.gain(i)]);
}

For periods from 1 to 20. Charting this gives the following:

From the chart, you can tell that cycles of between 1.2 and 6 periods are effectively removed from the data. There is a gradual increase from 6 to 14 where the gain goes above 0.9.

You can also see that if you only want to look at the gain of a filter you can do this without specifying any data, accessor functions or anything else.

More information (including the formulae) is on the ABS website. Source code is on github.






Asymmetric filters on time series data in JavaScript

I recently made a change to the ssci.smooth.filter() function in the ssci.js JavaScript library. Here is a lengthier explanation of the change.

One of the changes I’ve made is to add the ability to use asymmetric filters with this function. This is achieved via a setter function that defines the start and end index of the points to apply the filter with reference to the point being adjusted.

To give a concrete example, if we call the point being adjusted ‘n’ and the start is two points before this and the end is two points after then we would set this via:

var example = ssci.smooth.filter()
.data(data)
.filter([0.2,0.2,0.2,0.2,0.2])
.limits([-2,2]);

This is still a symmetric filter and is the default if you don’t use the limits() setter function given the filter used in the example.

However if you had quarterly data and wanted to take a moving average over the year you can now do this via the method used above. This time you will need to use:

var example = ssci.smooth.filter()
.data(data)
.filter([0.25,0.25,0.25,0.25])
.limits([-3,0]);

You can also difference the data using this method:

var example = ssci.smooth.filter()
.data(data)
.filter([-1,1])
.limits([-1,0]);

This is just taking the current point, multiplying it by 1 and subtracting the point before from it.

An explanation of the function can be found here. The source code is here.

Henderson Filters in JavaScript

I wrote in a recent post that I’d added a function to calculate Henderson filters to the ssci.js library. This post will expand on that (slightly).

To quote the Australian Bureau of Statistics:

Henderson filters were derived by Robert Henderson in 1916 for use in actuarial applications. They are trend filters, commonly used in time series analysis to smooth seasonally adjusted estimates in order to generate a trend estimate. They are used in preference to simpler moving averages because they can reproduce polynomials of up to degree 3, thereby capturing trend turning points.

The filters themselves have an odd number of terms and can be generated for sequences of 3 terms and more. To use the function contained within the JavaScript library you can just use:

ssci.smooth.henderson(term)

where term is the number of terms you want to return. These are returned as an array. So if term was 5 you would get:
[
-0.07342657342657342,
0.2937062937062937,
0.5594405594405595,
0.2937062937062937,
-0.07342657342657342
]

The equation to generate the filters was taken from here.

To actually filter a set of data this would be combined with the ssci.smooth.filter() function i.e.:

var ft = ssci.smooth.filter()
.filter(ssci.smooth.henderson(23))
.data(data);

And here’s an example using the data from the kernel smoothing post.






Change to Kernel Smoother

Introduction

I recently changed a whole load of the functions in the ssci javascript library. One of these changes was the way that the kernel smoothing function worked.

The previous function was fairly slow and didn’t scale particularly well. In fact, the main loop of the function would suggest that it scales as O(n^2).

I therefore decided to make a change to the function. Instead of looping over every point and then calculating the weight at every other point, I’ve changed it so that:

  • It loops through every point similarly to the previous function
  • Then for point n it calculates the weight at the central point (i.e. point n)
  • It then loops through every point lower (in the array) than n and calculates the weight. If the weight of this point is lower than a certain threshold to the central point, then the loop ends.
  • It then loops through every point higher (in the array) than n and calculates the weight. If the weight of this point is lower than a certain threshold to the central point, then the loop ends.

The default setting of the threshold is 0.001 (i.e. 0.1%). The way the function operates though does mean two assumptions have been made.

  • The data has already been ordered in the x-coordinate within the array.
  • The kernel function being used must always decrease from the central point beyond this threshold.

Example

Issue #43 of D3 shape gives an example of some data that has been smoothed. It’s the observed rate of tweets for Eric Fischer’s Twitter feed over the period from January 8, 2015 through November 30, 2015. I’ve taken the data and charted it below.

Performance

I’ve put together some figures detailing the performance of the various routines on my computer. To get the times of the function and how it scaled I used the following:

var iterations = 10;
for(var j=0; j<10; j++){
var temp2 = temp1.slice(j*temp1.length/10);

console.time('Function #1');
for(var i = 0; i < iterations; i++ ){
ssci.smoothKernel("Gaussian", temp2, 2000000);
};
console.timeEnd('Function #1')

console.time('Function #2');
for(var i = 0; i < iterations; i++ ){
var data4 = ssci.smoothKernel2()
.scale(2000000)
.kernel("Gaussian")
.diff(0.001)
.data(temp2);

data4();
};
console.timeEnd('Function #2')
}

In the code above temp1 is the array holding the points of data. The code basically takes the data, chops it into progressively smaller arrays and then runs the function 10 times.

This leads to the following times (in milliseconds) per run i,e, divided by ten:

Rows #1 (ms) #2 (ms) #1/#2
144 10 10 0.94
288 38 26 1.45
432 84 42 1.99
576 149 58 2.55
720 234 76 3.08
864 336 91 3.69
1008 457 107 4.26
1152 597 124 4.82
1296 756 140 5.41
1440 949 160 5.95

Let’s plot these numbers.

The differences are anything from the same to six times faster. With more data it becomes even faster than the corresponding O(n^2) function. It would appear to scale linearly.

Of course there is also the question of what this change does to the accuracy of the smoothing. It may be a slightly odd thing to say given that no smoothing algorithm can be said to be accurate but I’m looking at the differences here.

As you can see from the graph the differences in this example are of the order of 0.004% with a maximum of 0.03%. I think we can live with that.

The new function can be found on www.surveyscience.co.uk







Updated Holt-Winters function

Due to the complexity of the Holt-Winters function I’ve changed the way it works in the JavaScript Library.

Now instead of passing arguments to the function you chain functions to the Holt-Winters object.

So to give a concrete example, let’s take the monthly UK fish landing data from the ONS. I’ve stitched data from 2008 to 2015 together and filtered on just the amount landed in the UK by both UK and foreign vessels.

To perform the Holt-Winters exponential smoothing we first create a foreHoltWinter() object. Then we pass the parameters and data to the object via some methods/functions.

var fhw = ssci.fore.holtWinter()
                    .data(data)
                    .period(12)
                    .level(.14)
                    .trend(0)
                    .season(0);

After this the function needs to be called:

fhw.call();

To get the output you can then use:

fhw.output();

This will return an array of forecast points. To forecast further you use the forecast(value) function where value is the number of time periods ahead that you wish to forecast for. More detail is on the page for the function itself, including the other functions that can be called to control more aspects of the analysis.

Putting this all together, here is an example using the data specified previously. I’ve added the forecast data on this chart and predicted forwards for 12 months.






SurveyScience JavaScript Library

I’ve been converting some of the functions from the Excel add-in to Javascript functions within a library. It’s finally ready for release and can be found here.

It contains functions to:

  • Smooth data
  • Deseasonalise data
  • Perform least squares regression
  • Exponential smoothing
  • Create auto-correlation and partial auto-correlation plots
  • Market research functions
  • Utility functions to modify arrays

There’s still some work to do on it but it can be used as is – go to the above link for more details.

I originally started it to created some smoothed lines for some D3 charts.

Hopefully it will prove useful.