More OVER Function Examples: Multiple columns, Intersect function, and LeBron James

The OVER function is a very useful function in Spotfire that calculates expressions and aggregations across different groups of data. I’ve previously written about the OVER function, which you can check out here. This post has additional examples using the OVER function, focusing on multiple columns and expressions requiring the Intersect function.

1-Lebron_James_Scoring_Analysis_with_TIBCO_Spotfire

Click here to download the Spotfire file shown in these examples!!

OVER Statements with Multiple columns
Often times you’ll want to evaluate an aggregation or expression over just a subset of your data, and sometimes that subset is defined by multiple columns. For example: “Show me average production during the last 3 months for each region we operate.” This is where the OVER function comes in.

When writing custom a custom expression with the OVER function, in most cases you can list the columns in parentheses behind the OVER function (eg: Average([Production]) OVER [Region]). However, in some cases you’re required to use the Intersect function, which I describe a little further down the page.

I’ve listed several examples below to demonstrate these different methods.

Compare well performance based on completion parameters
Let’s say we’re looking at completion data for a group of oil wells. These wells are geographically located in different Fields (North, South, and West), and each well is completed using either Completion Type A or Completion Type B (Note: “Completion” in this context refers to the processes and operations required after drilling an oil / gas well to start producing the well. It’s the 2nd of 3 main phases: 1. Drilling > 2. Completion > 3. Production). A common performance metric is the 90 Day IP. This is the average oil production during the first 90 days of production history.

Here’s what the data looks like:

2_-_completions_data

We want to know: Does the Completion Type and geographic location (Field) affect the 90 Day IP?

To help answer this question, I’ll average the 90 day IP rate for each combination of Field and Completion Type. In this data, there are 3 Fields and 2 Completion Types, therefore we should end up with 6 unique values of Avg IP:

  • Avg IP for Completion Type A in the North Field
  • Avg IP for Completion Type A in the South Field
  • Avg IP for Completion Type A in the West Field
  • Avg IP for Completion Type B in the North Field
  • Avg IP for Completion Type B in the South Field
  • Avg IP for Completion Type B in the West Field

Here’s the custom expression to calculate this, inserted as a calculated column (Insert > Calculated Column):

Avg([90 Day IP]) OVER ([Completion Type], [Field])

Make sure to include the parentheses around the column names after the word OVER – if not you’ll get a strange error message.

Here’s the data after adding the new calculated column:

3-Completsion_Data_with_Calc_Column

Let’s take a closer look at this and manually calculate a few of the values to double-check the accuracy. To calculate Avg IP for Completion Type A in the North Field, let’s find well records (Well ID) that used Completion Type A and are located in the North Field (highlighted below):

4-First_3_records

Next, just take the average of the IP rate for these 3 values:

5-Showing_OVER_Calc

The next group to evaluate would include the rest of the wells in the North Field – those using Completion Type B:

6-Showing_OVER_Calc2

Of course, I could keep going with Completion Type A in the South Field, but we’ll let Spotfire do the math from here!

Summarizing all of the data into the 6 unique values in the calculated column:

  • Avg IP for Completion Type A in the North Field = 1,643
  • Avg IP for Completion Type A in the South Field = 2,157
  • Avg IP for Completion Type A in the West Field = 2,853
  • Avg IP for Completion Type B in the North Field = 1,992
  • Avg IP for Completion Type B in the South Field = 2,451
  • Avg IP for Completion Type B in the West Field = 3,174

We could get the same result (without inserting a calculated column) by using a Cross Table or Bar Chart (or some other type of visualization) as shown below. Thanks to the Marking feature in Spotfire, it’s really quick to check the calculated column by simply clicking on rows in the Cross Table or Bars in the chart:

8-Cross_Table_and_Bar_Chart_Spotfire

Comparing LeBron James’ Regular and Post Season Performance
If you’re not in oil and gas or that data doesn’t make sense to you, let’s look at a sports example. Since the NBA playoffs are going on right now, I’ll choose basketball.

LeBron James has played 13 NBA seasons with 2 different teams (Cleveland Cavaliers & Miami Heat) and has made 11 playoff appearances. Let’s say we want to compare James’ average scoring per game for each part of the Season (Regular or Playoffs).

The average scoring per game for each season has already been calculated for us, so we don’t have to worry about that. That leaves 2 groups of data that we want to compare (Team and Season), and each group has 2 unique values, for a total of 4 combinations. We should end up with 4 unique values in our calculated column:

  • Avg scoring for Cleveland during Regular Season
  • Avg scoring for Cleveland during Playoffs
  • Avg scoring for Miami during Regular Season
  • Avg scoring for Miami during Playoffs

Heres’ the data:

9-LeBron_Scoring_Data

Because we want to compare James’ performance for each Team and Type of season, we need to use both of these columns in the calculated column. Here’s the expression:

Avg([Avg Points per Game]) OVER ([Team], [Season Type])

The result:

10-LeBron_Scoring_Data_-_with_calc_column

Summarizing this into the 4 unique values mentioned above:

  • Avg scoring for Cleveland during Regular Season = 27.3 pts
  • Avg scoring for Cleveland during Playoffs = 28.9 pts
  • Avg scoring for Miami during Regular Season = 26.9 pts
  • Avg scoring for Miami during Playoffs = 26.8 pts

We could get the same result (without inserting a calculated column) by using a Cross Table:

11-Lebron_James_scoring_data_in_Cross_Table_Spotfire

And once again, it’s really quick to check the calculated column by simply Marking rows in the Cross Table:

12-calculated_column_check_with_cross_table-1 13-calculated_column_check_with_cross_table-2
14-calculated_column_check_with_cross_table-3 15-calculated_column_check_with_cross_table-4

The Intersect Function – Example 1

Now I’ll add a layer of complexity to our problem by adding the element of time. For example, what is LeBron’s regular season average points per game to date (including the current year)? In other words, what’s his historical running average?

Let’s say we’re evaluating this historical-to-date average for the 2008-09 season. To calculate that by hand, we’d average values during the regular season between for seasons between 2003-2008:

(20.9 + 27.2 + 31.4 + 27.3 + 30.0 + 28.4) / 6 = 27.5

16-Avg_points_per_game_to_date [16]

Thankfully Spotfire has a built-in function that does this for us. It’s called AllPrevious. To calculate the average points per game to date, use the following expression:

Avg([Avg Points per Game]) OVER Intersect([Season Type], AllPrevious([Season]))

You’ll notice that writing this expression is very similar to the previous one, the only catch is that we must use the Intersect function with AllPrevious.

This is because for a given row’s calculation, AllPrevious is not using all unique values from the Season column – only values that are less than or equal to the current season. So to calculate the average to date, Spotfire takes the average only when Season type (regular or playoffs) intersects with all previous seasons for a given row.

Without Intersect, you’ll get the error message: “This expression is not valid”.

Here’s a list of functions that require Intersect when combined with the OVER function. Note that Intersect is only required when there are 2 or more columns after the OVER function.

  • All
  • AllNext
  • AllPrevious
  • FirstNode
  • LastNode
  • LastPeriods
  • NavigatePeriod
  • Next
  • NextPeriod
  • Parent
  • Previous
  • PreviousPeriod

The Intersect Function – Example 2

Here’s another example of something we might want to evaluate: Was LeBron’s 28.4 PPG average scoring for the 2008-09 regular season better or worse than the average of all previous seasons (not including the 2008-09)?

AllPrevious does include the current row in the calculation (in our example above, the Avg PPG for the 2008-09 season). It gets a little trickier if we want to exclude this value from the calculation:

(Sum([Avg Points per Game]) OVER Intersect([Season Type], AllPrevious([Season])) – [Avg Points per Game]) / (Count([Season]) OVER Intersect([Season Type], AllPrevious([Season])) – 1)

We can check the result by comparing it to the previous season’s average to date:

17-Last_Seasons_Avg_points_per_game

Another way to write that complex expression, since we already calculated Last Seasons Avg Points per Game to Date, is to just use the Previous function:

Avg([Avg Points per Game to Date]) OVER Intersect([Season Type], Previous([Season]))

This will take the value from the Previous Season in the Avg Points per Game to Date column. This also demonstrates that sometimes its easier to do a calculation in 2 steps with 2 calculated columns, instead of 1 column with a complex expression.

Here’s the result, which we can verify matches the other calculated column with the more complex expression:

18-comparison_of_2_calc_columns

With this new column, Last Seasons Avg Points per Game to Date, now we can see for a given season whether his performance improved or not compared to the average of all previous seasons:

[Avg Points per Game] – [Last Seasons Avg Points per Game to Date]

19-Current_Year_Over_Under_Compared_to_Last_year

Looking at the data, we can see that LeBron’s scoring average started declining in the 2010-11 regular season when he moved to the Miami Heat. His playoff performance compared to previous years is up and down over time. This is more easily visualized with a Line Chart or Bar Chart, trellised by Season Type, as below:

20-Lebron_James_Current_Year_Scoring_average_PPG_compared_to_historical_avg

Questions / Comments
Hopefully these examples have helped you understand how to use the OVER function with multiple columns and the Intersect function. If you have any questions or comments, or if there’s something you’d like featured on this blog, feel free to email me at kyle@datafuel.co or leave a comment below.

Posted in Text and tagged , , , , , , , , , , , .

7 Comments

  1. I am trying to calculate the 90 day IP for oil for oil production. Can anyone help me in the right direction with the custom expression to do this?

  2. I have a question.
    I need sum of all -ve values and +ve values for a customer column/ product column distributed as based on 12 months columns calculated.
    Customer/Product/1/2/3/4/months columns.
    How many have sold from stock in a month is +ve values and left over are -ve values.
    Can I know like how to put this in SPF.

    • Whoops, forgot to embed the data in Spotfire. Thanks for pointing that out!! Just fixed it – you should be able to access that link again and get a new Zip folder with the updated Spotfire file with embedded data.

  3. The sample data file (LeBron James Scoring Data.xlsx) is missing from the zipped file. Can you please share that as well?

Leave a Reply

Your email address will not be published.