Advanced Tips And Tricks For Power Query

Chris Webbchris@crossjoin.co.uk

@technitrain

Who Am I?

• Chris Webb• chris@crossjoin.co.uk• Twitter @Technitrain

• UK-based consultant and trainer: • www.crossjoin.co.uk • www.technitrain.com

• Author of several books: • MDX Solutions• Expert Cube Development with SSAS 2008• Analysis Services 2012: The BISM Tabular Model• Power Query for Power BI and Excel

• SQL Server MVP• Blogger: http://blog.crossjoin.co.uk/

Agenda

• Power Query recap• Introduction to M• Parameterised queries• Privacy levels and the Formula Firewall• Query folding and performance• Creating and using M functions• Combining data from multiple data sources

Power Query Recap

• Power Query is a self-service data manipulation tool• Excel add-in for Excel 2010/2013• Native feature of Excel 2016 as “Get & Transform”• The “Get Data” part of Power BI Desktop• Integrated with SSIS, SSRS and SSAS at some point?

• It does three things:• Connects to multiple data sources• Applies transformations to that data• Loads data to Excel tables and/or the Data Model

• New builds of Power Query released almost every month

M

• Power Query is actually a UI on a language (unofficially) called M• M is a functional language like F#

• Nothing like Excel formula language, DAX or VBA

• Everything you do in the UI is translated to M• The M code is visible in the Formula Bar• You can edit the whole M script for a query in the Advanced Editor

• Understanding how to write M is the key to writing more complex transformations and calculations

Values and Expressions

• Expressions are central to the M language• Expressions can be calculated to return Values

• 1+1 is an expression• 1+1 returns the value 2

• Values can be:• Primitives like numbers or text• Tables, lists, records, functions…

• Each Power Query query is a single expression that returns a single value

Let Expressions

• Each Power Query query created by the user interface consists of a single Let expression

• Let expressions contain a series of named expressions that can reference other named expressions• These become the steps in a query

• The Let expression finishes with an In, where the final value of the expression is calculated• This is the output of the query• It can reference any of the named expressions

Parameterised Queries

• For large data sources, users should be prevented from downloading all data to Excel• Takes too long• May crash Excel

• Therefore you need to give users an easy way to select the data they want to download

• This involves• Giving them an easy-to-use UI• Writing M code to handle the parameterization

Parameterised Queries

• Power Query queries can reference other Power Query queries• Therefore you can use Power Query to load data on what to filter by

from an Excel table or named range• …and pass that data as a parameter to a second Power Query query

to load the main data set

Query Folding

• Fast data load performance is very important for users• For certain data sources (relational databases , OData and a few

others) Power Query can push logic back to the data source• Eg translate your Power Query transformations to SQL

• This is called Query Folding and can make a massive difference to performance

Query Folding

• There is no indication in the UI that Query Folding is happening• You need to use tools like SQL Server Profiler

• Certain operations prevent folding, such as using native SQL queries• A query may be partially folded – reordering steps can increase the

amount that is folded

Privacy Levels And The Formula Firewall• Power Query cares about data privacy!• Sending data from one source as parameter values to another source

could be dangerous• Power Query will prompt you to set data privacy levels to govern

whether this can happen• Turning on ‘Fast Combine’ stops Power Query from checking this• But this will need to be turned on every time the workbook is opened on a

new PC

M Functions

• Functions are expressions that take parameters to return a value• The Standard Library contains all of the built-in functions• You can define your own functions in the Advanced Editor• They can then be invoked in • Within the query you’re writing• Other queries in the same workbook

• Functions are a great way of:• Sharing logic between queries• Separating complex logic from otherwise simple queries• Controlling how end-users access data sources

Functions For Data Access

• M functions can be an alternative to parameterised queries• Use M functions when an end user needs to create multiple queries

against the same data source to get different slices of data• With more advanced M code you can provide:• Default values for parameters• A list of available values that can be selected from a dropdown box

Combining Data From Multiple Sources• Power Query makes it easy to combine data from multiple, identically

structured csv files• However end users often need to do the same thing with other data

sources, for example Excel files• The solution is to create an M function to do this

Thanks!