level up your biml: best practices and coding techniques (ntk 2016)
Post on 11-Jan-2017
165 Views
Preview:
TRANSCRIPT
Level Up Your Biml:Best Practices and Coding Techniques
Cathrine Wilhelmsen
Session Description
You already know how to use Biml to build a staging environment in an hour, so
let's dive straight into some of the more advanced features of Biml.
Attend this session for an overview of Biml best practices and coding techniques.
Learn how to centralize and reuse code with include files and the CallBimlScript
method. Make your code easier to read and write by utilizing LINQ (Language-
Integrated Queries). Share code between files by using Annotations and
ObjectTags. And finally, if standard Biml is not enough to solve your problems,
you can create your own C# helper classes and extension methods to implement
custom logic.
Start improving your code today and level up your Biml in no time!
Cathrine Wilhelmsen
@cathrinew
cathrinewilhelmsen.net
Data Warehouse Architect
Business Intelligence Developer
You…
Know basic Biml and BimlScript
Completed BimlScript.com lessons
Have created a staging environment
…?
Today…
Code Management
Practical Biml Programming
C# Classes and Methods
… :)
Quick Recap of
Basic Biml
What is Biml?
Business Intelligence Markup Language
Easy to read and write XML language
Describes business intelligence objects:
• Databases, Schemas, Tables, Views, Columns
• SSIS Packages
• SSAS Cubes
• Metadata
What do you need?
…or you can use the new Biml tools
How does it work?
dem
o t
ime!
Let's generate
some packages!
Ok, so we can go from Biml to SSIS…
…can we go from SSIS to Biml?
Yes! :)
dem
o t
ime!
Let's reverse-engineer
some packages!
The magic is in the BimlScript!
Extend Biml with C# or VB code blocks
Import database structure and metadata
Loop over tables and columns
Expressions replace static values
BimlScript allows you to control and manipulate Biml code
BimlScript Code Nuggets
<# … #> Control Nuggets (Control logic)
<#= … #> Text Nuggets (Returns string)
<#@ … #> Directives (Compiler instructions)
<#+ … #> Class Nuggets (Create C# classes)
How does it work?
Yes, but how does it work?
Yes, but how does it actually work?
<Biml xmlns="http://schemas.varigence.com/biml.xsd"><Packages>
<# foreach (var table in RootNode.Tables) { #><Package Name="Load_<#=table.Name#>"></Package>
<# } #></Packages>
</Biml> <Biml xmlns="http://schemas.varigence.com/biml.xsd"><Packages>
<Package Name="Load_Customer"/><Package Name="Load_Product"/><Package Name="Load_Sales"/>
</Packages></Biml>
Biml vs. BimlScript
Automate, control and
manipulate Biml with C#
Flat XML"Just text"
dem
o t
ime!
Let's generate
a lot of packages!
Code
Management
Don't Repeat Yourself
Move common code to separate files
Centralize and reuse in many projects
Update code once for all projects
1. Include files
2. CallBimlScript with Parameters
3. Tiered Biml files
BimlExpress vs. BimlOnline / BimlStudio
"Black Box"
Only SSIS packages visible
Visual Editors
All in-memory objects visible
Include Files
Include common code in multiple files and projects
Can include many file types: .biml .txt .sql .cs
Use the include directive
<#@ include file="CommonCode.biml" #>The directive will be replaced by the included file
Include pulls code from the included file into the main file
Works like an automated Copy & Paste
Include Files
Include Files
Include Files
CallBimlScript with Parameters
Works like a parameterized include
File to be called (callee) specifies input parameters it accepts
<#@ property name="Parameter" type="String" #>File that calls (caller) passes input parameters
<#=CallBimlScript("CommonCode.biml", Parameter)#>
CallBimlScript pushes parameters from the caller to the callee, and the
callee returns code
CallBimlScript with Parameters
CallBimlScript with Parameters
CallBimlScript with Parameters
CallBimlScript with Parameters
CallBimlScript with Parameters
Tiered Biml Files
Split Biml code in multiple files and use the template directive:
<#@ template tier="1" #>
Create objects in-memory from lowest to highest tier to:
• Solve logical dependencies
• Simulate manual workflows
In-memory objects are added to the RootNode
Higher tiers can get objects added to RootNode in lower tiers
What is this RootNode?
The RootNode contains all in-memory objects:
• Connections, Databases, Schemas, Tables
• Projects, Packages
• Annotations, Metadata
Query the RootNode to loop over collections:<# foreach (var table in RootNode.Tables) { #>
Query the RootNode to get specific objects:<#=RootNode.Tables["Product"].Schema#>
Inside the Black Box: Tiered Biml Files
<#@ template tier="1" #><Connections>...</Connections>
<#@ template tier="2" #><Packages>...</Packages>
<#@ template tier="3" #><Package>...</Package>
Inside the Black Box: Tiered Biml Files
<#@ template tier="1" #><Connections>...</Connections>
<#@ template tier="2" #><Packages>...</Packages>
<#@ template tier="3" #><Package>...</Package>
Inside the Black Box: Tiered Biml Files
<#@ template tier="1" #><Connections>...</Connections>
<#@ template tier="2" #><Packages>...</Packages>
<#@ template tier="3" #><Package>...</Package>
Inside the Black Box: Tiered Biml Files
<#@ template tier="1" #><Connections>...</Connections>
<#@ template tier="2" #><Packages>...</Packages>
<#@ template tier="3" #><Package>...</Package>
Inside the Black Box: Tiered Biml Files
<#@ template tier="1" #><Connections>...</Connections>
<#@ template tier="2" #><Packages>...</Packages>
<#@ template tier="3" #><Package>...</Package>
Inside the Black Box: Tiered Biml Files
<#@ template tier="1" #><Connections>...</Connections>
<#@ template tier="2" #><Packages>...</Packages>
<#@ template tier="3" #><Package>...</Package>
Inside the Black Box: Tiered Biml Files
<#@ template tier="1" #><Connections>...</Connections>
<#@ template tier="2" #><Packages>...</Packages>
<#@ template tier="3" #><Package>...</Package>
Inside the Black Box: Tiered Biml Files
<#@ template tier="1" #><Connections>...</Connections>
<#@ template tier="2" #><Packages>...</Packages>
<#@ template tier="3" #><Package>...</Package>
Inside the Black Box: Tiered Biml Files
<#@ template tier="1" #><Connections>...</Connections>
<#@ template tier="2" #><Packages>...</Packages>
<#@ template tier="3" #><Package>...</Package>
Inside the Black Box: Tiered Biml Files
<#@ template tier="1" #><Connections>...</Connections>
<#@ template tier="2" #><Packages>...</Packages>
<#@ template tier="3" #><Package>...</Package>
Inside the Black Box: Tiered Biml Files
How do you use Tiered Biml files?
1. Create Biml files with specified tiers
2. Select all the tiered Biml files
3. Right-click and click Generate SSIS Packages
1
2
3
dem
o t
ime!
How does this
actually work?
Debugging Biml
Debugging Biml
BimlExpress is a "black box":• You can only see the generated SSIS packages
• It is not possible to see the compiled Biml first
Add a high-tier helper file to save compiled, flat Biml to file• Check Biml For Errors to save flat Biml without generating packages
SaveFlatBimlToFile.biml
Add the helper file to your project…
<#@ template tier="999" #><# System.IO.File.WriteAllText(
@"C:\Biml\FlatBiml.xml", RootNode.GetBiml()
); #>
SaveFlatBimlToFile.biml
…with a high tier so it is executed as the last step
<#@ template tier="999" #><# System.IO.File.WriteAllText(
@"C:\Biml\FlatBiml.xml", RootNode.GetBiml()
); #>
SaveFlatBimlToFile.biml
It creates a file…
<#@ template tier="999" #><# System.IO.File.WriteAllText(
@"C:\Biml\FlatBiml.xml", RootNode.GetBiml()
); #>
SaveFlatBimlToFile.biml
…at the specified path…
<#@ template tier="999" #><# System.IO.File.WriteAllText(
@"C:\Biml\FlatBiml.xml", RootNode.GetBiml()
); #>
SaveFlatBimlToFile.biml
…with all the Biml for all the object in RootNode
<#@ template tier="999" #><# System.IO.File.WriteAllText(
@"C:\Biml\FlatBiml.xml", RootNode.GetBiml()
); #>
How do you use this helper file?
1. Create the helper file
2. Select all the Biml files and the helper file
3. Right-click and click Check Biml For Errors
1
2
3
dem
o t
ime!
How is this
helper file used?
Annotations and
ObjectTags
Annotations and ObjectTags
Biml Annotations != SSIS Annotations
Annotations are string/string Key/Value pairs
ObjectTags are string/object Key/Value pairs
Use Annotations and ObjectTags to pass code
between Biml files
Annotations
Create annotations:<OleDbConnection Name="Destination" ConnectionString="…"><Annotations><Annotation Tag="Schema">AW2014</Annotation>
</Annotations></OleDbConnection>
Use annotations:<# var destinationSchema =
RootNode.OleDbConnections["Destination"].GetTag("Schema"); #>
ObjectTags
Create ObjectTags:<# RootNode.OleDbConnections["Destination"].ObjectTag["TableFilter"] = new List<string> {"Product","ProductSubcategory","ProductCategory"};
#>
Use ObjectTags:<#var TableFilter = (List<string>)RootNode.OleDbConnections["Destination"].ObjectTag["TableFilter"];
#>
LINQ
LINQ (Language-Integrated Query)
One language to query:
SQL Server Databases
XML Documents
Datasets
Collections
Two ways to write queries:
SQL-like Syntax
Extension Methods
LINQ Extension Methods
..and many, many more!
Sort
OrderBy, ThenBy
Filter
Where, OfType
Group
GroupBy
Aggregate
Count, Sum
Check Collections
All, Any, Contains
Get Elements
First, Last, ElementAt
Project Collections
Select, SelectMany
LINQ Extension Methods
var numConnections = RootNode.Connections.Count()
foreach (var table in RootNode.Tables.Where(…))
if (RootNode.Packages.Any(…))
LINQ and Lambda expressions
Use lambda expressions to filter or specify values:
.Where(table => table.Schema.Name == "Production").OrderBy(table => table.Name)
LINQ and Lambda expressions
For each element in the collection…
.Where(table => table.Schema.Name == "Production").OrderBy(table => table.Name)
LINQ and Lambda expressions
…evaluate a criteria or get a value:
.Where(table => table.Schema.Name == "Production").OrderBy(table => table.Name)
LINQ: Filter collections
Where()Returns the filtered collection with all elements that meet the criteria
RootNode.Tables.Where(t => t.Schema.Name == "Production")
OfType()Returns the filtered collection with all elements of the specified type
RootNode.Connections.OfType<AstExcelOleDbConnectionNode>()
LINQ: Sort collections
OrderBy()Returns the collection sorted by key…
RootNode.Tables.OrderBy(t => t.Name)
ThenBy()…then sorted by secondary key
RootNode.Tables.OrderBy(t => t.Schema.Name).ThenBy(t => t.Name)
LINQ: Sort collections
OrderByDescending()Returns the collection sorted by key…
RootNode.Tables.OrderByDescending(t => t.Name)
ThenByDescending()…then sorted by secondary key
RootNode.Tables.OrderBy(t => t.Schema.Name).ThenByDescending(t => t.Name)
LINQ: Sort collections
Reverse()Returns the collection sorted in reverse order
RootNode.Tables.Reverse()
LINQ: Group collections
GroupBy()Returns a collection of key-value pairs where each value is a new collection
RootNode.Tables.GroupBy(t => t.Schema.Name)
LINQ: Aggregate collections
Count()Returns the number of elements in the collection
RootNode.Tables.Count()RootNode.Tables.Count(t => t.Schema.Name == "Production")
LINQ: Aggregate collections
Sum()Returns the sum of the (numeric) values in the collection
RootNode.Tables.Sum(t => t.Columns.Count)
Average()Returns the average value of the (numeric) values in the collection
RootNode.Tables.Average(t => t.Columns.Count)
LINQ: Aggregate collections
Min()Returns the minimum value of the (numeric) values in the collection
RootNode.Tables.Min(t => t.Columns.Count)
Max()Returns the maximum value of the (numeric) values in the collection
RootNode.Tables.Max(t => t.Columns.Count)
LINQ: Check collections
All()Returns true if all elements in the collection meet the criteria
RootNode.Databases.All(d => d.Name.StartsWith("A"))
Any()Returns true if any element in the collection meets the criteria
RootNode.Databases.Any(d => d.Name.Contains("DW"))
LINQ: Check collections
Contains()Returns true if collection contains element
RootNode.Databases.Contains(AdventureWorks2014)
LINQ: Get elements
First()Returns the first element in the collection (that meets the criteria)
RootNode.Tables.First()RootNode.Tables.First(t => t.Schema.Name == "Production")
FirstOrDefault()Returns the first element in the collection or default value (that meets the criteria)
RootNode.Tables.FirstOrDefault()RootNode.Tables.FirstOrDefault(t => t.Schema.Name == "Production")
LINQ: Get elements
Last()Returns the last element in the collection (that meets the criteria)
RootNode.Tables.Last()RootNode.Tables.Last(t => t.Schema.Name == "Production")
LastOrDefault()Returns the last element in the collection or default value (that meets the criteria)
RootNode.Tables.LastOrDefault()RootNode.Tables.LastOrDefault(t => t.Schema.Name == "Production")
LINQ: Get elements
ElementAt()Returns the element in the collection at the specified index
RootNode.Tables.ElementAt(42)
ElementAtOrDefault()Returns the element in the collection or default value at the specified index
RootNode.Tables.ElementAtOrDefault(42)
LINQ: Project collections
Select()Creates a new collection from one collection
A list of table names:
RootNode.Tables.Select(t => t.Name)
A list of table and schema names:
RootNode.Tables.Select(t => new {t.Name, t.Schema.Name})
LINQ: Project collections
SelectMany()Creates a new collection from many collections and merges the collections
A list of all columns from all tables:
RootNode.Tables.SelectMany(t => t.Columns)
dem
o t
ime!
How is LINQ used
in Biml projects?
C#
Classes and
Methods
C# Classes and Methods
BimlScript and LINQ not enough?
Need to reuse C# code?
Create your own classes and methods!
C# Classes and Methods: From this…
public static class HelperClass {public static bool AnnotationTagExists(AstNode node, string tag) {if (node.GetTag(tag) != "") {return true;
} else {return false;
}}
}
C# Classes and Methods: …to this
public static class HelperClass {public static bool AnnotationTagExists(AstNode node, string tag) {return (node.GetTag(tag) != "") ? true : false;
}} * For bools you can just use:
return (node.GetTag(tag) != "");
But in this example we'll use the verbose, SSIS-like syntax
because it can be reused with other data types, like…
C# Classes and Methods: …or this
public static class HelperClass {public static string AnnotationTagExists(AstNode node, string tag) {return (node.GetTag(tag) != "") ? "Yes" : "No";
}}
Where do you put your code?
Inline code nuggets
Included Biml files with code nuggets
Reference code files
C# Classes and Methods: Inline
<Biml xmlns="http://schemas.varigence.com/biml.xsd"><# foreach (var table in RootNode.Tables) { #> <# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #>...
<# } #><# } #>
</Biml>
<#+public static class HelperClass {public static bool AnnotationTagExists(AstNode node, string tag) {
return (node.GetTag(tag) != "") ? true : false;}
}#>
C# Classes and Methods: Included Files
<#@ include file="HelperClass.biml" #>
<Biml xmlns="http://schemas.varigence.com/biml.xsd"><# foreach (var table in RootNode.Tables) { #> <# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #>...
<# } #><# } #>
</Biml>
<#+public static class HelperClass {public static bool AnnotationTagExists(AstNode node, string tag) {
return (node.GetTag(tag) != "") ? true : false;}
}#>
C# Classes and Methods: Code Files
<#@ code file="HelperClass.cs" #>
<Biml xmlns="http://schemas.varigence.com/biml.xsd"><# foreach (var table in RootNode.Tables) { #> <# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #>...
<# } #><# } #>
</Biml>public static class HelperClass {public static bool AnnotationTagExists(AstNode node, string tag) {
return (node.GetTag(tag) != "") ? true : false;}
}
C#
Extension
Methods
Extension Methods
"Make it look like the method belongs to
an object instead of a helper class"
Extension Methods: From this…
<#@ code file="HelperClass.cs" #>
<Biml xmlns="http://schemas.varigence.com/biml.xsd"><# foreach (var table in RootNode.Tables) { #> <# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #>...
<# } #><# } #>
</Biml>public static class HelperClass {public static bool AnnotationTagExists(AstNode node, string tag) {
return (node.GetTag(tag) != "") ? true : false;}
}
Extension Methods: …to this
<#@ code file="HelperClass.cs" #>
<Biml xmlns="http://schemas.varigence.com/biml.xsd"><# foreach (var table in RootNode.Tables) { #> <# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #>...
<# } #><# } #>
</Biml>public static class HelperClass {public static bool AnnotationTagExists(this AstNode node, string tag) {
return (node.GetTag(tag) != "") ? true : false;}
}
Extension Methods: …to this
<#@ code file="HelperClass.cs" #>
<Biml xmlns="http://schemas.varigence.com/biml.xsd"><# foreach (var table in RootNode.Tables) { #> <# if (table.AnnotationTagExists("SourceSchema")) { #>...
<# } #><# } #>
</Biml>public static class HelperClass {public static bool AnnotationTagExists(this AstNode node, string tag) {
return (node.GetTag(tag) != "") ? true : false;}
}
Extension Methods: …to this :)
<#@ code file="HelperClass.cs" #>
<Biml xmlns="http://schemas.varigence.com/biml.xsd"><# foreach (var table in RootNode.Tables.Where(t =>
t.AnnotationTagExists("SourceSchema")) { #>...
<# } #><# } #>
</Biml>public static class HelperClass {public static bool AnnotationTagExists(this AstNode node, string tag) {
return (node.GetTag(tag) != "") ? true : false;}
}
Questions?
Get things done
Start small
Start simple
Start with ugly code
Keep going
Expand
Improve
Deliver often
Izpolnite anketo!
Vam je bilo predavanje všeč?
Ste se naučili kaj novega?
Vaše mnenje nam veliko pomeni!
Da bo NT konferenca prihodnje leto še boljša, vas
prosimo, da izpolnite anketo o zadovoljstvu, ki jo
najdete v svojem NTK spletnem profilu.
@cathrinew
cathrinewilhelmsen.net
linkedin.com/in/cathrinewilhelmsen
contact@cathrinewilhelmsen.net
slideshare.net/cathrinewilhelmsen
Biml resources and references:
cathrinewilhelmsen.net/biml
top related