great migrations technology · etl test cut-over conversion tuning legacy db the tool-assisted data...
TRANSCRIPT
gmStudio Workshop
Mark Juras
Great Migrations LLC
9/11/2016 Copyright Great Migrations LLC 1
Workshop Agenda
• Objectives and Expectations
• Methodology Concepts
• Technology Concepts
• Smart Start Demo
9/11/2016 Copyright Great Migrations LLC 2
Objectives and Expectations
• Objectives
– Successfully upgrade VB6/ASP systems to .NET
– Understand how Great Migrations can help you
• Expectations
– VB6/ASP systems = ?
– Successfully = ?
– Upgrade = ?
– .NET = ?
9/11/2016 Copyright Great Migrations LLC 3
Expectations: Estimation Factors
• What is your Business Case for upgrading
• What is your Source/Target Architecture?
• What is the Maintenance Process? (SDLC, schedule, team structure/size/capabilities)
• What is your Source/Target Platform? OS upgrade? Multiple-OS?
• What is your desired project Staffing model?
• What is your desired project Schedule?
• What is your Project Management process?
• What is your Acceptance Criteria for deliverables?
• What is your SCM Process? (Version control, tools, standards)
• What is your Deployment Process?
• What is your Test Process? (team, environment, data, automation)
• What Functional Changes are expected with the upgrade?
• What SDLC Changes are expected with the upgrade?
Planning Questions Setting Upgrade Project Scope and Budget
9/11/2016 Copyright Great Migrations LLC 4
Methodology: Concept
9/11/2016 Copyright Great Migrations LLC 5
Shiny
New
DB
Assess
Design
Prepare
Automated
ETLTest Cut-Over
Conversion
Tuning
Legacy
DB
The Tool-Assisted Data Conversion
9/11/2016 Copyright Great Migrations LLC 6
Methodology: Concept
Assess
Design
Prepare
Semantic
TranslationTest Cut-Over
VB6/
ASP/
COM
.NET
Translation
Tuning
The Tool-Assisted Rewrite
Agile Iterative Scalable Repeatable Measureable Improvable
Methodology: Balance
The Great Migrations Methodology balances
automated translation and other techniques
to create a custom upgrade solution that
delivers high quality results with less risk and less effort.
Convert or Rewrite?
9/11/2016 Copyright Great Migrations LLC 7
9/11/2016 Copyright Great Migrations LLC 8
Methodology: Phases
.NET ApplicationPhase 2: Advanced Upgrade
.NET BaselinePhase 1: Basic Upgrade
Re-engineer
gmStudiogmStudio
Integrate Solution to .NET
Resolve Runtime Issues
Build
Visual StudioVisual Studio
Test
Translate
gmStudiogmStudio
Resolve Translation Issues
Resolve Build Issues
Build
Visual StudioVisual Studio
Review
.NET Framework
.NET Externals
.NET Code
.NET Framework
.NET Stubs
.NET Code
VB6/ASP Application
VB6 Infrastructure
COM Externals
VB6 Code
VB6/ASP Application
VB6 Infrastructure
COM Externals
VB6 Code
Methodology: Milestones
0: Source Complete, Ready to Translate
1: Translate Complete, Ready to Assess
2: Build Complete, Ready for Reengineering
3: Reengineering Complete, Ready for Testing
4: Verification Complete, Ready for Cut-Over
5: .NET in Production
6: Goodbye VB6!
9/11/2016 Copyright Great Migrations LLC 9
Incremental Upgrade: Step 1
Code
gmStudio
.NET Project 1
Code
COM Stub A
COM Stub B
COM Stub C
Source Project 1
Standalone Translation
COM
B
COM
C COM
A
gmStudio
.NET Project 1
Code
COM Stub Code A
COM Stub Code B-1
COM Stub Code C
Code
Source Project 1
Code
Source Project 2
Code
.NET Project 2
COM Stub Code B-2
Incremental Upgrade: Step 2
Multiple Standalone Translations
COM
B
COM
C COM
A
gmStudio
Code
Source Project 1
Code
Source Project 2
COM
B
COM
C
Code
.NET Project 1
Code
.NET Project 2
Stub C
.NET Project 4
Stub B
(1 + 2)
.NET Project 3
COM
A .NET
Incremental Upgrade: Step 3
Multiple, Integrated Translations
gmStudio
Code
Source Project 1
COM
B
COM
C
Code
.NET Project 1
Code
Source Project 2
COM
A
Code
.NET Project 2
.NET
A
Incremental Upgrade: Step 4
Multiple, Integrated Upgraded Translations
.NET
B
.NET
C
Methodology: Functional Testing
9/11/2016 Copyright Great Migrations LLC 14
Parallel Acceptance Testing
9/11/2016 Copyright Great Migrations LLC 17
• Overview
– gmStudio
– gmBasic
• Demonstrations
– Preparation
– Translation
– Deployment
– Verification
– Refinement
Technology: Agenda
9/11/2016 Copyright Great Migrations LLC 19
Technology: gmBasic
What is gmBasic?
A highly configurable, robust
VB6/ASP/COM compiler
that produces source codes
instead of binaries.
How does it work?
– Compiler: Builds a comprehensive semantic model of the codebase implementation.
– Analyzer: evaluates and restructures the model to fit the desired architecture patterns.
– Author: processes the optimized model to generate clean, correct code that meets custom standards and conventions.
VB6 to .NET, ASP to .NET, IDL to XML, XML to .NET, Scripting, Reporting…
9/11/2016 Copyright Great Migrations LLC 20
Technology: Semantic Model
• Symbol Trees
– External Components
– Language Elements
– Source Structures
• P-Code Tables
– Operations
– Expressions
– Source Mappings
9/11/2016 Copyright Great Migrations LLC 21
Technology: gmStudio
What is gmStudio?
An extensible IDE
for designing, building, and improving
custom upgrade solutions
How does it work?
– Project Setup • Project Definition
• Basic Scripts
• COM Descriptions
– Workspace Management
– Process Orchestration
– Tool Integration
– Upgrade Solution Analysis
– Technical Analysis
Setup, Analyze, Orchestrate, Navigate, Verify, Report
9/11/2016 Copyright Great Migrations LLC 23
Technology: Upgrade Features
Types of
Upgrades • COM Replacement
• Code Structure
• Language
• Win32 Replacement
• Architecture
Upgrade
Process • Analysis (As-Is)
• Strategies (To-Be)
• Detail Design
• Implementation
• Iterative Refinement
Technology: Upgrade Solution
• Source Codes
• gmStudio Project File
• Translation Script Templates
• COM Interface Description Files (IDFs)
• Custom Script Templates
• Project-Specific Metalanguage Files
• User Batch Command Script Templates
• Custom code
• Custom tools
9/11/2016 Copyright Great Migrations LLC 24
Technology: Generated Files
• Process Artifacts
– Actual Translation Scripts
– Semantic Model Files (.VBI)
– Interface Description Files (.XML)
– Translation Logs
– Deployment Logs
– Build Logs
– Reports (.TAB, .TXT)
• .NET Code and Binaries
– Generated .NET Application Codes (Deploy Folder)
– Generated Stub Classes (instead of interop)
– Generated Stub Assemblies (instead of interop)
– Interop Assemblies (rarely used)
9/11/2016 Copyright Great Migrations LLC 25
9/11/2016 Copyright Great Migrations LLC 26
Technology: Standard Distribution
Compiler
• ASPLANG.XML
• DAO350.DLL.xml
• HTMLLANG.XML
• Language.std.xml
• mig.MSVBVM60.dll.xml
• OPCODES.XML
• stdole2.tlb.xml
• vb6control.dcl.xml
• vbconstants.xml
• vbcontrols.xml
• vbmethods.xml
• vbproject.xml
• vbscript_3.dll.xml
Meta-language
• GMSLANG.XML
• TOOLLANG.XML
• enumerations.xml
Utility
• auditvbi.gmsl
• gmslPatterns.xml
• gmSLtoNI.gmsl
• IdlAuthor.gmsl
• messages.xml
• SharedFile.gmsl
• utility.gmsl
• VB7Lang.xml
Author
• AspAuthor.gmsl
• authortext.gmsl
• cblpatterns.xml
• CodeStyle.std.xml
• locSubSystem.gmsl
• mig.MSVBVM60_3.dll.xml
• MigrationSupport.dll.xml
• netclasses.xml
• netControls.xml
• templates.xml
• VB6IDL.XML
• VBASIC.XML
• wpfSubsystem.gmsl
gmBasic Configuration Files
9/11/2016 Copyright Great Migrations LLC 27
Technology: Standard Distribution
Custom Meta-language
• VB7Lang.xml
• gmBasic.xml
Interop Generation
• GenInterop.cs
• GenInterop_VS.csproj.xml
• GenInterop_VS.ref.xml
Translation
• tran.asp_bld.xml
• tran.asp_std.xml
• tran.vbp_bld.xml
• tran.vbp_std.xml
• ScriptRules.xml
Reporting
• CodeScanList.txt
• rpt.ana.xml
• rpt.asp_ord.xml
• rpt.custom.xml
• rpt.sym.xml
• rpt.vbp_ord.xml
• SearchReportTemplate.txt
Utility
• DiffFiles.cmd
• GlobalSettings.xml
• GlobalStubs.xml
• GlobalIncludes.xml
• make.std.mak
• RunGmStudio.cmd
• SetEnv.cmd
• userCmd.std.cmd
• registry.xml
gmStudio Configuration Files
9/11/2016 Copyright Great Migrations LLC 28
Technology: Standard Distribution
1. ADODB_DataLib_Transform.gmsl
2. advapi32.dcl.xml
3. comdlg32.dcl.xml
4. gdi32.dcl.xml
5. GM.actrpt2.dll.xml
6. GM.ADODB_to_DataLib.dll.xml
7. GM.ADODB_to_System.Data.Odbc.dll.xml
8. GM.ADODB_to_System.Data.Sql.dll.xml
9. GM.COMDLG32.OCX.gmSl
10. GM.COMDLG32.OCX.xml
11. GM.COMDLG32_Remove.OCX.xml
12. GM.Comsvcs.dll.xml
13. GM.Comsvcs_to_EnterpriseServices.dll.xml
14. GM.DAO350.DLL.xml
15. GM.EXCEL_Interop.EXE.xml
16. GM.ieframe.dll.xml
17. GM.msado27.tlb.xml
18. GM.MSCAL.OCX.xml
19. GM.MSCOMCT2.OCX.xml
20. GM.MSComCt2_WinForms.OCX.XML
21. gm.mscomctl.gmsl
22. GM.MSCOMCTL.OCX.xml
23. gm.mscomctltransform.gmsl
24. GM.MSFLXGRD.OCX.xml
25. gm.msflxgrdtransform.gmsl
26. GM.MSHFLXGD.OCX.xml
27. GM.msinet.ocx.xml
28. GM.msmask32.ocx.xml
29. gm.msmasktransform.gmsl
30. GM.MSOUTL_Interop.OLB.xml
31. GM.MSRDO20.DLL.xml
32. GM.msrdo20Transform.gmsl
33. GM.MSVBVM60.dll.xml
34. GM.msvbvm60_3.dll.xml
35. GM.MSWINSCK.OCX.XML
36. GM.MSWORD_Interop.OLB.xml
37. GM.msxml6.dll.xml
38. GM.OIP11.tlb.xml
39. GM.ResizeKit.ocx.xml
40. GM.richtx32.ocx.xml
41. GM.scrrun.dll.xml
42. GM.shdocvw.dll.xml
43. GM.ssa3d30.ocx.xml
44. GM.SSLstBar.ocx.xml
45. GM.Stdole2.tlb.xml
46. GM.TABCTL32.OCX.xml
47. GM.tdbg6.ocx.gmsl
48. GM.tdbg6.ocx.xml
49. GM.TLBINF32.DLL.xml
50. GM.uiRAD.ocx.xml
51. GM.vbscript_3.dll.xml
52. GM.xarraydb.ocx.xml
53. instrument.gmsl
54. instrument.xml
55. IpHlpAPI.dcl.xml
56. kernel32.dcl.xml
57. mig.vb7lang.xml
58. mpr.dcl.xml
59. ole32.dcl.xml
60. olepro32.dcl.xml
61. ServiceClass.xml
62. shell32.dcl.xml
63. StatefulModules.xml
64. StubControlBaseClass.xml
65. TabctlTransform.gmsl
66. user32.dcl.xml
67. version.dcl.xml
Sample Rule Files
9/11/2016 Copyright Great Migrations LLC 29
Technology: gmStudio Upgrade Techniques
Basic Techniques
Select Settings
Registry Settings
Pre-Edit Source
Post-Edit Results
Refactoring
FixType
Remove
Reauthor
Overloads
Generic
Custom IDF XML
Intermediate Techniques
Custom Metalang Files
Custom SubSystem
Collections/Dictionary to Generics
GlobalStubs
GlobalSettings
Advanced Techniques
Custom IDF with gmSL
Translation Event Handlers
COM Event Handlers
Custom Analysis
Custom Upgrade Commands
Meta-Migration
9/11/2016 Copyright Great Migrations LLC 33
Demo: FileExplorer
• VB6 to .Net (externals stubbed)
• VB6 to .Net (externals upgraded)
• Look at Code, Build, Report, Run
Migration
Toolset
System
Config
Project
Config
VB6
COM
.NET
Stubs
.NET
Managed
• Standalone VB6/COM
– ListView
– TreeView
– ImageList
– StatusBar
– ToolBar
– RichTextBox
– Scripting
Technology: Workspace
9/11/2016 Copyright Great Migrations LLC 37
Workspace │ ├───deploy : Generated Content │ ├──bin : Runtime Assemblies │ ├──Task-1 : Generated code VBP-1 │ └──Task-2 : Generated code VBP-2 │ ... │ └──Task-N : Generated code VBP-N │ ├───idf │ ├──FromCode : COM Info from VBPs │ └──FromIdl : COM Info from binaries │ ├─log : detailed logs etc. │ ├─report : reports │ ├─resx : FRX/CTX as RESX │ └─usr : User-specific content
9/11/2016 Copyright Great Migrations LLC 38
Analytics-References Report
Technology: COM Upgrade Analysis
• Upgrade Process
– Metrics Analysis (As-Is)
– Strategies (To-Be)
– Detail Design
– Implementation
– Iterative Refinement
9/11/2016 Copyright Great Migrations LLC 39
Technology: COM Upgrades Implementation
Declarative Rules
– Assembly References
– Namespaces
– Classes, Structs
– Enums
– Enum Entries
– Members, Properties
– Control Initialization (Designer)
– Event Handlers
• Dynamic Rules
– gmSL Event Handlers
– Migration DLLs
Standard COM IDF
+
Your
Customizations
=
Custom COM IDF
Giving you
Automated, Repeatable
COM Replacement
9/11/2016 Copyright Great Migrations LLC 40
GM.Scrrun.dll.xml
Technology: COM Replacement, Sample
Managed Interface Descriptions
• Template Generated from COM
• Map source API to target API
GM.MsComCtLib.dll.xml
Migration
Toolset
System
Config
Project
Config
VB6
COM
.NET
Stubs
.NET
Managed
9/11/2016 Copyright Great Migrations LLC 41
Demo: ScanTool
• Two VBPs: UI.exe and LIB.dll
• Lib talks back to UI via events
• 4 External COM APIs
• Win32 APIs
• Many VB Intrinsics
• Object Polymorphism
• Error Handling
• Over 2000 LOC
ScanToolUI
ScanToolLib
COM Scripting
COM MSXML
COM Typelib
Info
COM Common
Dialog
• VB6 to .Net (externals stubbed)
– Local Stubs
• VB6 to .Net (externals upgraded)
– COM replacements
– Custom Runtime
• Look at Code, Build, Report, Run
Technology: Reporting
• Run
– Report menu (open after run)
– Report Panel (batch runs)
• Types
– Code Scans
– Project Reports
– Model-based
– Utilities
• Formats
– Tab-delimted
– Unformatteed
• Locations
– Workspace\log
– Workspace\report
9/11/2016 Copyright Great Migrations LLC 42
Title Output File
Code Scan Reports
Source Structure [MigName]-SrcStruct.tab
Source References [MigName]-SrcRef.tab
Source Members [MigName]-SrcMember.tab
Source GUI Scan [MigName]-SrcGUI.tab
Source Code Scan [MigName]-SrcScan.tab
Iceberg [MigName]-Iceberg.tab
Project Reports
Project Summary [MigName]-MigStat.txt
Metrics Summary [MigName]-Metrics.htm
Migration Set [MigName]-MigSet.tab
Code Bundles [MigName]-Bundle.tab
.NET Build Logs [MigName]-BldLog.tab
Translation Logs [MigName]-TranLog.tab
All Logs [MigName]-AllLog.txt
Interface File Headers [MigName]-LibHeaders.tab
Interface File ProgIds [MigName]-LibProgIds.tab
Semantic Model Reports
Semantic References [MigName]-AnaRef.tab
Semantic Definitions [MigName]-AnaDef.tab
Semantic Symbols [MigName]-AnaSym.tab
Semantic Audit [MigName]-Audit.txt
Utility Reports
Migration Project List gmProjects.tab
Multi-Unit Script tran.[MigName]_MultiJob.xml
Target Code Scan [MigName]-BndScan.tab
Target Code Changes [MigName]-BndChanges.txt
Technology: Searching
9/11/2016 Copyright Great Migrations LLC 43
• Powerful Search
– Code and Results Analysis
– Solution Management
– Custom Reports
Technology: SQL Analysis
• Bulk Insert to SQL
• Various Procs and Queries
• Load into Excel
9/11/2016 Copyright Great Migrations LLC 44
Title Output File
Code Scan Reports
Source Structure [MigName]-SrcStruct.tab
Source References [MigName]-SrcRef.tab
Source Members [MigName]-SrcMember.tab
Source GUI Scan [MigName]-SrcGUI.tab
Source Code Scan [MigName]-SrcScan.tab
Project Reports
Project Summary [MigName]-MigStat.txt
Semantic Model Reports
Semantic References [MigName]-AnaRef.tab
Semantic Definitions [MigName]-AnaDef.tab
Technology: More Reengineering
Basic Transformations
• Replace COM/Win32 APIs with .NET replacements
• Reauthor, Remove, or Stubout a member, class, file or entire component
• Control target file names, folder names, etc.
• Control target Visual Studio project files (resx, assemblyinfo, *proj)
• Control formatting – blank lines, comments, indenting, boilerplate code
• Specify settings that control internal translator operation
Advanced Transformations
• Generate a complete skeleton of all application and external components
• Consolidate of Shared files into a new or existing host assembly
• Convert of COM classes to WCF web services
• Convert shared module state to thread-isolated state
• Break build cycles: convert circular references to interface references
• Define rules to map ASP/VB6 language elements to .NET coding patterns
• Shared Files Consolidation
• Custom
9/11/2016 Copyright Great Migrations LLC 45
Technology: Select Command
• Select Identifier Attributes
• Select Value Attributes
• Select Enumerated Attributes
• Select Search String Attributes
• Select Location String Attributes
• Select ComputeConditional String Attribute
• Select Author Flag Attributes
• Select Compiler Flag Attributes
• Select Analyser Flag Attributes
• Select Process Flag Attributes
9/11/2016 Copyright Great Migrations LLC 46
Technology: Registry Command
• Dependency: Specify a possibly omitted include file dependency
• EditFile: Supply a set of Fix statements for a specified file
• FixType: Fix the type of a source component
• FixStatus: Specify an ASP page status
• Guid: Define the value of a GUID
• IdfStatus: Specify the Interface Description File status of an external
• Include: Specify the path to an include file
• LibName: Specify a library name or file name
• OverLoadArgument: Specify types for arguments to be overloaded
• ProgId: Resolve a ProgId
• RefactorFile: Supply a set of Refactor statements for a specified file
• SharedFile: Specifies that a file is shared by multiple VBPs
• UsesInterfaces: Specifies that a project file uses certain interfaces
9/11/2016 Copyright Great Migrations LLC 47
Technology: Fix/Replace
• When to use – Correcting source VB6 coding errors
– Correcting rare exceptions
– Work arounds
• Types of Fixes – Source Code Fix (Compile/Fix/Replace)
– Target Code Fix (Author/Fix/Replace@lang=“csh”)
– Target Project Fix (Author/Fix/Replace@lang=“csproj”)
– Whole File (Author/Fix/ReplaceFile)
– Target Stub Class (Author/Fix@FileFilter=“[lib.dll]”/Replace)
– Any File Fix (Fix@FileFilter=“path”/Replace)
– Regex Fix (bundle) (Author/Fix/Replace@status=“regex”)
9/11/2016 Copyright Great Migrations LLC 48
Technology: Refactor Command
CallByName Changes symbol-related code events that yield CallByName late
binding calls into direct boxed calls.
Extend Extends the content of a class by adding new components.
FixType Changes the binary type of a component or group of components
Implements Specifies that a VB6 class implements another class or interface.
MigClass Introduces a new class that contains related refactoring information
used for complex migration operations, especially as related to
designer code.
Migrate Specifies migration of a specific symbol introduced via an external
library description.
Reauthor Replaces the content of a subprogram with a completely rewritten
block of code
Remove Prevents a component from being authored
Rename Changes the authored name of components
Replace Replaces either the members of an external class or the patterns of
opcodes via replacement declarations.
9/11/2016 Copyright Great Migrations LLC 49
BEFORE public static string GetComputerName()
{
string GetComputerName = "";
// Set or retrieve the name of the computer.
string strBuffer = "";
int lngLen = 0;
strBuffer = VBNET.Strings.Space(255 + 1);
lngLen = VBNET.Strings.Len(strBuffer);
if (Convert.ToBoolean(GetComputerNameAPI( strBuffer,out lngLen)))
{
GetComputerName =
VBNET.Strings.Left(strBuffer,lngLen);
}
else
{
GetComputerName = "";
}
return GetComputerName;
}
Translation Script – Refactor/Reauthor
AFTER
public static string GetComputerName()
{
// UPGRADE_INFO: hand-coded
return System.Environment.MachineName;
}
Technology : Refactor/Reauthor
9/11/2016 Copyright Great Migrations LLC 50
9/11/2016 Copyright Great Migrations LLC 51
Sidebar: API Crossing
Be wary of Top-Down Migrations
(PiecePorting)
• COM interfaces have COM types as
member parameters and return types.
• If you interop COM components, your
.NET clients will end up straddling the
fence between COM and .NET and
this will require more interop code
which runs counter to the premise of
adopting .NET in first place.
PiecePort
Interoped
COM
Component
COM
Scripting
.NET
System.IO
.NET
Application
Scripting.File
System.IO.FileInfo
COM
Scripting
9/11/2016 Copyright Great Migrations LLC 52
.NET
Component
.NET
System.IO
.NET
Application
System.IO.FileInfo
.NET
System.IO
Sidebar: API Crossing
Generated Interface Description
Bottom-Up, SmartPort Migrations
• gmBasic remembers what it has translated and knows
which components are going to .NET
• gmBasic knows how interfaces are changing to use new
types
• gmBasic uses this information to generate clean, native
code in client applications.