data transfer from mysql database to ms sql database

7/22/2019 Data Transfer From MySQL Database to MS SQL Database

1/34

DATA TRANSFER FROM MYSQL DATABASE TO MS SQL DATABASE

Contents:

1. Prologue2. Terminology used3. Setting Up Machine 1

a. Setting up MySQL Server for exporting datab. Installing WinSCP on machine 1c. Creating a scheduled task on machine 1

4. Setting up Machine 2a. Installing and configuring freeSSHd on machine 2b. Creating database and tables in MS SQL server on machine 2c. Setting up MS SQL Server for importing data

PROLOGUE

We are gathering huge amount of data from Anasage website visitors, like which page are they viewing from our

website, how much they are spending, which visitors is most active etc. But cannot process this data unless we

transfer it to much more accessible database where we can perform data mining other required operations on it.

So in this document I am going to set-up a system using which we will transfer this data from MySQL database of

Piwik Web Analytics tool to MS SQL Server running on a remote machine.

Before continuing please make sure that you have access to all the required files and soft-wares. SQL files are

stored in TFS server under Document Analytics folder.

Terminology Used In This Manual

Terms used in this manual are explained below:

MACHINE 1:

This is the machine which is running MySQL Server and it holds data tracked by Piwik Web Analytics Tool from

Anasage website visitors. This machine will act as SSH client during transfer of CSV files to machine 2 (explained

below).

MACHINE 2:

This is the machine which is running MS SQL Server and our tracked data from MySQL database will be transferred

to MS SQL Server here. In transferring CSV files from machine 1 to machine 2 this machine will act as SSH Server.


2/34

SETTING UP MACHINE 2

I am assuming that MS SQL Server 2008 R2 is already installed on machine 2. If it is not then you can consult its

installation manual prepared by Waseem.

INSTALLING AND CONFIGURING FREESSHD ON MACHINE 2

We will first install freeSSHd software on this machine. This software will act as SFTP/SSH server so that client

running on machine 1 can connect to it.

1. From Extra software folder run freeSSHd.exe.

2. Click Yes. And then click Next on next window.


3/34

3. Keep clicking next until you reach install screen. There just click install and you are almost done. Clickclose in next screen:

4. Next you will be asked to create private key. Select Yes:

5. In next screen click Yes. This will make freeSSHd run in background as a service, so that even if itsInterface is not running we can still connect to it. NOTE: if you installed freeSSHd already and it did not

work please select NO here. It will solve the problem but you will have to start this application everytime


4/34


5/34

7. This software is installed. Now we need to make some configurations in it. Right Click FreeSSHd ondesktop to run this program as an administrator. Click OK. A new icon will appear in tray area of task bar.

Click on its icon.

8. A new window will appear on desktop.

9. Create a new folderin C:\ drive named remote_ftp_folder. Then goto SFTP tab on freeSSHd Settingwindow select this folder as home folder for SFTP. Our CSV files transferred by client machine 1 will be


6/34

stored here.


7/34

10. Next go to SSH tab and make sure Listen Address matches IP address of machine 2.


8/34

11. Create a new folder Public Keys insideremote_ftp_folder installation directory, Copy file user_key tothis directory.


9/34

12. Go to Authentication Tab and updates settings as follow:


10/34

13. Next click Users tab and select Add.

14. Enter user name as shown in above picture. Make sure you entered these values as it is and case of everyletter is same. Also Authorization should be selected as Public Key. Click OK.


11/34

15. Next go to logging tab and entered following

16. Next go to Automatic Update and turn off check box for Check for newer version on startup.Next clickapply from below. You will be presented with a message box.

17. Click Yesand then click OK from bottom navigation bar.CREATING DATABASE AND TABLES IN MS SQL SERVER ON MACHINE 2

Now we will create a database in MS SQL Server and create tables in it.


12/34

1. First run Microsoft SQL Server Management Studio.


13/34

2. Enter your credentials and Select Connect.

3. Press Ctrl+O to open file and select database.sql from folder Machine 2-> SQL Server Tables


14/34

4. Then press F5 to execute query. Repeat this step again with files actions_table.sql, site_table.sql,visit_actions_table.sql and visit_table.sql.And execute all of them. This will create four tables inside

this database.

SETTING UP MS SQL SERVER FOR I MPORTING DATA

1. Now we need to create stored procedure in this database to import data from CSV files and insert it intotables. Press Ctrl+O to open stored procedure file this file is located under folder Machine 2-> Stored

Procedure->ImportTrackingData.sql. After opening it just execute it.

2. Now one last step is remaining: Creating SQL Agent Job to execute this stored procedure at regularintervals to import data from CSV files.

3. Right Click on SQL Server Agent-> Jobs and select create new job.


15/34

4. Enter following information in window which will appear:


16/34

5. Go to Steps from left side-bar and then click new. Enter information as follows: (Text inside Commandsection is [tracking_data_db].[dbo].[import_tracking_data])


17/34

6. Next go to Advanced from left side bar and make sure all values are as follows: Then press OK.


18/34

7. Next Click Schedules and then click new. Input information as follows. And Press ok.


19/34

8. Press ok to create this job.


20/34

9. Now one last step is remaining. Create a folder named formats inside C:\remote_ftp_folder wepreviously create and move these four file to it from Machine 2->MS SQL Format Files folder.

SETTING UP MACHINE 1

SETTING UP MYSQL SERVER FOR EXPORTING DATA

Setting up MySQL Server for exporting data involves following steps:

1. Creating a stored procedure to export data2. Creating an event to execute stored procedure regularly at specific intervals

SQL File for creating a stored procedure and event can be found under directory Machine 1->MySQL Files.

1. First make sure that wamp server is running and it is online. Steps to start wamp server are:


21/34

a. Click Start Menu and type wamp. Select start WampServer.

b. Then in tray click on and click Restart all Services.2. Now start your browser and go to this address: http://localhost/phpmyadmin/. You will see following

page:


22/34

3. Enter credentials and click Go.


23/34

4. Now click on piwik_db from left side-bar. You should see following page:


24/34

5. Click Import from top menu-bar.

6. Click Browse and select file Export Tracked Data.sql and click G0. You will see following messeage :Import has been successfully finished, 2 queries executed. (Export Tracked Data.sql)

7. Now click Import and select Start Procedure Event.sql and click Go. You will see following message:Import has been successfully finished, 2 queries executed. (Export Tracked Data.sql)

8. Now stored procedure and event both are created in MySQL database.SETTINGUP SSHCLIENT ON MACHINE 1

Next we will install a SSH client on this machine to transfer CSV files to machine 2.

1. Create a folder piwik_tracking_data inside drive where wamp server is installed like C:\ drive.


25/34

2. Copy scripts folder from Machine 1->SSH Client to this folder.


26/34

3. This script folder contains a custom built SSH client which uses SharpSSH at its backend. Now go insidescripts folder and open SSHClient.exe.xml in your favorite text editor.

4. This file contains settings for our SSH server and files to be transferred. Enter correct values Also if youneed to add more files make sure to append key name with correct number like file1, file2 etc. Also

do not remove any key from this configuration file otherwise this client may not work properly.5. Save the file and close text editor.

CREATING A SCHEDULED TASK ON MACHINE 1

1. Now we will create a scheduled task in maschie 1 to run SSHClient.exe file at regular intervals.


27/34

2. Go to Control Panel and click on Administrative Tools.


28/34

3. Double click on Task Scheduler.


29/34

4. You should see a window like this:


30/34

5. Click on Create Task from right side-bar. And fill up the form which will appear as follows:


31/34

6. Now go to Triggers tab.Click New enter following data. Then click Ok once.


32/34

7. Next go to Actions tab. Click New and enter following data. Note that text entered in Start In field ispath to SSHClient.exe file e.g. in my case it is C:\piwik_tracking_data\scripts. Click Ok.


33/34

8. Now go to Conditions tab and make sure your selections match this image:


34/34

9. Next go to Settings tab and repeat previous step.

10. Now click OK. Enter you credentials if it asks:

11. Close Task Sheduler.Now MACHINE 1 is completely setup.

Now this is it Whole setup should be working fine now

data transfer from mysql database to ms sql database

Documents