Selene BainumRiteTech LLC

Doing ColdFusion & SQL development for more than 1/3 of my lifetime

Chief Architect @ RiteTech RiteTech is my company – completely

independent for the first time! Likes:

SQL and databases! Love making money on the internet▪ www.smartsupplies.net

Dislikes: DC Traffic Rude People

Intro Importing Data

Tools Examples *

Exporting Data Tools Examples *

Resources

* Any custom tags/functions shown in examples are free!

SIMILARITIES

Data stored in tabular format

Both contains rows and columns

Headers/column names denote data

DIFFERENCES

Spreadsheets often contain repeated rows of data

Database tables should be normalized – repeated data among rows kept to a minimum

Spreadsheets used column delimiters – not as reliable as database columns

Spreadsheets often contain no unique key

IMPORT

Data only available as: Web pages Text files Excel spreadsheets Etc…

Need data in your database!

EXPORT

Export raw data so that others can view/manipulate it in Excel

Create reports

Bad/non-standard data Spreadsheets may take hours and

hours to clean up the data manually Ill-placed commas and quotes can

wreak havoc Data types can get misread/lost Matching imported data can be

daunting It helps to have good knowledge of SQL

Import CSV or Excel files via SQL Server

Loop over file line by line * CFHTTP * Custom Tags

cfx_text2Query cfx_Excel2Query cfx_Excel

* Problematic as CF doesn’t recognize empty string elements

Find a site that contains regularly formatted data Using list of US Representatives as an

exampleCopy the data and paste into

notepadPerform any formatting necessary

May not work on all databases Using SQL Server 2005 in example

Not all installs of databases may support this type of import

May not have access to database

Insert all rows first Start with query result set Create/modify table as needed in

database Loop over query results Insert results

Perform other database inserts/updates as needed

-- Declare the variablesDECLARE @RepID INT, @StateID INT, @DistrictID INT, @LastNameTx VARCHAR(50), @FirstNameTx VARCHAR(50),

@PhoneTx VARCHAR(25), @RoomNb INT, @StateCd VARCHAR(2), @DistrictTx VARCHAR(10)-- Create a cursorDECLARE cRep CURSOR FOR SELECT FirstName, LastName, State, District, Phone, Room FROM RepListImport WITH (NOLOCK)-- Open the cursorOPEN cRep-- Loop over the cursorFETCH NEXT FROM cRep INTO @LastNameTx, @FirstNameTx, @StateCd, @DistrictTx, @PhoneTx, @RoomNbWHILE @@FETCH_STATUS = 0 BEGIN

SET @StateID = NULL SET @DistrictID = NULL-- State exist?SELECT @StateID = StateID FROM State WITH (NOLOCK) WHERE StateCd = @StateCdIF @StateID IS NULL BEGIN

SET NOCOUNT OFFINSERT INTO State (StateCd) VALUES (@StateCd) SELECT @StateID = Scope_Identity()SET NOCOUNT ON

END-- District exist?SELECT @DistrictID = DistrictID FROM District WITH (NOLOCK) WHERE DistrictTx = @DistrictTxIF @DistrictID IS NULL BEGIN

SET NOCOUNT OFFINSERT INTO District (DistrictTx) VALUES (@DistrictTx) SELECT @DistrictID = Scope_Identity()SET NOCOUNT ON

END-- Insert the repINSERT INTO Rep (FirstNameTx, LastNameTx, PhoneTx, RoomNb, StateID, DistrictID)VALUES (@FirstNameTx, @LastNameTx, @PhoneTx, @RoomNb, @StateID, @DistrictID)-- Get the next recordFETCH NEXT FROM cRep INTO @LastNameTx, @FirstNameTx, @StateCd, @DistrictTx, @PhoneTx, @RoomNb

END-- Close the cursorCLOSE cRep DEALLOCATE cRep

Create table structures and import scripts

Schedule retrieval of files Email attachments External files

Process import when new files available Run additional database scripts

ColdFusion Database scheduler

Much easier than imports Data is already cleaned and normalized

Tricky part is formatting

Manually create a CSV file via ColdFusion

Custom Tags/Functions generateExcel cf_Excel_XML cfx_Excel cfx_Query2Excel

Manually created CSV file Created by looping over the query results Can add custom column headings Can format data in cells – date, number,

etc…generateExcel Easier than manual creation Cannot add custom column headings▪ Column heading is the same as the column name

Automatically formats numbers and dates

<cfsetting showdebugoutput="no" />

<!--- Query the reps. ---><cfquery datasource="Mayhem_SQL" name="qryGetRepList">SELECT R.RepID, R.FirstNameTx, R.LastNameTx, R.PhoneTx, R.RoomNb,

S.StateCd, D.DistrictTxFROM Preso..Rep R WITH (NOLOCK) INNER JOIN Preso..State S WITH

(NOLOCK) ON R.StateID = S.StateIDINNER JOIN Preso..District D WITH (NOLOCK) ON R.DistrictID = D.DistrictID

ORDER BY S.StateCd, D.DistrictTx</cfquery>

<cfheader name="Content-Disposition" value="inline; filename=export.csv">

<cfcontent reset="yes" type="text/comma-separated-values" />"State","District","LastName","FirstName","Phone","Room"

<cfoutput query="qryGetRepList">"#StateCd#","#DistrictTx#","#LastNameTx#","#FirstNameTx#","#PhoneTx#","#RoomNb#"

</cfoutput><cfabort />

<cfsetting showdebugoutput="no" />

<!--- Query the reps. ---><cfquery datasource="Mayhem_SQL" name="qryGetRepList">SELECT R.RepID, R.FirstNameTx, R.LastNameTx, R.PhoneTx, R.RoomNb,

S.StateCd, D.DistrictTxFROM Preso..Rep R WITH (NOLOCK) INNER JOIN Preso..State S WITH

(NOLOCK) ON R.StateID = S.StateIDINNER JOIN Preso..District D WITH (NOLOCK) ON R.DistrictID = D.DistrictID

ORDER BY S.StateCd, D.DistrictTx</cfquery>

<cfinclude template="inc_fnGenerateExcel.cfm" />

<cfscript>generateExcel(qryGetRepList, "StateCd,DistrictTx,LastNameTx,FirstNameTx,PhoneTx,RoomNb");

</cfscript><cfabort />

Creates real Excel file Uses Excel’s XML format

Custom Settings Header/column names Multiple data formatting options Multiple sheets per file

Advanced Formatting Header color/text Alternating row colors Column width

<cfsetting showdebugoutput="no" />

<!--- Query the reps. ---><cfquery datasource="Mayhem_SQL" name="qryGetRepList">SELECT R.RepID, R.FirstNameTx, R.LastNameTx, R.PhoneTx, R.RoomNb,

S.StateCd, D.DistrictTxFROM Preso..Rep R WITH (NOLOCK) INNER JOIN Preso..State S WITH

(NOLOCK) ON R.StateID = S.StateIDINNER JOIN Preso..District D WITH (NOLOCK) ON R.DistrictID = D.DistrictID

ORDER BY S.StateCd, D.DistrictTx</cfquery>

<cf_excel_xml filename="export"sheets="1"

collist1="StateCd,DistrictTx,LastNameTx,FirstNameTx,PhoneTx,RoomNb"

headerlist1="State,District,Last Name,First Name,Phone,Room" colwidths1="50,75,100,100,75,50" query1="#qryGetRepList#"> <cfabort />

<!--- Create the attribute collection. ---><cfset ac = StructNew() /> <cfset counter = 0 /><cfoutput query="qryGetRepList" group="StateCd"> <cfset counter = counter + 1 /> <cfquery dbtype="query" name="qryGetReps"> SELECT FirstNameTx, LastNameTx, PhoneTx, RoomNb, DistrictTx, StateCd FROM qryGetRepList WHERE StateCd = '#StateCd#' ORDER BY DistrictTx </cfquery> <cfoutput><cfscript> ac["collist#counter#"] =

"StateCd,DistrictTx,LastNameTx,FirstNameTx,PhoneTx,RoomNb"; ac["headerlist#counter#"] =

"State,DistrictTx,LastNameTx,FirstNameTx,PhoneTx,RoomNb"; ac["colwidths#counter#"] = "50,75,100,100,75,50"; ac["query#counter#"] = "#qryGetReps#"; ac["sheetname#counter#"] = "#StateCd#"; </cfscript></cfoutput></cfoutput>

<cf_excel_xml filename="export" sheets="#counter#" attributecollection="#ac#"> <cfabort />

Importing and Exporting can be: Easy Difficult Both

Great tools exist – use them!Search regularly for new tags & code If you know ASP.NET, even better

Adobe ColdFusion Exchange cf_Excel_XML▪ http://www.adobe.com/cfusion/exchange

Ryan Emerle cfx_text2query cfx_excel2query cfx_query2excel▪ http://www.emerle.net/old/programming/

cfTopper Generate Excel Function▪ http://www.cftopper.com/index.cfm/blogId/1/tag/

ColdFusion

selene@ritetech.net