Control SQL Import Processes
Steps to Import Customer Part Spreadsheets
Description
This document describes the process of importing a customer parts list from the template developed during the 4DX process. The document may be a Google Document and shared with the customer while they are filling it in, but will be exported to Excel to work with it.
The file format is very specifically defined. Among the rules are:
- The first row is the column names.
- The spelling of the column names must match exactly, including spaces.
- The following column names are required in every tab:
- Import
- Active
- Part Name
- Part Description
- Unit Cost
- The following columns are supported if present but are not required:
- Part To Clone
- Part Category
- Parent Part Category
- Units
- SKU
- Any UDF
- If a Parent Part Category is given and the Part Category is not found under it, it will be created during the import.
- If a Parent Part Category is given it must exist.
- If a Part Category is given and no Parent Part Category is specified, the Part Category must exist.
- The Units must match one of the existing standard or abbreviation unit formats found in the _Unit table. Each is used if no unit is given.
- Avoid formatting with $.
Creating Import Files from The Spreadsheet
For the import, we will create a separate comma delimited file for each Tab in the spreadsheet. To make this simpler, follow these steps:
- If you are using a Google Spreadsheet, export it to Excel.
- Open the spreadsheet in Excel
- Make sure the Spreadsheet is not in ReadOnly or Restricted mode.
- Press Alt + F11 keys simultaneously to open the Microsoft Visual Basic Application window.
- In the Microsoft Visual Basic Application window, click Insert > Module.
- Copy and paste the following code routines into the Module window.
Sub ExportSheetsToCSV() Dim xWs As Worksheet Dim xPath As String xPath = "<<path>>" Dim xFile As String For Each xWs In ThisWorkbook.Worksheets xWs.Copy xFile = xPath & "\" & xWs.Name & ".csv" Application.ActiveWorkbook.SaveAs Filename:=xFile, FileFormat:=xlCSV, CreateBackup:=False Application.ActiveWorkbook.Saved = True Application.ActiveWorkbook.Close Next End Sub
For some reason, this VBA does not always work. If you get an error message, you can use this version
Sub ExportSheetsToCSV() Dim xPath As String Dim xFile As String xPath = "C:\Users\scott\Desktop\Customer Data\Tabs" 'Application.ActiveWorkbook.Path For Each xWs In ThisWorkbook.Worksheets xFile = xPath & "\" & xWs.Name & ".csv" xWs.SaveAs Filename:=xFile, FileFormat:=xlCSV, CreateBackup:=False Next Application.ActiveWorkbook.Close End Sub
- Set the directory you want the Tab-delimited files created in the XPath variable. I recommend you create a sub-directory called Tabs for these files
- Press the F5 key to run the code.
- The screen will blink a bunch. Don't worry, this is just old 1993 code so it's not the prettiest.
- You will see all exported CSV files in the specified folder.
Importing into SQL
Now that you have the CSV files, you need to get them into the Database. This routine brings in all of the CSV files into their own tables called PartImport.{filename}.
You'll have to add a lot of stored procedures and functions to the database for this to run. See Scott or a consultant for the latest versions.
Enabling CMD Shell and Ad Hoc Distributed Query Calls in SQL
In order to retrieve the list of files, SQL must make a directory call to read the contents of the folder. By default, this access is denied in SQL since it could be used by malicious software to access the local drive.
To enable these features, run these SQL commands
EXEC sp_configure 'show advanced options', 1 GO RECONFIGURE; GO sp_configure 'xp_cmdshell', 1; GO RECONFIGURE; sp_configure 'Ad Hoc Distributed Queries', 1; GO RECONFIGURE;
You also need to download and install the Access Database Engine(AccessDatabaseEngine_X64.exe). You can get the file using the link below.
https://www.microsoft.com/en-us/download/details.aspx?id=39358.
Importing the CSV Files
To import the files as parts involve these steps:
- Add the following procedures and functions to SQL if they don't exist (they probably won't)
- csp_ImportPartSpreadsheetTabs
- csp_ImportPart
- csp_ImportPartCategory
- csf_MergeXML
- csf_String_Split
- csf_TextToBit
- csf_TextToDecimal
- csf_TextToUnitID
- csf_TextToUnitType
- csp_BulkFilesToTables
- BACKUP AND RESTORE A TEST DATABASE. While I show SQLs below to delete the import, it will delete a lot of other things to and you should only plan to import on the live database after you have successfully tested everything on the backup.
- Run the following SQL command to execute the import. (You'll need to fill in the actual path first)
DECLARE @Path VARCHAR(255) = '{fill in}' , @SkipRows VARCHAR(255) = 3 ; EXEC csp_ImportPartSpreadsheetTabs @Path = @Path , @SkipRows = 3 , @DontCommit = 0 -- 1 to test only but not save, 0 to save
- Note that the import produces a lot of single row table outputs (one for each part added).
- You MUST click on the Messages tab in SQL Management Studio and scroll down the list to see any errors. Most errors will not appear on the table output and you will only know if there are problems if you scroll through that list looking for red output.
- If the UDF names are not correct, there is no warning. You'll have to check the parts to find that out.
Backing Out the Import
If you made a mistake and want to re-import, you can delete all of the parts that were imported that day with this query, fix the data, and then run it again. This will delete anything done that day, whether by you or a user so only run this on a backup of the data.
DELETE FROM Part WHERE ModifiedbyUser = 'csp_ImportPartSpreadsheet' DELETE FROM PricingElement WHERE ModifiedbyUser = 'csp_ImportPartSpreadsheet' DELETE FROM PartInventoryConversion WHERE ModifiedbyUser = 'csp_ImportPartSpreadsheet' DELETE FROM Inventory WHERE ModifiedbyUser = 'csp_ImportPartSpreadsheet' DELETE FROM InventoryLog WHERE ModifiedbyUser = 'csp_ImportPartSpreadsheet'
Notes and Areas Not Implemented
Not all areas are currently implemented. These include:
- Default Units. All units must be specified on every column. A quick copy-and-paste is usually all that is needed for this. Any units not specified are treated as Each.
- Setting products Inactive with the Active = No value.
Other Notes:
- The routine does not delete the import files when it is done (though it will overwrite them if you re-import). You may want to clean them up when you are done.