SSIS Incremental Load

1. Have a lastUpdatedTime column in the destination table and the source table.

2. Create a string variable as LastUpdatedTime.

3. Create an SQL task pointing to the destination table and get the maximum value of the lastUpdatedTime:

SELECT max(UpdatedAt) as LastUpdated FROM DestinationTable

4. In the result set, point the LastUpdated result in the query to the LastUpdatedTime variable.

5. Create a data flow task.

6. Create another variable (sqlValue) with the following value :
\”SELECT        *
FROM            SourceTable
WHERE LastUpdatedTime >\’\”+   @[User::LastUpdatedTime]   +\”\’\”

7. Create a OLE DB source editor and select the created sqlValue variable. Select the columns from the column table that needs mapping.

8. Do the necessary data conversions and transformations.

9. Create a OLE DB destination to update the destination table with getDate() as the lastUpdatedTime column.



Leave a comment