Tech Stack - Azure Data Factory, Data warehouse, HDInsights, Data Lake, SQL Server Automation - Rest Apis, ARM templates Overview of Steps-- -->Prepared source and destination data stores. -->Created a data factory. -->Created a self-hosted integration runtime. -->Created source, sink, and watermark datasets. -->Created linked services for respective datasets. -->Created a ForEach activity that iterates through a list of source table names that is passed as a parameter to the pipeline -->Created two lookup activities. Use the first Lookup activity to retrieve the last watermark value. Use the second Lookup activity to retrieve the new watermark value -->Created a Copy activity that copies rows from the source data store with the value of the watermark column greater than the old watermark value and less than the new watermark value -->Created a StoredProcedure activity that updates the watermark value for the pipeline that runs next time. --> Monitoring the pipelines for any changes or updates --> Automating the process using Rest APIs and ARM templates