Skip to main content
Question

Best practices in transitioning to Replication Cockpit real-time data pipeline

  • June 26, 2023
  • 1 reply
  • 17 views
mario.aires
Level 1

Hello everyone,

 

We are in the process of implementing Replication Cockpit and have several questions about the best way to adapt our current pipelines to this new paradigm:

 

1- Our business uses multiple SAP systems and we need to unify tables across them. Is there a way to do this in real time through Replication Cockpit or should we keep the unification scripts on a scheduled data job that runs before the data load?

 

2- For optimization purposes, we make extensive use of filter joins during extraction to restrict our tables to just the relevant data. Outside initialization joins, what is the recommended way to maintain this behavior for replications? Should we create a transformation that deletes all rows from Table A where there are no matching entries in Table B?

 

3- It is our understanding that once a table is being extracted through Replication Cockpit we should remove any scheduled extraction involving it to avoid conflicts. Is this understanding correct?

 

4- Once a table is actively replicating, is there any reason to reinitialize it regularly unless the extraction parameters change or an error occurs?

 

Thanks in advance!

    1 reply

    V
    Celonaut

    Hi Mario:

     

    Regarding your questions:

     

    1- Our business uses multiple SAP systems and we need to unify tables across them. Is there a way to do this in real time through Replication Cockpit or should we keep the unification scripts on a scheduled data job that runs before the data load?

    Did you try the "Rename target table" option in the Replication configuration? 

     

    Screen Shot 2023-08-10 at 17.33.38If this feature does not fulfill your request, could you please share with us more details regarding the request?

     

    2- For optimization purposes, we make extensive use of filter joins during extraction to restrict our tables to just the relevant data. Outside initialization joins, what is the recommended way to maintain this behavior for replications? Should we create a transformation that deletes all rows from Table A where there are no matching entries in Table B?

    There are two kinds of filters, like we have in regular data job extractions. Replication Filters and Initialization Join Filters. Replication Filters are applied to every replication, and can only be done on the basis of the table being replicated (i.e. when Replicating VBAK, you can only filter on a VBAK column). Initialization Join Filters allow you to pre-filter the first (full) load of a table on the basis of another (i.e. when initializing VBAK, you can join to VBAP and filter on PSTYV, for example). You cannot join two (or more) tables together to filter at replication time. This can only be done at initialization time

     

    3- It is our understanding that once a table is being extracted through Replication Cockpit we should remove any scheduled extraction involving it to avoid conflicts. Is this understanding correct?

    Make sure to remove the tables added to the Replication Cockpit tables in the Delta Data Jobs. There should be no conflict between Replication Cockpit and data jobs, i.e. the same table should not be replicated via Replication Cockpit and delta loaded via data jobs. Otherwise, execution conflicts may occur if these two tools try to write data to the same EMS table concurrently.

     

    4- Once a table is actively replicating, is there any reason to reinitialize it regularly unless the extraction parameters change or an error occurs?

    Our recommendation is that you should run initializations if

    1. You are replicating a table for the first time
    2. You are adding columns to the table being replicated

     

    THank you for your understanding and patience.

     

    Best regards,

    Valmira Abdiu

    Community Code of ConductAccessibility statement