Skip to main content

Hello,

 

how do you deal best with deleted records in Delta extractions? For a set of tables, we see, that delta jobs do not account for records that are deleted in the data source.

E.g. when we extract a table and at that specific point in time it has - let's say - 50 records, but in the meantime, 25 of them get deleted, we have different data in Celonis and in the data source. The next delta job will not remove those records in Celonis, but only add changes or new records. Only a full extraction will overwrite the table and in the process of doing so, remove the deleted records.

The question therefore is, what is best practice here?

What options other than the Real-Time-Extractor and running full jobs daily do we have?

 

Thanks!

Hi Thomas, You can select between 2 different deletion options on table level: Directly delete data from the Data Pool table: In case a deletion in an SAP table happens it will be captured in the changelog via the trigger. By selecting this option, the deleted records will directly be deleted from the table in Vertica after the Extraction. Please note, that data that is archived in SAP is also captured as a deletion and therefore deleted in Celonis. With this option you will not be able to do transformations with the deleted data (for this you can refer to option 2). When should I choose this option? This option allows you to exactly mirror the table of your source system in Event Collection. If you do not regularly archive data in SAP and it is fine to have archived data deleted in Celonis, then choose this option. Store deleted data in staging table TABLE_DELETED_DATA: This option pushes all records that are captured as deletion to a separate staging table 'TABLE_DELETED_DATA'. The records are not directly deleted from the table in Vertica. There is no mechanism to clean the staging table automatically. When should I choose this option? This option allows to keep track of the deletions without directly deleting all the respective records. Please find this online page for more details: https://help.celonis.cloud/help/display/CIBC/Replication+Cockpit+FAQ Best regards, Blerta

Hi Blerta,

 

thanks for the feedback! I see, with the Real-Time Extractor & Replication Cockpit this would be perfect.

Are there any other options for data pools that cannot use the Real Time Extractor?

 

Appreciate the feedback!

Thomas

Community Code of ConductAccessibility statement