Skip to main content
Question

Best way for Delta Load from HANA Database to avoid any historical data added at a later date on HANA Database

  • November 1, 2022
  • 4 replies
  • 15 views
D
Level 5

Hello,

Could you please advise how to avoid missing on data which was loaded to HANA Database after the extraction was done? In my case, Data is being loaded and consolidated onto HANA (no direct connection to SAP but through HANA database) and a delta load is run for Celonis as per timestamps. Now if some new business segment is added, any data prior to last "delta load" is not getting ingested to Celonis as it belongs to older dates. I know a full data load for a period can do the trick but that's not the efficient way. Any suggestions please?

4 replies

G
Level 12

I would say that Realtime ingestion with Realtime Cockpit. It will send you all changes detected on the triggers you deploy. And, in theory, that will include those "old prior data"

 

Also, if you have some "updated timestamp" in the HANA Schema that will solve your problem, using that field for the delta loads.

 

HTH

D
Level 5

I would say that Realtime ingestion with Realtime Cockpit. It will send you all changes detected on the triggers you deploy. And, in theory, that will include those "old prior data"

 

Also, if you have some "updated timestamp" in the HANA Schema that will solve your problem, using that field for the delta loads.

 

HTH

Thanks Guillermo, will need more info. Is there any link I can refer to?

G
Level 12

Thanks Guillermo, will need more info. Is there any link I can refer to?

Replication Cockpit - https://docs.celonis.com/en/replication-cockpit.html

Realtime for SAP/HANA -

 

In short you define triggers and log tables in SAP so any change is replicated there. Then Celonis picks that info and via Replication Cockpit you get a near-realtime upload of the changes.

 

Regarding delta dataloads, for example in our current customer we have modifyts (when has been changed the record). Also our CASE_ID is composed of the timestamp when that case was created, plus some additional codes.

 

HTH

C
Level 2
  • carmo.melo
  • Level 2
  • June 28, 2023

Thanks Guillermo, will need more info. Is there any link I can refer to?

Hi!

I liked the use of Data Integration to replicate the data, but it makes me understand the difficulty that SAP/HANA has today in having a column for the date of change in the record with a timestamp. This would solve the problem of delta loads; whenever this record was included, the change date would be the same too, and then it would be incremented with each change.

Community Code of ConductAccessibility statement