What kind of filters/ offset are you using for the exraction?
Good morning Paul,
- For CDPOS we have a Join filter with CDHDR + a Date Filter (2020+), while restricting OBJECTCLAS to what is relevant
- For /COCKPIT/THDRV we have a Join filter with /COCKPIT/ + a Date Filter (2020+)
Not sure what you meant by offset by I hope I replied accordingly
For the delta extraction to work, you should add a time filter. this you can do in your extraction settings.
Additionally, a full load is mainly a 'SELECT * FROM ...' which is simple to execute for the DB, transporting all the data is what takes time.
Delta loads fire one or several smaller more complex queries to the DB. This might be difficult for the DB to perform, hence the increase in duration. Beeing smart about your delta load settings, as Paul recommends, is key.
For the delta extraction to work, you should add a time filter. this you can do in your extraction settings.
Hello everyone, I'm part of Miriam's (OP) team
We've since applied this solution (Time Filters) to all eligible tables and it has worked great, so thank you for your answers!
We've now found ourselves in a serious bottleneck situation where two tables (/COCKPIT/THDRV and CDPOS) are holding up the whole Data Reload process (Extraction->Transformation->Data Model Reload) due to their very lengthy extraction times (6h to 11hrs).
These tables are unfortunately not eligible for a time filter due to the absence of a reliable Date field.
Is there any other way to reduce the timing of extraction?
FYI, our current filters for both these tables are essentially joins to sibling tables that do have these data fields:
- "/COCKPIT/THDRV" joined with "/COCKPIT/THDR"
- Joined Table Filter:
- "/COCKPIT/THDR"."DOC_DATE" >= <%=startDate%> OR "/COCKPIT/THDR"."PSTNG_DATE" >= <%=startDate%>
- "CDPOS" joined with CDHDR
- Joined Table Filter:
- "CDHDR"."UDATE" >= <%=startDate%> AND "CDHDR"."OBJECTCLAS" IN <%=objectclasses%>
Thank you in advance for your time!
Have I understood this correctly? A delta load without a time filter enabled is a full load?
Have I understood this correctly? A delta load without a time filter enabled is a full load?
Hello Sasa,
I'm not privy to the back-end of how delta loads work, what I can say from experience is that, depending on the size of your data set, a Delta Load without Time-Filters can take as long as (or longer) than a full load.
Have I understood this correctly? A delta load without a time filter enabled is a full load?
Hi @sasa.redze12 ,
This is not totally the same. A full load captures the entire table at that moment, while a delta load appends all the rows that are new based on the defined primary keys (if no primary keys are selected, all the rows are appended).
If you have selected delta-loads and rows are deleted from you source table, these rows are still visible in Celonis over time using delta loads, but will get lost once a full load is executed.
Kind regards,
Jan-peter
That makes it very clear. Thanks Jan-peter.