Good afternoon everyone,
We've been looking at our Delta Loads and Full Loads runtimes to try and optimize our Data Consumption and we've noticed that the Delta Loads always take more time than the Full Loads.
Is there a reason for this? Is there something we can do to have them be faster?
In the below Image you can see some examples of /COCKPIT/THDRV and CDPOS tables which are quite large
Please advise
Thank you in advance for your support
What kind of filters/ offset are you using for the exraction?
Good morning Paul,
Not sure what you meant by offset by I hope I replied accordingly
For the delta extraction to work, you should add a time filter. this you can do in your extraction settings.
Additionally, a full load is mainly a 'SELECT * FROM ...' which is simple to execute for the DB, transporting all the data is what takes time.
Delta loads fire one or several smaller more complex queries to the DB. This might be difficult for the DB to perform, hence the increase in duration. Beeing smart about your delta load settings, as Paul recommends, is key.
For the delta extraction to work, you should add a time filter. this you can do in your extraction settings.
Hello everyone, I'm part of Miriam's (OP) team
We've since applied this solution (Time Filters) to all eligible tables and it has worked great, so thank you for your answers!
We've now found ourselves in a serious bottleneck situation where two tables (/COCKPIT/THDRV and CDPOS) are holding up the whole Data Reload process (Extraction->Transformation->Data Model Reload) due to their very lengthy extraction times (6h to 11hrs).
These tables are unfortunately not eligible for a time filter due to the absence of a reliable Date field.
Is there any other way to reduce the timing of extraction?
FYI, our current filters for both these tables are essentially joins to sibling tables that do have these data fields:
Thank you in advance for your time!
Have I understood this correctly? A delta load without a time filter enabled is a full load?
Have I understood this correctly? A delta load without a time filter enabled is a full load?
Hello Sasa,
I'm not privy to the back-end of how delta loads work, what I can say from experience is that, depending on the size of your data set, a Delta Load without Time-Filters can take as long as (or longer) than a full load.
Have I understood this correctly? A delta load without a time filter enabled is a full load?
Hi @sasa.redze12 ,
This is not totally the same. A full load captures the entire table at that moment, while a delta load appends all the rows that are new based on the defined primary keys (if no primary keys are selected, all the rows are appended).
If you have selected delta-loads and rows are deleted from you source table, these rows are still visible in Celonis over time using delta loads, but will get lost once a full load is executed.
Kind regards,
Jan-peter
That makes it very clear. Thanks Jan-peter.
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
