Dear data engineers in the Celonis community,
At the beginning of this year, we have released a feature that allows you to specify which Data Models are to be loaded after a successful Data Job execution: https://help.celonis.cloud/help/display/CIBC/Specify+Data+Model+Loads+in+Data+Jobs+to+make+the+pipeline+more+efficient
This feature allows you to have more control on which Data Models are loaded in your pipelines. This feature is used by many of you already which is great. However, they are still a lot of teams on our platform which choose to load all Data Models after a schedule execution which was the previous behaviour, but one which can be completely replaced with the new feature.
The old feature to reload all Data Models after a schedule execution will no longer be available after January 15th, 2021. Therefore, we encourage everyone to change their pipelines as soon as possible in the following way according to your situation:
Case 1 You have already adopted the feature to choose the Data Models in Data Jobs and you no longer use Reload all Data Models after a schedule execution
Action: No action is required.
Case 2 You have one or more schedules that reloads all Data Models after a schedule execution and you want to keep the pipelines exactly as they are.
Action: Our recommendation is to create a new Data Job that contains all the Data Models of the Data Pool to which the schedule belongs and add this Data Model Data Job to the schedule. Afterward, remove the Reload all Data Models after execution flag within the schedule configuration.
Case 3 You have one or more schedules that reloads all Data Models after a schedule execution and you actually want to reload only some Data Models after these schedule executions.
Action: Add the desired Data Models to the last Data Job of the schedule or create a new Data Job with only the desired Data Model loads and add it to the schedule in the last position. Afterward, remove the Reload all Data Models after execution flag within the schedule configuration.
In case you do not change your pipelines until January 15th, 2021 the schedules will still run, but they will not reload Data Models anymore unless they are specified within Data Jobs.
Best regards,
Leonid