Success Hub
Product Updates
Groups

Home
Community
Product Q&A
Data Ingestion & Infrastructure

Data Ingestion & Infrastructure

Discuss and ask questions about data integration, task mining, admin settings, and PQL extractors to optimize your infrastructure.

781 Topics
2,074 Replies

When you subscribe we will email you when there is a new topic in this category

781 Topics

Most replies

Most views

Newest first

haruka.yanoLevel 2

asked in Data Ingestion & Infrastructure

Object transformation preview once showed and disappers

HiI’m curently working on OCPM Data Pool and have trouble in an object transformation.In the Currecny Conversion object(o_celonis_CurrencyConversion),I checked that I can see data record in Preview,and executed publish to development.Then I ran the Data Job Transformation(full load), succeeded,ran the Data Model Load, succeeded,but at the end o_celonis_CurrencyConversion has no data.I don’t know why but when I go see the object preview again, the preview doesn’t show anything.Has anyone had this kind of experience ever?If yes please kindly help. FYI as the prerequisite, - the relevant sap table data is sycronized from another data pool - the object and its transformation script is celonis default one - Data job is done by the duplicate of test:ocpm-data-job(celonis default test data job) - the reason why I use the duplicate data job is that the original one returns an error for schema.

3 days ago

PhuongBHNLevel 3

asked in Data Ingestion & Infrastructure

How to enable the ‘Push data into Celonis’ Data connection option within my Data pool

Hello everyone, I am unable to locate the ‘Push data into Celonis’ Data connection option within my Data pool.Could you guide me on how to enable this feature?!--endfragment>My Celonis I want the ‘Push data into Celonis’ as the image below

413

19 days ago

MrAjLevel 3

asked in Data Ingestion & Infrastructure

How to Trigger a Machine Learning Script from a Button in Celonis?

Hi everyone,I need help figuring out how to execute a Machine Learning script directly from a button in my Celonis app in views (dashboard). I believe I might need to use Action Flows, but I’m unsure about the exact steps to make this work.Here’s what I want to achieve: When a user clicks a button in the Celonis app, it should trigger an ML script. The script could be hosted externally (e.g., a REST API) or executed within Celonis.( atm its not hosted, there is just a file in ipynb format, stored in ml workbench in analysis. I need guidance on how to set up the Action Flow and link it to the button. Has anyone implemented something similar? If so, could you share the best approach and any potential challenges I should be aware of?Thank you in advance!

140

1 month ago

MrAjLevel 3

posted in Data Ingestion & Infrastructure

How to Trigger a Machine Learning Script from a Button in Celonis?

110

1 month ago

.14600833Level 4

asked in Data Ingestion & Infrastructure

Source und Target with PROCESS_ORDER

Hello everyone,i need a throughput between two consequitive activities and want to implement it via source and target.Die idee wre source(activity_table.activity), target(activity_table.activity, process_order(target)=process_order(source)+1))would it work?Best,Maria

105

1 month ago

farouk.abdelLevel 2

asked in Data Ingestion & Infrastructure

CPM4 sizing

what is the celonis sizing guidelines for CPM4?

1 month ago

maxime.psbpmLevel 3

asked in Data Ingestion & Infrastructure

Issue with Many-to-Many (N:N) Relationships in Celonis

Hi everyone,I'm facing an issue with how Celonis handles many-to-many (N:N) relationships. It seems like Celonis treats these relationships as 1:N → 1:N instead of supporting 0:N → 0:N, which leads to missing data when filtering.Example Issue:I have a many-to-many relationship between contracts and assets, modeled with an intermediate table:Contract Asset C1 A2 C2 A1 C3 NULL C4 NULL NULL A3 NULL A4 The issue occurs when I apply a filter like "Asset ≠ A4". Instead of just removing A4, I also lose contracts C3 and C4, which should still be visible since they are not linked to any asset.My Understanding of the Issue:Celonis seems to implicitly treat N:N relationships as 1:N → 1:N, enforcing a strict link between contracts and assets. It does not seem to support a true N:N model, where both sides can have optional (0:N) relationships.Question:Has anyone faced this issue before?Is there a recommended modeling approach to ensure that filtering on assets does not remove u

805

1 month ago

lllllizLevel 2

asked in Data Ingestion & Infrastructure

How to use data pool parameter?

Dear community, I would like to set up a data pool parameter to allow me to switch between test and prod environment? So I would need two global variables for my data jobs:var_data_source: to allow me switch between test schema and prod schema var_table_name: to allow me when creating table to add _test at the end of the table nameI have only found the way to create parameters in EACH data job under transformation and refer to the data pool parameter. Is there a smarter way? I am still using the data integration (not OCPM). Thanks a lot!

100

1 month ago

lize.erasm11Level 3

asked in Data Ingestion & Infrastructure

Adding attributes to a event in a OCPM model

Hi, has anyone found a easy or quick way to add a large list of attributes to a event?

191

1 month ago

PhuongBHNLevel 3

asked in Data Ingestion & Infrastructure

How to connect S3 without using access key?

I need to create a connection from Celonis - Data Integration to AWS S3 without using an access key, as our policy requires password rotation every 90 days, and I prefer to avoid manual updates. Do you have any suggestions on how to achieve this?

2 months ago

Juan PalenciaLevel 2

asked in Data Ingestion & Infrastructure

How to Reduce Data Job Execution Time for Large Volumes (e.g., 824M Rows) in SAP Saturn?

Hi Celonis Community, I’m working on optimizing data jobs in Celonis Data Integration for a client with an extremely heavy SAP system (Saturn). Our data jobs currently take around 1.5 hours to process, and the client wants to reduce this to 0.5 hours. Despite optimizing SQL queries (e.g., early filtering, minimizing joins) and disabling unnecessary objects, the massive data volume seems to be the main bottleneck. Here are some examples of the volumes we’re handling: - SalesOrderScheduleLine Transformation: 119M rows in CDPOS, 59M in CDHDR, cost of 24M.- CreditMemoItem Transformation: 123M in VBAP, 824M in VBFA, 178M in VBRP, cost of 83M.- CustomerInvoice Transformation: Similar volumes, 824M in VBFA, 178M in VBRP, cost of 83M. We’ve explored strategies like incremental loads and partitioning, but the sheer volume from Saturn (e.g., 824M rows in VBFA) still slows us down. I found in the documentation that the Payment Term Checker slows with 100M+ invoices, suggesting infrastructure migh

2 months ago

HafsaLevel 2

asked in Data Ingestion & Infrastructure

Preview not available "data model load" -

Hello, I'm working on connecting serviceNow to Celonis, I did Task Extraction (OK) → Task Transformation (OK) and when I loaded data it works (loaded rows OK), but when I press "set as activity table" it gives me the message : "Preview not available". A table preview is currently not available. The cause may be due to a missing data pool or connectivity issues” without columns to select “Case ID”, “brief description”, “date”.I need to know why the columns are not showing up in views (after transformation task) but showing up in the original tables….for information: I tested the data in task extraction, task transformation and loaded data model and they contain all data of the views. I also execute data job for all, but the columns still don't appear.Thank you in advance for your help,

583

2 months ago

marietta.kallaLevel 2

asked in Data Ingestion & Infrastructure

Data Model load failed

Hi everyone,I received the following error message during the last few data uploads:The load failed: Load failed. Error Message: Could not execute foreign key join: there are duplicates on both sides of the specified key relationship. Table "PM_AFRU": key columns ("MANDT", "AUFPL", "APLZL"), key ('100', '1016228353', '00000001'), rows 0 and 1. Table "AFVC": key columns ("MANDT", "AUFPL", "APLZL"), key ('100', '1018097598', '00000001'), rows 1603274 and 1608762.Everything was working fine about a year ago, and I have checked the table relationships—they are correct. What could be the issue?Can anyone help?THX, M

322

2 months ago

herique.sarmeLevel 3

asked in Data Ingestion & Infrastructure

Every Excel module in Action flow stopped working, help

Out of the blue, every single Excel module in Action Flows has stopped working. Has anyone gone through something similar recently? Has any announcement on this topic been done by Celonis?The screenshot of the error is meant to help better understand the issue as it seems to be a simple ‘access’ problem, but every account that’s been used within those modules has the same issue and all of then started at the same time, so it feels something structural is going on…Either way, if anyone has some insight on this it would be greatly appreciated! Thanks!

272

2 months ago

prasanth.rajaLevel 2

asked in Data Ingestion & Infrastructure

N:1:N connection fix

I am trying to retrieve a count of a specific column based on a **CASE WHEN** condition using PQL. However, I am encountering the following error: *"A PQL error is preventing this query from executing. No common table could be found. The tables ["LINK"] and ["JS_PGT_Resolved"] are connected, but have no common table. This means that they do not have a direct (or indirect) 1:N or N:1 relationship. Join path: [LINK]N <-- 1![SUPPORTCASE]!1 --> N[JS_PGT_Resolved]. For more information on the join path, search for 'Join functionality' in PQL documentation."* I would appreciate any guidance on how to resolve this issue. Thank you!

752

2 months ago

robin.pagteLevel 2

asked in Data Ingestion & Infrastructure

Use pyCelonis to back up entire datapool

Hi,I was wondering the following: I want to export my entire data pool (extraction, transformations and data models) using the pyCelonis API. I have been trying to code it myself but there doesn't seem to be a straight forward way to do it. I was wondering if any of you have been using this method and can help me find the right code to use. The use case is to back up our work once per day. A very nice to have would be an additional script to use the exported data pools to automatically recreate the back-upped work. In other words: Run script > back up written to files > run script > recreate celonis environment in case of total loss of data

270

2 months ago

shaifLevel 3

asked in Data Ingestion & Infrastructure

Amazon connectivity best practices

Dear Experts, If there is decision to be made in terms of best practices on connectivity to S3 , could you comment on option 1 : connecting S3 bucket directly which contains source tables option 2 : transforming data from S3 using databricks converting into activity and case tables and placing it to seperate bucket and then data shared with celonis ? Thanks, Shaif

2 months ago

rishabh.agrawalLevel 2

asked in Data Ingestion & Infrastructure

Delta Extraction

Hey all looking for some insights whether this will work or not if I want a previous years data e.g ( I have 2023-2025 data in CDHDR but I also want 2022 ) can I change the UDATE filter from ( 2023 to NOW ) to ( 2022 to NOW) and disable the change date check and do a DELTA load to only extract fresh records that are not present in celonis ( even if it’s from the previous year of already ingested data )

311

3 months ago

madhava.pemmaLevel 2

asked in Data Ingestion & Infrastructure

Mapping of Celonis Activities with SAP activities

Can anyone has this mapping information?I would like to see the list that shows the mapping of Celonis activities (eg: AP process) to SAP tables.Like while calculating the clear invoice steps which tables, timestamps are used in SAP. Similarly for other activities. This will be a good reference document to begin with and help to customize the activities in Celonis if we need to do so.Appreciate your inputs here.

142

3 months ago

daniel.ziolkLevel 2

asked in Data Ingestion & Infrastructure

Creating CSV file from Query as output SKILL has been used with email by celonis added attachmet file from Action Flow

SKILL:OUTPUT ATTACHMENT:Can someone explain why this is happening that encoding is not perform - what do I missed?

293

3 months ago

manuel.wetzeLevel 5

posted in Data Ingestion & Infrastructure

Analyze / Visualize Data Lineage of SQL backend

Hi all,I am lately thinking of how to make my life as Data Engineer easier and get a up-to-date documentation of the current backend relationships. The goal is to see: which tables from source are transformed into which VIEWS / TABLES downstream. I want to have a quick and accurate answer to:Which tables are sources for a given table in the datamodel and how are they connected? What is the potential impact on downstream tables if I change something on a given extraction. Which tables are not in use? Potentially even better would be a granularity of column and not only View/TableI believe those questions belong all to the concept of Data Lineage. Which in my opinion is currently very poorly supported within Celonis.I would like to open the discussion on how to best do this and if this is a relevant problem to more people:What is you workflow when you face the questions above? (How) are you guys documenting the SQL Backend? Is it done manually?

584

3 months ago

rafael.souza11Level 2

asked in Data Ingestion & Infrastructure

Unable to recreate Postman API configuration Bearer Authentication on a custom extractor builder

Hi, I'm using an API to extract datafor this purpose, the scope of Celonis, I am trying to adapt the configuration parameters that are 100% functional in POSTMAN, but when building Extractor I am finding it difficult to adapt 1) The API Resource first requires a POST request to generate the Bearer Token: Request POST: https://prod.api.wehandle.com.br/loginParamsAuthorizationHeader:Body:Scripts:Result Body Response:2) The second request uses the generated Token to GET data from the API Request GET: https://prod.api.wehandle.com.br/integrations/main/employees?page=1&limit=50&changes_to=2025-01-14T23:59:59.000Z&changes_from=2025-01-01T00:00:00.000Z&employee_address=true&employee_services=true&employee_manager=true&employee_supplier_company=trueParamsAuthorizationHeadersResult Body ResponseI'm having trouble recreating this integration successfully via Extractor BuilderPerhaps my starting point would be to decide the Authentication Method betweenBearer Authentic

140

3 months ago

koert.kritz11Level 3

asked in Data Ingestion & Infrastructure

Previous month KPI filter

Hi everyone, I need help with a filter.I want to put a filter on a kpi that only allows it to show the previous months data. The filter must be dynamic so that when I select a month from the date picker that it would always calulate the previous months data.I am currently using this PQL: FILTER "CEL_ACTIVITIES_OTT"."first_eventtime" >=ADD_MONTHS(PU_MAX ( CONSTANT ( ), FILTER_TO_NULL(ROUND_MONTH("CEL_ACTIVITIES_OTT"."first_eventtime"))),-1)AND"CEL_ACTIVITIES_OTT"."first_eventtime"<PU_MAX ( CONSTANT ( ),FILTER_TO_NULL( ROUND_MONTH("CEL_ACTIVITIES_OTT"."first_eventtime")))The problem is that I have to select both months from the date picker and Ideally I would like to only select one month.

493

3 months ago

thomas.guerrLevel 3

asked in Data Ingestion & Infrastructure

Extractor configuration for data transfer from MS Sharepoint to Celonis

Hello,I am trying to get data from a Microsoft sharepoint list to the Celonis Platform.According to the Flowchart provided by the API-Team - When to use the Standard Data Ingestion API (and other methods) - the extractor builder should be used for a Sharepoint data transfer, since the REST API is exposed.I tried to set up the connection, but I fail when configuring the endpoint, since I get xml-data even if I specify that I would like to have a json structure (Request Parameter - Accept: application/json;odata=verbose)Does anyone have experience with connecting the MS universe with the Celonis universe?I also found a workaround that works (Push CSV to Data Integration), but I would rather orchestrate all the data loads in the Data Integration environment instead of using Process Automation modules to make a data transfer.Thanks for your support!BR THomas

510

3 months ago

jakob.hanseLevel 2

asked in Data Ingestion & Infrastructure

Staging Table for inserts in Replication Cockpit

Hi,We have a scenario where records can be created/deleted in source system multiple times.We are currently using the replication cockpit for extracting changes, but for deleted records we do not get the full information of the deleted records, we only get the primary key of the table and a change indicator = ‘D’ (Deleted) and we would like to know what the values of all the attributes for the specific records were prior to the deletion. We have located the following staging tables and views:_CELONIS_TMP_{TableName}_NEW_DATA _CELONIS_TMP_{TableName}_TRANSFORM_DATA {TableName}_DELETED_DATA Unfortunately, it seems like the _CELONIS_TMP_{TableName}_NEW_DATA table only includes records with change indicator = ‘U’ (Updates) and not change indicator = ‘I’ (Inserts). This makes it difficult for us to map what the historical values were, if we have both inserts and deletions within the same execution of the replication cockpit.Does anyone know of a staging table for inserts we can use, befor

231

3 months ago

Page 1 / 32

Badge winners

Vladimir Smoleyhas earned the badge Value Realization Expert
Vladimir Smoleyhas earned the badge Qualified to Deliver Business Value with Celonis
zayneb.alsum11has earned the badge Qualified to Manage Change and Scale Adoption
karin.khourhas earned the badge Qualified to Create and Deliver Demos
karamdeep.singhhas earned the badge Celonis Certified Data Engineer