Skip to main content

Data Ingestion & Infrastructure

Discuss and ask questions about data integration, task mining, admin settings, and PQL extractors to optimize your infrastructure.

  • 774 Topics
  • 2,062 Replies
774 Topics
maxime.psbpm
Level 2
maxime.psbpmLevel 2
asked in Data Ingestion & Infrastructure
Issue with Many-to-Many (N:N) Relationships in Celonis

Hi everyone,I'm facing an issue with how Celonis handles many-to-many (N:N) relationships. It seems like Celonis treats these relationships as 1:N → 1:N instead of supporting 0:N → 0:N, which leads to missing data when filtering.Example Issue:I have a many-to-many relationship between contracts and assets, modeled with an intermediate table:Contract Asset C1 A2 C2 A1 C3 NULL C4 NULL NULL A3 NULL A4 The issue occurs when I apply a filter like "Asset ≠ A4". Instead of just removing A4, I also lose contracts C3 and C4, which should still be visible since they are not linked to any asset.My Understanding of the Issue:Celonis seems to implicitly treat N:N relationships as 1:N → 1:N, enforcing a strict link between contracts and assets. It does not seem to support a true N:N model, where both sides can have optional (0:N) relationships.Question:Has anyone faced this issue before?Is there a recommended modeling approach to ensure that filtering on assets does not remove u

80
maxime.psbpm
Level 2
PhuongBHN
Level 2
PhuongBHNLevel 2
asked in Data Ingestion & Infrastructure
How to connect S3 without using access key?

I need to create a connection from Celonis - Data Integration to AWS S3 without using an access key, as our policy requires password rotation every 90 days, and I prefer to avoid manual updates. Do you have any suggestions on how to achieve this?

70
PhuongBHN
Level 2
Juan Palencia
Level 2
Juan PalenciaLevel 2
asked in Data Ingestion & Infrastructure
How to Reduce Data Job Execution Time for Large Volumes (e.g., 824M Rows) in SAP Saturn?

Hi Celonis Community, I’m working on optimizing data jobs in Celonis Data Integration for a client with an extremely heavy SAP system (Saturn). Our data jobs currently take around 1.5 hours to process, and the client wants to reduce this to 0.5 hours. Despite optimizing SQL queries (e.g., early filtering, minimizing joins) and disabling unnecessary objects, the massive data volume seems to be the main bottleneck. Here are some examples of the volumes we’re handling: - SalesOrderScheduleLine Transformation: 119M rows in CDPOS, 59M in CDHDR, cost of 24M.- CreditMemoItem Transformation: 123M in VBAP, 824M in VBFA, 178M in VBRP, cost of 83M.- CustomerInvoice Transformation: Similar volumes, 824M in VBFA, 178M in VBRP, cost of 83M. We’ve explored strategies like incremental loads and partitioning, but the sheer volume from Saturn (e.g., 824M rows in VBFA) still slows us down. I found in the documentation that the Payment Term Checker slows with 100M+ invoices, suggesting infrastructure migh

60
Juan Palencia
Level 2
Hafsa
Level 2
HafsaLevel 2
asked in Data Ingestion & Infrastructure
Preview not available "data model load" -

Hello, I'm working on connecting serviceNow to Celonis, I did Task Extraction (OK) → Task Transformation (OK) and when I loaded data it works (loaded rows OK), but when I press "set as activity table" it gives me the message :  "Preview not available". A table preview is currently not available. The cause may be due to a missing data pool or connectivity issues” without columns to select “Case ID”, “brief description”, “date”.I need to know why the columns are not showing up in views (after transformation task) but showing up in the original tables….for information: I tested the data in task extraction, task transformation and loaded data model and they contain all data of the views. I also execute data job for all, but the columns still don't appear.Thank you in advance for your help,

463
abhishek.chatu14
Level 7
marietta.kalla
Level 2
marietta.kallaLevel 2
asked in Data Ingestion & Infrastructure
Data Model load failed

Hi everyone,I received the following error message during the last few data uploads:The load failed: Load failed. Error Message: Could not execute foreign key join: there are duplicates on both sides of the specified key relationship. Table "PM_AFRU": key columns ("MANDT", "AUFPL", "APLZL"), key ('100', '1016228353', '00000001'), rows 0 and 1. Table "AFVC": key columns ("MANDT", "AUFPL", "APLZL"), key ('100', '1018097598', '00000001'), rows 1603274 and 1608762.Everything was working fine about a year ago, and I have checked the table relationships—they are correct. What could be the issue?Can anyone help?THX, M

202
marietta.kalla
Level 2
herique.sarme
Level 3
herique.sarmeLevel 3
asked in Data Ingestion & Infrastructure
Every Excel module in Action flow stopped working, help

Out of the blue, every single Excel module in Action Flows has stopped working. Has anyone gone through something similar recently? Has any announcement on this topic been done by Celonis?The screenshot of the error is meant to help better understand the issue as it seems to be a simple ‘access’ problem, but every account that’s been used within those modules has the same issue and all of then started at the same time, so it feels something structural is going on…Either way, if anyone has some insight on this it would be greatly appreciated! Thanks! 

262
abhishek.chatu14
Level 7
prasanth.raja
Level 2
prasanth.rajaLevel 2
asked in Data Ingestion & Infrastructure
N:1:N connection fix

I am trying to retrieve a count of a specific column based on a **CASE WHEN** condition using PQL. However, I am encountering the following error:   *"A PQL error is preventing this query from executing. No common table could be found. The tables ["LINK"] and ["JS_PGT_Resolved"] are connected, but have no common table. This means that they do not have a direct (or indirect) 1:N or N:1 relationship. Join path: [LINK]N <-- 1![SUPPORTCASE]!1 --> N[JS_PGT_Resolved]. For more information on the join path, search for 'Join functionality' in PQL documentation."*   I would appreciate any guidance on how to resolve this issue. Thank you! 

542
prasanth.raja
Level 2
robin.pagte
Level 2
robin.pagteLevel 2
asked in Data Ingestion & Infrastructure
Use pyCelonis to back up entire datapool

Hi,I was wondering the following: I want to export my entire data pool (extraction, transformations and data models) using the pyCelonis API. I have been trying to code it myself but there doesn't seem to be a straight forward way to do it. I was wondering if any of you have been using this method and can help me find the right code to use.  The use case is to back up our work once per day. A very nice to have would be an additional script to use the exported data pools to automatically recreate the back-upped work.  In other words: Run script > back up written to files > run script > recreate celonis environment in case of total loss of data

220
robin.pagte
Level 2
shaif
Level 3
shaifLevel 3
asked in Data Ingestion & Infrastructure
Amazon connectivity best practices

Dear Experts, If there is decision to be made in terms of best practices on connectivity to S3  , could you comment on option 1 : connecting S3 bucket directly which contains source tables option 2 : transforming data from S3 using databricks converting into activity and case tables and placing it to seperate bucket and then data shared with celonis ? Thanks, Shaif 

70
shaif
Level 3
rishabh.agrawal
Level 2
rishabh.agrawalLevel 2
asked in Data Ingestion & Infrastructure
Delta Extraction

Hey all looking for some insights whether this will work or not if I want a previous years data e.g ( I have 2023-2025 data in CDHDR but I also want 2022 ) can I change the UDATE filter from ( 2023 to NOW ) to ( 2022 to NOW) and disable the change date check  and do a DELTA load to only extract fresh records that are not present in celonis ( even if it’s from the previous year of already ingested data )

221
kailash.potha12
Level 5
M
Level 2
madhava.pemmaLevel 2
asked in Data Ingestion & Infrastructure
Mapping of Celonis Activities with SAP activities

Can anyone has this mapping information?I would like to see the list that shows the mapping of Celonis activities (eg: AP process) to SAP tables.Like while calculating the clear invoice steps which tables, timestamps are used in SAP. Similarly for other activities. This will be a good reference document to begin with and help to customize the activities in Celonis if we need to do so.Appreciate your inputs here.

132
Adam Smith
Level 2
daniel.ziolk
Level 2
daniel.ziolkLevel 2
asked in Data Ingestion & Infrastructure
Creating CSV file from Query as output SKILL has been used with email by celonis added attachmet file from Action Flow

SKILL:OUTPUT ATTACHMENT:Can someone explain why this is happening that encoding is not perform - what do I missed?

253
matt.witty13
Level 5
manuel.wetze
Level 4
manuel.wetzeLevel 4
posted in Data Ingestion & Infrastructure
Analyze / Visualize Data Lineage of SQL backend

Hi all,I am lately thinking of how to make my life as Data Engineer easier and get a up-to-date documentation of the current backend relationships. The goal is to see: which tables from source are transformed into which VIEWS / TABLES downstream. I want to have a quick and accurate answer to:Which tables are sources for a given table in the datamodel and how are they connected? What is the potential impact on downstream tables if I change something on a given extraction. Which tables are not in use? Potentially even better would be a granularity of column and not only View/TableI believe those questions belong all to the concept of Data Lineage. Which in my opinion is currently very poorly supported within Celonis.I would like to open the discussion on how to best do this and if this is a relevant problem to more people:What is you workflow when you face the questions above? (How) are you guys documenting the SQL Backend? Is it done manually?

504
manuel.wetze
Level 4
rafael.souza11
Level 2
rafael.souza11Level 2
asked in Data Ingestion & Infrastructure
Unable to recreate Postman API configuration Bearer Authentication on a custom extractor builder

Hi, I'm using an API to extract datafor this purpose, the scope of Celonis, I am trying to adapt the configuration parameters that are 100% functional in POSTMAN, but when building Extractor I am finding it difficult to adapt 1) The API Resource first requires a POST request to generate the Bearer Token: Request POST: https://prod.api.wehandle.com.br/loginParamsAuthorizationHeader:Body:Scripts:Result Body Response:2) The second request uses the generated Token to GET data from the API Request GET: https://prod.api.wehandle.com.br/integrations/main/employees?page=1&limit=50&changes_to=2025-01-14T23:59:59.000Z&changes_from=2025-01-01T00:00:00.000Z&employee_address=true&employee_services=true&employee_manager=true&employee_supplier_company=trueParamsAuthorizationHeadersResult Body ResponseI'm having trouble recreating this integration successfully via Extractor BuilderPerhaps my starting point would be to decide the Authentication Method betweenBearer Authentic

130
rafael.souza11
Level 2
koert.kritz11
Level 3
koert.kritz11Level 3
asked in Data Ingestion & Infrastructure
Previous month KPI filter

Hi everyone, I need help with a filter.I want to put a filter on a kpi that only allows it to show the previous months data. The filter must be dynamic so that when I select a month from the date picker that it would always calulate the previous months data.I am currently using this PQL: FILTER "CEL_ACTIVITIES_OTT"."first_eventtime" >=ADD_MONTHS(PU_MAX ( CONSTANT ( ), FILTER_TO_NULL(ROUND_MONTH("CEL_ACTIVITIES_OTT"."first_eventtime"))),-1)AND"CEL_ACTIVITIES_OTT"."first_eventtime"<PU_MAX ( CONSTANT ( ),FILTER_TO_NULL( ROUND_MONTH("CEL_ACTIVITIES_OTT"."first_eventtime")))The problem is that I have to select both months from the date picker and Ideally I would like to only select one month. 

403
A
Celonaut
thomas.guerr
Level 3
thomas.guerrLevel 3
asked in Data Ingestion & Infrastructure
Extractor configuration for data transfer from MS Sharepoint to Celonis

Hello,I am trying to get data from a Microsoft sharepoint list to the Celonis Platform.According to the Flowchart provided by the API-Team - When to use the Standard Data Ingestion API (and other methods) - the extractor builder should be used for a Sharepoint data transfer, since the REST API is exposed.I tried to set up the connection, but I fail when configuring the endpoint, since I get xml-data even if I specify that I would like to have a json structure (Request Parameter - Accept: application/json;odata=verbose)Does anyone have experience with connecting the MS universe with the Celonis universe?I also found a workaround that works (Push CSV to Data Integration), but I would rather orchestrate all the data loads in the Data Integration environment instead of using Process Automation modules to make a data transfer.Thanks for your support!BR THomas

380
thomas.guerr
Level 3
jakob.hanse
Level 2
jakob.hanseLevel 2
asked in Data Ingestion & Infrastructure
Staging Table for inserts in Replication Cockpit

Hi,We have a scenario where records can be created/deleted in source system multiple times.We are currently using the replication cockpit for extracting changes, but for deleted records we do not get the full information of the deleted records, we only get the primary key of the table and a change indicator = ‘D’ (Deleted) and we would like to know what the values of all the attributes for the specific records were prior to the deletion. We have located the following staging tables and views:_CELONIS_TMP_{TableName}_NEW_DATA  _CELONIS_TMP_{TableName}_TRANSFORM_DATA {TableName}_DELETED_DATA Unfortunately, it seems like the _CELONIS_TMP_{TableName}_NEW_DATA table only includes records with change indicator = ‘U’ (Updates) and not change indicator = ‘I’ (Inserts). This makes it difficult for us to map what the historical values were, if we have both inserts and deletions within the same execution of the replication cockpit.Does anyone know of a staging table for inserts we can use, befor

181
jakob.hanse
Level 2
christoph.kronb
Level 2
christoph.kronbLevel 2
asked in Data Ingestion & Infrastructure
Celonis STXL converter program for SAP R/3

We need for a use case the text which is stored in the SAP table STXL.I know that Celonis provides a converter program for the table STXL.https://docs.celonis.com/en/extracting-the-table-stxl.html We currently using SAP R/3 and S4H. Form the IT Department I got the message that the Celonis program only works for S4H. My question is, if some knows if it is possible to use the program for R/3 and what for changes we must do that it works for R/3. I hope someone can help me.Regards, Christoph

150
christoph.kronb
Level 2
raman.silot11
Level 3
raman.silot11Level 3
asked in Data Ingestion & Infrastructure
Delta filter using joined table

Hi everyone, I am currently working on optimising extractions in Celonis and applying delta for tables. But tables like BSEG, RSEG and CDPOS are not having the relevant date fields nor I can apply header table filters which I joined with these tables in delta filters. Is there any possible solution to apply delta on these tables ?

371
ilker.karak11
Level 3
celopeers.musup
Level 3
celopeers.musupLevel 3
asked in Data Ingestion & Infrastructure
Knowledge Model | APC [Data Consumption) Space tequirements / impact.

Knowledge Model | APC [Data Consumption) Space requirements / impact.Does a Knowledge Model have any APC [Data Consumption) Space requirements / impact?If yes, how can this be determined?  The space size is required to be known as to discern if we’ll exceed APC space.

261
manuel.wetze
Level 4
A
Level 3
ayushi.ade12Level 3
asked in Data Ingestion & Infrastructure
Hi All,Any best practices on optimizing the APC consumption or any solutions.

Hi All, Any best practices on optimizing the APC consumption or any solutions.

514
celopeers.musup
Level 3
johann.riede
Level 3
johann.riedeLevel 3
asked in Data Ingestion & Infrastructure
Setting up a OAuth 2.0 connection to SuccessFactors

Hi community, has anyone managed to set up a OAuth 2.0 connection to SAPs Success Factors yet?We're running in some issues there and I'd be happy to know if there are users out there who were able to successfully connect the two systems. Best regards!Johann

134
K
Level 2
J
Level 2
jonas.venneLevel 2
asked in Data Ingestion & Infrastructure
Hi Everyone,I am trying to Filter my data/table for entries, if two columns are empty. If they are empty I don't want to see them at all (Filter them from all my data).But the Filter (X!='' and Y!='') is not working.

This Screenshot show my unfilteres table:As you can see I got a lot of data where my resultsqueeze and resultvalidation is empty. I want to filter those entries. What i did is Filter the whole sheet with the following statement:FILTER ("reportdataextraction"."resultsqueeze" !='' and "reportdataextraction"."resultvalidation" !=''); But the statement is filtering every entry where either one of the columns is empty. This is my result:  What am i doing wrong? Thank you for the help! Bests,Jonas Vennemann

173
dxb.miner
Level 3
henning.gertz
Level 3
henning.gertzLevel 3
asked in Data Ingestion & Infrastructure
PQL Formula

The data model looks like this. (1) is VBAP(2) is VBEP(3) is MARA(4) is MVKE   Now I created an OLAP table with columns from different SAP table. When I now want to include the SAP field MVGR1 from table MVKE is receive following error message.  How should the formula look like by using e.g. DOMAIN_TABLE or BIND together with PU-Function so that I can display the field MVKE.MVGR1 in the OLAP-Table?

869
dxb.miner
Level 3
victor.diens
Level 2
victor.diensLevel 2
asked in Data Ingestion & Infrastructure
How to use a table twice in an extraction

Hi everybody! I’m pretty new to celonis development and never ask a question here before, but I hope you can help me! I’m extracting data from SAP for Production data and now have the issue that I want to join the JEST table to two other tables (AUFK and AFVC) on the column AUFK.OBJNR and AFVC.OBJNR. How can I do that since downloading the whole JEST table means like 200 million entries :-( I hope you can help me!  Thank you very much!

323
florian.mrkvi
Level 4

Badge winners

Show all badges
Terms & Conditions

ANY QUESTION?

Talk to an expert
FAQs
Contact us

MORE CELONIS

Academy arrow_outward
Documentation arrow_outward
Support Center arrow_outward
Partner Portal arrow_outward
Marketplace arrow_outward

CHECK US IN

LinkedIn
YouTube
Instagram
Facebook
Imprint
Disclaimer
Privacy Notice
Cookie Notice
Terms and Conditions
Celonis © 2025