Skip to main content

Data Ingestion & Infrastructure

Discuss and ask questions about data integration, task mining, admin settings, and PQL extractors to optimize your infrastructure.

  • 801 Topics
  • 2,108 Replies
801 Topics
M
Level 2
mel.schic11Level 2
asked in Data Ingestion & Infrastructure
I would like to connect two fields that have an indirect N:N relation. Is this something which I can do via a PU_Function?

Within an OLAP table, I would like to use both the Delivery Line Item Date ("VBEP"."EDATU") and the shipment gross weight ("LIPS"."BRGEW") Does anyone know how I can apply a PQL function to combine these tables? Please find the error below. I looked into the PQL documentation but I can't find the solution. Error in formulaNo common table could be found. The tables ["VBEP"] and ["LIPS"] are connected, but have no common table. This means that they do not have a direct (or indirect) 1:N or N:1 relationship. Join path: [VBEP]N <-- 1![VBAP]!1 --> N[VBFA]N <-- 1[LIPS]. For more information on the join path, search for "Join functionality" in PQL documentation.

162
kandi.naidu
Level 4
carmen.kaluz
Level 3
carmen.kaluzLevel 3
asked in Data Ingestion & Infrastructure
Use App Key for content-cli in ML workbench

Hi all,I would like to export datapools as json using content-cli in the ML workbench in order to backup them in Git. I have created an API key in my profile and used it in a python script with content-cli pull data-pool --id data-pool-id, which successfully generated a json of the datapool.However, I don't want to do this with a user-based API key but with an Application key (created under admin&settings).When I run the same command using the Application key, I get the error "error: The provided team or api key is wrong."With get_celonis(api_token=“applicationkey”, key_type="APP_KEY") however, I can see the Application key and all its permissions.Is content-cli possible with an Application key and if so, how can I use it?

30
carmen.kaluz
Level 3
janpeter.van.d
Level 8
janpeter.van.dLevel 8
asked in Data Ingestion & Infrastructure
What does the Eventlog Automerge option in Data Model overview do?

Hi all, Since a short while we see the option eventlog automerge in the data model overview (see attached screenshot). However, this is not mentioned in the Release Notes or online documentation. Does somebody knows what this option is for? Thanks in advance! Bests, Jan-peter! Celonis_Automerge|690x150

63
Ushmita Patil
Level 2
thomas.guerr
Level 4
thomas.guerrLevel 4
asked in Data Ingestion & Infrastructure
Extractor configuration for data transfer from MS Sharepoint to Celonis

Hello,I am trying to get data from a Microsoft sharepoint list to the Celonis Platform.According to the Flowchart provided by the API-Team - When to use the Standard Data Ingestion API (and other methods) - the extractor builder should be used for a Sharepoint data transfer, since the REST API is exposed.I tried to set up the connection, but I fail when configuring the endpoint, since I get xml-data even if I specify that I would like to have a json structure (Request Parameter - Accept: application/json;odata=verbose)Does anyone have experience with connecting the MS universe with the Celonis universe?I also found a workaround that works (Push CSV to Data Integration), but I would rather orchestrate all the data loads in the Data Integration environment instead of using Process Automation modules to make a data transfer.Thanks for your support!BR THomas

692
thomas.guerr
Level 4
thomas.keim
Level 4
thomas.keimLevel 4
asked in Data Ingestion & Infrastructure
Usage of PQL Function QUANTITY_CONVERT

Hello Community,I need you help in using the PQL Function QUANTITY_CONVERTMy objective is simple: I want to convert the SAP Field LIPS.BTGEW to Tons (TO). The original Unit is stored in field LIPS.GEWEI.With this try I get an errorQUANTITY_CONVERT("LIPS"."BRGEW", FROM ("LIPS"."GEWEI"), TO ('TO'))A PQL error is preventing this query from executing.Syntax error near [)] after reading [QUANTITY_CONVERT("LIPS"."BRGEW", FROM ("LIPS"."GEWEI"), TO ('TO')] at line 32. Please refer to PQL documentation for available syntax. What's the problem? If I understand the docu (https://docs.celonis.com/en/quantity_convert.html) correctly, I must not provide a conversation rates table?This operator converts quantity units based on a QUANTITY_CONVERSION_RATES table, which is optionally provided by the user. If not provided by the user, a default QUANTITY_CONVERSION_RATES table will be used. Where can I see the contents of the default QUANTITY_CONVERSION_RATES table?Thanks for helpBest, Thomas 

20
thomas.keim
Level 4
swapnil.karad12
Level 2
swapnil.karad12Level 2
asked in Data Ingestion & Infrastructure
Linking tables for O2C and P2P in SAP

I want to visualize end to end process of a buisness; order received → purchasing begin→ production → payments to vendors → order shipped to customer → cutomer payment receivedhow i can connect in SAP all this O2C to P2P; what are common tables to link all this

90
swapnil.karad12
Level 2
gauri.pai11
Level 3
gauri.pai11Level 3
asked in Data Ingestion & Infrastructure
Content-cli Profile creation

Hi team,I am using this command content-cli profile create.I am asked for name,url and api key.I have added all the three, but it gives me error as - The provided team or api key is wrong.Name of the profile: XYZYour team (please provide the full url): https://XYZ.eu-5.celonis.cloudProfile type: OAuth Device Code (1), OAuth Client Credentials (2) or Application Key / API Key (3): 3Your api token: XYZ*API token is directly fetched from celonis-edit profile-api key.Is it necessary to have admin role for this.Currently i have analyst access.Please help me with this.

211
abdulnour.duale11
Celonaut
haruka.yano
Level 2
haruka.yanoLevel 2
asked in Data Ingestion & Infrastructure
Object transformation preview once showed and disappers

HiI’m curently working on OCPM Data Pool and have trouble in an object transformation.In the Currecny Conversion object(o_celonis_CurrencyConversion),I checked that I can see data record in Preview,and executed publish to development.Then I ran the Data Job Transformation(full load), succeeded,ran the Data Model Load, succeeded,but at the end o_celonis_CurrencyConversion has no data.I don’t know why but when I go see the object preview again, the preview doesn’t show anything.Has anyone had this kind of experience ever?If yes please kindly help. FYI as the prerequisite, - the relevant sap table data is sycronized from another data pool - the object and its transformation script is celonis default one​​​​​​​ - Data job is done by the duplicate of test:ocpm-data-job(celonis default test data job) - the reason why I use the duplicate data job is that the original one returns an error for schema.

90
haruka.yano
Level 2
PhuongBHN
Level 3
PhuongBHNLevel 3
asked in Data Ingestion & Infrastructure
How to enable the ‘Push data into Celonis’ Data connection option within my Data pool

Hello everyone, I am unable to locate the ‘Push data into Celonis’ Data connection option within my Data pool.Could you guide me on how to enable this feature?!--endfragment>My Celonis  I want the ‘Push data into Celonis’ as the image below 

423
abdulnour.duale11
Celonaut
Suveesh_adessi
Level 2
Suveesh_adessiLevel 2
asked in Data Ingestion & Infrastructure
Celonis Audit API - Group & Group Members

I’ve been extracting Celonis audit logs API for user-platform analysis. I couldn’t find the right field for group name and group members in the extracted data. If you are someone done this already, please share your knowledge to get these data.Also, please mention if there is any dedicated API endpoint is available to extract those data.

151
jonas.hess
Level 3
MrAj
Level 3
MrAjLevel 3
asked in Data Ingestion & Infrastructure
How to Trigger a Machine Learning Script from a Button in Celonis?

Hi everyone,I need help figuring out how to execute a Machine Learning script directly from a button in my Celonis app in views (dashboard). I believe I might need to use Action Flows, but I’m unsure about the exact steps to make this work.Here’s what I want to achieve: When a user clicks a button in the Celonis app, it should trigger an ML script. The script could be hosted externally (e.g., a REST API) or executed within Celonis.( atm its not hosted, there is just a file in ipynb format, stored in ml workbench in  analysis. I need guidance on how to set up the Action Flow and link it to the button. Has anyone implemented something similar? If so, could you share the best approach and any potential challenges I should be aware of?Thank you in advance!

170
MrAj
Level 3
MrAj
Level 3
MrAjLevel 3
posted in Data Ingestion & Infrastructure
How to Trigger a Machine Learning Script from a Button in Celonis?

 Hi everyone,I need help figuring out how to execute a Machine Learning script directly from a button in my Celonis app in views (dashboard). I believe I might need to use Action Flows, but I’m unsure about the exact steps to make this work.Here’s what I want to achieve: When a user clicks a button in the Celonis app, it should trigger an ML script. The script could be hosted externally (e.g., a REST API) or executed within Celonis.( atm its not hosted, there is just a file in ipynb format, stored in ml workbench in  analysis. I need guidance on how to set up the Action Flow and link it to the button. Has anyone implemented something similar? If so, could you share the best approach and any potential challenges I should be aware of?Thank you in advance!

140
MrAj
Level 3
MrAj
Level 3
MrAjLevel 3
asked in Data Ingestion & Infrastructure
Configuration Management for Process Flow in Celonis by Gui or flat files

Current State:Our Celonis process mining implementation includes configuration settings for process flow and status updates in our data transformation pipeline Currently, only developers can modify these configurations through SQL queries and code changes End users (procurement specialists/buyers) have no way to modify process flow configurations without technical knowledgeRequirement Question: What is the most effective way to enable non-technical end users to update process flow configurations in Celonis without requiring knowledge of SQL, PQL, YAML, or coding?Specifically:What data format should be used for configuration (CSV, Excel, UI form, etc.)? Where should this configuration be stored in the Celonis architecture? How will the configuration files be uploaded/modified by end users? How can we ensure the uploaded configurations are properly integrated with our data transformation steps?The solution should accommodate users with no programming knowledge while maintaining data inte

40
MrAj
Level 3
.
Level 4
.14600833Level 4
asked in Data Ingestion & Infrastructure
Source und Target with PROCESS_ORDER

Hello everyone,i need a throughput between two consequitive activities and want to implement it via source and target.Die idee wre source(activity_table.activity), target(activity_table.activity, process_order(target)=process_order(source)+1))would it work?Best,Maria

105
sudharani.perum
Level 2
andres.naran12
Level 4
andres.naran12Level 4
asked in Data Ingestion & Infrastructure
How to get the timestamp for an specific activity?

Hi community, I have the following challenge that I’m trying to figure out.I need to calculate how many hours were spent on a holiday, since the connection is not through SAP, I cannot use the functions that PQL provides for that.My idea is to get the timestamp related to the last activity that happened on holiday and calculate the throughput time between 9:00 AM and that specific time in order to get how many hours were spent.Example:Activity 1 → Day 1 Activity 2 → Day 2 (holiday) Activity 3 → Day 3 Activity 4 → Day 4In this example let’s suppose activity 2 occurred at 13:00, so in this case I should do the following calculation. 13 - 9 = 4 hours → This is the number of hours that were spent on holiday. Does anyone have a similar experience?

272
andres.naran12
Level 4
farouk.abdel
Level 2
farouk.abdelLevel 2
asked in Data Ingestion & Infrastructure
CPM4 sizing

what is the celonis sizing guidelines for CPM4?

81
A
Celonaut
maxime.psbpm
Level 3
maxime.psbpmLevel 3
asked in Data Ingestion & Infrastructure
Issue with Many-to-Many (N:N) Relationships in Celonis

Hi everyone,I'm facing an issue with how Celonis handles many-to-many (N:N) relationships. It seems like Celonis treats these relationships as 1:N → 1:N instead of supporting 0:N → 0:N, which leads to missing data when filtering.Example Issue:I have a many-to-many relationship between contracts and assets, modeled with an intermediate table:Contract Asset C1 A2 C2 A1 C3 NULL C4 NULL NULL A3 NULL A4 The issue occurs when I apply a filter like "Asset ≠ A4". Instead of just removing A4, I also lose contracts C3 and C4, which should still be visible since they are not linked to any asset.My Understanding of the Issue:Celonis seems to implicitly treat N:N relationships as 1:N → 1:N, enforcing a strict link between contracts and assets. It does not seem to support a true N:N model, where both sides can have optional (0:N) relationships.Question:Has anyone faced this issue before?Is there a recommended modeling approach to ensure that filtering on assets does not remove u

905
maxime.psbpm
Level 3
Bharath
Level 3
BharathLevel 3
asked in Data Ingestion & Infrastructure
PQL For distinct activities in a case, happening at the same time.

I have a table, for CaseIDs (1,2,3...) and some Activities (A,B,C….). The activities might be repeated in a case or they can be unique. and a TimeStamp column corresponding to the caseid and activity. i need to write a query where in a case there are 7 distinct activities and all of them must happen in the same timestamp. kindly help me with this.

80
Bharath
Level 3
llllliz
Level 2
lllllizLevel 2
asked in Data Ingestion & Infrastructure
How to use data pool parameter?

Dear community, I would like to set up a data pool parameter to allow me to switch between test and prod environment? So I would need two global variables for my data jobs:var_data_source: to allow me switch between test schema and prod schema var_table_name: to allow me when creating table to add _test at the end of the table nameI have only found the way to create parameters in EACH data job under transformation and refer to the data pool parameter. Is there a smarter way? I am still using the data integration (not OCPM). Thanks a lot!   

150
llllliz
Level 2
lize.erasm11
Level 3
lize.erasm11Level 3
asked in Data Ingestion & Infrastructure
Adding attributes to a event in a OCPM model

Hi, has anyone found a easy or quick way to add a large list of attributes to a event? 

261
abdulnour.duale11
Celonaut
thomas.keim
Level 4
thomas.keimLevel 4
asked in Data Ingestion & Infrastructure
Check if a Variable (New Views) is null

Hello Community,how can I check if a variable from the new views is NULL? My challenge is to not use an filter in a PU_FUNCTION when the variable is empty.Variable: VAR_CONDITION_FILTERExample Value with multiple selection: ‘Y001’, ‘Y020’, ‘Y300’I want to use the variable in a PU_SUM Filter statement like in this way.PU_SUM("VTTK", "FREIGHT"."FREIGHT_CONVERTED", CASE WHEN '${var_condition_filter}' IS NOT NULL THEN "FREIGHT"."CONDITION" IN (${var_condition_filter}) END )But this don’t work. Error:A PQL error is preventing this query from executing.Syntax error near [IN] after reading [PU_SUM("VTTK", "FREIGHT"."FREIGHT_CONVERTED", CASE WHEN '${var_conditon_filter}' IS NOT NULL THEN "FREIGHT"."CONDITION" ] at line 9. Please refer to PQL documentation for available syntax. This here works, but I need the check if the variable is empty to avoid errors and to avoid that users has to select all the possible values.PU_SUM("VTTK", "FREIGHT"."FREIGHT_CONVERTED", "FREIGHT"."CONDITION" IN (${var_c

341
thomas.keim
Level 4
kandi.naidu
Level 4
kandi.naiduLevel 4
asked in Data Ingestion & Infrastructure
To get Count of Activities in Process Explorer

Hi Team, I am trying to write a pql query to extract value of activity count between two activities one of them is either process start or process end.Like I highlighted the value in the screenshot. please let me know, how to achieve it. Regards,Ajith Naidu.

202
Sverre Klein
Level 6
PhuongBHN
Level 3
PhuongBHNLevel 3
asked in Data Ingestion & Infrastructure
How to connect S3 without using access key?

I need to create a connection from Celonis - Data Integration to AWS S3 without using an access key, as our policy requires password rotation every 90 days, and I prefer to avoid manual updates. Do you have any suggestions on how to achieve this?

90
PhuongBHN
Level 3
Juan Palencia
Level 2
Juan PalenciaLevel 2
asked in Data Ingestion & Infrastructure
How to Reduce Data Job Execution Time for Large Volumes (e.g., 824M Rows) in SAP Saturn?

Hi Celonis Community, I’m working on optimizing data jobs in Celonis Data Integration for a client with an extremely heavy SAP system (Saturn). Our data jobs currently take around 1.5 hours to process, and the client wants to reduce this to 0.5 hours. Despite optimizing SQL queries (e.g., early filtering, minimizing joins) and disabling unnecessary objects, the massive data volume seems to be the main bottleneck. Here are some examples of the volumes we’re handling: - SalesOrderScheduleLine Transformation: 119M rows in CDPOS, 59M in CDHDR, cost of 24M.- CreditMemoItem Transformation: 123M in VBAP, 824M in VBFA, 178M in VBRP, cost of 83M.- CustomerInvoice Transformation: Similar volumes, 824M in VBFA, 178M in VBRP, cost of 83M. We’ve explored strategies like incremental loads and partitioning, but the sheer volume from Saturn (e.g., 824M rows in VBFA) still slows us down. I found in the documentation that the Payment Term Checker slows with 100M+ invoices, suggesting infrastructure migh

70
Juan Palencia
Level 2
Hafsa
Level 2
HafsaLevel 2
asked in Data Ingestion & Infrastructure
Preview not available "data model load" -

Hello, I'm working on connecting serviceNow to Celonis, I did Task Extraction (OK) → Task Transformation (OK) and when I loaded data it works (loaded rows OK), but when I press "set as activity table" it gives me the message :  "Preview not available". A table preview is currently not available. The cause may be due to a missing data pool or connectivity issues” without columns to select “Case ID”, “brief description”, “date”.I need to know why the columns are not showing up in views (after transformation task) but showing up in the original tables….for information: I tested the data in task extraction, task transformation and loaded data model and they contain all data of the views. I also execute data job for all, but the columns still don't appear.Thank you in advance for your help,

623
abhishek.chatu14
Level 7

Badge Winners

  • Qualified to Deliver Business Value with Celonis
    Graham Holmeshas earned the badge Qualified to Deliver Business Value with Celonis
  • Qualified to Create and Deliver Demos
    Graham Holmeshas earned the badge Qualified to Create and Deliver Demos
  • Value Realization Expert
    ismael.jimenhas earned the badge Value Realization Expert
  • Qualified to Deliver Business Value with Celonis
    ismael.jimenhas earned the badge Qualified to Deliver Business Value with Celonis
  • Qualified to Get Data Into Celonis
    yuri.fujimhas earned the badge Qualified to Get Data Into Celonis
VIEW ALL
Community Code of Conduct