Data Ingestion & Infrastructure

Hi everyone,During data extraction, I would like to convert a string field into a datetime format.The original field format is: 2024-12-09I would like to convert it to the following format: 2024-12-09 00:00:00.Does anyone have any idea how I could do this?I would really appreciate your help. Thank you! 😊   

Hi everyone,I’m trying to connect MongoDB to Celonis. At first, I assumed that I’d need to use a custom JDBC driver (based on a thread that I’ve read here and the Celonis documentation).However, when I reached the step to select the “Database Type” in the connector, I couldn’t find a fitting option for MongoDB.Has anyone successfully set this up?Should I use a specific Database Type in this case? Or is there another workaround to get MongoDB connected?Thanks for any tips or guidance! 

Hitting a wall with an extraction of tens of millions of rows from Amazon Athena via the SQL editor. The pull itself seems to finish without complaint, but the pipeline then breaks at the Data Push Job step with an InternalException.A couple of things I've noticed:A similar extraction through the visual editor (same columns, but a much smaller row count) runs without any issue as part of the daily load, so I don't think it's the source system itself causing this. I only moved to the SQL editor because I needed JOIN and CAST logic that the visual editor doesn't support. Not sure yet whether it's the volume or the JOIN/CAST operations that's actually the trigger.Has anyone hit Data Push failures at this scale, or specifically with JOIN/CAST in SQL editor extractions? Any insight into what usually causes this would help.

Hi! I tried to add a new table to my data model (EORD) linked to EINE vía WERKS, MATNR, SYS_ID, MANDT, EKORG, ESOKZ, LIFNR.If I check this unique key in EINE ( Using: select WERKS,MATNR,SYS_ID,MANDT,EKORG,ESOKZ,LIFNR, count(*) AS Amount FROM ALL_NPM_EINE group by WERKS,MATNR,SYS_ID,MANDT,EKORG,ESOKZ,LIFNR order by Amount DESC ) and there is only one case per combination. So in a 1:N relationship it should work or? (EINE 1 → N EORD)Error Message: Case insensitive duplicated joins detected between tables: [EORD] and [EINE].

I'm working with a REST API that returns doubly-nested JSON (think: a top-level array of objects, each containing a nested array, which itself contains another nested array). Celonis splits this correctly into three separate tables, and the parent key from the top level propagates fine into the first nested table. The problem is that it stops there.The second-level nested table has no reference back to the top-level parent. The intermediate ID that's available at that level isn't unique (it's a type identifier that's the same across all top-level records), so I have no way to join the second-level table back to a specific top-level record without getting a cartesian product.The root issue as far as I can tell: in the Connector configurator, you can only choose from fields that exist in the API response as Primary Key. The propagated parent key that Celonis adds to the first nested table isn't available as a selectable column when configuring the second nested table, so there's no way t

Large history table extraction keeps hitting an Athena query timeout.I'm stuck on a recurring issue when pulling a large history table through Data Integration → Extractions. No matter whether I build it in the visual editor or the SQL editor, the run dies after a long-running query with:[Simba]AthenaJDBC An error has been thrown from the AWS Athena client. Query timeoutI already tried cutting the row count by joining the history table against a header table first, no change. The only version that actually finishes is one where I add a hard date cutoff, but that obviously leaves out a chunk of history I actually need.Anyone dealt with this on larger Athena-backed extractions? Looking for a way around the timeout that doesn't mean sacrificing part of the dataset.

I asked a similar question some time ago, still looking for an elegant solution:Context:Currently we do all the extraction within one Data pool, that does also contain the OCPM objects and events. Since we start to connect additional systems, we want to use the approach “1 dedicated Data pool per system, 1 common Data pool for all OCPM objects and events data”Problem:I hoped to simply rebuild the existing extractions in a new Data pool and then switch the Data source within the OCPM-transformations to the new Data pool. This works totally fine for custom objects, but for Celonis objects the Data source is read-only:  What is the most “elegant” solution?One approach would be to create a view for each table: view in the old data source pointing to the underlaying table of the new data pool. ...quite a lot of (maintenance) effort and not sure whether this would work.Another approach is to create full overwrites of all transformations. But by this you loose the connection to the official c

Hi everyone,I'm working with Celonis Machine Learning Workbench (MLWB) and a Knowledge Model as the data source.Current setup:Data is queried from a Knowledge Model into MLWB. A Python/scikit-learn model is trained successfully. Predictions are generated and stored in a pandas DataFrame. The goal is to visualize those prediction results in a Studio View/dashboard.The challenge:I can generate the prediction table in MLWB, but I have not found a way to make the prediction output directly available in Studio.I'm specifically looking for a solution that does not require:Exporting a CSV from MLWB Uploading the CSV into Data Integration Manually creating new data tablesInstead, I'm trying to understand whether there is a more native integration pattern between MLWB and Studio.QuestionsIs it possible to expose a DataFrame or ML prediction output from MLWB directly to a Studio View? Can MLWB prediction results be persisted as a Knowledge Model asset, View asset, or another object that Studio c

Hi everyone,I need help figuring out how to execute a Machine Learning script directly from a button in my Celonis app in views (dashboard). I believe I might need to use Action Flows, but I’m unsure about the exact steps to make this work.Here’s what I want to achieve: When a user clicks a button in the Celonis app, it should trigger an ML script. The script could be hosted externally (e.g., a REST API) or executed within Celonis.( atm its not hosted, there is just a file in ipynb format, stored in ml workbench in  analysis. I need guidance on how to set up the Action Flow and link it to the button. Has anyone implemented something similar? If so, could you share the best approach and any potential challenges I should be aware of?Thank you in advance!

Hi everyone, I'm facing an issue with Machine Learning Workbench and would appreciate some guidance.I cannot access any resources from within the notebook environment: - get_data_pools() returns 0 data pools- get_spaces() returns 0 Studio spaces- Team connection appears to work- Authentication succeeds without errors- The notebook can connect to Celonis successfully Additionally, when checking permissions through PyCelonis, all services seem to return empty permission lists. In the browser UI, I can access my Studio project and Knowledge Model without any issues, but the same resources are not visible from ML Workbench. My questions are: 1. Does ML Workbench require separate permissions or resource assignments to access Studio Spaces or Knowledge Models?2. Can ML Workbench access Knowledge Models directly, or does it only work through Data Pools/Data Models?3. Has anyone encountered a situation where authentication succeeds but get_data_pools() and get_spaces() both return empty result

Hi Guys,I am currently facing issues with the datapipeline between celonis studio and the machine learning workbench. The ML Workbench is unable to access Celonis data due to outdated celonis_ml dependencies that rely on deprecated sklearn functions. This breaks the built-in data pipeline and prevents direct data model access. As a workaround, data had to be exported manually as CSV and processed externally. An update to the ML environment and data connectors is required to restore full functionality.Is there a fix or a workaround for this?

Hi Community,I'm getting the following error while loading my data model:Could not execute foreign key join: there are duplicates on both sides of the specified key relationship.Table EKKO, key columns (MANDT, LIFNR), key ('011', '0000805024').What is the recommended best practice for resolving this issue?Should the duplicate records be removed, or should the join relationship be adjusted? If removing duplicates is the preferred approach, how do you typically handle it? I have already identified the duplicate records, so the only remaining step would be how to delete them.Thanks in advance for your insights and recommendations!

  If you have built any Object-Centric Data Models (OCDM) recently, you know the pain of handling tables that use composite primary keys.Previously, Celonis only allowed a single string ID as the primary key. If you were pulling tables from SAP (like VBAP), you had to write custom SQL transformations to concatenate keys (like VBELN and POSNR) into a single string ID just to make joins work.With the June 15, 2026 release, Celonis natively supports composite primary keys. You can now select multiple columns directly in the Objects and Events UI to form your primary key, completely bypassing the old SQL concatenation step.This change fixes three major issues.First, you don't have to write redundant CONCAT SQL code in your data jobs anymore. Your SQL transformations stay clean.Second, database joins run faster on raw composite columns than on concatenated string columns, especially when you are querying millions of rows.Third, your Celonis data model now matches your source ERP schema exac

hi all,Context :We have been using Celonis for many years connected to SAP ECC.To limit our data consumption + keep our jobs running as efficiently as possible, we have been extracting our tables based on a 24 months rolling basis.For that, we have a small job running every month setting the start date of all the extractions. This start date is used in the tables extraction filters. This has been running smoothly so far.What is happening :Our company is going to transition one ERP from SAP ECC to SAP S4.in SAP S4, the price conditions table PRCD_ELEMENTS (KONV table in SAP ECC) and more specifically the date field KDATU, is setup in CHAR data type. This is not compatible with our extraction setup.Assuming I wont be able to get this data type change, any recommendation on how to handle that?Thanks

We have multiple cases when we want to make sure there is an explicit correlation between two data series apart from visually representing those series on a graph. Is there any simple KPI/PQL formula we can use, or do we need to go with a workaround deriving corr from calculating standard deviations etc?

Hi, I’m currently facing an issue while setting up an Action Flow / Machine Learning Workbench solution to download and merge multiple Excel/CSV files from a SharePoint site.I can see that it’s possible to download a single file using the Microsoft SharePoint Online module in Action Flow. However, my problem is that the folder contains multiple files with different names, and I would like to handle all of them within one solution.Has anyone worked on something similar and could share some advice on how to approach this?

Dear experts , What is the size limit of Objects and Events in OCPM and  so far we have not consider archiving the older records from OCPM is it advisable ?  

I’m currently facing a challenge with copying objects and events created in one data pool to another data pool.How can this be done efficiently without recreating everything manually? Is there a recommended or best‑practice approach to reuse objects/events across data pools? Any tips to ensure consistency and minimal rework?If you’ve solved this before or know a better way, I’d really appreciate your insights. Thanks in advance!

Hi everyone,has anyone here worked with Coupa’s APIs — especially the requisition_revision_records and requisition_revision_record$requisition-lines APIs?Does anyone know if these are still supported by Coupa?Many thanks.

Hi - I am trying to write back the Variables I used in Studio View to Data Integration via an Action Flow but the attribute where I am storing the variable is showing as blank. I read Action Flows can’t read Variables form the Studio View. Is this true?

I’m trying to connect Celonis in Power BII created an Application Key but still don’t see any tables loaded after connectionWhat could be the issue? 

For testing some data, I just imported some manual files from my computer.After I click the ‘Add data connection’ button, and add the 2 files from my PC, I dont see anything in the window below.   

Hello everyone, As part of setting up a control tower, I am trying to gather as much information as possible to retrieve data from the platform via API calls. After combing through most of the existing documentation on the Celonis website, the pycelonis library documentation, and the freely accessible GitHub repositories, I am still having trouble identifying how to retrieve certain elements. For example, is there a way to get a list of all the pool ids of your environment (I know it’s possible using pycelonis but I’m not able to use private library in the external scheduler I’m using)?. Do you have comprehensive documentation of all possible API calls (in my case, with OAuth authentication. This is also one of my main concern: lot of API calls working with App Key are not working with OAuth2 identification, which is preferable)? I am interested in any resources I may have overlooked. Thank you for your help,Best regards, Alex

Hi everyone,I’m currently reviewing our extraction setup for Accounts Payable (AP) data and would love to hear from others about best practices that have proven effective across different organizations.Of course, I fully understand that every company’s landscape, compliance requirements, and data models are unique. Still, I believe there are common patterns and techniques that tend to work well in most AP extraction scenarios. I’d be very interested in your experience around the following topics:Extraction FrequencyHow often do you typically run delta extractions for AP (e.g., hourly, daily, near‑real‑time)? How frequently do you schedule full loads?Delta vs. Full Load StrategyWhich tables do you usually extract in full, and which ones are better handled via delta? Do you rely on timestamps such as creation date or change date for deltas?FiltersWhich common and robust filters do you use to reduce data volume without losing relevant AP data? Examples: company codes, fiscal years, docume

We can update augmented attributes via an Action flow, but is there a way to mass-update directly from PyCelonis?