Skip to main content

Data Ingestion & Infrastructure

Discuss and ask questions about data integration, task mining, admin settings, and PQL extractors to optimize your infrastructure.

  • 806 Topics
  • 2,121 Replies
806 Topics
M
Level 2
markus.wagneLevel 2
asked in Data Ingestion & Infrastructure
Using PyCelonis "export_data_frame" with Query got from TableComponent, which uses "variables" & "saved formulas" ?

Hello @All i'm trying to export data from an ready/productive Analysis with pycelonis. My Analysis has a table, from which i got the query (compontent.get_query()). With this query i wanted to export the dataframes (data_model.export_data_frame(query)).I'm getting some errors on all variables and saved formulas.e.g: PQLColumn(name='<%=ZZ_ABC1%>', query='<%=ZZ_GHJ2%>'), Is there an option to use variables & saved formulas in pycelonis export? Thanks in advance!  

500
jonas.schei12
Level 3
P
Level 2
p.fLevel 2
asked in Data Ingestion & Infrastructure
Unknown User Error in Action Flow for USER_NAME()

I want to be able to get the username of someone that executes a skill and update an augmented attribute with the username. In Knowledge Model, I created a KPI with the PQL formula - USER_NAME(). This works fine in the knowledge model and returns the user. However, in action flow when I use Get Rows to get the KPI, it returns Unknown User. Does anyone have an idea why this is the case or perhaps any other idea to achieve what I want?

120
neil.whita
Level 3
S
Level 3
sam.weyel11Level 3
asked in Data Ingestion & Infrastructure
What are considered to be the 'best in class', currently, Extract, Transform & Load' (ETL) applications, which support Users with the general data preparation activities?

I'd suspect a lot of activities happens outside of Celonis, initially to 'prepare' the data quality, ensuring its reliability and integrity once in Celonis. I'm sure there's a lot of common challenges when uploading into Celonis, and have experienced some myself.

30
K
Level 1
D
Level 3
dilyar.iskan11Level 3
asked in Data Ingestion & Infrastructure
DATA INTEGRATION - DELTA EXTRACTIONS.

I want to extract a table "SRRELROLES" from SAP system. For this table, I have to use dynamic and private parameters on the column "UTCTIME" UTC Timestamp in Short Form (YYYYMMDDhhmmss).  I get an error for delta extraction since the data type I use is date (I have tried integer and double but neither worked)and the data type of UTCTIME is float according to SQL vertica in Celonis and UTCTIME has data type DEC in the SAP system.  My question is which datatype can I use for my dynamic and private parameters? Or how can I define a variable inside the FILTER STATMENT or DELTA FILTER STATMENT in order to get the max of the UTCTIME and use it for my delta extractions?

190
D
Level 3
patrik.patri
Level 2
patrik.patriLevel 2
asked in Data Ingestion & Infrastructure
Pycelonis to extract and pull data from analysis/datapool to external MySQL or similar

Hallo, as pycelonis states: Using this package you can programmatically interact with Analyses, Workspaces, Data Models, Data Pools and other Celonis objects. The package also allows pushing and pulling data to and from Data Pools and Data Models. I would like to programmmatically interact with celonis analysis (mainly) and extract the data for other purposes and to other sources. That means, the statement suggests to me the possibility to extract the data from Celonis into an external database like say MySQL. I dont find a supporting material on this in the documentation. Can someone shed a bit more light into the darkness here?

110
D
Level 3
.
Level 2
.1463214Level 2
asked in Data Ingestion & Infrastructure
What is the meaning of this error ?

30
abhishek.chatu14
Level 7
A
Level 1
alicja.baran12Level 1
asked in Data Ingestion & Infrastructure
Calculating Month-to-Month/Day-to-day/Week-to-week/Interval-to-Interval change on a Graph

Hello friends! :) I have a bar graph with ROUND_MONTH("ACTIVITIES"."EVENTTIME") set as a dimension and a requirement for it to display the following: -For each month, we want to get a difference of new cases between said month and the previous one-It would be ideal to also get a percentage change compared to the previous month. Have you guys ever dealt with something like this? It seems simple enough, but I've been scratching my head with this one. Thank you for your help in advance!

10
S
Level 1
H
Level 2
hannes.geyerLevel 2
asked in Data Ingestion & Infrastructure
How to join between different data sources within the same datapool

Joining between different data sources are possible in global data jobs, e.g.  select *from <%=DATASOURCE:MDT_100%>.EKPO A join <%=DATASOURCE:MDT_200%>.EKPO B ON ... But is it possible to join each other within the scoped data jobs (MDT_100, MDT_200), that means Scope MDT_100:select *from EKPO A join <%=DATASOURCE:MDT_200%>.EKPO B ON ...

180
H
Level 2
S
Level 3
sudhanshu.kumar15Level 3
asked in Data Ingestion & Infrastructure
I push data in celonis using data push api (python methods), but column width of some of the is so high that it is showing entire content .is there any way to increase the column with from 80 to 240 in data push api methods

I push data in celonis using data push api (python methods), but column width of some of the is so high that it is showing entire content . is there any way to increase the column with from 80 to 240 in data push api methods

10
mateusz.dudek14
Level 7
A
Level 1
artur.krask12Level 1
asked in Data Ingestion & Infrastructure
Is it possible to have a timestamp based process model? By this I mean seeing process steps in time as they happened in reality and were recorded in eventlog.

When I try to mine a process from an event log I see wrong activity in the beginning. And can't understand why is it there. As well, BPMN model is built incorrectly. I can modify it, but I don't see it in process or variant explorer in a modified way.  I use Python simulation to generate data. it simulates facility with several rooms where users can move and drink coffee. Coffee increases their heart rate. As well, the system starts displaying messages and offers users to drink water.  In the process model I get users have increased heart rate in the beginning and their moves between facilities are happening in one direction, while in reality they move randomly.  I try to understand what do I need to modify (Python code for data generation or something inside Celonis) in order to see the real model and a sphagetti diagram showing all possible users' movements and events.   

10
marcin.konop13
Level 3
enrique.sanch13
Level 3
enrique.sanch13Level 3
asked in Data Ingestion & Infrastructure
How can you pass the proxy configuration to a dockerized extractor?

 The Celonis documentation does not specify passing the proxy configuration to docker. I don't know if the way to pass it to docker is to add one more line-e PROXY_ENABLED=true \\ -e PROXY_HOST=<host>, etc. Or how it would be since it depends on how the Docker image has been mounted. 

20
R
Celonaut
G
Level 2
gregor.schei12Level 2
asked in Data Ingestion & Infrastructure
Is it possible to push an excel file via Action Flows to Celonis (for CSV it is working)

Dear all, I would like to use Action flows to push an excel file (xslx) to Celonis using Action Flows.  Use Case: Excel file is received via email and up until now manualy uploaded to a data pool. I am successful in accessing this email, downloading the excel file.  I am following this tutorial to learn how to push a CSV file to a data pool. Create and Push CSV to Data Integration (celonis.com) However, it does not work for XSLX files for me. I changed the file type already to "XLSX" and "EXCEL". Error is: "something went wrong while handling the request. Please contact Celonis customer support. Celonis Support ID" I also tried to find a module that would help me translate an excel to CSV but could not found anything. Thank you,Gregor

170
carlos.taleg11
Level 5
florian.mrkvi
Level 4
florian.mrkviLevel 4
posted in Data Ingestion & Infrastructure
www.celopeers.com

Documentation of conditional filters Hi @mentor.vucit​ , I just came across your answer (link) and I am wondering where I can find the documentation of conditional filters? They seem very powerful but I can't find any of this in the celonis documentation. BR,Florian 

02
florian.mrkvi
Level 4
ravi.kumar23
Level 4
ravi.kumar23Level 4
posted in Data Ingestion & Infrastructure
Hi Team,

Hi Team,We are working on reviewing our landscape with respect to Celonis best practices.Currently we have multiple data pools and extracting data from multiple connections.In many cases we can have a situation where:1)Getting data from same connection but using extracted data for different data models.2)Getting data from different connections and combining extracted for single data models.3)Often times buiness requirement for same process may vary for different units and they would want to see different views / analysis / actions for them in those case how the data pool should be structured. Are already there best practices along these lines with pros and cons?

21
M
Level 1
C
Level 2
c.mLevel 2
asked in Data Ingestion & Infrastructure
How to combine two number logics with a trasnformation to see one process?

In one system I have two processes with two number logics. First one is called S_ID and shows Process A ("From Receiving Technical data to releasing technical data for invoicing". 2nd one is called "L_ID" and shows process B "Creating data set to invoice sent"  If data has status 4 in the system, the data has been transformed from process a to process b and every s_id is combined with a L_id.  I checked with a table if ir possible to see status4, s_id and L-id in one table and cheked in the source system if the data is correct- it is.  In the Data model I created two activity tables and zwo cases tables, one with S_ID logic, the other one with L_id logic.  The last step should be to combine the data so the customer is able to see the whole process from receiveing the technical data to invoice sent.  I tried it with this script, and I know it is not correct yet:  case when"TableS_id"."STATUS"= 4then Bind("TableS_id", PU_first(&

10
G
Level 8
A
Level 1
anagha.shara11Level 1
asked in Data Ingestion & Infrastructure
Hi, can anyone please help me with a PQL query.

I am trying to extract the date from a string. the date comes after '-'. The challenge is that there can be more than one instance of '- '.So, tried to find the '-'followed by a space and '/'after 3characterString sample: $$$$$ $$$$$$$: $$~$$$$ - $$$$ $$$$$$$$ $$$ $$$ ~$$$$ $$$$$$$$ $$/$$/$$$$ - 07/01/2022 $$$:$ $$$:$ $$$:$.$ ( i want to extract the date after the second '- ') Code I used:(code gives me error)CASE WHENMATCH_PROCESS_REGEX ( column, '.*- .{3,}/' )=1THEN TO_DATE(RTRIM(RIGHT(STRING_SPLIT (column, '- ' , 0),11)),FORMAT ( '%d/%m/%Y'))ELSE NULLEND Thanks In Advance

20
G
Level 8
P
Level 1
pranjali.bhattLevel 1
asked in Data Ingestion & Infrastructure
Hi, I'm unable to upload dataset in in the tool, ERROR- Remote application onhttp://integration/integration/api/internal/data-pools/data-models/655b7a20-e1d7-4c50-b4d6-328a000bb992/load-info-sync[GET]responded wi

Hi, I'm unable to upload dataset in in the tool, ERROR- Remote application on http://integration/integration/api/internal/data-pools/data-models/655b7a20-e1d7-4c50-b4d6-328a000bb992/load-info-sync[GET] responded wi

09552
B
Level 1
A
Level 3
abhinav.singa11Level 3
asked in Data Ingestion & Infrastructure
Formation of temporary Domain table when we specify multiple tables that are with N- 1relation with other tables in the data model.

In the given data tables below -  I used the statement - 'PU_SUM ( DOMAIN_TABLE ( “Chapters”.” Title”, “Book_Author_Mapping”.” Book ID” ), “Chapters”.” Number of Pages” )' Here I have a couple of questions in understanding the functionality:- If the Domain Table is formed based on the specified column/tables - To which Specified Table will the Damon Table be linked to? - as both the tables 'Chapters and Book_Author_Mapping' have an N:1 relationship with other tables?Will the Domain table link to the 'specified tables - Chapters; Book_Author_Mapping' based on the Table we mention in the SourceTable?Column field of PU_X Functions -i.e. “Chapters”.” Number of Pages” in this case ?'  Thanks in Advance. Kind regardsAbhinav

120
M
Level 1
M
Level 1
mudaga.pooji11Level 1
asked in Data Ingestion & Infrastructure
How many cases are covered by the five most common variants.

How many cases are covered by the five most common variants.

10
K
Level 2
T
Level 1
tim.tak12Level 1
posted in Data Ingestion & Infrastructure
Help needed with a % connection path KPI

Help needed with a % connection path KPI I am trying to figure out how i can make a connection KPI which shows the path percentage of cases that passed through. now i want to see the path percentage flow from each individual Activity. exmaple: Start -->Activity A (branches pathways) ---55% for a path---> Activity B (branches Pathways) --23% for a path--> activity Z. so from each Acitivy i want to calculate how much of the (total) cases that landed there go a certain path % wise. Now i figured out by doing: COUNT (DISTINCT SOURCE ("Table"."colum")) / GLOBAL(COUNT (DISTINCT SOURCE ("Table"."colum"))) I can filter the path connections to % based on the Total amount of cases present in the dataset. This only shows me how cases % wise flow through the whole process. but not from an indivual activity.  In my mind somehow every Activity point needs to be seen as the "new" total 100% so a calculation can be made for the paths. but i

51
T
Level 1
S
Level 3
sudhanshu.kumar15Level 3
asked in Data Ingestion & Infrastructure
i want change the dimension of olap table using dropdown in views, How to display name like display name in dropdown in views

i want change the dimension of olap table using dropdown in views, How to display name like display name in dropdown in views

40
S
Level 3
M
Level 3
matt.slocu12Level 3
asked in Data Ingestion & Infrastructure
Need help with calculating times between consecutive activities

Hi! can anyone please help me figure out the following PQL? I have cases that have a lot of the same activity.  Imagine the activity is "Walk The Dog" and there are activities with the following dates: 1/1/20231/8/20231/15/20231/25/2023 I would like a query that returns the largest number of days between consecutive events  So in this example: 1/1/2023->1/8/2023 = 7 days1/8/2023->1/15/2023 = 7 days1/15/2023-1/25/2023 = 10 days I would like the query to return '10'  I am having trouble with trying to connect the dots with the documentation on calc throughput. Not sure how to approach it. 

110
M
Level 3
W
Level 3
windu.hardiLevel 3
asked in Data Ingestion & Infrastructure
Dear All & Celonis expertI would like to get advice, on how to convert the date format DD-MM-YYYY HH:MI:SS to YYYY-MM-DD HH:MI:SS?Your advice and support is appreciated. Thank YouBest Regards,Windu

Detail:in our data model, the event time must with format YYYY-MM-DD HH:MI:SS. But my data source is DD-MM-YYYY HH:MI:SS. so this data is not available as event time. I tried to follow : https://www.vertica.com/docs/9.2.x/HTML/Content/Authoring/SQLReferenceManual/Functions/Formatting/TemplatePatternsForDateTimeFormatting.htm use query : TO_TIMESTAMP(Date_column, 'YYYY-MM-DD HH:MI:SS'); (attachment 5) but the date format is not correct as below: 0028-01-20 00:00:00 (Attachment 3) The original date is 28-01-2022 13:51:24 (as attachment 4) 

40
Z
Level 3
P
Level 2
philip.schre12Level 2
asked in Data Ingestion & Infrastructure
Usage of Pull-Up Functions in data integration

Can Pull-up functions also be used in the context of transformations on Data integration side or is the usage only possible in Celonis Studio components etc.?

10
V
Level 3
kane.sher
Level 2
kane.sherLevel 2
posted in Data Ingestion & Infrastructure
Hi All,

Hi All, I am trying to take my activity table and break it out into different subactivtiies groupings. Basically trying to create a counter calculation by case whenever the activity is 'Start' and have it copy all the way down until the next 'Start' activity or using a sort of lead object to distinguish the beginning of a sub activity. Example:Case Activitiy Counter1 Start 11 Activity 11 Activity 1 1 Activity 11 Start 21 Activity 22 Start 12 Activity 1 I have tried to do this with a variety of formulas, but have been unable to recreate this. I can do this in the backend transformation, but would prefer to see if it can be done within a calc from the front end. Please advise whenever you get a chance. Thanks for reading! 

04
V
Level 3

Badge Winners

  • Celonis Certified Data Engineer
    linda.xianghas earned the badge Celonis Certified Data Engineer
  • Qualified to Prove Celonis Value
    gabriela.cruzhas earned the badge Qualified to Prove Celonis Value
  • Qualified to Write PQL Queries
    adrian.michlhas earned the badge Qualified to Write PQL Queries
  • Qualified to Identify and Drive Opportunities
    luis.montehas earned the badge Qualified to Identify and Drive Opportunities
  • First Post
    Taylor Khas earned the badge First Post
VIEW ALL
Community Code of Conduct