Skip to main content

Data Ingestion & Infrastructure

Discuss and ask questions about data integration, task mining, admin settings, and PQL extractors to optimize your infrastructure.

  • 764 Topics
  • 2,049 Replies
764 Topics
daniel.ziolk
Level 2
daniel.ziolkLevel 2
asked in Data Ingestion & Infrastructure
Creating CSV file from Query as output SKILL has been used with email by celonis added attachmet file from Action Flow

SKILL:OUTPUT ATTACHMENT:Can someone explain why this is happening that encoding is not perform - what do I missed?

111
matt.witty13
Level 5
9 hours ago
manuel.wetze
Level 4
manuel.wetzeLevel 4
posted in Data Ingestion & Infrastructure
Analyze / Visualize Data Lineage of SQL backend

Hi all,I am lately thinking of how to make my life as Data Engineer easier and get a up-to-date documentation of the current backend relationships. The goal is to see: which tables from source are transformed into which VIEWS / TABLES downstream. I want to have a quick and accurate answer to:Which tables are sources for a given table in the datamodel and how are they connected? What is the potential impact on downstream tables if I change something on a given extraction. Which tables are not in use? Potentially even better would be a granularity of column and not only View/TableI believe those questions belong all to the concept of Data Lineage. Which in my opinion is currently very poorly supported within Celonis.I would like to open the discussion on how to best do this and if this is a relevant problem to more people:What is you workflow when you face the questions above? (How) are you guys documenting the SQL Backend? Is it done manually?

254
manuel.wetze
Level 4
1 day ago
rafael.souza11
Level 2
rafael.souza11Level 2
asked in Data Ingestion & Infrastructure
Unable to recreate Postman API configuration Bearer Authentication on a custom extractor builder

Hi, I'm using an API to extract datafor this purpose, the scope of Celonis, I am trying to adapt the configuration parameters that are 100% functional in POSTMAN, but when building Extractor I am finding it difficult to adapt 1) The API Resource first requires a POST request to generate the Bearer Token: Request POST: https://prod.api.wehandle.com.br/loginParamsAuthorizationHeader:Body:Scripts:Result Body Response:2) The second request uses the generated Token to GET data from the API Request GET: https://prod.api.wehandle.com.br/integrations/main/employees?page=1&limit=50&changes_to=2025-01-14T23:59:59.000Z&changes_from=2025-01-01T00:00:00.000Z&employee_address=true&employee_services=true&employee_manager=true&employee_supplier_company=trueParamsAuthorizationHeadersResult Body ResponseI'm having trouble recreating this integration successfully via Extractor BuilderPerhaps my starting point would be to decide the Authentication Method betweenBearer Authentic

90
rafael.souza11
Level 2
6 days ago
koert.kritz11
Level 3
koert.kritz11Level 3
asked in Data Ingestion & Infrastructure
Previous month KPI filter

Hi everyone, I need help with a filter.I want to put a filter on a kpi that only allows it to show the previous months data. The filter must be dynamic so that when I select a month from the date picker that it would always calulate the previous months data.I am currently using this PQL: FILTER "CEL_ACTIVITIES_OTT"."first_eventtime" >=ADD_MONTHS(PU_MAX ( CONSTANT ( ), FILTER_TO_NULL(ROUND_MONTH("CEL_ACTIVITIES_OTT"."first_eventtime"))),-1)AND"CEL_ACTIVITIES_OTT"."first_eventtime"<PU_MAX ( CONSTANT ( ),FILTER_TO_NULL( ROUND_MONTH("CEL_ACTIVITIES_OTT"."first_eventtime")))The problem is that I have to select both months from the date picker and Ideally I would like to only select one month. 

263
A
Level 3
12 days ago
thomas.guerr
Level 3
thomas.guerrLevel 3
asked in Data Ingestion & Infrastructure
Extractor configuration for data transfer from MS Sharepoint to Celonis

Hello,I am trying to get data from a Microsoft sharepoint list to the Celonis Platform.According to the Flowchart provided by the API-Team - When to use the Standard Data Ingestion API (and other methods) - the extractor builder should be used for a Sharepoint data transfer, since the REST API is exposed.I tried to set up the connection, but I fail when configuring the endpoint, since I get xml-data even if I specify that I would like to have a json structure (Request Parameter - Accept: application/json;odata=verbose)Does anyone have experience with connecting the MS universe with the Celonis universe?I also found a workaround that works (Push CSV to Data Integration), but I would rather orchestrate all the data loads in the Data Integration environment instead of using Process Automation modules to make a data transfer.Thanks for your support!BR THomas

60
thomas.guerr
Level 3
13 days ago
jakob.hanse
Level 2
jakob.hanseLevel 2
asked in Data Ingestion & Infrastructure
Staging Table for inserts in Replication Cockpit

Hi,We have a scenario where records can be created/deleted in source system multiple times.We are currently using the replication cockpit for extracting changes, but for deleted records we do not get the full information of the deleted records, we only get the primary key of the table and a change indicator = ‘D’ (Deleted) and we would like to know what the values of all the attributes for the specific records were prior to the deletion. We have located the following staging tables and views:_CELONIS_TMP_{TableName}_NEW_DATA  _CELONIS_TMP_{TableName}_TRANSFORM_DATA {TableName}_DELETED_DATA Unfortunately, it seems like the _CELONIS_TMP_{TableName}_NEW_DATA table only includes records with change indicator = ‘U’ (Updates) and not change indicator = ‘I’ (Inserts). This makes it difficult for us to map what the historical values were, if we have both inserts and deletions within the same execution of the replication cockpit.Does anyone know of a staging table for inserts we can use, befor

131
jakob.hanse
Level 2
15 days ago
christoph.kronb
Level 2
christoph.kronbLevel 2
asked in Data Ingestion & Infrastructure
Celonis STXL converter program for SAP R/3

We need for a use case the text which is stored in the SAP table STXL.I know that Celonis provides a converter program for the table STXL.https://docs.celonis.com/en/extracting-the-table-stxl.html We currently using SAP R/3 and S4H. Form the IT Department I got the message that the Celonis program only works for S4H. My question is, if some knows if it is possible to use the program for R/3 and what for changes we must do that it works for R/3. I hope someone can help me.Regards, Christoph

140
christoph.kronb
Level 2
2 months ago
raman.silot11
Level 3
raman.silot11Level 3
asked in Data Ingestion & Infrastructure
Delta filter using joined table

Hi everyone, I am currently working on optimising extractions in Celonis and applying delta for tables. But tables like BSEG, RSEG and CDPOS are not having the relevant date fields nor I can apply header table filters which I joined with these tables in delta filters. Is there any possible solution to apply delta on these tables ?

251
ilker.karak11
Level 2
2 months ago
celopeers.musup
Level 3
celopeers.musupLevel 3
asked in Data Ingestion & Infrastructure
Knowledge Model | APC [Data Consumption) Space tequirements / impact.

Knowledge Model | APC [Data Consumption) Space requirements / impact.Does a Knowledge Model have any APC [Data Consumption) Space requirements / impact?If yes, how can this be determined?  The space size is required to be known as to discern if we’ll exceed APC space.

191
manuel.wetze
Level 4
2 months ago
A
Level 3
ayushi.ade12Level 3
asked in Data Ingestion & Infrastructure
Hi All,Any best practices on optimizing the APC consumption or any solutions.

Hi All, Any best practices on optimizing the APC consumption or any solutions.

294
celopeers.musup
Level 3
2 months ago
johann.riede
Level 3
johann.riedeLevel 3
asked in Data Ingestion & Infrastructure
Setting up a OAuth 2.0 connection to SuccessFactors

Hi community, has anyone managed to set up a OAuth 2.0 connection to SAPs Success Factors yet?We're running in some issues there and I'd be happy to know if there are users out there who were able to successfully connect the two systems. Best regards!Johann

94
K
Level 2
2 months ago
J
Level 2
jonas.venneLevel 2
asked in Data Ingestion & Infrastructure
Hi Everyone,I am trying to Filter my data/table for entries, if two columns are empty. If they are empty I don't want to see them at all (Filter them from all my data).But the Filter (X!='' and Y!='') is not working.

This Screenshot show my unfilteres table:As you can see I got a lot of data where my resultsqueeze and resultvalidation is empty. I want to filter those entries. What i did is Filter the whole sheet with the following statement:FILTER ("reportdataextraction"."resultsqueeze" !='' and "reportdataextraction"."resultvalidation" !=''); But the statement is filtering every entry where either one of the columns is empty. This is my result:  What am i doing wrong? Thank you for the help! Bests,Jonas Vennemann

173
dxb.miner
Level 3
2 months ago
henning.gertz
Level 3
henning.gertzLevel 3
asked in Data Ingestion & Infrastructure
PQL Formula

The data model looks like this. (1) is VBAP(2) is VBEP(3) is MARA(4) is MVKE   Now I created an OLAP table with columns from different SAP table. When I now want to include the SAP field MVGR1 from table MVKE is receive following error message.  How should the formula look like by using e.g. DOMAIN_TABLE or BIND together with PU-Function so that I can display the field MVKE.MVGR1 in the OLAP-Table?

779
dxb.miner
Level 3
2 months ago
victor.diens
Level 2
victor.diensLevel 2
asked in Data Ingestion & Infrastructure
How to use a table twice in an extraction

Hi everybody! I’m pretty new to celonis development and never ask a question here before, but I hope you can help me! I’m extracting data from SAP for Production data and now have the issue that I want to join the JEST table to two other tables (AUFK and AFVC) on the column AUFK.OBJNR and AFVC.OBJNR. How can I do that since downloading the whole JEST table means like 200 million entries :-( I hope you can help me!  Thank you very much!

263
florian.mrkvi
Level 4
2 months ago
xiangxiang.chen11
Level 2
xiangxiang.chen11Level 2
asked in Data Ingestion & Infrastructure
How can I have a common table on the temporary table from GENERATE_RANGE function and the case table

Hello,  I need a line chart showing created/open/resolved cases count every day. ​For x axis, I used GENERATE_RANGE function to generate DATE values as below. GENERATE_RANGE ( '1D' , TO_DATE ('2024-01-01', FORMAT('%Y-%m-%d')), TODAY()) For y axis(KPI), I use: COUNT_TABLE ( "casetable" ) However, the problem is there's no common table between this generated temporary table with the case table or activity table, so the case counts cannot be calculated based on this temporary table.   ​Does anyone know a good way to create that analysis, ideally only in frontend and not change the data model?

262
dxb.miner
Level 3
2 months ago
M
Level 3
matt.slocu12Level 3
asked in Data Ingestion & Infrastructure
How can I get a single value from a PU Aggregation?

Imagine I have two tables CasesCase IDCustomerActivitiesCase IDActivityEvent Time I would like to get the earliest "Upload" activity event time across all cases for a given customer. It is easy for me to get the earliest "Upload" activity for each Case with: PU_FIRST(Cases, Activities.Eventtime, Activities.Activity = 'Upload) But this returns a row for every case. I cannot figure out how to get the first of all these results. I tried domain tables, but couldn't figure out how to set it up. Any thoughts?

263
deepu.sebas11
Level 2
3 months ago
irfan.suria12
Level 3
irfan.suria12Level 3
asked in Data Ingestion & Infrastructure
Jira - Extraction Filter Statements

Hello community,I am struggling to set extraction filter statements for Jira tables. Some column names has a $ character that causes the following error:Extraction failed to start. Error for table 'issue': Field 'project$id' does not exist or you do not have permission to view it.With the INFO:additionalFilter: "\"project$id\" IN ('1234', '5678')" Furthermore, operators such as LIKE do not work. Error executing extraction. Unknown operator: LIKE. Any hints or best practices to share when configuring Jira extraction filter statements? CheersIrfan

150
irfan.suria12
Level 3
3 months ago
K
Level 1
kong.yangLevel 1
asked in Data Ingestion & Infrastructure
Is on-premise Celonis products still supported by SAP?

SAP Note: 3345663 - Termination of reseller agreement: On-premise solutions by Celonis  I read SAP Support note 3345663 and it says "SAP Support for the on-premise Celonis products will end." Is Celonis no longer supported on the SAP platform to extract data?

90
Sverre Klein
Level 6
3 months ago
E
Level 2
eduardo.horstLevel 2
asked in Data Ingestion & Infrastructure
Process copilot security settings

Is there a way to centrally managed the creation of process copilots? I know I can do it with the permissions to "create and maintain an studio package" but that is for everything within the package. Isn't there something that can be used to choose the studio packages where the process copilot can be created? https://docs.celonis.com/en/process-copilot.html

40
E
Level 2
4 months ago
E
Level 2
eduardo.horstLevel 2
asked in Data Ingestion & Infrastructure
How to restrict the creation of new data connections inside a data pool?

Would any of the possible permissions in here can help me with that:https://docs.celonis.com/en/managing-your-data-integration-service-permissions.htmlEdit Data Pool or Manage Data Pool I want to only let some groups have that possibility (creating new data connections)

00
E
Level 2
4 months ago
A
Level 2
abhishek.kumar35Level 2
asked in Data Ingestion & Infrastructure
Task Mining Query

Hi team, wanted to know does task mining recorder works properly and give accurate results for Windows 11? I've windows 11, but I was able to install the recorder, however few things didn't worked, like screenshots didn't got captured and I was unable to get process flow view of the task even after uploading the configuration file from task mining documentation app.

10
A
Level 2
4 months ago
M
Level 1
marcel.klinkLevel 1
asked in Data Ingestion & Infrastructure
How can I sum up the days where a target date is between 2 date fields. The summation should happen per day. A more detailed description can be found in the details.

I have a “Case Table” table. This has 2 columns “Case ID” and “Target date”. I also have a 2 table “Horizon” with 3 columns “Case ID”, “Start date” and “End date”. The relationship between the tables is as follows: “Case Table” N : 1 "Horizon". The primary key is “Case ID” Target with the help of a PQL:I want to build a graph where on the X axis are the days and on the Y axis are the number of target dates.If the “target date” from the “Case Table” is between the “start date” (exclusive) and “end date” (inclusive) from the “Horizon” table, it should be counted.  Example (The case ID of both tables is the same “Case ID” = 1): Target date = 20.09.2024Start date = 18.09.2024End date = 22.09.2024 Result:X axis: 17.09.2024; Y axis: 0X axis: 18.09.2024; Y axis: 0X axis: 19.09.2024; Y axis: 1X axis: 20.09.2024; Y axis: 1X axis: 21.09.2024; Y axis: 1X axis: 22.09.2024; Y axis: 1X axis: 23.09.2024; Y axis: 0 A 1 is added for all days where the target date lies between the start date (

100
manuel.wetze
Level 4
4 months ago
M
Level 3
margaret.daven12Level 3
posted in Data Ingestion & Infrastructure
Removal of SAP on-premise extraction (OPE) client for data integration (2024-08-20)

Removal of SAP on-premise extraction (OPE) client for data integration (2024-08-20) We removed the SAP on-premise extraction client from the Celonis Platform Download Portal on 2024-08-20. While any OPE instances that are already in use are unaffected, you now won't be able to set up a new SAP extractor or upgrade any existing SAP extractors. From 2024-08-20, you should use our on-premise client (OPC) instead. For information about accessing the OPC, see Installing on-prem clients. For information about the benefits of using OPC, see On-premise clients (OPC).

00
M
Level 3
4 months ago
eby.t11
Level 1
eby.t11Level 1
asked in Data Ingestion & Infrastructure
Is there a possibility to integrate data from Elasticsearch(where relevant journey data is present in an index) with Celonis. Also if new data can be pulled via a schedule again if the source is Elasticsearch

Is there a possibility to integrate data from Elasticsearch(where relevant journey data is present in an index) with Celonis. Also if new data can be pulled via a schedule again if the source is Elasticsearch

30
eby.t11
Level 1
4 months ago
S
Level 2
shubham.gupta20Level 2
posted in Data Ingestion & Infrastructure
Is there a way to extract data from image or PDF and relate it with respect to the document number and

Is there a way to extract data from image or PDF and relate it with respect to the document number and We need to visit external URL and provide the document number. Each document will have its attachment which can be in PDF/Image format. We need to extract data from PDF/Image and prepare a table of the details (tabular format output) This table needs to be push to backend data model. Overall, we want to extract data from image/PDF relate with the PO Number/Accounting document number and push this data to backend data model.

33
mateusz.dudek14
Level 7
4 months ago

Badge winners

Show all badges
Terms & Conditions

ANY QUESTION?

Talk to an expert
FAQs
Contact us

MORE CELONIS

Academy arrow_outward
Documentation arrow_outward
Support Center arrow_outward
Partner Portal arrow_outward
Marketplace arrow_outward

CHECK US IN

LinkedIn
YouTube
Instagram
Facebook
Imprint
Disclaimer
Privacy Notice
Cookie Notice
Terms and Conditions
Celonis © 2024