[Tips] Data Push API : pushing CSV file

Dear all,

(You may already know it but it is not explicitly written in swagger documentation, so)
I would like to share the point of pushing CSV file by Data Push API.

The difference between default Parquet file and CSV file is only one.
When you create job by posting /jobs/, add “fileType”:“CSV” to your request body (JSON).
In case of Python implementation, it is like
payload = {‘targetName’: targetName, ‘type’: job_type, ‘dataPoolId’: pool_id, ‘fileType’: ‘CSV’}

Best regards,
Kazuhiko

2 Likes

FYI
Regarding this topic, I implemented Microsoft Power Automate that start email receiving with CSV file then trigger REST API to push this file content.

1 Like

Hi Kazuhiko,

This is really helpful and I am still wondering how I missed this post for so long! :slight_smile:

I have a question. We have a sharepoint location where CSV files are uploaded, can the API would work with the sharepoint too?
My expectation from this that Celonis would pick the CSV file from sharepoint location at specific time and load into Celonis.

Thanks,
Amogh

Dear Amogh,

Setting file in Sharepoint is the one of the trigger point for Power Automate, so you can implement like mine and it will work.

Thanks, will explore this option.

Hi Takata-san,

What does “type” :“DELTA” means here? for example, if there are duplication entries in the csv file, would these be ignored if the data exists in target table?

also, is the “delta” option also supported if I use data_pool.push_table in pycelonis?

Thanks,
Denny

Dear Denny,

You can choose either REPLACE or DELTA for type attribute.
As you imagine, DELTA option tries to update duplicated entries, whereas REPLACE is just delete old data.
In my experience, DELTA is inserting data even if same CSV is pushed, because one more column _CELONIS_CHANGE_DATE is added and it is recorded time stamp when CSV was pushed. After data is stored in Celonis, I will remove duplicated record by SELECT DISTINCT SQL.

FYI, You can check below link to understand what is required for Data Push API.
https://help.celonis.cloud/help/display/CIBC/Data+Push+API+documentation

Best regards,
Kazuhiko