Skip to main content

I will have periodic extractions from the transactional systems once a day into a bunch of text files (.csv)

 

How can I set up extraction jobs to do that? Is it a specific connector?

 

Because it seems the "File uploads" is only for manually upload files. But I need to upload them from a shared folder, and do it periodically and automatically.

 

TIA

 

Hello,

 

I think that the only possibility is to use ML workbench and PyCelonis to loading specific files into data frames and then upload into Celonis, file by file. Python scripts upload should be easily automated, however it will require some coding.

 

Best Regards,

Mateusz Dudek


Thx. I can't believe it... I double checked all the connectors and ...no, I can't find anything....

 

So yes, I will use pycelonis either from a customer server or from the ML workbench....

 

 


Hello,

 

I think that the only possibility is to use ML workbench and PyCelonis to loading specific files into data frames and then upload into Celonis, file by file. Python scripts upload should be easily automated, however it will require some coding.

 

Best Regards,

Mateusz Dudek

BTW I found some ideas in docs.celonis.com....

- DataPush API . https://docs.celonis.com/en/data-integration/data-pools/data-input/data-push-api/data-push-api---python-example.html

 

 

also a colleague told me about a "Hybrid Loader" - that looks like a vm with a "standalone" database and the standard DB extractor. Files are UPLOADED into that database, then uploaded to Celonis using the standard jdbc extractor.


BTW I found some ideas in docs.celonis.com....

- DataPush API . https://docs.celonis.com/en/data-integration/data-pools/data-input/data-push-api/data-push-api---python-example.html

 

 

also a colleague told me about a "Hybrid Loader" - that looks like a vm with a "standalone" database and the standard DB extractor. Files are UPLOADED into that database, then uploaded to Celonis using the standard jdbc extractor.

  1. Interesting, exactly that's the feature I've had in my mind. However then there's problem - how to schedule getting data into ML workbench VM? Schedule some bash script with cron to get file from remote location? In that case it would be rather dangerous to upload those file outside the secured company infrastructure even with password, login etc. :)
  2. Can you provide you any details about that Hybrid loader? If case it doesn't solve the problem of scheduling file upload using only Celonis, however may be good workaround. Thanks for your ideas!

 

 


Hi @Guillermo Gost,

 

There are some use cases that extract files from a shared folder such as SharePoint. Please have a check at this page at the Celonis Help pages: Microsoft SharePoint Connection (celonis.com). It needs some permissions, but then you can schedule the notebook and setup a data pipeline.


BTW I found some ideas in docs.celonis.com....

- DataPush API . https://docs.celonis.com/en/data-integration/data-pools/data-input/data-push-api/data-push-api---python-example.html

 

 

also a colleague told me about a "Hybrid Loader" - that looks like a vm with a "standalone" database and the standard DB extractor. Files are UPLOADED into that database, then uploaded to Celonis using the standard jdbc extractor.

You can schedule a Jupyter notebook, or trigger it with an action. It is in the first screen of the ML Workbench

The scheduler has the possibility to use a Custom Cron scheduling.

 

Also, you can invoke a Jupiter notebook from Action Flows or Skills, and they have their own schedulers/triggers.

 

On the hybrid loader, I saw it from a Celonis pdf from d.navarro-galizi@celonis.com , it looks like something ad-hoc (no documentation in docs.celonis.com)

 

image


Hi @Guillermo Gost,

 

There are some use cases that extract files from a shared folder such as SharePoint. Please have a check at this page at the Celonis Help pages: Microsoft SharePoint Connection (celonis.com). It needs some permissions, but then you can schedule the notebook and setup a data pipeline.

Yep, found the Sharepoint. It is a bit limited (only Sharepoint Online and it is just a python script so you need ML)... but thx!


Reply