Ingestion for files - sftp. - how to schedule it?

Hello,

I think that the only possibility is to use ML workbench and PyCelonis to loading specific files into data frames and then upload into Celonis, file by file. Python scripts upload should be easily automated, however it will require some coding.

Best Regards,

Mateusz Dudek

Thx. I can't believe it... I double checked all the connectors and ...no, I can't find anything....

So yes, I will use pycelonis either from a customer server or from the ML workbench....

BTW I found some ideas in docs.celonis.com....

- DataPush API . https://docs.celonis.com/en/data-integration/data-pools/data-input/data-push-api/data-push-api---python-example.html

also a colleague told me about a "Hybrid Loader" - that looks like a vm with a "standalone" database and the standard DB extractor. Files are UPLOADED into that database, then uploaded to Celonis using the standard jdbc extractor.

1. Interesting, exactly that's the feature I've had in my mind. However then there's problem - how to schedule getting data into ML workbench VM? Schedule some bash script with cron to get file from remote location? In that case it would be rather dangerous to upload those file outside the secured company infrastructure even with password, login etc. :)
2. Can you provide you any details about that Hybrid loader? If case it doesn't solve the problem of scheduling file upload using only Celonis, however may be good workaround. Thanks for your ideas!

Hi @Guillermo Gost,

There are some use cases that extract files from a shared folder such as SharePoint. Please have a check at this page at the Celonis Help pages: Microsoft SharePoint Connection (celonis.com). It needs some permissions, but then you can schedule the notebook and setup a data pipeline.

You can schedule a Jupyter notebook, or trigger it with an action. It is in the first screen of the ML Workbench

The scheduler has the possibility to use a Custom Cron scheduling.

Also, you can invoke a Jupiter notebook from Action Flows or Skills, and they have their own schedulers/triggers.

On the hybrid loader, I saw it from a Celonis pdf from d.navarro-galizi@celonis.com , it looks like something ad-hoc (no documentation in docs.celonis.com)

Yep, found the Sharepoint. It is a bit limited (only Sharepoint Online and it is just a python script so you need ML)... but thx!