Hello,
I am investigating the data pool connection via the Python API.
I am following the guide in : tenant.realm.celonois.cloud/documentation/data-push-api-python/
If I then try to enter the datapool according to the documentation in:
tenant.realm.celonois.cloud/documentation/data-push-api-python/
I get the following error:
dataPoolId.Pool does not exist
If I try it via the curl command:
curl -X GET --header Content-Type: application/jso --header Authorization: Bearer <YOUR_API_KEY> https://<YOUR_DOMAIN>.celonis.cloud/integration/api/v1/data-push/<DATA_POOL_ID>/jobs/ -X POST -d '{targetName:<TABLE_NAME>",type:REPLACE}
I get the following error:
dataPoolId.Pool id may not be empty
For the data pool id I entered the datapool id of my connection.
In the picture attached you can see that the 1st mark is the data pool id. I took this as my data pool id.
image2019-8-22_17-1-111197512 101 KB
Page 1 / 1
small update:
The push API only works for the integration, so not integration-hybrid.
Unfortunately once I push something I cannot find my file, moreover, if I try to push something else or try to push it again I get a 400 bad request error.
The push API only works for the integration, so not integration-hybrid.
Unfortunately once I push something I cannot find my file, moreover, if I try to push something else or try to push it again I get a 400 bad request error.
Hi Paul,
To upload a parquet file or dataframe using the Python API I would recommend following this example:
https://python.celonis.cloud/docs/pycelonis/en/latest/notebooks/02_Pushing_Data.html
In my opinion this is the simplest way of uploading data, it supports Hybrid too.
Do you think this would be a solution for you?
Best regards,
Simon Riezebos
To upload a parquet file or dataframe using the Python API I would recommend following this example:
https://python.celonis.cloud/docs/pycelonis/en/latest/notebooks/02_Pushing_Data.html
In my opinion this is the simplest way of uploading data, it supports Hybrid too.
Do you think this would be a solution for you?
Best regards,
Simon Riezebos
Hello Simon,
The challenge here is that my Hybrid is not a parquet endpoint, the PUSH API pushes to my understanding Parquet files, this thus means that I cannot find my table anymore, since the Hybrid connector is a JDBC connector which does not support parquet.
Hopes this helps to understand the challenge, let me know if I can push something else then parquet.
Best,
Paul
The challenge here is that my Hybrid is not a parquet endpoint, the PUSH API pushes to my understanding Parquet files, this thus means that I cannot find my table anymore, since the Hybrid connector is a JDBC connector which does not support parquet.
Hopes this helps to understand the challenge, let me know if I can push something else then parquet.
Best,
Paul
Hi Paul,
I believe the parquet files are translated to a format that works with your database over JDBC, this is done by the extractor application that is running in your network. We found a small bug in
https://python.celonis.cloud/docs/pycelonis/en/latest/installation.html
Does this solve your problem?
Best regards,
Simon Riezebos
I believe the parquet files are translated to a format that works with your database over JDBC, this is done by the extractor application that is running in your network. We found a small bug in
pycelonis when pushing to a hybrid data pool. If you install the latest version of pycelonis (currently 1.1.9) and follow the example shown above it should work:https://python.celonis.cloud/docs/pycelonis/en/latest/installation.html
Does this solve your problem?
Best regards,
Simon Riezebos
Hello,
I tried pushing with the new API, I received the following error:
The datamodel I push to is connected to SAP.
Please let me know how I can fix this if possible.
I have a additional question regarding to the push pull topic. When I pull data, I cannot pull a KPI when it has a variable in it. Is there any way to pull KPIs with a variable in their calculation?
I tried pushing with the new API, I received the following error:
HTTPError: File upload problem: The file </tmp/celonis_push_Table_NAME_1574786623.9736688_0.parquet> is not a supported parquet/csv file.The datamodel I push to is connected to SAP.
Please let me know how I can fix this if possible.
I have a additional question regarding to the push pull topic. When I pull data, I cannot pull a KPI when it has a variable in it. Is there any way to pull KPIs with a variable in their calculation?
Hi Paul,
It is possible that this has something to do with the data, could you try this example? Assuming there is a data pool object in
This also return extra logging which we can use to check what happened to the uploaded file. You can find the id of your data push job at in a request that looks like:
If it doesnt work, could you go back after a few hours and run the following? Here
Does the output of this show any uploaded files?
Regards,
Simon
It is possible that this has something to do with the data, could you try this example? Assuming there is a data pool object in
hybrid_pool.import logging
logger = logging.getLogger('pycelonis')
logger.setLevel(logging.DEBUG)
import pandas as pd
df = pd.util.testing.makeMixedDataFrame()
hybrid_pool.push_table(df,"DATA_PUSH_TEST")
This also return extra logging which we can use to check what happened to the uploaded file. You can find the id of your data push job at in a request that looks like:
/integration-hybrid/api/v1/data-push/{pool_id}/jobs/{job_id}If it doesnt work, could you go back after a few hours and run the following? Here
job_id is the id of the job.hybrid_pool.celonis.api_request(f"{hybrid_pool.url_data_push}{job_id}/chunks"})Does the output of this show any uploaded files?
Regards,
Simon
Hello,
I was checking out the API again, this time with the result coming from the duplicate_checker notebook build by Celonis. When I here try to push Data back to the SAP Database, I found the following error:
So I think sometimes it works when pushing a specific format, but for other formats it doesnt show any clear errors.
I was checking out the API again, this time with the result coming from the duplicate_checker notebook build by Celonis. When I here try to push Data back to the SAP Database, I found the following error:
file upload problem, not a supported parquet/csv file.
from pycelonis.utils.paquet_utils import read_parquet
So I think sometimes it works when pushing a specific format, but for other formats it doesnt show any clear errors.
Reply
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.