i want to push data using the continuous-batch-processing/api using this function
def push_new_chunk(self, dataPoolId, tablename, file_path, primaryKeys, table_schema, fallbackVarcharLength = None):
api = self.get_jobs_api(dataPoolId) + "/items"
payload = {'targetName': tablename, 'primaryKeys': primaryKeys
,'connectionId': self.connection_id, 'fallbackVarcharLength': fallbackVarcharLength
,'upsertStrategy':'UPSERT_WITH_NULLIFICATION', 'tableSchema':table_schema}
headers = self.get_auth()
file = {'file': open(file_path, 'rb')}
return requests.post(url=api, params = payload, files=file, headers=headers)
tableSchema is defined as per the celonis documentation here:
e.g. thats how my tableSchema looks like in python before pushing:
table_schema = {
"columns": >
{"columnName": "CreatedAt", "columnType": "DATETIME", "decimals": 0, "fieldLength": 0, "pkField": False},
# Add more columns as needed
],
"tableName": "ev_16_activities"
}
however i am always getting an error that the schema is not in the expected format:
b'Validation failed for argument e0] in public java.util.concurrent.CompletableFuture<cloud.celonis.continuousbatchprocessing.client.BucketItemTransport> cloud.celonis.continuousbatchprocessing.web.ContinuousBatchProcessingController.storeBucketItem(cloud.celonis.continuousbatchprocessing.client.StoreBucketItemRequest): nField error in object \\'storeBucketItemRequest\\' on field \\'tableSchema\\': rejected value e{"columns": #{"columnName": "CreatedAt", "columnType": "DATETIME", "decimals": 0, "fieldLength": 0, "pkField": false}], "tableName": "ev_16_activities"}]; codes typeMismatch.storeBucketItemRequest.tableSchema,typeMismatch.tableSchema,typeMismatch.cloud.celonis.parquet.meta.Table$TableTransport,typeMismatch]; arguments iorg.springframework.context.support.DefaultMessageSourceResolvable: codes astoreBucketItemRequest.tableSchema,tableSchema]; arguments s]; default message qtableSchema]]; default message aFailed to convert property value of type \\'java.lang.String\\' to required type \\'cloud.celonis.parquet.meta.Table$TableTransport\\' for property \\'tableSchema\\'; Cannot convert value of type \\'java.lang.String\\' to required type \\'cloud.celonis.parquet.meta.Table$TableTransport\\' for property \\'tableSchema\\': no matching editors or conversion strategy found]]
Can anyone help what 'cloud.celonis.parquet.meta.Table$TableTransport is , since this seems to be the format Celonis API is expecting.