i want to push data using the continuous-batch-processing/api using this function

 

  def push_new_chunk(self, dataPoolId, tablename, file_path, primaryKeys, table_schema, fallbackVarcharLength = None):

    api = self.get_jobs_api(dataPoolId) + "/items"

    payload = {'targetName': tablename, 'primaryKeys': primaryKeys  

         ,'connectionId': self.connection_id, 'fallbackVarcharLength': fallbackVarcharLength

         ,'upsertStrategy':'UPSERT_WITH_NULLIFICATION', 'tableSchema':table_schema}

    headers = self.get_auth() 

    file = {'file': open(file_path, 'rb')}

    return requests.post(url=api, params = payload, files=file, headers=headers)

 

tableSchema is defined as per the celonis documentation here:

e.g. thats how my tableSchema looks like in python before pushing:

 

table_schema = {

  "columns": >

    {"columnName": "CreatedAt", "columnType": "DATETIME", "decimals": 0, "fieldLength": 0, "pkField": False},

    # Add more columns as needed

  ],

  "tableName": "ev_16_activities"

}

 

however i am always getting an error that the schema is not in the expected format:

b'Validation failed for argument e0] in public java.util.concurrent.CompletableFuture<cloud.celonis.continuousbatchprocessing.client.BucketItemTransport> cloud.celonis.continuousbatchprocessing.web.ContinuousBatchProcessingController.storeBucketItem(cloud.celonis.continuousbatchprocessing.client.StoreBucketItemRequest): nField error in object \\'storeBucketItemRequest\\' on field \\'tableSchema\\': rejected value e{"columns": #{"columnName": "CreatedAt", "columnType": "DATETIME", "decimals": 0, "fieldLength": 0, "pkField": false}], "tableName": "ev_16_activities"}]; codes typeMismatch.storeBucketItemRequest.tableSchema,typeMismatch.tableSchema,typeMismatch.cloud.celonis.parquet.meta.Table$TableTransport,typeMismatch]; arguments iorg.springframework.context.support.DefaultMessageSourceResolvable: codes astoreBucketItemRequest.tableSchema,tableSchema]; arguments s]; default message qtableSchema]]; default message aFailed to convert property value of type \\'java.lang.String\\' to required type \\'cloud.celonis.parquet.meta.Table$TableTransport\\' for property \\'tableSchema\\'; Cannot convert value of type \\'java.lang.String\\' to required type \\'cloud.celonis.parquet.meta.Table$TableTransport\\' for property \\'tableSchema\\': no matching editors or conversion strategy found]]

 

Can anyone help what 'cloud.celonis.parquet.meta.Table$TableTransport is , since this seems to be the format Celonis API is expecting.