Cannot able to extract tables from MongoDB using Python script/Python Connector

Hi @shritej.thorv11,

In my Celoxtractor script, I just return the extraction_data object, without any transformations. Does this solve your problem?

If not, please find some debugging tips below:

• Enable debug mode in the extraction settings (see screenshot below). This will provide more logging information within Celonis.
• You can also add logging to you script, that will become visible in the log in Celonis. If you're not familiar with logging, have a look at logging — Logging facility for Python — Python 3.10.2 documentation
• Last but not least, I saw that you already have some debugging lines in your script to print to the console. This won't work in Celonis, but you can save your extracted data to a file. By doing this, you can see if the problem lies in your script or in the transportation towards Celonis.

If you have additional questions with the debugging insights, feel free to contact me.

Kind regards,

Jan-peter

Hi @janpeter.van.d ,

Thank you for the suggesting.

If I return extraction_data object, without any transformations the script will not run as you have seen I am having MongoDB as my source.

In Celonis help : "response = requests.get('https://dev80458.service-now.com/api/now/table/sys_db_object', auth = HTTPBasicAuth(parameters["username"], parameters["password"])).json()

"

But I cant use this syntax as I need to use MongoClient. So I am using.

"myclient = pymongo.MongoClient("mongodb://un-cosmos-a:TkImJbcF6joLVq49D@un-cosmos-a.mongo.cosmos.azure.com:10255/?authSource=admin&replicaSet=globaldb&maxIdleTimeMS=120000&readPreference=primary&appname=MongoDB%20Co&retryWrites=false&ssl=true")"

& then I reading it & converting it into json format.

If I run this script, it is giving me nice JSON Format output. But when I use this in Celonis it just doesn't work.

I think there is problem with transportation to Celonis or It cant read "myclient".

Can you please suggest?

I think you're close, but should swap the creation of a json object and the extraction_data appending. Would be something like this:

from bson.json_util import dumps, loads

r = mycol.find()

l = list(r) # Converts object to list

d = dumps(l) # Converts to String

data = loads(d[0]) # Creates JSON object, not sure if the [0] is needed

for item in data:

extraction_data.records.append([data['_id'], data['entityId'], data['entityType']])

return extraction_data

I don't have a MongoDB server to test this, but hopefully it works.

source: https://stackoverflow.com/a/61259655/6901541

I have tried to update the code as you suggested. But got some new errors. Attaching screenshots.

Hi @shritej.thorv11,

I don't have access to a MongoDB instance, so it is hard for me to see what's happening. Two things you can try though are:

• In your first example that worked, I see that you use 'dumps(list_cur, indent = 2)'. Maybe that would help.
• Try to visualize what your data looks like after the manipulations by adding logging statements. In order to let your logging explode, I would change the 'find()' statement with 'find_one()', to limit the extracted rows to only one.

from bson.json_util import dumps, loads

import logging

r = mycol.find_one()

l = list(r) # Converts object to list

d = dumps(l) # Converts to String

data = loads(d, indent =2) # Creates JSON object

logging.info(str(data)) # Create a string from the JSON data, and show it in the logs.  

for item in data:

extraction_data.records.append([data['_id'], data['entityId'], data['entityType']])

return extraction_data