were currently working on connecting a Sybase ASE database via the IBC Uplink Connector.
Weve noticed, that the configuration of the tables in the extraction in some times takes ages.
In the extraction configuration, clicking on a large table with lots of rows the settings pane for that table (like filter settings and column selection) takes minutes to load or doesnt load at all.
Turning on the Debug Mode for the extraction reveals why that is.
Looking at the log files on the uplink server, the following lines apper once I select a table in the IBC frontend:
2020-07-22 15:33:21.777 INFO 4508 --- [ol-2-thread-105] c.c.c.j.s.DatabaseConnectorService : Metadata source being used is DRIVER_METADATA --EOM--
2020-07-22 15:33:21.777 INFO 4508 --- [ol-2-thread-105] c.c.c.j.s.DatabaseConnectorService : Calling DatabaseMetaDataService#getMetadata for table dbo$_CELONIS_wo_transfer --EOM--
2020-07-22 15:33:21.777 INFO 4508 --- [ol-2-thread-105] .c.c.j.s.m.DatabaseDriverMetaDataService : Getting metadata for table dbo$_CELONIS_wo_transfer --EOM--
2020-07-22 15:33:21.777 INFO 4508 --- [ol-2-thread-105] .c.c.j.s.m.DatabaseDriverMetaDataService : Executing getMetadata with the following statement 'SELECT * FROM "dbo"."wo_transfer"' --EOM--
2020-07-22 15:41:17.241 INFO 4508 --- [ol-2-thread-105] c.c.c.j.s.DatabaseConnectorService : Metadata call finished and table structure retrieved with 44 columns for table dbo$_CELONIS_wo_transfer --EOM--
2020-07-22 15:41:17.241 INFO 4508 --- [ol-2-thread-105] c.c.c.j.s.DatabaseConnectorService : Returning getMetadata response --EOM--
You can see that the Uplink executes a
SELECT * FROM [table], causing a full table scan.That will of course take time for a table containing >10 Million rows. In this case, the SELECT took 8 Minutes to finish. I wonder why theres no LIMIT (Or TOP)-Clause for this query.
Is there a workaround to avoid such behaviour? Furthermore, is this meta data check also during the actual extraction? Because that would make it pretty much unusable.
Thank you for your help.
Best
Max