How SAP add-on transaction info are extracted?


#1

Hello
When we extract the logging data from SAP, how can we extract the information about add-on transactions?
To analyze all the variant of business process, we need to gather not only the info about SAP standard transactions but also the info about add-on transactions which also create SAP standard documents (e.g. Purchase order, Production order, etc.).
I’d like to know where we need to register the info about add-on transactions ?
Do we need to register the info about add-on transactions (e.g. TR-CD, TR-Name, Program Name, Function Name etc.) into Configuration Store before we extract the log data from SAP ?

Thank you for youe help in advance.

Best Regards,
Hiroomi


#2

Dear Hiroomi,

Thanks for the question.
I am not so sure what you want to do with this information and what the user story is here.
The logging of Transaction usage can be done on the SAP side to see which transaction are used and which not. This will not telll anything about the quality of your process as far as I can tell.

The answer on your question will depend.
In some SAP Tables the transaction name is a Field within the table Example BKPF-TCODE. So you can read it out while filling the event table. Normally Add-on transaction have there own Z-tables etc. In case you get data from there you probably can say for 100% which Add-on transaction was used?
If all not possible a tough solution would to extend “your” SAP table with a new Field ZZ-Transaction, which if filled by the SAP program and which you can extract and use for Celonis.
My suggestion: You will need to talk with your SAP consultant.

Maybe others have a better idea…

Good luck and stay :grinning:!

Hans.


#3

Dear Hans,

Thank you so much for your answer !

I assumed that we need to have “metadata” somewhere to specify what add-on transaction code is used for what “business process name” which is shown on Celonis “Overview” sheet such as “Create Purchase Requisition item”. But I think my guess was not correct.

I think the table VBAK(Sales document header) and VBAP(Sales document item data) does not have a TCODE field. So do we need to extend the table VBAK to add a new field ZZ-Transaction and do wee need to add an EXIT program to store the transaction code into this new field ?

I’d like to know how we can create the input data file for Celonis.
Is there a “Celonis agent program” that directly read the data from SAP tables (such as VBAK, MKPF, BKPF) to creat a input data file for Celonis ? And is the “Celonis agent program” run periodically to extract the differnce data from last time extraction ?
And do we need to import these input data (extraced from SAP) into Celonis database?
Or does the Celonis process mining program directly reads SAP tables evey time to analayze “process variant” on a realtime basis?

Thank you for your help in advance.

Best Regards,
Hiroomi


#4

Dear Hiroomi,

Thanks for the explanation; I am still not sure if the TA-code will give you any usefull information for quality of the process beside people use different TA’s as thought. As said for the usages of a TA you can track this in SAP by activating the logging (without any table extensions).
You are right many tables do not give you the TA code and indeed you will need to use an User Exit / enhancement point to get that information into the ZZ-Fields (again programming).

The normal way to get the table content to Celonis is in three steps:

  1. You normally would use SAP SLT to get the SAP data into SAP BI on Hana (sidecare scenario).
    This SAP Programm indeed does a delta load of all SAP tables.
    Of course this step is theoretically not nessecary when your SAP system is already on HANA.
    (We still do the transfer since we have some information not on ERP but in MS Acces and we do not want to put any load on the ERP system). But in case you can avoid it, it is of course simpler.

  2. In a second step you will use the SAP HANA studio for all the needed SQL scripts /joins to fill the event table and provide the data structure. This event table is created on the SAP HANA side by using a job.
    In our case this table is renewed once a night with the orders of the last day.

  3. We load once a night all tables into Celonis.
    Meaning real time you will actually never have since two steps are not real time: creation of new entries in the event table, secondly read the content into Celonis.

Maybe Celonis has some techniques to minimise the last two steps.

Good luck and have :grinning:!

Hans


#5

Dear Hiromi.

I believe there is some misunderstanding on the boundary between event log preparation and analysis. Celonis per se has no idea about source system, number of source systems, transaction codes, etc. In Celonis data model you specify link to event log tables which already contains proper activity names.
In order to create event log you should either follow the path described by Hans or use “extractor” provided by Celonis free of charge as part of IBC or at cost for on premise solution. Extractor has limited configuration options and can deal only with “standard” sap templates.
Any z transaction means that you have to use path described by Hans and write whole ETL thing from scratch.
You will need somebody to pass Celonis data scientist training and together with you sap consultant they will be able to define optimal path to incorporate new anslytics/new process steps into event log.
There are multiple options based on you particular sap config.


#6

Dean Hans,

Thank you so much ! I really appreciate your help.
I think I’m getting an understanding of how it works little by little. :smiley:

The only reason I was mentioning about TR(transaction)code is that I thought it (TR-code) is neccesary to retrieve “the business process name” from somwhere and by some way.

With the help on your answer and Nicks’s, my understanding is the following. Am I understanding correctly ?

(1)SAP S4HANA tables --> (2) Event log tables (in SAP side) --> (3) Celonis tables (in Celonis side)

To creat “(2) Event log tables (in SAP side)”, we need to create SQL scripts.
In your case, SAP job is scheduled at night to update event log tables daily from S4HANA tables.
(in your case, SAP BI on HANA tables from SAP and MS-Access).

“(2) Event log tables” shoud contain “the business process name” information at this point.
This is done be SQL scripts.

May I ask you few more questions ?

How do you load “(3) Celonis tables” from “(2) Event log tables” ?
Is there any program or function provided by Celonis for this task ?
or Do you need to create SQL scripts for this part too ?
Do these 2 sets (i.e. (3) and (2)) of tables have the idenctical table structure ?

Thank you for your help.

Best Regards
Hiroomi


#7

Hi, Dina Hiromi

Step 2 is on “celonis” side. On step one you copy “raw data” from sap DB to Celonis DB and use SQL scripts to transform raw data into event log.
Tools you use to create event log depend on the version of Celonis you are using - is it IBC, SAP Process Mining or “plain” Celonis

RE transaction codes. Z codes can mean very different things thats why you need input from guys who implemented them. Sometimes they only mean new screen but standard sap objects and you can use Celonis extractors, or it can be new fields in standard tables in this case standard extractors will work as well. If z code means new sap object then you have to create new SQL script to handle them.


#8

Dear Nicks,

Thank you so much! I really appreciate your help.
I think I’m getting an understanding of how it works little by little. :smiley:

Please refer to my reply to Hans’s answer.

How do we specify the link to event log tables?
Is “Celonis data model” a fixed tables structure provided by Celonis?

Do we run “extractor” tool provided by Celonis on S4HANA side ?

What does IBC stand for ?

Thank you for your help.

Best Regards
Hiroomi


#9

Hi, Hiroomi.

IBC stands for intelligent business cloud - new Celonis cloud based option (they have regular webinars about it - find nearest one on their site).

Data model is rather flexible - there are several mandatory fields and unlimited number of extra field you can load and use as report measures/analytics. Check help for more details.

I have not used standard extractors as in all my projects sap was heavily customized. I hope someone from Celonis can give you more details.


#10

Dear Nicks,

Is the followings correct ?

<Step 1>
SAP S4HANA tables (sap DB) --> by SQL Script on “celonis” side --> Event log tables (Celonis DB)
<Step 2>
Event log tables (Celonis DB) --> by SQL Script on “celonis” side --> Celonis tables (Celonis DB)

Thank you for your support.

Best Regards
Hiroomi


#11

Event log tables = Celonis tables


#12

Dear Nicks

Thank you for your answer.

So <Step 2> is not neccesary. We only need <Step 1>
<Step 1>
SAP S4HANA tables (sap DB) --> by SQL Script on “celonis” side --> Event log tables (Celonis DB)

Is my understanding correct ?

Thank you for your help in advance.

Best Regards,
Hiroomi


#13

Hi, Hiroomy

Yes, you are right


#14

Dear Nicks,

Thank you for your support!

Hans answerd me the above process. So is the follwoing pattern possible too ?
SQL script can be executed on SAP side a a SAP JOB ?

<Step 1 >
SAP S4HANA tables (sap DB) --> by SQL Script on “SAP” side --> Event log tables (Celonis DB)

Best Regards,
Hiroomi


#15

Hi,

There are multitude of options available, based on your specific case. You can do it on SAP side, you can use external ETL product, and etc, etc. Bottom line is that you have to somehow transform you raw data from sap and/or other systems into event log. Sometimes it can be easy and straightforward, sometimes require pretty tricky data transformation


#16

Dear Nicks,

Thank you for your answer.

Best Regards,
Hiroomi


#17

Dear Nicks,

Could you show me why did your project decide not to use the standard extractor in case of SAP is heavily customized ?

And did you use any ETL tools ? Or did you write SQLs from scratch ?

Could you show me What SAP tables did you use as a source data in your previous project ?

e.g.1) Document header tables (such as VBAK, MKPF, BKPF)
e.g.2) Document item tables (such as VBAP,MSEG, BSEG )
e.g.3) Add-on tables
e.g.4) Change history table (CDHDR)
e.g.5) Audit log file (YYYYMMDD.AUD) OS file
e.g.6) ???

Best Regards,
Hiroomi Nakazawa


#18

Hi Hiroomi,

I had to smile when I saw your question. The reason for that is , that you seem to have a reverse aproach from what we are doing.
We decide in our Process Mining projects which SAP Processes makes sense to follow (Business and SAP Consultants) and also also how “far” do we go. Meaning e.g. including FI transactions in O2C; MTO Process in O2C; or include Service related processes (also in VBAK) and for us very meaningfull: include O2C releated purchasing (since the vast majority of our products are directly ordered from a DC by the Sales company).
In my world the including of a new SAP table into the SAP ETL Process (towards SAP BI on HANA) takes only 30 min.
In case you are interested about EDI and printing process you will need additinally the NAST.
In case you have have Change activities on Position level you will need the CDPOS.
For Purchase related selling you will need all P2P Tables ( EKKO, EKPO, LFA1; EKKN)
For MTO you will need additionally the AFPO and JEST
Of course this goes deeper and deeper: Are you looking for open invoices? Too late payments? Use of Credit blocks; Price Changes, Automation and Rework?
Of course this is a very interativ process at least 30% of the tables where added after the first approach due to new questions of Business / SAP Consultants.

You could also look at the Celonis Standard cockpits to start with. To be honest we had to change alsmost 95% of the business logics (SQL- PQL Code), but still you can use the graphs as a very good basement.

To setup the right logic (with process types) and the setup and testing of the SQL and PQL statements can be very complex…

Good luck!

Have :grinning:!

Hans


#19

Dear Hans,

Thnak you for your answer.
I think I could get an image about how you developed ETL process.

Best Regards,
Hiroomi Nakazawa


#20

Dear Hans,

We’d lik to do process mining for “Purchase related selling”.
My understanding is that “case id” is one of a key fields of event log table.

What “unit” did you use to assign “case id” for a Purchase related selling process mining ?

Thank you for your help.

Best Regards,
Hiroomi Nakazawa