University of Twente Student Theses
Classification of anomalous traces in audit logs using next activity prediction
Cutinha, Jennifer Alice (2022) Classification of anomalous traces in audit logs using next activity prediction.
|
PDF
2MB |
| Abstract: | The auditing profession is an increasingly complex one. One of the main tasks of an auditor is to ensure that the financial statements produced by organisations meet legal standards. Financial statements can be considered a summary of business transactions that are encapsulated in business process, and therefore, monitoring or inspecting these processes can help validate the credibility of these statements. Given the advancements in the current technological space, the financial information of most organisations are captured by some form of information system. Process mining seeks to leverage the data recorded in these systems in the form of event logs or audit trails. For this reason, process mining as an analytical tool for auditing has gained traction in the auditing community. However, despite the growing interest, the use cases for process mining, especially for predictive audit are fairly limited in practice. This study acknowledges this gap and demonstrates a practical use case of classifying anomalous traces in audit logs. To guide the research process, the Design Science Research Methodology (DSRM) has been chosen as an appropriate research methodology. According to DSRM, the entry point for this research is the identification of a body of research, namely, Predictive Business Process Monitoring (PBPM) which is based on the continuous generation of predictions from the analysis of past execution traces. Following the identification of this research domain, the use case for predictive audit is formulated. As part of the design & development phase of the DSRM, a procedural method (the artefact) is developed which incorporates elements from both traditional data science pipelines as well as the generic approach adopted in PBPM literature. The use of this procedural method is demonstrated by classifying anomalous traces in a loan application process, wherein the classification task is formulated as a next activity prediction problem which is a supervised multi-class classification problem. Accordingly, seven representative classifiers have been chosen from PBPM literature : Random Forest (RF), Extreme Gradient Boosting (XGBOOST), Multinomial Logistic Regression (MLR), Support Vector Machines (SVM), Long Short Term Memory (LSTM) and Bidirectional Long Short Term Memory (BI-LSTM). The performances of the classifiers are evaluated using confusion matrices. Based on the confusion matrices, the precision, recall and f1-score of each classifier is also computed. Additionally, the classifiers are evaluated across four different configurations to understand the effects of hyperparameter tuning and class weighting for class imbalance on the base models. The findings indicate that RF performs the best across all configurations achieving an f1-score of 0.69 for detecting the minority class (the anomalous class). Additionally, the experiments also indicate that some models are more sensitive to hyperparameter tuning than others such as XGBOOST, CNN, BI-LSTM and LSTM with a 6-7% increase in f1-scores from their respective base models. In general, all the models have a better performance when both hyperparameter tuning and class weighting are applied together. This research has relevance for both predictive audit and PBPM research domains. In this research, a procedural method is developed that demonstrates how predictive audit can be enabled in practice through a practical use case. This is especially important considering that the adoption of advanced data and analytics in audit has been generally low. This study serves as a first step towards that acceptance. |
| Item Type: | Essay (Master) |
| Clients: | Deloitte Risk Advisory B.V. |
| Faculty: | BMS: Behavioural, Management and Social Sciences |
| Subject: | 85 business administration, organizational science |
| Programme: | Industrial Engineering and Management MSc (60029) |
| Link to this item: | https://purl.utwente.nl/essays/93398 |
| Export this item as: | BibTeX EndNote HTML Citation Reference Manager |
Show download statistics for this publication
Show download statistics for this publicationRepository Staff Only: item control page