# Niki van Stein: publications > Niki van Stein is an Associate Professor in explainable AI for machine learning and optimisation at LIACS, Leiden University, and leads the XAI group there. This file lists all 148 publications on https://www.nikivanstein.nl, newest first, with authors, venue, links and abstract. A shorter overview is at https://www.nikivanstein.nl/llms.txt. ## Large language models and evolutionary computation: A critical review of bidirectional interaction, automated algorithm design, and co-adaptive systems - Authors: Dikshit Chauhan, Bapi Dutta, Indu Bala, Niki van Stein, Thomas Bäck, Anupam Yadav - Venue: Computer Science Review - Date: 2027-02-01 - Page: https://www.nikivanstein.nl/publication/2027-02-01-Large-language-models-and-evolutionary-computation-A-critical-review-of-bidirectional-interaction-automated-algorithm-design-and-co-adaptive-systems - DOI: https://doi.org/10.1016/j.cosrev.2026.101050 - arXiv: https://arxiv.org/abs/2505.15741 - PDF: https://arxiv.org/pdf/2505.15741 Large Language Models (LLMs) and Evolutionary Computation (EC) are increasingly being combined to support automated optimization, algorithm design, and adaptive decision-making. This survey reviews the bidirectional interaction between these two paradigms and examines how their complementary strengths can be leveraged in hybrid intelligent systems. First, we analyze how EC can enhance LLM-based systems through prompt optimization, hyperparameter tuning, and architecture search. Second, we review how LLMs can im- prove EC by supporting metaheuristic design, surrogate reasoning, adaptive operator control, and heuristic generation. We further discuss emerging co-adaptive frameworks in which LLMs and EC interact through iterative feedback loops. Beyond summarizing recent developments, the survey provides a structured perspec- tive on interaction mechanisms, application patterns, and methodological challenges, including computational cost, reproducibility, interpretability, benchmarking, and generalization. The paper concludes by outlining open research questions and future directions for developing more robust, transparent, and scalable LLM-EC systems. ## Agentic Large Language Models - Authors: Max van Duijn, Michiel van der Meer, Aske Plaat, Niki van Stein - Venue: Springer (book, coming soon) - Date: 2026-10-09 - Page: https://www.nikivanstein.nl/publication/2026-10-09-Agentic-Large-Language-Models - arXiv: https://arxiv.org/abs/2610.13455 - PDF: https://arxiv.org/pdf/2610.13455 Large language models (LLMs) have taken the world by storm. Their ability to communicate with us in our own language, and to reason about our world, has made them a favorite tool for writing, answering questions, and much more. Their answers already influence our decisions and our actions. What would happen if LLMs themselves could act in the world directly? What would happen if LLMs could interact with each other? Agentic LLMs combine the power of large language models with the ability to act in the real world. In this book we will discuss how agentic LLMs work, how they act, how they interact, and what this means for us, as users. ## Breaking Bureaucracy: Evaluating open-source LLMs for legal document review - Authors: Farrukh Baratov, Niki van Stein, Suzan Verberne - Venue: arXiv preprint arXiv:2610.06345 - Date: 2026-10-05 - Page: https://www.nikivanstein.nl/publication/2026-10-05-Breaking-Bureaucracy-Evaluating-open-source-LLMs-for-legal-document-review - arXiv: https://arxiv.org/abs/2610.06345 - PDF: https://arxiv.org/pdf/2610.06345 In this paper, we evaluate open-source generative LLMs on legal Natural Language Inference (NLI). Legal inspectorial processes take place in specific domains and often deal with confidential data. This creates a need for working with local models that do not require labeled training data. We evaluate our models on the ContractNLI benchmark and two NLI4Wills datasets. We successfully reproduce the baseline for the task (Span NLI BERT) and we evaluate multiple open-source LLMs on the same task. We analyze the invalid rate of the models, and their stability across temperature settings and domains. Among the generative models, Gemma-4 26B performs the best, reaching an accuracy of 81.2%, even outperforming the supervised model on one metric. On accuracy, it is not possible to beat the supervised model with zero-shot approaches. Qwen-3.6 35B performs well on both ContractNLI and additional datasets in the legal wills domain. Our findings indicate that zero-shot, open-source, generative LLMs are a viable alternative for real-world legal NLI when no supervised data is available. Our code is available at https://github.com/fbaratov/contractnli-llms. ## EvoMO-SR: Multiobjective LLM-based Evolution of Symbolic Expressions with substructure guidance - Authors: Cristina Rossetti, Anna V. Kononova, Thomas Bäck, Fei Liu, Niki van Stein - Venue: arXiv preprint arXiv:2609.36187 - Date: 2026-09-28 - Page: https://www.nikivanstein.nl/publication/2026-09-28-EvoMO-SR-Multiobjective-LLM-based-Evolution-of-Symbolic-Expressions-with-substructure-guidance - arXiv: https://arxiv.org/abs/2609.36187 - PDF: https://arxiv.org/pdf/2609.36187 Symbolic Regression (SR) is a data-driven method for scientific discovery which searches for interpretable analytical relationships within data. Recently, Large Language Models (LLMs) have also had a significant impact on scientific discovery, enabling the automation of various stages of the process. For these reasons, the possibility of harnessing the embedded scientific knowledge and programming capabilities of LLMs to solve SR tasks has emerged, showing promising performance compared with traditional methods. We propose EvoMO-SR, a novel LLM-driven SR framework in which the LLM generates equation skeletons, with their coefficients fitted separately by an external optimizer. The framework includes a multi-objective survival selection which controls bloating by balancing accuracy and complexity, and a substructure guidance mechanism which mutates expressions with candidate reusable building blocks. EvoMO-SR achieves the best accuracy in seven of the eight in-domain and out-of-domain settings for LSR-Synth, using a small LLM model, i.e., Llama-3.1-8B-Instruct. We also evaluated structural recovery through two symbolic accuracy metrics based on canonicalized subtree overlap and term matching, showing that our method has a greater probability of recovering highly accurate symbolic structures. ## Detecting and Repairing Hallucinations in Retrieval-Augmented Generation - Authors: Sai Krishna Reddy Mulakkayala, Niki van Stein, Aske Plaat - Venue: arXiv preprint arXiv:2608.29307 - Date: 2026-08-29 - Page: https://www.nikivanstein.nl/publication/2026-08-29-Detecting-and-Repairing-Hallucinations-in-Retrieval-Augmented-Generation - arXiv: https://arxiv.org/abs/2608.29307 - PDF: https://arxiv.org/pdf/2608.29307 Language models increasingly answer questions by consulting retrieved documents rather than memory alone, a design now common in search assistants and enterprise knowledge tools. Grounding a model in retrieved text reduces unsupported statements but does not eliminate them, and a reader cannot tell a grounded sentence from an invented one. Most research on this problem stops at detection, yet flagging a faulty answer changes nothing for the person reading it, and little is known about which action should follow. Using RAGTruth, a benchmark whose unsupported passages are annotated by hand, we split each flagged answer into individual factual claims, check each against the retrieved source, and compare leaving the answer untouched with three repair strategies of increasing richness: deleting an unsupported claim, replacing it with source text, and rewriting it. Three language models from different families judge the 916 repaired answers. Every strategy reduces the proportion of answers judged to contain unsupported content, and all three judges agree on the ordering. Deletion achieves the largest reduction while retaining least of the original answer, at 64.3% of the text, whereas rewriting retains 80.1% and reduces least. Repair is not confined to faulty answers: 83.5% of answers annotated clean are edited too. The strategies occupy different points on a grounding preservation trade-off rather than forming a quality ranking, and choosing between them needs evidence about answer usefulness that automatic metrics cannot supply. ## Lens-descriptor guided evolutionary algorithm for optimization of complex optical systems with glass choice - Authors: Kirill Antonov, Teus Tukker, Tiago Botari, Thomas Bäck, Anna V. Kononova, Niki van Stein - Venue: International Conference on Parallel Problem Solving from Nature - Date: 2026-08-25 - Page: https://www.nikivanstein.nl/publication/2026-08-25-Lens-descriptor-guided-evolutionary-algorithm-for-optimization-of-complex-optical-systems-with-glass-choice - DOI: https://doi.org/10.1007/978-3-032-36229-2_25 - arXiv: https://arxiv.org/abs/2601.22075 - PDF: https://arxiv.org/pdf/2601.22075 Designing high-performance optical lenses entails exploring a high-dimensional, tightly constrained space of surface curvatures, glass choices, element thicknesses, and spacings. In practice, standard optimizers (e.g., gradient-based local search and evolutionary strategies) often converge to a single local optimum, overlooking many comparably good alternatives that matter for downstream engineering decisions. We propose the Lens Descriptor-Guided Evolutionary Algorithm (LDG-EA), a two-stage framework for multimodal lens optimization. LDG-EA first partitions the design space into behavior descriptors defined by curvature-sign patterns and material indices, then learns a probabilistic model over descriptors to allocate evaluations toward promising regions. Within each descriptor, LDG-EA applies the Hill-Valley Evolutionary Algorithm with covariance-matrix self-adaptation to recover multiple distinct local minima, optionally followed by gradient-based refinement. On a 24-variable (18 continuous and 6 integer), six-element Double-Gauss topology, LDG-EA generates on average around 14500 candidate minima spanning 636 unique descriptors, an order of magnitude more than a CMA-ES baseline, while keeping wall-clock time at one hour scale. Although the best LDG-EA design is slightly worse than a fine-tuned reference lens, it remains in the same performance range. Overall, the proposed LDG-EA produces a diverse set of solutions while maintaining competitive quality within practical computational budgets and wall-clock time. ## Evolving Executable Pipeline Programs for AutoML with Language Models - Authors: Sofoklis Kitharidis, Cor J. Veenman, Jan N. van Rijn, Thomas Bäck, Niki van Stein - Venue: arXiv preprint arXiv:2608.16416 - Date: 2026-08-17 - Page: https://www.nikivanstein.nl/publication/2026-08-17-Evolving-Executable-Pipeline-Programs-for-AutoML-with-Language-Models - arXiv: https://arxiv.org/abs/2608.16416 - PDF: https://arxiv.org/pdf/2608.16416 Automated machine learning (AutoML) systems search for pipelines within a space of preprocessing operators, learners, and hyper-parameters specified in advance: they can select and tune known components, but cannot produce structure outside that space. We present LACE, an AutoML framework that instead searches over complete executable pipeline programs: an evolutionary loop maintains a population of scikit-learn-compatible Python classes, and a large language model acts as the variation operator. To our knowledge, LACE is the first to formulate general tabular pipeline AutoML this way, evaluated on standardized OpenML tasks under a leakage-controlled protocol that withholds dataset identity from the generator. Because every candidate is ordinary Python, the returned pipeline and the search that produced it can be inspected and edited directly, rather than only through a framework's model objects. On 68 OpenML classification tasks, LACE with GPT-5.4-mini significantly outperforms auto-sklearn, H2O, and a fixed XGBoost baseline, with no detectable difference against AutoGluon, the strongest search-based system evaluated, while covering the full benchmark. Newer tabular foundation models are more accurate on the subset of tasks they support, but apply a fixed pretrained predictor rather than returning an editable task-specific program. LACE's contribution is therefore not raw accuracy but a search space defined by code: complete coverage, pipelines practitioners can reuse directly, and a component set extended by editing the prompt rather than the framework. ## Pruning Federated Models through Loss Landscape Analysis and Client Agreement Scoring - Authors: Christian Internò, Elena Raponi, Markus Olhofer, Ali Raza, Thomas Bäck, Niki van Stein, Yaochu Jin, Barbara Hammer - Venue: IEEE Internet of Things Journal - Date: 2026-07-15 - Page: https://www.nikivanstein.nl/publication/2026-07-15-Pruning-Federated-Models-through-Loss-Landscape-Analysis-and-Client-Agreement-Scoring - DOI: https://doi.org/10.1109/jiot.2026.3686028 - arXiv: https://arxiv.org/abs/2405.10271 - PDF: https://arxiv.org/pdf/2405.10271 The practical deployment of Federated Learning (FL) on resource-constrained devices is fundamentally limited by the high cost of training large models and the instability caused by heterogeneous (non-IID) client data. Conventional pruning methods often treat data heterogeneity as a problem to be mitigated. In this work, we introduce a paradigm shift: we reframe client diversity as a feature to be harnessed. We propose AutoFLIP, a framework that begins not with training, but with a one-time federated loss exploration. During this phase, clients collaboratively build a map of the collective loss landscape, using their diverse data to reveal the problem's essential structure. This shared intelligence then guides an adaptive pruning strategy that is dynamically refined by client agreement throughout training. This approach allows AutoFLIP to identify robust and efficient sub-networks from the outset. Our extensive experiments show that AutoFLIP reduces computational overhead by an average of 52% and communication costs by over 65% while simultaneously achieving state-of-the-art accuracy in challenging non-IID settings. ## Optimizing for Difference: LLM-Driven Benchmark Design for Maximum Solver Discriminability - Authors: Ananta A. Shahane, Niki van Stein - Venue: Proceedings of the Genetic and Evolutionary Computation Conference Companion - Date: 2026-07-13 - Page: https://www.nikivanstein.nl/publication/2026-07-13-Optimizing-for-Difference-LLM-Driven-Benchmark-Design-for-Maximum-Solver-Discriminability - DOI: https://doi.org/10.1145/3795101.3815555 The GECCO Benchmark Design Challenge formulates benchmark construction as the problem of maximizing performance differences between optimizers, measured via pairwise rank distances. We approach this as a meta-optimization task and propose an automated framework that generates and refines benchmark suites using large language models (LLMs) and evolutionary search. Candidate functions are synthesized through LLM-driven evolution and evaluated directly on their ability to discriminate between five standard optimizers under a fixed evaluation budget. We further optimise the suite by combinatorial optimization of all generated functions over all considered dimensionalities in the range of [2,80). Our results show that this combined approach substantially increases discriminability, improving scores from approximately 2.04 to 2.58. These findings highlight that effective benchmark design requires not only diverse function structures but also careful control of problem dimensionality, and demonstrate the potential of learning-driven methods for constructing targeted benchmarking suites. ## Large Language Models for Metaheuristic Design: Exploring Challenges, Limitations, Tools, Benchmarking and Future Opportunities - Authors: Roman Senkerik, Niki van Stein, Adam Viktorin, Michal Pluhacek, Thomas Bäck - Venue: Proceedings of the Genetic and Evolutionary Computation Conference Companion - Date: 2026-07-13 - Page: https://www.nikivanstein.nl/publication/2026-07-13-Large-Language-Models-for-Metaheuristic-Design-Exploring-Challenges-Limitations-Tools-Benchmarking-and-Future-Opportunities - DOI: https://doi.org/10.1145/3795101.3807753 ## EvoHIIT: Preference-Based Selection in LLM-Assisted Evolutionary Design of Natural Language Solutions - Authors: Johana Chen, Niki van Stein, Robert Cabri, Thomas Bäck, Anna V. Kononova - Venue: Proceedings of the Genetic and Evolutionary Computation Conference Companion - Date: 2026-07-13 - Page: https://www.nikivanstein.nl/publication/2026-07-13-EvoHIIT-Preference-Based-Selection-in-LLM-Assisted-Evolutionary-Design-of-Natural-Language-Solutions - DOI: https://doi.org/10.1145/3795101.3814653 ## Structural Bias in Multi-objective Optimization - Authors: Jakub Kudela, Niki van Stein, Thomas Bäck, Anna Kononova - Venue: Proceedings of the Genetic and Evolutionary Computation Conference - Date: 2026-07-10 - Page: https://www.nikivanstein.nl/publication/2026-07-10-Structural-Bias-in-Multi-objective-Optimization - DOI: https://doi.org/10.1145/3795095.3805108 Structural bias (SB) refers to systematic preferences of an optimisation algorithm for particular regions of the search space that arise independently of the objective function. While SB has been studied extensively in single-objective optimisation, its role in multi-objective optimisation remains largely unexplored. This is problematic, as dominance relations, diversity preservation and Pareto-based selection mechanisms may introduce or amplify structural effects. ## Landscape-aware Automated Algorithm Design: An Efficient Framework for Real-world Optimization - Authors: Haoran Yin, Shuaiqun Pan, Zhao Wei, Jian Cheng Wong, Yew-Soon Ong, Anna Kononova, Thomas Bäck, Niki van Stein - Venue: Proceedings of the Genetic and Evolutionary Computation Conference - Date: 2026-07-10 - Page: https://www.nikivanstein.nl/publication/2026-07-10-Landscape-aware-Automated-Algorithm-Design-An-Efficient-Framework-for-Real-world-Optimization - DOI: https://doi.org/10.1145/3795095.3805094 - arXiv: https://arxiv.org/abs/2602.04529 - PDF: https://arxiv.org/pdf/2602.04529 The advent of Large Language Models (LLMs) has opened new frontiers in automated algorithm design, giving rise to numerous powerful methods. However, these approaches retain critical limitations: they require extensive evaluation of the target problem to guide the search process, making them impractical for real-world optimization tasks, where each evaluation consumes substantial computational resources. This research proposes an innovative and efficient framework that decouples algorithm discovery from high-cost evaluation. Our core innovation lies in combining a Genetic Programming (GP) function generator with an LLM-driven evolutionary algorithm designer. The evolutionary direction of the GP-based function generator is guided by the similarity between the landscape characteristics of generated proxy functions and those of real-world problems, ensuring that algorithms discovered via proxy functions exhibit comparable performance on real-world problems. Our method enables deep exploration of the algorithmic space before final validation while avoiding costly real-world evaluations. We validated the framework's efficacy across multiple real-world problems, demonstrating its ability to discover high-performance algorithms while substantially reducing expensive evaluations. This approach shows a path to apply LLM-based automated algorithm design to computationally intensive real-world optimization challenges. ## LLaMEA-BO: A Large Language Model Evolutionary Algorithm for Automatically Generating Bayesian Optimization Algorithms - Authors: Wenhu Li, Niki van Stein, Thomas Bäck, Elena Raponi - Venue: Proceedings of the Genetic and Evolutionary Computation Conference - Date: 2026-07-10 - Page: https://www.nikivanstein.nl/publication/2026-07-10-LLaMEA-BO-A-Large-Language-Model-Evolutionary-Algorithm-for-Automatically-Generating-Bayesian-Optimization-Algorithms - DOI: https://doi.org/10.1145/3795095.3805059 - arXiv: https://arxiv.org/abs/2505.21034 - PDF: https://arxiv.org/pdf/2505.21034 Bayesian optimization (BO) is a powerful class of algorithms for optimizing expensive black-box functions, but designing effective BO algorithms remains a manual, expertise-driven task. Recent advancements in Large Language Models (LLMs) have opened new avenues for automating scientific discovery, including the automatic design of optimization algorithms. While prior work has used LLMs within optimization loops or to generate non-BO algorithms, we tackle a new challenge: Using LLMs to automatically generate full BO algorithm code. Our framework uses an evolution strategy to guide an LLM in generating Python code that preserves the key components of BO algorithms: An initial design, a surrogate model, and an acquisition function. The LLM is prompted to produce multiple candidate algorithms, which are evaluated on the established Black-Box Optimization Benchmarking (BBOB) test suite from the COmparing Continuous Optimizers (COCO) platform. Based on their performance, top candidates are selected, combined, and mutated via controlled prompt variations, enabling iterative refinement. Despite no additional fine-tuning, the LLM-generated algorithms outperform state-of-the-art BO baselines in 19 (out of 24) BBOB functions in dimension 5 and generalize well to higher dimensions, and different tasks (from the Bayesmark framework). This work demonstrates that LLMs can serve as algorithmic co-designers, offering a new paradigm for automating BO development and accelerating the discovery of novel algorithmic combinations. The source code is provided at https://github.com/Ewendawi/LLaMEA-BO. ## Block-Bench: A Framework for Controllable and Transparent Discrete Optimization Benchmarking - Authors: Furong Ye, Frank Neumann, Thomas Bäck, Niki van Stein - Venue: Proceedings of the Genetic and Evolutionary Computation Conference - Date: 2026-07-10 - Page: https://www.nikivanstein.nl/publication/2026-07-10-Block-Bench-A-Framework-for-Controllable-and-Transparent-Discrete-Optimization-Benchmarking - DOI: https://doi.org/10.1145/3795095.3805071 - arXiv: https://arxiv.org/abs/2604.06973 - PDF: https://arxiv.org/pdf/2604.06973 We present a novel approach for constructing discrete optimization benchmarks that enables fine-grained control over problem properties, and such benchmarks can facilitate analyzing discrete algorithm behaviors. We build benchmark problems based on a set of block functions, where each block function maps a subset of variables to a real value. Problems are instantiated through a set of block functions, weight factors, and an adjacency graph representing the dependency among the block functions. Through analyzing intermediate block values, our framework allows to analyze algorithm behavior not only in the objective space but also at the level of variable representations in the obtained solutions. This capacity is particularly useful for analyzing discrete heuristics in large-scale multi-modal problems, thereby enhancing the practical relevance of benchmark studies. We demonstrate how the proposed approach can inspire the related work in self-adaptation and diversity control in evolutionary algorithms. Moreover, we explain that the proposed benchmark design enables explicit control over problem properties, supporting research in broader domains such as dynamic algorithm configuration and multi-objective optimization. ## Assessing Reproducibility in Evolutionary Computation: A Case Study using Human-and LLM-based Assessment - Authors: Francesca Da Ros, Tarik Začiragić, Aske Plaat, Thomas Bäck, Niki van Stein - Venue: Proceedings of the Genetic and Evolutionary Computation Conference - Date: 2026-07-10 - Page: https://www.nikivanstein.nl/publication/2026-07-10-Assessing-Reproducibility-in-Evolutionary-Computation-A-Case-Study-using-Human-and-LLM-based-Assessment - arXiv: https://arxiv.org/abs/2602.07059 - PDF: https://arxiv.org/pdf/2602.07059 Reproducibility is an important requirement in evolutionary computation, where results largely depend on computational experiments. In practice, reproducibility relies on how algorithms, experimental protocols, and artifacts are documented and shared. Despite growing awareness, there is still limited empirical evidence on the actual reproducibility levels of published work in the field. In this paper, we study the reproducibility practices in papers published in the Evolutionary Combinatorial Optimization and Metaheuristics track of the Genetic and Evolutionary Computation Conference over a ten-year period. We introduce a structured reproducibility checklist and apply it through a systematic manual assessment of the selected corpus. In addition, we propose RECAP (REproducibility Checklist Automation Pipeline), an LLM-based system that automatically evaluates reproducibility signals from paper text and associated code repositories. Our analysis shows that papers achieve an average completeness score of 0.62, and that 36.90% of them provide additional material beyond the manuscript itself. We demonstrate that automated assessment is feasible: RECAP achieves substantial agreement with human evaluators (Cohen's k of 0.67). Together, these results highlight persistent gaps in reproducibility reporting and suggest that automated tools can effectively support large-scale, systematic monitoring of reproducibility practices. ## LLM-Driven Evolutionary Generation of Multi-Objective Bayesian Optimization Algorithms - Authors: Georgios Laskaris, Reuben Brasher, Niki van Stein, Elena Raponi, Thomas Bäck, Florian Neukart - Venue: arXiv preprint arXiv:2607.08791 - Date: 2026-07-06 - Page: https://www.nikivanstein.nl/publication/2026-07-06-LLM-Driven-Evolutionary-Generation-of-Multi-Objective-Bayesian-Optimization-Algorithms - arXiv: https://arxiv.org/abs/2607.08791 - PDF: https://arxiv.org/pdf/2607.08791 Designing effective multi-objective Bayesian optimization (MOBO) algorithms requires balancing many interdependent design choices whose optimal configuration is problem-dependent and typically demands deep expertise. We extend the LLaMEA framework to MOBO, using large language models as mutation and crossover operators within evolutionary strategies to generate complete algorithm implementations, with SMAC hyperparameter optimization integrated into the evolutionary loop. Across nine evolutionary runs we generated approximately 900 algorithms and benchmarked them on twelve synthetic problems (ZDT, DTLZ, WFG) and three real-world engineering problems (RE), using a BoFire qParEGO implementation as a state-of-the-art Bayesian-optimization baseline. On the synthetic suite the strongest generated algorithm attains the highest mean normalized hypervolume (0.971, vs. 0.869 for qParEGO) while requiring roughly 60x less wall-clock time; a Friedman test with post-hoc analysis places the two in a single top-performing group, and per-problem tests find the generated algorithm significantly better than qParEGO on 7 of the 12 problems and never worse, matching state-of-the-art accuracy at an order-of-magnitude lower cost. On the three unseen real-world engineering problems a generated algorithm attains the best mean normalized hypervolume (0.985, vs. 0.971 for qParEGO)--significantly better than qParEGO on two of the three problems--at roughly 3.4x lower wall-clock cost, confirming that the gains transfer beyond the synthetic regime. LLM-driven evolutionary search can thus discover algorithm designs that achieve Pareto-efficient trade-offs difficult to reach through manual design. ## Computational Intelligence: 16th International Joint Conference, IJCCI 2024 Porto, Portugal, November 20-22, 2024 Revised Selected Papers - Authors: Niki van Stein, Francesco Marcelloni, Kurosh Madani, Joaquim Filipe (Eds.) - Venue: Springer Nature - Date: 2026-07-04 - Page: https://www.nikivanstein.nl/publication/2026-07-04-Computational-Intelligence-16th-International-Joint-Conference-IJCCI-2024-Porto-Portugal-November-20-22-2024-Revised-Selected-Papers ## Maintenance Planning for Prognostic Health Management of Multi-Component Systems: A Case Study of Multi-Objective Optimisation for Railway Vehicles - Authors: Bernd Wagner, Sofoklis Kitharidis, Furong Ye, Thomas Bäck, Niki van Stein - Venue: PHM Society European Conference - Date: 2026-07-03 - Page: https://www.nikivanstein.nl/publication/2026-07-03-Maintenance-Planning-for-Prognostic-Health-Management-of-Multi-Component-Systems-A-Case-Study-of-Multi-Objective-Optimisation-for-Railway-Vehicles - DOI: https://doi.org/10.36001/phme.2026.v9i1.5006 - PDF: https://papers.phmsociety.org/index.php/phme/article/download/5006/2866 Condition-based maintenance (CBM) offers an opportunity to improve classical preventive maintenance strategies by transitioning to a dedicated prognostics & health management (PHM) strategy for individual components. This means that time-based fixed maintenance intervals are being replaced by condition-dependent and component-specific interventions. For multi-component, complex systems, such a component-condition-based maintenance strategy yields multiple remaining useful life (RUL) prognostics. Maintainers must combine these into optimal maintenance plans that account for reliability, availability, maintainability, safety and costs (RAMS-C). Combining multiple RUL prognostics with the system's real-world dynamics, including architecture, anticipated future operating conditions, economic interdependence, stochastic interdependence and structural interdependence, yields complex maintenance planning decisions. To address this problem, several studies propose solutions based on multi-objective optimisation algorithms. However, these are often based on a priori knowledge and do not account for the day-to-day dynamics of operational variations. This paper presents a methodology for managing the complexity of maintenance scheduling under these conditions. The model is applied to the maintenance history of a Voith Maxima 30CC cargo locomotive. A priori maintenance models are compared with real-life implementation and optimised models using evolutionary multi-objective optimisation (EMO) to assess the rigour of the a priori plans under varying operating conditions. The study shows that applying such algorithms can significantly reduce costs while increasing overall system reliability and availability. ## 3DMorph: Single-Image-Guided Local 3D Shape Editing and Morphing - Authors: Tobias Preintner, Yunfei Deng, Phillip Müller, Sebastian Illing, Adrian König, Thomas Bäck, Elena Raponi, Niki van Stein - Venue: arXiv preprint arXiv:2606.07115 - Date: 2026-06-05 - Page: https://www.nikivanstein.nl/publication/2026-06-05-3DMorph-Single-Image-Guided-Local-3D-Shape-Editing-and-Morphing - DOI: https://doi.org/10.1109/ijcnn69854.2026.11728815 - arXiv: https://arxiv.org/abs/2606.07115 - PDF: https://arxiv.org/pdf/2606.07115 Despite recent progress in 3D generation, intuitive editing of existing shapes remains limited. Unlike images, which benefit from well-established inpainting tools, general 3D objects such as meshes still lack simple and effective methods for local shape editing. Existing approaches are often global, domain-specific, require complex user interaction, or focus on appearance (color and texture) rather than geometry. We introduce 3DMorph, a training-free framework for single-image-guided local 3D shape editing and morphing. Given an edited image showing a desired shape modification, our method automatically localizes the relevant 3D region and transfers 2D modifications to 3D while preserving unmodified areas. 3DMorph also enables intermediate shape generation between the original and edited objects, facilitating design exploration. To benchmark editing quality, we introduce Delta3D, an image-guided local 3D editing benchmark with paired ground-truth edits. Experimental results show that 3DMorph translates intuitive 2D edits into 3D, outperforming state-of-the-art generative and editing methods. ## Every Component is a Lookup: Token Attribution and Composition from a Single Decomposition - Authors: Po-Kai Chen, Aske Plaat, Niki van Stein - Venue: arXiv preprint arXiv:2605.23393 - Date: 2026-05-22 - Page: https://www.nikivanstein.nl/publication/2026-05-22-Every-Component-is-a-Lookup-Token-Attribution-and-Composition-from-a-Single-Decomposition - arXiv: https://arxiv.org/abs/2605.23393 - PDF: https://arxiv.org/pdf/2605.23393 Interpretability methods for transformers are typically built around separate questions: which components interact, how information routes to the output, and which input tokens contribute. Because these methods rely on different assumptions, their answers are difficult to relate. We argue that two architecturally motivated assumptions suffice to address all three questions: attention and MLPs share a key-value form, $φ(S)\,U$, in which $φ(S)$ selects over values $U$, and components read from an additive residual stream, the sum of component outputs. Holding these selections at their forward-pass values turns the model into a computational graph, of which component interactions, composition paths, and token attribution are different readouts. We develop Unpack, a backward attribution procedure over this graph, and validate each readout against the corresponding established test: interaction scores predict ablation effects across models from 160M to 6.9B parameters, recovered routes reproduce established circuits down to the key, query, or value branch the circuit specifies, and token attribution passes the same faithfulness test as dedicated attribution methods. The results suggest that these two assumptions suffice for the interpretability questions above. On a task with a known circuit, we find that contribution and causal effect can differ, and that the difference has a recognisable signature: components that matter for the task change their contribution when the task is removed from the input, while components that act like a bias term do not. Code is available at https://github.com/Fun-Cry/unpacklm. ## MM-OptBench: A Solver-Grounded Benchmark for Multimodal Optimization Modeling - Authors: Zhong Li, Qi Huang, Yuxuan Zhu, Mohammad Mohammadi Amiri, Niki van Stein, Thomas Bäck, Matthijs van Leeuwen, Zaiwen Wen, Lincen Yang - Venue: arXiv preprint arXiv:2605.12154 - Date: 2026-05-12 - Page: https://www.nikivanstein.nl/publication/2026-05-12-MM-OptBench-A-Solver-Grounded-Benchmark-for-Multimodal-Optimization-Modeling - arXiv: https://arxiv.org/abs/2605.12154 - PDF: https://arxiv.org/pdf/2605.12154 Optimization modeling translates real decision-making problems into mathematical optimization models and solver-executable implementations. Although language models are increasingly used to generate optimization formulations and solver code, existing benchmarks are almost entirely text-only. This omits many optimization-modeling tasks that arise in operational practice, where requirements are described in text but instance information is conveyed through visual artifacts such as tables, graphs, maps, schedules, and dashboards. We introduce multimodal optimization modeling, a benchmark setting in which models must construct both a mathematical formulation and executable solver code from a text-and-visual problem specification. To evaluate this setting, we develop a solver-grounded framework that generates structured optimization instances, verifies each with an exact solver, and builds both the model-facing inputs and hidden reference files from the same verified source. We instantiate the framework as MM-OptBench, a benchmark of 780 solver-verified instances spanning 6 optimization families, 26 subcategories, and 3 structural difficulty levels. We evaluate 9 multimodal large language models (MLLMs), including 6 frontier general-purpose models and 3 math-specialized models, with aggregate, family-level, difficulty-level, and failure-mode analyses. The results show that the task remains far from solved: the best two models reach 52.1% and 51.3% pass@1, while on average across the six general-purpose MLLMs, pass@1 is 43.4% on easy instances and 15.9% on hard instances. All three math-specialized MLLMs solve 0/780 instances. Failure attribution shows that errors arise both when extracting instance data from text and visuals and when turning extracted data into solver-correct formulations and code. MM-OptBench provides a testbed for solver-grounded, decision-oriented multimodal intelligence. ## LLM Driven Design of Continuous Optimization Problems with Controllable High-Level Properties - Authors: Urban Skvorc, Niki van Stein, Moritz Seiler, Britta Grimme, Thomas Bäck, Heike Trautmann - Venue: International Conference on the Applications of Evolutionary Computation (Part of EvoStar) - Date: 2026-04-29 - Page: https://www.nikivanstein.nl/publication/2026-04-29-LLM-Driven-Design-of-Continuous-Optimization-Problems-with-Controllable-High-Level-Properties - DOI: https://doi.org/10.1007/978-3-032-23607-4_12 - arXiv: https://arxiv.org/abs/2601.18846 - PDF: https://arxiv.org/pdf/2601.18846 Benchmarking in continuous black-box optimisation is hindered by the limited structural diversity of existing test suites such as BBOB. We explore whether large language models embedded in an evolutionary loop can be used to design optimisation problems with clearly defined high-level landscape characteristics. Using the LLaMEA framework, we guide an LLM to generate problem code from natural-language descriptions of target properties, including multimodality, separability, basin-size homogeneity, search-space homogeneity and globallocal optima contrast. Inside the loop we score candidates through ELA-based property predictors. We introduce an ELA-space fitness-sharing mechanism that increases population diversity and steers the generator away from redundant landscapes. A complementary basin-of-attraction analysis, statistical testing and visual inspection, verifies that many of the generated functions indeed exhibit the intended structural traits. In addition, a t-SNE embedding shows that they expand the BBOB instance space rather than forming an unrelated cluster. The resulting library provides a broad, interpretable, and reproducible set of benchmark problems for landscape analysis and downstream tasks such as automated algorithm selection. ## Investigating the Interplay of Parameterization and Optimizer in Gradient-Free Topology Optimization: A Cantilever Beam Case Study - Authors: Jelle Westra, Ivan Olarte Rodriguez, Niki van Stein, Thomas Bäck, Elena Raponi - Venue: International Conference on the Applications of Evolutionary Computation (Part of EvoStar) - Date: 2026-04-29 - Page: https://www.nikivanstein.nl/publication/2026-04-29-Investigating-the-Interplay-of-Parameterization-and-Optimizer-in-Gradient-Free-Topology-Optimization-A-Cantilever-Beam-Case-Study - DOI: https://doi.org/10.1007/978-3-032-23607-4_29 - arXiv: https://arxiv.org/abs/2601.22241 - PDF: https://arxiv.org/pdf/2601.22241 Gradient-free black-box optimization (BBO) is widely used in engineering design and provides a flexible framework for topology optimization (TO), enabling the discovery of high-performing structural designs without requiring gradient information from simulations. Yet, its success depends on two key choices: the geometric parameterization defining the search space and the optimizer exploring it. This study investigates this interplay through a compliance minimization problem for a cantilever beam subject to a connectivity constraint. We benchmark three geometric parameterizations, each combined with three representative BBO algorithms: differential evolution, covariance matrix adaptation evolution strategy, and heteroscedastic evolutionary Bayesian optimization, across 10D, 20D, and 50D design spaces. Results reveal that parameterization quality has a stronger influence on optimization performance than optimizer choice: a well-structured parameterization enables robust and competitive performance across algorithms, whereas weaker representations increase optimizer dependency. Overall, this study highlights the dominant role of geometric parameterization in practical BBO-based TO and shows that algorithm performance and selection cannot be fairly assessed without accounting for the induced design space. ## From Performance to Understanding: A Vision for Explainable Automated Algorithm Design - Authors: Niki van Stein, Anna V. Kononova, Thomas Bäck - Venue: International Conference on the Applications of Evolutionary Computation (Part of EvoStar) - Date: 2026-04-29 - Page: https://www.nikivanstein.nl/publication/2026-04-29-From-Performance-to-Understanding-A-Vision-for-Explainable-Automated-Algorithm-Design - DOI: https://doi.org/10.1007/978-3-032-23607-4_21 - arXiv: https://arxiv.org/abs/2511.16201 - PDF: https://arxiv.org/pdf/2511.16201 Automated algorithm design is entering a new phase: Large Language Models can now generate full optimisation (meta)heuristics, explore vast design spaces and adapt through iterative feedback. Yet this rapid progress is largely performance-driven and opaque. Current LLM-based approaches rarely reveal why a generated algorithm works, which components matter or how design choices relate to underlying problem structures. This paper argues that the next breakthrough will come not from more automation, but from coupling automation with understanding from systematic benchmarking. We outline a vision for explainable automated algorithm design, built on three pillars: (i) LLM-driven discovery of algorithmic variants, (ii) explainable benchmarking that attributes performance to components and hyperparameters and (iii) problem-class descriptors that connect algorithm behaviour to landscape structure. Together, these elements form a closed knowledge loop in which discovery, explanation and generalisation reinforce each other. We argue that this integration will shift the field from blind search to interpretable, class-specific algorithm design, accelerating progress while producing reusable scientific insight into when and why optimisation strategies succeed. ## Benchmarking that Matters: Rethinking Benchmarking in Continuous Optimisation for Practical Impact - Authors: Anna V. Kononova, Niki van Stein, Olaf Mersmann, Thomas Bäck, Thomas Bartz-Beielstein, Tobias Glasmachers, Michael Hellwig, Sebastian Krey, Jakub Kůdela, Boris Naujoks, Leonard Papenmeier, Elena Raponi, Quentin Renau, Jeroen Rook, Lennart Schäpermeier, Diederick Vermetten, Daniela Zaharie - Venue: International Conference on the Applications of Evolutionary Computation (Part of EvoStar) - Date: 2026-04-29 - Page: https://www.nikivanstein.nl/publication/2026-04-29-Benchmarking-that-Matters-Rethinking-Benchmarking-in-Continuous-Optimisation-for-Practical-Impact - DOI: https://doi.org/10.1007/978-3-032-23607-4_20 ## Bridging Racial and Age Gaps in Face Recognition: A Data-Augmentation Framework for Fair AI - Authors: Sofoklis Kitharidis, Anthonie Schaap, Thomas Bäck, Niki van Stein - Venue: SN Computer Science - Date: 2026-04-04 - Page: https://www.nikivanstein.nl/publication/2026-04-04-Bridging-Racial-and-Age-Gaps-in-Face-Recognition-A-Data-Augmentation-Framework-for-Fair-AI - DOI: https://doi.org/10.1007/s42979-026-04885-x - PDF: https://link.springer.com/content/pdf/10.1007/s42979-026-04885-x.pdf This work addresses both race- and age-based imbalances in facial recognition, aiming to mitigate disparate error rates across underrepresented demographics. We refine our original framework through iterative data augmentation, combining standard transformations and GAN-based synthesis while introducing an embedding-based filter to discard artifacts or near-duplicates. Fairness is measured using per-subgroup Equalized Odds and F1-scores, comparing incremental retraining strategies, targeting the worst-performing age-ethnicity subgroups, to larger static merges of UTKFace and FairFace. Results show that boosting a single subgroup can yield local gains but often triggers a cyclical “whack-a-mole” pattern, wherein new disparities emerge elsewhere. Although StyleGAN-generated samples can narrow bias for specific subgroups, they sometimes inherit generative limitations and may not stabilize global performance. In contrast, merging balanced datasets or introducing large, diverse synthetic sets leads to more consistent fairness gains, though complete parity remains elusive. Overall, our findings demonstrate that while targeted data injections can alleviate certain imbalances, true equity requires broader measures such as robust data balancing and refined generation techniques. Future work should further explore adversarial debiasing, conditional GAN training, and dynamic reweighting to sustain fairness improvements across varied demographic settings. ## How does chain of thought think? mechanistic interpretability of chain-of-thought reasoning with sparse autoencoding - Authors: Xi Chen, Aske Plaat, Niki van Stein - Venue: Proceedings of the AAAI Conference on Artificial Intelligence - Date: 2026-03-14 - Page: https://www.nikivanstein.nl/publication/2026-03-14-How-does-chain-of-thought-think-mechanistic-interpretability-of-chain-of-thought-reasoning-with-sparse-autoencoding - DOI: https://doi.org/10.1609/aaai.v40i36.40281 - arXiv: https://arxiv.org/abs/2507.22928 - PDF: https://arxiv.org/pdf/2507.22928 Chain-of-thought (CoT) prompting boosts Large Language Models accuracy on multi-step tasks, yet whether the generated "thoughts" reflect the true internal reasoning process is unresolved. We present the first feature-level causal study of CoT faithfulness. Combining sparse autoencoders with activation patching, we extract monosemantic features from Pythia-70M and Pythia-2.8B while they tackle GSM8K math problems under CoT and plain (noCoT) prompting. Swapping a small set of CoT-reasoning features into a noCoT run raises answer log-probabilities significantly in the 2.8B model, but has no reliable effect in 70M, revealing a clear scale threshold. CoT also leads to significantly higher activation sparsity and feature interpretability scores in the larger model, signalling more modular internal computation. For example, the model's confidence in generating correct answers improves from 1.2 to 4.3. We introduce patch-curves and random-feature patching baselines, showing that useful CoT information is not only present in the top-K patches but widely distributed. Overall, our results indicate that CoT can induce more interpretable internal structures in high-capacity LLMs, validating its role as a structured prompting method. ## From heuristic selection to automated algorithm design: Llms benefit from strong priors - Authors: Qi Huang, Furong Ye, Ananta Shahane, Thomas Bäck, Niki van Stein - Venue: arXiv preprint arXiv:2603.02792 - Date: 2026-03-03 - Page: https://www.nikivanstein.nl/publication/2026-03-03-From-heuristic-selection-to-automated-algorithm-design-Llms-benefit-from-strong-priors - arXiv: https://arxiv.org/abs/2603.02792 - PDF: https://arxiv.org/pdf/2603.02792 Large Language Models (LLMs) have already been widely adopted for automated algorithm design, demonstrating strong abilities in generating and evolving algorithms across various fields. Existing work has largely focused on examining their effectiveness in solving specific problems, with search strategies primarily guided by adaptive prompt designs. In this paper, through investigating the token-wise attribution of the prompts to LLM-generated algorithmic codes, we show that providing high-quality algorithmic code examples can substantially improve the performance of the LLM-driven optimization. Building upon this insight, we propose leveraging prior benchmark algorithms to guide LLM-driven optimization and demonstrate superior performance on two black-box optimization benchmarks: the pseudo-Boolean optimization suite (pbo) and the black-box optimization suite (bbob). Our findings highlight the value of integrating benchmarking studies to enhance both efficiency and robustness of the LLM-driven black-box optimization methods. ## Multi-subspace SVD Generators for Continual Learning - Authors: Christiaan Lamers, Ahmed Nabil Belbachir, Thomas Bäck, Niki van Stein - Venue: International Joint Conference on Computational Intelligence - Date: 2026-02-11 - Page: https://www.nikivanstein.nl/publication/2026-02-11-Multi-subspace-SVD-Generators-for-Continual-Learning - DOI: https://doi.org/10.1007/978-3-032-15638-9_42 ## Mechanistic interpretability for transformer-based time series classification - Authors: Matīss Kalnāre, Sofoklis Kitharidis, Thomas Bäck, Niki van Stein - Venue: International Joint Conference on Computational Intelligence - Date: 2026-02-11 - Page: https://www.nikivanstein.nl/publication/2026-02-11-Mechanistic-interpretability-for-transformer-based-time-series-classification - DOI: https://doi.org/10.1007/978-3-032-15638-9_15 - arXiv: https://arxiv.org/abs/2511.21514 - PDF: https://arxiv.org/pdf/2511.21514 Transformer-based models have become state-of-the-art tools in various machine learning tasks, including time series classification, yet their complexity makes understanding their internal decision-making challenging. Existing explainability methods often focus on input-output attributions, leaving the internal mechanisms largely opaque. This paper addresses this gap by adapting various Mechanistic Interpretability techniques; activation patching, attention saliency, and sparse autoencoders, from NLP to transformer architectures designed explicitly for time series classification. We systematically probe the internal causal roles of individual attention heads and timesteps, revealing causal structures within these models. Through experimentation on a benchmark time series dataset, we construct causal graphs illustrating how information propagates internally, highlighting key attention heads and temporal positions driving correct classifications. Additionally, we demonstrate the potential of sparse autoencoders for uncovering interpretable latent features. Our findings provide both methodological contributions to transformer interpretability and novel insights into the functional mechanics underlying transformer performance in time series classification tasks. ## Computational Intelligence: 17th International Joint Conference, IJCCI 2025, Marbella, Spain, October 22–24, 2025, Proceedings, Part III - Authors: Francesco Marcelloni, Kurosh Madani, Niki van Stein, Joaquim Filipe (Eds.) - Venue: Springer Nature - Date: 2026-02-11 - Page: https://www.nikivanstein.nl/publication/2026-02-11-Computational-Intelligence-17th-International-Joint-Conference-IJCCI-2025-Marbella-Spain-October-22-24-2025-Proceedings-Part-III ## Behaviour space analysis of llm-driven meta-heuristic discovery - Authors: Niki van Stein, Haoran Yin, Anna V. Kononova, Thomas Bäck, Gabriela Ochoa - Venue: International Joint Conference on Computational Intelligence - Date: 2026-02-01 - Page: https://www.nikivanstein.nl/publication/2026-02-01-Behaviour-space-analysis-of-llm-driven-meta-heuristic-discovery - DOI: https://doi.org/10.1007/978-3-032-15635-8_23 - arXiv: https://arxiv.org/abs/2507.03605 - PDF: https://arxiv.org/pdf/2507.03605 We investigate the behaviour space of meta-heuristic optimisation algorithms automatically generated by Large Language Model driven algorithm discovery methods. Using the Large Language Evolutionary Algorithm (LLaMEA) framework with a GPT o4-mini LLM, we iteratively evolve black-box optimisation heuristics, evaluated on 10 functions from the BBOB benchmark suite. Six LLaMEA variants, featuring different mutation prompt strategies, are compared and analysed. We log dynamic behavioural metrics including exploration, exploitation, convergence and stagnation measures, for each run, and analyse these via visual projections and network-based representations. Our analysis combines behaviour-based projections, Code Evolution Graphs built from static code features, performance convergence curves, and behaviour-based Search Trajectory Networks. The results reveal clear differences in search dynamics and algorithm structures across LLaMEA configurations. Notably, the variant that employs both a code simplification prompt and a random perturbation prompt in a 1+1 elitist evolution strategy, achieved the best performance, with the highest Area Over the Convergence Curve. Behaviour-space visualisations show that higher-performing algorithms exhibit more intensive exploitation behaviour and faster convergence with less stagnation. Our findings demonstrate how behaviour-space analysis can explain why certain LLM-designed heuristics outperform others and how LLM-driven algorithm discovery navigates the open-ended and complex search space of algorithms. These findings provide insights to guide the future design of adaptive LLM-driven algorithm generators. ## A survey of features used for representing black-box single-objective continuous optimization - Authors: Gjorgjina Cenikj, Ana Nikolikj, Gašper Petelin, Niki van Stein, Carola Doerr, Tome Eftimov - Venue: Swarm and Evolutionary Computation - Date: 2026-02-01 - Page: https://www.nikivanstein.nl/publication/2026-02-01-A-survey-of-features-used-for-representing-black-box-single-objective-continuous-optimization - DOI: https://doi.org/10.1016/j.swevo.2026.102288 - arXiv: https://arxiv.org/abs/2406.06629 - PDF: https://arxiv.org/pdf/2406.06629 This survey examines key advancements in designing features to represent optimization problem instances, algorithm instances, and their interactions within the context of single-objective continuous black-box optimization. These features support machine learning tasks such as algorithm selection, algorithm configuration, and problem classification, and they are also used to evaluate the complementarity of benchmark problem sets. We provide a comprehensive overview of problem landscape features, algorithm features, high-level problem-algorithm interaction features, and trajectory features, including the latest works from the past five years. We also point out limitations of the current state-of-the-art and suggest directions for future research. ## LLaMEA-SAGE: Guiding Automated Algorithm Design with Structural Feedback from Explainable AI - Authors: Niki van Stein, Anna V. Kononova, Lars Kotthoff, Thomas Bäck - Venue: arXiv preprint arXiv:2601.21511 - Date: 2026-01-29 - Page: https://www.nikivanstein.nl/publication/2026-01-29-LLaMEA-SAGE-Guiding-Automated-Algorithm-Design-with-Structural-Feedback-from-Explainable-AI - arXiv: https://arxiv.org/abs/2601.21511 - PDF: https://arxiv.org/pdf/2601.21511 Large language models have enabled automated algorithm design (AAD) by generating optimization algorithms directly from natural-language prompts. While evolutionary frameworks such as LLaMEA demonstrate strong exploratory capabilities across the algorithm design space, their search dynamics are entirely driven by fitness feedback, leaving substantial information about the generated code unused. We propose a mechanism for guiding AAD using feedback constructed from graph-theoretic and complexity features extracted from the abstract syntax trees of the generated algorithms, based on a surrogate model learned over an archive of evaluated solutions. Using explainable AI techniques, we identify features that substantially affect performance and translate them into natural-language mutation instructions that steer subsequent LLM-based code generation without restricting expressivity. We propose LLaMEA-SAGE, which integrates this feature-driven guidance into LLaMEA, and evaluate it across several benchmarks. We show that the proposed structured guidance achieves the same performance faster than vanilla LLaMEA in a small controlled experiment. In a larger-scale experiment using the MA-BBOB suite from the GECCO-MA-BBOB competition, our guided approach achieves superior performance compared to state-of-the-art AAD methods. These results demonstrate that signals derived from code can effectively bias LLM-driven algorithm evolution, bridging the gap between code structure and human-understandable performance feedback in automated algorithm design. ## State Design Matters: How Representations Shape Dynamic Reasoning in Large Language Models - Authors: Annie Wong, Aske Plaat, Thomas Bäck, Niki van Stein, Anna V. Kononova - Venue: arXiv preprint arXiv:2602.15858 - Date: 2026-01-25 - Page: https://www.nikivanstein.nl/publication/2026-01-25-State-Design-Matters-How-Representations-Shape-Dynamic-Reasoning-in-Large-Language-Models - arXiv: https://arxiv.org/abs/2602.15858 - PDF: https://arxiv.org/pdf/2602.15858 As large language models (LLMs) move from static reasoning tasks toward dynamic environments, their success depends on the ability to navigate and respond to an environment that changes as they interact at inference time. An underexplored factor in these settings is the representation of the state. Holding model parameters fixed, we systematically vary three key aspects: (1) state granularity (long form versus summary), (2) structure (natural language versus symbolic), and (3) spatial grounding (text-only versus images or textual map encodings) across sequential decision-making benchmarks. We find that trajectory summarisation improves performance by reducing noise and stabilising long-horizon reasoning. Second, natural language representations are the most robust across models, whereas structured encodings help mainly for models with strong code or structured output priors, such as JSON schemas. Third, while image-inputs show some benefit, text-based spatial encodings prove most effective. This advantage stems not from the spatial information itself, but from the act of construction, which compels the model to perform the spatial reasoning that static input does not elicit. Overall, we demonstrate that design choices for representing state are a decisive factor in performance, distinct from the availability of information itself. We note, however, that even with improved representations, current LLMs and VLMs remain brittle over long horizons, particularly when they must synthesise information to manage multiple subtasks to reach a goal. ## Agentic large language models, a survey - Authors: Aske Plaat, Max van Duijn, Niki van Stein, Mike Preuss, Peter van der Putten, Kees Joost Batenburg - Venue: Journal of Artificial Intelligence Research - Date: 2025-12-30 - Page: https://www.nikivanstein.nl/publication/2025-12-30-Agentic-large-language-models-a-survey - arXiv: https://arxiv.org/abs/2503.23037 - PDF: https://arxiv.org/pdf/2503.23037 Background: There is great interest in agentic LLMs, large language models that act as agents. Objectives: We review the growing body of work in this area and provide a research agenda. Methods: Agentic LLMs are LLMs that (1) reason, (2) act, and (3) interact. We organize the literature according to these three categories. Results: The research in the first category focuses on reasoning, reflection, and retrieval, aiming to improve decision making; the second category focuses on action models, robots, and tools, aiming for agents that act as useful assistants; the third category focuses on multi-agent systems, aiming for collaborative task solving and simulating interaction to study emergent social behavior. We find that works mutually benefit from results in other categories: retrieval enables tool use, reflection improves multi-agent collaboration, and reasoning benefits all categories. Conclusions: We discuss applications of agentic LLMs and provide an agenda for further research. Important applications are in medical diagnosis, logistics and financial market analysis. Meanwhile, self-reflective agents playing roles and interacting with one another augment the process of scientific research itself. Further, agentic LLMs provide a solution for the problem of LLMs running out of training data: inference-time behavior generates new training states, such that LLMs can keep learning without needing ever larger datasets. We note that there is risk associated with LLM assistants taking action in the real world-safety, liability and security are open problems-while agentic LLMs are also likely to benefit society. ## Multi-step reasoning with large language models, a survey - Authors: Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, Thomas Bäck - Venue: ACM Computing Surveys - Date: 2025-12-09 - Page: https://www.nikivanstein.nl/publication/2025-12-09-Multi-step-reasoning-with-large-language-models-a-survey - DOI: https://doi.org/10.1145/3774896 - arXiv: https://arxiv.org/abs/2407.11511 - PDF: https://arxiv.org/pdf/2407.11511 Large language models (LLMs) with billions of parameters exhibit in-context learning abilities, enabling few-shot learning on tasks that the model was not specifically trained for. Traditional models achieve breakthrough performance on language tasks, but do not perform well on basic reasoning benchmarks. However, a new in-context learning approach, Chain-of-thought, has demonstrated strong multi-step reasoning abilities on these benchmarks. The research on LLM reasoning abilities started with the question whether LLMs can solve grade school math word problems, and has expanded to other tasks in the past few years. This article reviews the field of multi-step reasoning with LLMs. We propose a taxonomy that identifies different ways to generate, evaluate, and control multi-step reasoning. We provide an in-depth coverage of core approaches and open problems, and we propose a research agenda for the near future. We find that multi-step reasoning approaches have progressed beyond math word problems, and can now successfully solve challenges in logic, combinatorial games, and robotics, sometimes by first generating code that is then executed by external tools. Many studies in multi-step methods use reinforcement learning for finetuning, external optimization loops, in-context reinforcement learning, and self-reflection. ## Gradient Free Multi-Objective Counterfactual Explainability for Multivariate Time Series Classification - Authors: Qi Huang, Sofoklis Kitharidis, Furong Ye, Marius Ottolini, Thomas Bäck, Niki van Stein - Venue: 2025 IEEE International Conference on Big Data (BigData) - Date: 2025-12-08 - Page: https://www.nikivanstein.nl/publication/2025-12-08-Gradient-Free-Multi-Objective-Counterfactual-Explainability-for-Multivariate-Time-Series-Classification - DOI: https://doi.org/10.1109/bigdata66926.2025.11400892 - PDF: https://scholarlypublications.universiteitleiden.nl/download/1887/4306668/1?col=81522199340002711 ## Reasoning capabilities of large language models on dynamic tasks - Authors: Annie Wong, Thomas Bäck, Aske Plaat, Niki van Stein, Anna V. Kononova - Venue: 2025 3rd International Conference on Foundation and Large Language Models (FLLM) - Date: 2025-11-25 - Page: https://www.nikivanstein.nl/publication/2025-11-25-Reasoning-capabilities-of-large-language-models-on-dynamic-tasks - DOI: https://doi.org/10.1109/fllm67465.2025.11391251 - arXiv: https://arxiv.org/abs/2505.10543 - PDF: https://arxiv.org/pdf/2505.10543 Large language models excel on static benchmarks, but their ability as self-learning agents in dynamic environments remains unclear. We evaluate three prompting strategies: self-reflection, heuristic mutation, and planning across dynamic tasks with open-source models. We find that larger models generally outperform smaller ones, but that strategic prompting can close this performance gap. Second, an overly long prompt can negatively impact smaller models on basic reactive tasks, while larger models show more robust behaviour. Third, advanced prompting techniques primarily benefit smaller models on complex games, but offer less improvement for already high-performing large language models. Yet, we find that advanced reasoning methods yield highly variable outcomes: while capable of significantly improving performance when reasoning and decision-making align, they also introduce instability and can lead to big performance drops. Compared to human performance, our findings reveal little evidence of true emergent reasoning. Instead, large language model performance exhibits persistent limitations in areas like planning and spatial coordination, suggesting that large language models still suffer fundamental shortcomings that may not be fully overcome through self-reflective prompting alone. Reasoning is a multi-faceted task, and while methods like Chain-of-thought improve multi-step reasoning on math word problems, our findings using dynamic benchmarks highlight important shortcomings in general reasoning capabilities, indicating a need to move beyond static benchmarks to capture the complexity of reasoning. ## REMAINING USEFUL LIFE IN COMPLEX MULTI-COMPONENT SYSTEMS: TAXONOMY, REVIEW, AND RESEARCH DIRECTIONS - Authors: Bernd Wagner, Qi Huang, Lucas Correia, Thomas Bäck, Anna Kononova, Niki van Stein - Venue: Authorea Preprints - Date: 2025-11-11 - Page: https://www.nikivanstein.nl/publication/2025-11-11-REMAINING-USEFUL-LIFE-IN-COMPLEX-MULTI-COMPONENT-SYSTEMS-TAXONOMY-REVIEW-AND-RESEARCH-DIRECTIONS - DOI: https://doi.org/10.36227/techrxiv.176288069.90065228/v1 - PDF: https://www.techrxiv.org/doi/pdf/10.36227/techrxiv.176288069.90065228/v1 Current research into remaining useful life (RUL) has a strong emphasis on single-component or simple multi-component systems. As such, it often fails to account for the interactions between components and systems, as well as the impact of high fluctuations in operating conditions. The existing body of literature on RUL is therefore only marginally applicable to researchers and asset managers dealing with complex, multi-component systems, such as cars, trains, or ships. At the same time, the substantial costs involved in acquiring and maintaining these complex systems make optimising maintenance management through a condition-based approach a potentially valuable avenue for improving life-cycle performance. In this paper, we firstly define and clarify the distinction between simple and complex multi-component systems. Secondly, we explore the consequences of this distinction for RUL estimation. Next, we review existing literature on RUL in the context of complex systems, and finally, we distil opportunities and gaps for further research into such systems. Our goal is to provide a reference point for more comprehensive research and better application development in the domain of RUL calculations for complex systems. ## EvoCAD: Evolutionary CAD Code Generation with Vision Language Models - Authors: Tobias Preintner, Weixuan Yuan, Adrian König, Thomas Bäck, Elena Raponi, Niki van Stein - Venue: 2025 IEEE 37th International Conference on Tools with Artificial Intelligence (ICTAI) - Date: 2025-11-03 - Page: https://www.nikivanstein.nl/publication/2025-11-03-EvoCAD-Evolutionary-CAD-Code-Generation-with-Vision-Language-Models - DOI: https://doi.org/10.1109/ictai66417.2025.00075 - arXiv: https://arxiv.org/abs/2510.11631 - PDF: https://arxiv.org/pdf/2510.11631 Combining large language models with evolutionary computation algorithms represents a promising research direction leveraging the remarkable generative and in-context learning capabilities of LLMs with the strengths of evolutionary algorithms. In this work, we present EvoCAD, a method for generating computer-aided design (CAD) objects through their symbolic representations using vision language models and evolutionary optimization. Our method samples multiple CAD objects, which are then optimized using an evolutionary approach with vision language and reasoning language models. We assess our method using GPT-4V and GPT-4o, evaluating it on the CADPrompt benchmark dataset and comparing it to prior methods. Additionally, we introduce two new metrics based on topological properties defined by the Euler characteristic, which capture a form of semantic similarity between 3D objects. Our results demonstrate that EvoCAD outperforms previous approaches on multiple metrics, particularly in generating topologically correct objects, which can be efficiently evaluated using our two novel metrics that complement existing spatial metrics. ## Visual Model Selection using Feature Importance Clusters in Fairness-Performance Similarity Optimized Space - Authors: Sofoklis Kitharidis, Cor J. Veenman, Thomas Bäck, Niki van Stein - Venue: arXiv preprint arXiv:2510.22209 - Date: 2025-10-25 - Page: https://www.nikivanstein.nl/publication/2025-10-25-Visual-Model-Selection-using-Feature-Importance-Clusters-in-Fairness-Performance-Similarity-Optimized-Space - arXiv: https://arxiv.org/abs/2510.22209 - PDF: https://arxiv.org/pdf/2510.22209 In the context of algorithmic decision-making, fair machine learning methods often yield multiple models that balance predictive fairness and performance in varying degrees. This diversity introduces a challenge for stakeholders who must select a model that aligns with their specific requirements and values. To address this, we propose an interactive framework that assists in navigating and interpreting the trade-offs across a portfolio of models. Our approach leverages weakly supervised metric learning to learn a Mahalanobis distance that reflects similarity in fairness and performance outcomes, effectively structuring the feature importance space of the models according to stakeholder-relevant criteria. We then apply clustering technique (k-means) to group models based on their transformed representations of feature importances, allowing users to explore clusters of models with similar predictive behaviors and fairness characteristics. This facilitates informed decision-making by helping users understand how models differ not only in their fairness-performance balance but also in the features that drive their predictions. ## Scalable, Explainable and Provably Robust Anomaly Detection with One-Step Flow Matching - Authors: Zhong Li, Qi Huang, Yuxuan Zhu, Lincen Yang, Mohammad Mohammadi Amiri, Niki van Stein, Matthijs van Leeuwen - Venue: Advances in Neural Information Processing Systems - Date: 2025-10-21 - Page: https://www.nikivanstein.nl/publication/2025-10-21-Scalable-Explainable-and-Provably-Robust-Anomaly-Detection-with-One-Step-Flow-Matching - DOI: https://doi.org/10.52202/085713-2936 - arXiv: https://arxiv.org/abs/2510.18328 - PDF: https://arxiv.org/pdf/2510.18328 We introduce Time-Conditioned Contraction Matching (TCCM), a novel method for semi-supervised anomaly detection in tabular data. TCCM is inspired by flow matching, a recent generative modeling framework that learns velocity fields between probability distributions and has shown strong performance compared to diffusion models and generative adversarial networks. Instead of directly applying flow matching as originally formulated, TCCM builds on its core idea -- learning velocity fields between distributions -- but simplifies the framework by predicting a time-conditioned contraction vector toward a fixed target (the origin) at each sampled time step. This design offers three key advantages: (1) a lightweight and scalable training objective that removes the need for solving ordinary differential equations during training and inference; (2) an efficient scoring strategy called one time-step deviation, which quantifies deviation from expected contraction behavior in a single forward pass, addressing the inference bottleneck of existing continuous-time models such as DTE (a diffusion-based model with leading anomaly detection accuracy but heavy inference cost); and (3) explainability and provable robustness, as the learned velocity field operates directly in input space, making the anomaly score inherently feature-wise attributable; moreover, the score function is Lipschitz-continuous with respect to the input, providing theoretical guarantees under small perturbations. Extensive experiments on the ADBench benchmark show that TCCM strikes a favorable balance between detection accuracy and inference cost, outperforming state-of-the-art methods -- especially on high-dimensional and large-scale datasets. The source code is available at our GitHub repository. ## Automated Algorithm Design for Auto-Tuning Optimizers - Authors: Floris-Jan Willemsen, Niki van Stein, Ben van Werkhoven - Venue: Proceedings of Machine Learning and Systems - Date: 2025-10-19 - Page: https://www.nikivanstein.nl/publication/2025-10-19-Automated-Algorithm-Design-for-Auto-Tuning-Optimizers - arXiv: https://arxiv.org/abs/2510.17899 - PDF: https://arxiv.org/pdf/2510.17899 Automatic performance tuning (auto-tuning) is essential for optimizing high-performance applications, where vast and irregular search spaces make manual exploration infeasible. While auto-tuners traditionally rely on classical approaches such as evolutionary, annealing, or surrogate-based optimizers, designing algorithms that efficiently find near-optimal configurations robustly across diverse tasks is challenging. We propose a new paradigm: using large language models (LLMs) to automatically generate optimization algorithms tailored to auto-tuning problems. We introduce a framework that prompts LLMs with problem descriptions and search space characteristics to synthesize, test, and iteratively refine specialized optimizers. These generated algorithms are evaluated on four real-world auto-tuning applications across six hardware platforms and compared against the state-of-the-art in two contemporary auto-tuning frameworks. The evaluation demonstrates that providing additional application- and search space-specific information in the generation stage results in an average performance improvement of 30.7% and 14.6%, respectively. In addition, our results show that LLM-generated optimizers can rival, and in various cases outperform, existing human-designed algorithms, with our best-performing generated optimization algorithms achieving an average 72.4% improvement over state-of-the-art optimizers for auto-tuning. ## Leveraging Lightweight Generators for Memory Efficient Continual Learning - Authors: Christiaan Lamers, Ahmed Nabil Belbachir, Thomas Bäck, Niki van Stein - Venue: Intelligent Systems Conference - Date: 2025-09-03 - Page: https://www.nikivanstein.nl/publication/2025-09-03-Leveraging-Lightweight-Generators-for-Memory-Efficient-Continual-Learning - DOI: https://doi.org/10.1007/978-3-031-99958-1_21 - arXiv: https://arxiv.org/abs/2506.19692 - PDF: https://arxiv.org/pdf/2506.19692 Catastrophic forgetting can be trivially alleviated by keeping all data from previous tasks in memory. Therefore, minimizing the memory footprint while maximizing the amount of relevant information is crucial to the challenge of continual learning. This paper aims to decrease required memory for memory-based continuous learning algorithms. We explore the options of extracting a minimal amount of information, while maximally alleviating forgetting. We propose the usage of lightweight generators based on Singular Value Decomposition to enhance existing continual learning methods, such as A-GEM and Experience Replay. These generators need a minimal amount of memory while being maximally effective. They require no training time, just a single linear-time fitting step, and can capture a distribution effectively from a small number of data samples. Depending on the dataset and network architecture, our results show a significant increase in average accuracy compared to the original methods. Our method shows great potential in minimizing the memory footprint of memory-based continual learning algorithms. ## CoComposer: LLM Multi-agent Collaborative Music Composition - Authors: Peiwen Xing, Aske Plaat, Niki van Stein - Venue: arXiv preprint arXiv:2509.00132 - Date: 2025-08-29 - Page: https://www.nikivanstein.nl/publication/2025-08-29-CoComposer-LLM-Multi-agent-Collaborative-Music-Composition - arXiv: https://arxiv.org/abs/2509.00132 - PDF: https://arxiv.org/pdf/2509.00132 Existing AI Music composition tools are limited in generation duration, musical quality, and controllability. We introduce CoComposer, a multi-agent system that consists of five collaborating agents, each with a task based on the traditional music composition workflow. Using the AudioBox-Aesthetics system, we experimentally evaluate CoComposer on four compositional criteria. We test with three LLMs (GPT-4o, DeepSeek-V3-0324, Gemini-2.5-Flash), and find (1) that CoComposer outperforms existing multi-agent LLM-based systems in music quality, and (2) compared to a single-agent system, in production complexity. Compared to non- LLM MusicLM, CoComposer has better interpretability and editability, although MusicLM still produces better music. ## Quality--diversity-driven robust evolutionary optimization of optical designs - Authors: Kirill Antonov, Teus Tukker, Tiago Botari, Thomas Bäck, Anna V. Kononova, Niki van Stein - Venue: Optical Engineering - Date: 2025-07-31 - Page: https://www.nikivanstein.nl/publication/2025-07-31-Quality-diversity-driven-robust-evolutionary-optimization-of-optical-designs - DOI: https://doi.org/10.1117/1.oe.64.7.075106 Developing optical systems, particularly those consisting of spherical lenses, is relevant for various applications such as lithographic scanners and metrology equipment. The design process of an optical system typically involves the optimization of specific objectives to ensure the best performance. As a common example of such an objective, we consider the problem of determining the lens curvatures that result in a sufficiently small root mean square spot size. Optimization algorithms are commonly employed to solve this problem by heuristically eliminating suboptimal optical designs. This class of algorithms includes the damped least squares widely applied in commercial software and advanced methods such as saddle-point construction. However, within a restricted computational budget, these optimizers are limited in exploring potentially promising novel solutions because they heavily rely on the initial specific designs that must conform to complex or unknown requirements. We address the considered problem with a modified Hill Valley Evolutionary Algorithm, which has proved itself as one of the best state-of-the-art metaheuristics for multimodal black-box optimization. We demonstrate that our algorithm locates a diverse set of high-quality optical designs with four lenses in a single run, even when initialized with random starting curvatures. This is the first result in this domain when an optimization algorithm that does not take specific optical properties into account can still generate relevant and high-performing optical systems. Furthermore, we show the benefits of the proposed methodology for the diversity of the obtained set of solutions while maintaining a solution of the same quality as the one found by the most prominent algorithm in the domain. We provide analyses of the obtained solutions according to: (1) tolerance to the alignment of lenses and (2) susceptibility to small variations of lens curvatures. ## Development of a photonic crystal spectrometer for greenhouse gas measurements - Authors: Marijn Siemons, Martijn Veen, Irina Malysheva, Johannes Algera, Stefan Philippi, Kirill Antonov, Niki van Stein, Jérôme Loicq, Nandini Bhattacharya, Rene Berlich, Anna V. Kononova, Ralf Kohlhaas - Venue: International Conference on Space Optics—ICSO 2024 - Date: 2025-07-28 - Page: https://www.nikivanstein.nl/publication/2025-07-28-Development-of-a-photonic-crystal-spectrometer-for-greenhouse-gas-measurements - DOI: https://doi.org/10.1117/12.3071597 - arXiv: https://arxiv.org/abs/2411.02056 - PDF: https://arxiv.org/pdf/2411.02056 The need of atmospheric information with a higher spatial and temporal resolution drives the development of small satellites and satellite constellations to complement satellite flagship missions. Since optical systems are a main contributor to the satellite size, these are the prime candidate for their miniaturization. We present here a novel optical system where the complete spectrometer part of the optical system is compressed in one flat optical element. The element consists of an array of photonic crystals which is directly placed on a detector. The photonic crystals act as optical filters with a tunable spectral transmission response. From the integrated optical signals per filter and the atmosphere model, greenhouse gas concentrations are obtained using computational inversion. We present in this article the instrument concept, the manufacturing and measurement of the photonic crystals, methods for the filter array optimization, and discuss the predicted retrieval performance for the detection of methane and carbon dioxide. ## Structural bias in optimization algorithms - Authors: Anna V. Kononova, Niki van Stein - Venue: Proceedings of the Genetic and Evolutionary Computation Conference Companion - Date: 2025-07-14 - Page: https://www.nikivanstein.nl/publication/2025-07-14-Structural-bias-in-optimization-algorithms - DOI: https://doi.org/10.1145/3712255.3716527 ## Optimizing photonic structures with large language model driven algorithm discovery - Authors: Haoran Yin, Anna V. Kononova, Thomas Bäck, Niki van Stein - Venue: Proceedings of the Genetic and Evolutionary Computation Conference Companion - Date: 2025-07-14 - Page: https://www.nikivanstein.nl/publication/2025-07-14-Optimizing-photonic-structures-with-large-language-model-driven-algorithm-discovery - DOI: https://doi.org/10.1145/3712255.3734288 - arXiv: https://arxiv.org/abs/2503.19742 - PDF: https://arxiv.org/pdf/2503.19742 We study how large language models can be used in combination with evolutionary computation techniques to automatically discover optimization algorithms for the design of photonic structures. Building on the Large Language Model Evolutionary Algorithm (LLaMEA) framework, we introduce structured prompt engineering tailored to multilayer photonic problems such as Bragg mirror, ellipsometry inverse analysis, and solar cell antireflection coatings. We systematically explore multiple evolutionary strategies, including (1+1), (1+5), (2+10), and others, to balance exploration and exploitation. Our experiments show that LLM-generated algorithms, generated using small-scale problem instances, can match or surpass established methods like quasi-oppositional differential evolution on large-scale realistic real-world problem instances. Notably, LLaMEA's self-debugging mutation loop, augmented by automatically extracted problem-specific insights, achieves strong anytime performance and reliable convergence across diverse problem scales. This work demonstrates the feasibility of domain-focused LLM prompts and evolutionary approaches in solving optical design tasks, paving the way for rapid, automated photonic inverse design. ## Neighborhood Adaptive Differential Evolution - Authors: Niki van Stein - Venue: Proceedings of the Genetic and Evolutionary Computation Conference Companion - Date: 2025-07-14 - Page: https://www.nikivanstein.nl/publication/2025-07-14-Neighborhood-Adaptive-Differential-Evolution - DOI: https://doi.org/10.1145/3712255.3735092 - PDF: https://scholarlypublications.universiteitleiden.nl/download/1887/4294834/1?col=81522199340002711 We introduce Neighborhood Adaptive DE, an automatically designed variant of Differential Evolution (DE) tuned specifically for Many Affine Black-Box Optimization benchmark functions. The algorithm was generated by the Large Language Model Evolutionary Algorithm LLaMEA approach, which employed a (4,12) Evolution Strategy in combination with the large language model Gemini-2.0-Flash to generate new optimization algorithms. Key features of Neighborhood Adaptive DE include sampling a small, randomly chosen subset of 'neighbors' for each individual and using the best among those as the base vector in mutation (instead of the global best). This neighborhood-based strategy, along with adaptive tuning of scale factor F and crossover rate CR, leads to greater diversity and mitigates premature convergence. Additionally, an automatic restart mechanism re-initializes the population upon detecting stagnation. Experimental benchmarking indicates that Neighborhood Adaptive DE outperforms other algorithms on the benchmark, including the winner of last-years Many Affine competition, suggesting that automated design with LLaMEA can yield novel, high-performance algorithms for challenging black-box optimization tasks. ## BLADE: Benchmark suite for LLM-driven Automated Design and Evolution of iterative optimisation heuristics - Authors: Niki van Stein, Anna V. Kononova, Haoran Yin, Thomas Bäck - Venue: Proceedings of the Genetic and Evolutionary Computation Conference Companion - Date: 2025-07-14 - Page: https://www.nikivanstein.nl/publication/2025-07-14-BLADE-Benchmark-suite-for-LLM-driven-Automated-Design-and-Evolution-of-iterative-optimisation-heuristics - DOI: https://doi.org/10.1145/3712255.3734347 - arXiv: https://arxiv.org/abs/2504.20183 - PDF: https://arxiv.org/pdf/2504.20183 The application of Large Language Models (LLMs) for Automated Algorithm Discovery (AAD), particularly for optimisation heuristics, is an emerging field of research. This emergence necessitates robust, standardised benchmarking practices to rigorously evaluate the capabilities and limitations of LLM-driven AAD methods and the resulting generated algorithms, especially given the opacity of their design process and known issues with existing benchmarks. To address this need, we introduce BLADE (Benchmark suite for LLM-driven Automated Design and Evolution), a modular and extensible framework specifically designed for benchmarking LLM-driven AAD methods in a continuous black-box optimisation context. BLADE integrates collections of benchmark problems (including MA-BBOB and SBOX-COST among others) with instance generators and textual descriptions aimed at capability-focused testing, such as generalisation, specialisation and information exploitation. It offers flexible experimental setup options, standardised logging for reproducibility and fair comparison, incorporates methods for analysing the AAD process (e.g., Code Evolution Graphs and various visualisation approaches) and facilitates comparison against human-designed baselines through integration with established tools like IOHanalyser and IOHexplainer. BLADE provides an `out-of-the-box' solution to systematically evaluate LLM-driven AAD approaches. The framework is demonstrated through two distinct use cases exploring mutation prompt strategies and function specialisation. ## Code evolution graphs: Understanding large language model driven design of algorithms - Authors: Niki van Stein, Anna V. Kononova, Lars Kotthoff, Thomas Bäck - Venue: Proceedings of the Genetic and Evolutionary Computation Conference - Date: 2025-07-13 - Page: https://www.nikivanstein.nl/publication/2025-07-13-Code-evolution-graphs-Understanding-large-language-model-driven-design-of-algorithms - DOI: https://doi.org/10.1145/3712256.3726328 - arXiv: https://arxiv.org/abs/2503.16668 - PDF: https://arxiv.org/pdf/2503.16668 Large Language Models (LLMs) have demonstrated great promise in generating code, especially when used inside an evolutionary computation framework to iteratively optimize the generated algorithms. However, in some cases they fail to generate competitive algorithms or the code optimization stalls, and we are left with no recourse because of a lack of understanding of the generation process and generated codes. We present a novel approach to mitigate this problem by enabling users to analyze the generated codes inside the evolutionary process and how they evolve over repeated prompting of the LLM. We show results for three benchmark problem classes and demonstrate novel insights. In particular, LLMs tend to generate more complex code with repeated prompting, but additional complexity can hurt algorithmic performance in some cases. Different LLMs have different coding ``styles'' and generated code tends to be dissimilar to other LLMs. These two findings suggest that using different LLMs inside the code evolution frameworks might produce higher performing code than using only one LLM. ## Why Are You Wrong? Counterfactual Explanations for Language Grounding with 3D Objects - Authors: Tobias Preintner, Weixuan Yuan, Qi Huang, Adrian König, Thomas Bäck, Elena Raponi, Niki van Stein - Venue: 2025 International Joint Conference on Neural Networks (IJCNN) - Date: 2025-06-30 - Page: https://www.nikivanstein.nl/publication/2025-06-30-Why-Are-You-Wrong-Counterfactual-Explanations-for-Language-Grounding-with-3D-Objects - DOI: https://doi.org/10.1109/ijcnn64981.2025.11227256 - arXiv: https://arxiv.org/abs/2505.06030 - PDF: https://arxiv.org/pdf/2505.06030 Combining natural language and geometric shapes is an emerging research area with multiple applications in robotics and language-assisted design. A crucial task in this domain is object referent identification, which involves selecting a 3D object given a textual description of the target. Variability in language descriptions and spatial relationships of 3D objects makes this a complex task, increasing the need to better understand the behavior of neural network models in this domain. However, limited research has been conducted in this area. Specifically, when a model makes an incorrect prediction despite being provided with a seemingly correct object description, practitioners are left wondering: "Why is the model wrong?". In this work, we present a method answering this question by generating counterfactual examples. Our method takes a misclassified sample, which includes two objects and a text description, and generates an alternative yet similar formulation that would have resulted in a correct prediction by the model. We have evaluated our approach with data from the ShapeTalk dataset along with three distinct models. Our counterfactual examples maintain the structure of the original description, are semantically similar and meaningful. They reveal weaknesses in the description, model bias and enhance the understanding of the models behavior. Theses insights help practitioners to better interact with systems as well as engineers to improve models. ## XAI for Benchmarking Black-Box Metaheuristics - Authors: Anna V. Kononova, Diederick Vermetten, Niki van Stein - Venue: Explainable AI for Evolutionary Computation - Date: 2025-05-03 - Page: https://www.nikivanstein.nl/publication/2025-05-03-XAI-for-Benchmarking-Black-Box-Metaheuristics - DOI: https://doi.org/10.1007/978-981-96-2540-6_5 - PDF: https://scholarlypublications.universiteitleiden.nl/download/1887/4291381/1?col=81522199340002711 ## The Synergy of Explainable AI and Evolutionary Computation: Real-World Applications - Authors: Niki van Stein, Qi Huang, Elena Raponi - Venue: Explainable AI for Evolutionary Computation - Date: 2025-05-03 - Page: https://www.nikivanstein.nl/publication/2025-05-03-The-Synergy-of-Explainable-AI-and-Evolutionary-Computation-Real-World-Applications - DOI: https://doi.org/10.1007/978-981-96-2540-6_8 ## Global Sensitivity Analysis Is Not Always Beneficial for Evolutionary Computation: A Study in Engineering Design - Authors: Elena Raponi, Ivan Olarte Rodriguez, Niki van Stein - Venue: Explainable AI for Evolutionary Computation - Date: 2025-05-03 - Page: https://www.nikivanstein.nl/publication/2025-05-03-Global-Sensitivity-Analysis-Is-Not-Always-Beneficial-for-Evolutionary-Computation-A-Study-in-Engineering-Design - DOI: https://doi.org/10.1007/978-981-96-2540-6_2 - PDF: https://scholarlypublications.universiteitleiden.nl/download/1887/4294891/1?col=81522199340002711 ## Explainable AI for Evolutionary Computation - Authors: Niki van Stein, Anna V Kononova (Eds.) - Venue: Springer - Date: 2025-05-03 - Page: https://www.nikivanstein.nl/publication/2025-05-03-Explainable-AI-for-Evolutionary-Computation - DOI: https://doi.org/10.1007/978-981-96-2540-6 ## An Introduction to the Crossroads of Explainable Artificial Intelligence and Evolutionary Computation - Authors: Niki van Stein, Anna V. Kononova - Venue: Explainable AI for Evolutionary Computation - Date: 2025-05-03 - Page: https://www.nikivanstein.nl/publication/2025-05-03-An-Introduction-to-the-Crossroads-of-Explainable-Artificial-Intelligence-and-Evolutionary-Computation - DOI: https://doi.org/10.1007/978-981-96-2540-6_1 ## In-the-loop hyper-parameter optimization for llm-based automated design of heuristics - Authors: Niki van Stein, Diederick Vermetten, Thomas Bäck - Venue: ACM Transactions on Evolutionary Learning - Date: 2025-04-23 - Page: https://www.nikivanstein.nl/publication/2025-04-23-In-the-loop-hyper-parameter-optimization-for-llm-based-automated-design-of-heuristics - DOI: https://doi.org/10.1145/3731567 - arXiv: https://arxiv.org/abs/2410.16309 - PDF: https://arxiv.org/pdf/2410.16309 Large Language Models (LLMs) have shown great potential in automatically generating and optimizing (meta)heuristics, making them valuable tools in heuristic optimization tasks. However, LLMs are generally inefficient when it comes to fine-tuning hyper-parameters of the generated algorithms, often requiring excessive queries that lead to high computational and financial costs. This paper presents a novel hybrid approach, LLaMEA-HPO, which integrates the open source LLaMEA (Large Language Model Evolutionary Algorithm) framework with a Hyper-Parameter Optimization (HPO) procedure in the loop. By offloading hyper-parameter tuning to an HPO procedure, the LLaMEA-HPO framework allows the LLM to focus on generating novel algorithmic structures, reducing the number of required LLM queries and improving the overall efficiency of the optimization process. We empirically validate the proposed hybrid framework on benchmark problems, including Online Bin Packing, Black-Box Optimization, and the Traveling Salesperson Problem. Our results demonstrate that LLaMEA-HPO achieves superior or comparable performance compared to existing LLM-driven frameworks while significantly reducing computational costs. This work highlights the importance of separating algorithmic innovation and structural code search from parameter tuning in LLM-driven code optimization and offers a scalable approach to improve the efficiency and effectiveness of LLM-based code generation. ## Surrogate-based automated hyperparameter optimization for expensive automotive crashworthiness optimization - Authors: Fu Xing Long, Niki van Stein, Moritz Frenzel, Peter Krause, Markus Gitterle, Thomas Bäck - Venue: Structural and Multidisciplinary Optimization - Date: 2025-04-18 - Page: https://www.nikivanstein.nl/publication/2025-04-18-Surrogate-based-automated-hyperparameter-optimization-for-expensive-automotive-crashworthiness-optimization - DOI: https://doi.org/10.1007/s00158-025-03989-x - PDF: https://link.springer.com/content/pdf/10.1007/s00158-025-03989-x.pdf In the automotive industry, solving crashworthiness optimization problems efficiently is crucial to minimize time and cost investment on expensive function evaluations, e.g., using simulation runs. Nonetheless, automotive crashworthiness optimization is time-consuming and challenging even with domain knowledge, due to the fact that crash problems are typically high-dimensional, nonlinear, and discontinuous. In this work, we propose an automated hyperparameter optimization (HPO) approach for expensive black-box optimization (BBO) problems that can assist practitioners to solve automotive crash problems efficiently using optimally configured optimization algorithms. Precisely, the landscape characteristics of BBO problems, e.g., quantified using exploratory landscape analysis (ELA), are analyzed to identify cheap-to-evaluate representative functions that belong to the same optimization problem class. Based on these representative functions, algorithm configurations can be optimally fine-tuned at a relatively low computational cost. Using three optimization algorithms, consisting of modular covariance matrix adaptation evolutionary strategy (CMA-ES), modular differential evolution (DE), and Bayesian optimization (BO), we evaluate the potential of our approach based on the black-box optimization benchmarking (BBOB) suite and an automotive side crash problem. Since the optimal configurations identified using our approach can perform well on most of the BBOB functions, we believe that our approach can generalize well to BBO problems with similar optimization complexity. For the automotive side crash problem, the BO configuration fine-tuned using our approach can outperform the default BO configuration as well as the conventional response surface method (RSM), in terms of the best-found-solution and convergence speed. Furthermore, better solutions can be identified using the proposed approach compared to successive RSM (SRSM), when dealing with complex crash functions and a limited function evaluation budget. With appropriate extensions, we are confident that our approach can be applied to other real-world expensive BBO domains beyond automotive crashworthiness optimization. ## Stalling in Space: Attractor Analysis for any Algorithm - Authors: Sarah L. Thomson, Quentin Renau, Diederick Vermetten, Emma Hart, Niki van Stein, Anna V. Kononova - Venue: International Conference on the Applications of Evolutionary Computation (Part of EvoStar) - Date: 2025-04-17 - Page: https://www.nikivanstein.nl/publication/2025-04-17-Stalling-in-Space-Attractor-Analysis-for-any-Algorithm - DOI: https://doi.org/10.1007/978-3-031-90062-4_32 - arXiv: https://arxiv.org/abs/2412.15848 - PDF: https://arxiv.org/pdf/2412.15848 Network-based representations of fitness landscapes have grown in popularity in the past decade; this is probably because of growing interest in explainability for optimisation algorithms. Local optima networks (LONs) have been especially dominant in the literature and capture an approximation of local optima and their connectivity in the landscape. However, thus far, LONs have been constructed according to a strict definition of what a local optimum is: the result of local search. Many evolutionary approaches do not include this, however. Popular algorithms such as CMA-ES have therefore never been subject to LON analysis. Search trajectory networks (STNs) offer a possible alternative: nodes can be any search space location. However, STNs are not typically modelled in such a way that models temporal stalls: that is, a region in the search space where an algorithm fails to find a better solution over a defined period of time. In this work, we approach this by systematically analysing a special case of STN which we name attractor networks. These offer a coarse-grained view of algorithm behaviour with a singular focus on stall locations. We construct attractor networks for CMA-ES, differential evolution, and random search for 24 noiseless black-box optimisation benchmark problems. The properties of attractor networks are systematically explored. They are also visualised and compared to traditional LONs and STN models. We find that attractor networks facilitate insights into algorithm behaviour which other models cannot, and we advocate for the consideration of attractor analysis even for algorithms which do not include local search. ## Controlling the mutation in large language models for the efficient evolution of algorithms - Authors: Haoran Yin, Anna V. Kononova, Thomas Bäck, Niki van Stein - Venue: International Conference on the Applications of Evolutionary Computation (Part of EvoStar) - Date: 2025-04-17 - Page: https://www.nikivanstein.nl/publication/2025-04-17-Controlling-the-mutation-in-large-language-models-for-the-efficient-evolution-of-algorithms - DOI: https://doi.org/10.1007/978-3-031-90065-5_25 - arXiv: https://arxiv.org/abs/2412.03250 - PDF: https://arxiv.org/pdf/2412.03250 The integration of Large Language Models (LLMs) with evolutionary computation (EC) has introduced a promising paradigm for automating the design of metaheuristic algorithms. However, existing frameworks, such as the Large Language Model Evolutionary Algorithm (LLaMEA), often lack precise control over mutation mechanisms, leading to inefficiencies in solution space exploration and potentially suboptimal convergence. This paper introduces a novel approach to mutation control within LLM-driven evolutionary frameworks, inspired by theory of genetic algorithms. Specifically, we propose dynamic mutation prompts that adaptively regulate mutation rates, leveraging a heavy-tailed power-law distribution to balance exploration and exploitation. Experiments using GPT-3.5-turbo and GPT-4o models demonstrate that GPT-3.5-turbo fails to adhere to the specific mutation instructions, while GPT-4o is able to adapt its mutation based on the prompt engineered dynamic prompts. Further experiments show that the introduction of these dynamic rates can improve the convergence speed and adaptability of LLaMEA, when using GPT-4o. This work sets the starting point for better controlled LLM-based mutations in code optimization tasks, paving the way for further advancements in automated metaheuristic design. ## Computational Intelligence: 14th and 15th International Joint Conference on Computational Intelligence (IJCCI 2022 and IJCCI 2023) Revised Selected Papers - Authors: Thomas Bäck, Niki van Stein, Christian Wagner, Jonathan M Garibaldi, Francesco Marcelloni, Hak-Keung Lam, Marie Cottrell, Faiyaz Doctor, Joaquim Filipe, Kevin Warwick, Janusz Kacprzyk (Eds.) - Venue: Springer Nature - Date: 2025-03-26 - Page: https://www.nikivanstein.nl/publication/2025-03-26-Computational-Intelligence-14th-and-15th-International-Joint-Conference-on-Computational-Intelligence-IJCCI-2022-and-IJCCI-2023-Revised-Selected-Papers ## Diffusion and flow matching models for tabular data: A survey - Authors: Zhong Li, Qi Huang, Lincen Yang, Jiayang Shi, Zhao Yang, Niki van Stein, Thomas Bäck, Matthijs van Leeuwen - Venue: arXiv preprint arXiv:2502.17119 - Date: 2025-02-24 - Page: https://www.nikivanstein.nl/publication/2025-02-24-Diffusion-and-flow-matching-models-for-tabular-data-A-survey - arXiv: https://arxiv.org/abs/2502.17119 - PDF: https://arxiv.org/pdf/2502.17119 Deep generative models have made rapid progress in image, text, audio, and video generation, and are increasingly being applied to structured records. For tabular data, however, generative modeling remains difficult: a dataset may contain numerical and categorical attributes, missing values, sensitive fields, imbalanced categories, complex feature dependencies, and domain constraints. Earlier tabular data modeling methods based on GANs or VAEs have achieved useful results, but they can suffer from unstable training, mode collapse, weak modeling of multimodal distributions, and fragile handling of mixed-type features. Diffusion models have therefore attracted growing interest because their noising-and-denoising formulation provides a flexible and stable way to model complex data distributions, and has been adapted to tabular synthesis, missing-value imputation, trustworthy data generation, and anomaly detection. Flow matching offers a closely related route by learning transport vector fields along probability paths, often with more direct control over path design and sampling efficiency. Despite this progress, the literature on diffusion and flow matching models for tabular data remains difficult to compare because methods target different tasks and rely on different representations, objectives, evaluation protocols, and domain assumptions. To the best of our knowledge, this is the first survey dedicated specifically to diffusion and flow matching models for tabular data. We review work from June 2015 to May 2026, organize it around data-engineering challenges, tasks, design choices, and evaluation dimensions, and discuss open problems in scalability, feature dependency modeling, privacy, fairness, benchmarking, and constraint-aware generation. We maintain updates in a GitHub repository. ## Explainable benchmarking for iterative optimization heuristics - Authors: Niki van Stein, Diederick Vermetten, Anna Kononova, Thomas Bäck - Venue: ACM Transactions on Evolutionary Learning - Date: 2025-02-11 - Page: https://www.nikivanstein.nl/publication/2025-02-11-Explainable-benchmarking-for-iterative-optimization-heuristics - DOI: https://doi.org/10.1145/3712255.3734245 - arXiv: https://arxiv.org/abs/2401.17842 - PDF: https://arxiv.org/pdf/2401.17842 Benchmarking heuristic algorithms is vital to understand under which conditions and on what kind of problems certain algorithms perform well. In most current research into heuristic optimization algorithms, only a very limited number of scenarios, algorithm configurations and hyper-parameter settings are explored, leading to incomplete and often biased insights and results. This paper presents a novel approach we call explainable benchmarking. Introducing the IOH-Xplainer software framework, for analyzing and understanding the performance of various optimization algorithms and the impact of their different components and hyper-parameters. We showcase the framework in the context of two modular optimization frameworks. Through this framework, we examine the impact of different algorithmic components and configurations, offering insights into their performance across diverse scenarios. We provide a systematic method for evaluating and interpreting the behaviour and efficiency of iterative optimization heuristics in a more transparent and comprehensible manner, allowing for better benchmarking and algorithm design. ## Towards Fairness in Machine Learning: Balancing Racially Imbalanced Datasets Through Data Augmentation and Generative AI - Authors: Anthonie Schaap, Sofoklis Kitharidis, Niki van Stein - Venue: IJCCI - Date: 2024-11-24 - Page: https://www.nikivanstein.nl/publication/2024-11-24-Towards-Fairness-in-Machine-Learning-Balancing-Racially-Imbalanced-Datasets-Through-Data-Augmentation-and-Generative-AI - DOI: https://doi.org/10.5220/0013002600003837 ## TX-Gen: multi-objective optimization for sparse counterfactual explanations for time-series classification - Authors: Qi Huang, Sofoklis Kitharidis, Thomas Bäck, Niki van Stein - Venue: arxiv - Date: 2024-11-24 - Page: https://www.nikivanstein.nl/publication/2024-11-24-TX-Gen-multi-objective-optimization-for-sparse-counterfactual-explanations-for-time-series-classification - DOI: https://doi.org/10.5220/0013066400003886 - arXiv: https://arxiv.org/abs/2409.09461 - PDF: https://arxiv.org/pdf/2409.09461 In time-series classification, understanding model decisions is crucial for their application in high-stakes domains such as healthcare and finance. Counterfactual explanations, which provide insights by presenting alternative inputs that change model predictions, offer a promising solution. However, existing methods for generating counterfactual explanations for time-series data often struggle with balancing key objectives like proximity, sparsity, and validity. In this paper, we introduce TX-Gen, a novel algorithm for generating counterfactual explanations based on the Non-dominated Sorting Genetic Algorithm II (NSGA-II). TX-Gen leverages evolutionary multi-objective optimization to find a diverse set of counterfactuals that are both sparse and valid, while maintaining minimal dissimilarity to the original time series. By incorporating a flexible reference-guided mechanism, our method improves the plausibility and interpretability of the counterfactuals without relying on predefined assumptions. Extensive experiments on benchmark datasets demonstrate that TX-Gen outperforms existing methods in generating high-quality counterfactuals, making time-series models more transparent and interpretable. ## Llamea: A large language model evolutionary algorithm for automatically generating metaheuristics - Authors: Niki van Stein, Thomas Bäck - Venue: IEEE Transactions on Evolutionary Computation - Date: 2024-11-13 - Page: https://www.nikivanstein.nl/publication/2024-11-13-Llamea-A-large-language-model-evolutionary-algorithm-for-automatically-generating-metaheuristics - DOI: https://doi.org/10.1109/tevc.2024.3497793 - arXiv: https://arxiv.org/abs/2405.20132 - PDF: https://arxiv.org/pdf/2405.20132 Large Language Models (LLMs) such as GPT-4 have demonstrated their ability to understand natural language and generate complex code snippets. This paper introduces a novel Large Language Model Evolutionary Algorithm (LLaMEA) framework, leveraging GPT models for the automated generation and refinement of algorithms. Given a set of criteria and a task definition (the search space), LLaMEA iteratively generates, mutates and selects algorithms based on performance metrics and feedback from runtime evaluations. This framework offers a unique approach to generating optimized algorithms without requiring extensive prior expertise. We show how this framework can be used to generate novel black-box metaheuristic optimization algorithms automatically. LLaMEA generates multiple algorithms that outperform state-of-the-art optimization algorithms (Covariance Matrix Adaptation Evolution Strategy and Differential Evolution) on the five dimensional black box optimization benchmark (BBOB). The algorithms also show competitive performance on the 10- and 20-dimensional instances of the test functions, although they have not seen such instances during the automated generation process. The results demonstrate the feasibility of the framework and identify future directions for automated generation and optimization of algorithms via LLMs. ## EconoJax: A Fast & Scalable Economic Simulation in Jax - Authors: Koen Ponse, Aske Plaat, Niki van Stein, Thomas M. Moerland - Venue: arXiv preprint arXiv:2410.22165 - Date: 2024-10-29 - Page: https://www.nikivanstein.nl/publication/2024-10-29-EconoJax-A-Fast-Scalable-Economic-Simulation-in-Jax - DOI: https://doi.org/10.65109/lmzy7606 - arXiv: https://arxiv.org/abs/2410.22165 - PDF: https://arxiv.org/pdf/2410.22165 Accurate economic simulations often require many experimental runs, particularly when combined with reinforcement learning. Unfortunately, training reinforcement learning agents in multi-agent economic environments can be slow. This paper introduces EconoJax, a fast simulated economy, based on the AI economist. EconoJax, and its training pipeline, are completely written in JAX. This allows EconoJax to scale to large population sizes and perform large experiments, while keeping training times within minutes. Through experiments with populations of 100 agents, we show how real-world economic behavior emerges through training within 15 minutes, in contrast to previous work that required several days. We additionally perform experiments in varying sized action spaces to test if some multi-agent methods produce more diverse behavior compared to others. Here, our findings indicate no notable differences in produced behavior with different methods as is sometimes suggested in earlier works. To aid further research, we open-source EconoJax on Github. ## Evolutionary computation and explainable AI: A roadmap to understandable intelligent systems - Authors: Ryan Zhou, Jaume Bacardit, Alexander Edward Ian Brownlee, Stefano Cagnoni, Martin Fyvie, Giovanni Iacca, John McCall, Niki van Stein, David Walker, Ting Hu - Venue: IEEE Transactions on Evolutionary Computation - Date: 2024-10-23 - Page: https://www.nikivanstein.nl/publication/2024-10-23-Evolutionary-computation-and-explainable-AI-A-roadmap-to-understandable-intelligent-systems - DOI: https://doi.org/10.1109/tevc.2024.3476443 - arXiv: https://arxiv.org/abs/2406.07811 - PDF: https://arxiv.org/pdf/2406.07811 Artificial intelligence methods are being increasingly applied across various domains, but their often opaque nature has raised concerns about accountability and trust. In response, the field of explainable AI (XAI) has emerged to address the need for human-understandable AI systems. Evolutionary computation (EC), a family of powerful optimization and learning algorithms, offers significant potential to contribute to XAI, and vice versa. This paper provides an introduction to XAI and reviews current techniques for explaining machine learning models. We then explore how EC can be leveraged in XAI and examine existing XAI approaches that incorporate EC techniques. Furthermore, we discuss the application of XAI principles within EC itself, investigating how these principles can illuminate the behavior and outcomes of EC algorithms, their (automatic) configuration, and the underlying problem landscapes they optimize. Finally, we discuss open challenges in XAI and highlight opportunities for future research at the intersection of XAI and EC. Our goal is to demonstrate EC's suitability for addressing current explainability challenges and to encourage further exploration of these methods, ultimately contributing to the development of more understandable and trustworthy ML models and EC algorithms. ## Selection of filters for photonic crystal spectrometer using domain-aware evolutionary algorithms - Authors: Kirill Antonov, Marijn Siemons, Niki van Stein, Thomas Bäck, Ralf Kohlhaas, Anna V. Kononova - Venue: arXiv preprint arXiv:2410.13657 - Date: 2024-10-17 - Page: https://www.nikivanstein.nl/publication/2024-10-17-Selection-of-filters-for-photonic-crystal-spectrometer-using-domain-aware-evolutionary-algorithms - DOI: https://doi.org/10.2139/ssrn.5984923 - arXiv: https://arxiv.org/abs/2410.13657 - PDF: https://arxiv.org/pdf/2410.13657 This work addresses the critical challenge of optimal filter selection for a novel trace gas measurement device. This device uses photonic crystal filters to retrieve trace gas concentrations affected by photon and read noise. The filter selection directly influences the accuracy and precision of the gas retrieval and, therefore, is a crucial performance driver. We formulate the problem as a stochastic combinatorial optimization problem and develop a simulator modeling gas retrieval with noise. Metaheuristics representing various families of optimizers are used to minimize the retrieval error objective function. We improve the top-performing algorithms using our novel distance-driven extensions, which employ metrics on the space of filter selections. This leads to a new adaptation of the Univariate Marginal Distribution Algorithm (UMDA), called the Univariate Marginal Distribution Algorithm Unified by Probabilistic Logic Sampling driven by Distance (UMDA-U-PLS-Dist), equipped with one of the proposed distance metrics as the most efficient and robust solver among the considered ones. We apply this algorithm to obtain a diverse set of high-performing solutions and analyze them to draw general conclusions about better combinations of transmission profiles. The analysis reveals that filters with large local differences in transmission improve the device performance. Moreover, the obtained top-performing solutions show significant improvement compared to the baseline. ## ACO-NSGAII: A Novel Metaheuristics for Bi-Objective Electric Vehicle Routing Problems - Authors: Suzan Al-Nassar, Niki van Stein, Yingjie Fan - Venue: 2024 IEEE 27th International Conference on Intelligent Transportation Systems (ITSC) - Date: 2024-09-24 - Page: https://www.nikivanstein.nl/publication/2024-09-24-ACO-NSGAII-A-Novel-Metaheuristics-for-Bi-Objective-Electric-Vehicle-Routing-Problems - DOI: https://doi.org/10.1109/itsc58415.2024.10919987 This study introduces ACO-NSGAII, a hybrid metaheuristic algorithm that integrates ant colony optimization (ACO) and non-dominated sorting genetic algorithm II (NSGA-II), to address the bi-objective Electric Vehicle Routing Problem with Time Windows (EVRPTW). The algorithm aims to minimize both the overall travel distances and the number of vehicles required in last-mile delivery. Our approach starts with ACO to generate an initial solution focusing on distance mini-mization, followed by NSGA-II to optimize the dual objectives efficiently. Extensive computational experiments demonstrate that ACO-NSGAII significantly outperforms existing methods like Random-NSGAII and NN-NSGAII (an integration of near-est neighbor and NSGA-II), offering promising solutions for sustainable urban logistics. The findings contribute valuable insights into the trade-offs between minimizing distance and vehicle usage. Moreover, our research extensively studies the effectiveness of ACO-NSGAII for complex routing problems and highlights the impact of search budget allocation on the convergence of ACO-NSGAII. ## Landscape-aware automated algorithm configuration using multi-output mixed regression and classification - Authors: Fu Xing Long, Moritz Frenzel, Peter Krause, Markus Gitterle, Thomas Bäck, Niki van Stein - Venue: International Conference on Parallel Problem Solving from Nature - Date: 2024-09-07 - Page: https://www.nikivanstein.nl/publication/2024-09-07-Landscape-aware-automated-algorithm-configuration-using-multi-output-mixed-regression-and-classification - DOI: https://doi.org/10.1007/978-3-031-70068-2_6 - arXiv: https://arxiv.org/abs/2409.01446 - PDF: https://arxiv.org/pdf/2409.01446 In landscape-aware algorithm selection problem, the effectiveness of feature-based predictive models strongly depends on the representativeness of training data for practical applications. In this work, we investigate the potential of randomly generated functions (RGF) for the model training, which cover a much more diverse set of optimization problem classes compared to the widely-used black-box optimization benchmarking (BBOB) suite. Correspondingly, we focus on automated algorithm configuration (AAC), that is, selecting the best suited algorithm and fine-tuning its hyperparameters based on the landscape features of problem instances. Precisely, we analyze the performance of dense neural network (NN) models in handling the multi-output mixed regression and classification tasks using different training data sets, such as RGF and many-affine BBOB (MA-BBOB) functions. Based on our results on the BBOB functions in 5d and 20d, near optimal configurations can be identified using the proposed approach, which can most of the time outperform the off-the-shelf default configuration considered by practitioners with limited knowledge about AAC. Furthermore, the predicted configurations are competitive against the single best solver in many cases. Overall, configurations with better performance can be best identified by using NN models trained on a combination of RGF and MA-BBOB functions. ## A deep dive into effects of structural bias on cma-es performance along affine trajectories - Authors: Niki van Stein, Sarah L. Thomson, Anna V. Kononova - Venue: International Conference on Parallel Problem Solving from Nature - Date: 2024-09-07 - Page: https://www.nikivanstein.nl/publication/2024-09-07-A-deep-dive-into-effects-of-structural-bias-on-cma-es-performance-along-affine-trajectories - DOI: https://doi.org/10.1007/978-3-031-70068-2_3 - arXiv: https://arxiv.org/abs/2404.17323 - PDF: https://arxiv.org/pdf/2404.17323 To guide the design of better iterative optimisation heuristics, it is imperative to understand how inherent structural biases within algorithm components affect the performance on a wide variety of search landscapes. This study explores the impact of structural bias in the modular Covariance Matrix Adaptation Evolution Strategy (modCMA), focusing on the roles of various modulars within the algorithm. Through an extensive investigation involving 435,456 configurations of modCMA, we identified key modules that significantly influence structural bias of various classes. Our analysis utilized the Deep-BIAS toolbox for structural bias detection and classification, complemented by SHAP analysis for quantifying module contributions. The performance of these configurations was tested on a sequence of affine-recombined functions, maintaining fixed optimum locations while gradually varying the landscape features. Our results demonstrate an interplay between module-induced structural bias and algorithm performance across different landscape characteristics. ## Reasoning with large language models, a survey - Authors: Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, Thomas Bäck - Venue: arXiv preprint arXiv:2407.11511 - Date: 2024-07-16 - Page: https://www.nikivanstein.nl/publication/2024-07-16-Reasoning-with-large-language-models-a-survey - DOI: https://doi.org/10.20944/preprints202510.1024.v1 - arXiv: https://arxiv.org/abs/2407.11511 - PDF: https://arxiv.org/pdf/2407.11511 Large language models (LLMs) with billions of parameters exhibit in-context learning abilities, enabling few-shot learning on tasks that the model was not specifically trained for. Traditional models achieve breakthrough performance on language tasks, but do not perform well on basic reasoning benchmarks. However, a new in-context learning approach, Chain-of-thought, has demonstrated strong multi-step reasoning abilities on these benchmarks. The research on LLM reasoning abilities started with the question whether LLMs can solve grade school math word problems, and has expanded to other tasks in the past few years. This article reviews the field of multi-step reasoning with LLMs. We propose a taxonomy that identifies different ways to generate, evaluate, and control multi-step reasoning. We provide an in-depth coverage of core approaches and open problems, and we propose a research agenda for the near future. We find that multi-step reasoning approaches have progressed beyond math word problems, and can now successfully solve challenges in logic, combinatorial games, and robotics, sometimes by first generating code that is then executed by external tools. Many studies in multi-step methods use reinforcement learning for finetuning, external optimization loops, in-context reinforcement learning, and self-reflection. ## Modular optimization framework for mixed expensive and inexpensive real-world problems - Authors: Roy de Winter, Thomas Bäck, Niki van Stein - Venue: Proceedings of the Genetic and Evolutionary Computation Conference Companion - Date: 2024-07-14 - Page: https://www.nikivanstein.nl/publication/2024-07-14-Modular-optimization-framework-for-mixed-expensive-and-inexpensive-real-world-problems - DOI: https://doi.org/10.1145/3638530.3664133 A modular optimization framework is proposed that can deal with different problem characteristics encountered in real-world design problems. The common denominator of problems considered in this work is the computationally demanding objective function used to evaluate the designs, while some of the constraint functions can be inexpensive. The framework uses radial basis functions as surrogates for the computationally expensive objectives while inexpensive functions can directly be used for searching promising solutions on the surrogates. The framework is adjustable for the number of continuous design parameters, the number of constraints, the number of solutions that can be evaluated in parallel, and different strategies can be selected on how to deal with the computationally inexpensive functions. We propose guidelines on how to set up the modular framework depending on the problem characteristics one has to deal with. To verify the working of the optimization framework, two different real-world ship design optimization problems have been optimized with different characteristics such as the number of constraints and how expensive they are. The results show that it is beneficial to directly exploit the inexpensive functions when possible and that parameterization of the problem is the most important step of the process. ## A functional analysis approach to symbolic regression - Authors: Kirill Antonov, Roman Kalkreuth, Kaifeng Yang, Thomas Bäck, Niki van Stein, Anna Kononova - Venue: Proceedings of the Genetic and Evolutionary Computation Conference - Date: 2024-07-14 - Page: https://www.nikivanstein.nl/publication/2024-07-14-A-functional-analysis-approach-to-symbolic-regression - DOI: https://doi.org/10.1145/3638529.3654079 - arXiv: https://arxiv.org/abs/2402.06299 - PDF: https://arxiv.org/pdf/2402.06299 Symbolic regression (SR) poses a significant challenge for randomized search heuristics due to its reliance on the synthesis of expressions for input-output mappings. Although traditional genetic programming (GP) algorithms have achieved success in various domains, they exhibit limited performance when tree-based representations are used for SR. To address these limitations, we introduce a novel SR approach called Fourier Tree Growing (FTG) that draws insights from functional analysis. This new perspective enables us to perform optimization directly in a different space, thus avoiding intricate symbolic expressions. Our proposed algorithm exhibits significant performance improvements over traditional GP methods on a range of classical one-dimensional benchmarking problems. To identify and explain limiting factors of GP and FTG, we perform experiments on a large-scale polynomials benchmark with high-order polynomials up to degree 100. To the best of the authors' knowledge, this work represents the pioneering application of functional analysis in addressing SR problems. The superior performance of the proposed algorithm and insights into the limitations of GP open the way for further advancing GP for SR and related areas of explainable machine learning. ## A Critical Analysis of Raven Roost Optimization - Authors: Martijn Halsema, Diederick Vermetten, Thomas Bäck, Niki van Stein - Venue: Proceedings of the Genetic and Evolutionary Computation Conference Companion - Date: 2024-07-14 - Page: https://www.nikivanstein.nl/publication/2024-07-14-A-Critical-Analysis-of-Raven-Roost-Optimization - DOI: https://doi.org/10.1145/3638530.3664124 This study critically examines the Raven Roost Optimization (RRO) algorithm within the broader context of nature-inspired metaheuristics, challenging its novelty and efficacy in the field of black-box optimization. Many similar methods use ideas from nature, but it's important to see if they really bring something new to the table. We compared RRO with another well-known method called Particle Swarm Optimization (PSO) to see how well RRO works and if it's truly a new idea. Through comprehensive analysis and benchmarking, we reveal that RRO's purported novelty largely recapitulates existing strategies under a new metaphorical guise. The algorithm's performance is systematically evaluated across various dimensions, revealing inherent limitations and biases introduced by its metaphorical foundation. Our findings advocate for a critical reassessment of metaphor-based heuristics, urging the computational intelligence community to prioritize substantive algorithmic advancements over superficial novelty. The call for rigorous evaluation and validation of new optimization methods is underscored, emphasizing the need for transparency, reproducibility, and genuine innovation in algorithmic design. This work contributes to the ongoing discourse on the validation and merit of bio-inspired algorithms, providing insights that may guide future research towards more meaningful and empirically justified contributions. ## A Corridor Model Evolutionary Algorithm for Fast Converging Green Vehicle Routing Problem - Authors: Ananta Shahane, Niki van Stein, Yingjie Fan - Venue: Proceedings of the Genetic and Evolutionary Computation Conference Companion - Date: 2024-07-14 - Page: https://www.nikivanstein.nl/publication/2024-07-14-A-Corridor-Model-Evolutionary-Algorithm-for-Fast-Converging-Green-Vehicle-Routing-Problem - DOI: https://doi.org/10.1145/3638530.3654323 In this paper, we introduce an innovative evolutionary algorithm for the Green Vehicle Routing Problem (GVRP). We use the relative location of the customers with respect to the depot to estimate whether a mutation is exploitation or exploration. By introducing a probabilistic corridor model, our mutation strategy converges faster to optimal solutions compared to random mutation strategies in existing research. We then introduce our evolutionary algorithms that use this mutation strategy to reduce the number of hyper-parameters to two or three. We test our algorithms across 92 benchmarks. Results show that our algorithms are robust and efficient. ## Beyond the veil of similarity: Quantifying semantic continuity in explainable AI - Authors: Qi Huang, Emanuele Mezzi, Osman Mutlu, Miltiadis Kofinas, Vidya Prasad, Shadnan Azwad Khan, Elena Ranguelova, Niki van Stein - Venue: World Conference on Explainable Artificial Intelligence - Date: 2024-07-10 - Page: https://www.nikivanstein.nl/publication/2024-07-10-Beyond-the-veil-of-similarity-Quantifying-semantic-continuity-in-explainable-AI - DOI: https://doi.org/10.1007/978-3-031-63787-2_16 - arXiv: https://arxiv.org/abs/2407.12950 - PDF: https://arxiv.org/pdf/2407.12950 We introduce a novel metric for measuring semantic continuity in Explainable AI methods and machine learning models. We posit that for models to be truly interpretable and trustworthy, similar inputs should yield similar explanations, reflecting a consistent semantic understanding. By leveraging XAI techniques, we assess semantic continuity in the task of image recognition. We conduct experiments to observe how incremental changes in input affect the explanations provided by different XAI methods. Through this approach, we aim to evaluate the models' capability to generalize and abstract semantic concepts accurately and to evaluate different XAI methods in correctly capturing the model behaviour. This paper contributes to the broader discourse on AI interpretability by proposing a quantitative measure for semantic continuity for XAI methods, offering insights into the models' and explainers' internal reasoning processes, and promoting more reliable and transparent AI systems. ## Landscape analysis based vs. domain-specific optimization for engineering design applications: a clear case - Authors: Roy de Winter, Fu Xing Long, Andre Thomaser, Thomas Bäck, Niki van Stein, Anna V. Kononova - Venue: 2024 IEEE Conference on Artificial Intelligence (CAI) - Date: 2024-06-25 - Page: https://www.nikivanstein.nl/publication/2024-06-25-Landscape-analysis-based-vs-domain-specific-optimization-for-engineering-design-applications-a-clear-case - DOI: https://doi.org/10.1109/cai59869.2024.00148 - PDF: https://scholarlypublications.universiteitleiden.nl/download/1887/4254666/1?col=81522199340002711 Traditional methods in black-box optimization often prescribe general-purpose algorithms for broad problem categories, which overlooks the significant variability in optimization landscapes that can exist even within a single domain. We address this gap by demonstrating the efficacy of landscape analysis techniques in guiding the choice and tuning of optimization algorithms for distinct landscape features. This study focuses on two real-world problem domains characterized by black-box simulation-based optimization: automotive control system calibration (anti-lock braking system) and automotive crashworthiness optimization. Through comparative analysis of computer experiments in these domains, landscape analysis techniques are employed for both visualization and explanation purposes. The results reveal substantial variations in landscape features across different instances within the same problem domain. Consequently, we advocate for a paradigm shift towards learning and applying optimizers tailored to specific landscape feature spaces rather than applying a one-size-fits-all approach to an entire problem domain. This research not only enhances the understanding of landscape variability in optimization problems but also paves the way for more efficient and effective optimization processes in complex, real-world scenarios. ## Quality-diversity driven robust evolutionary optimization of optical designs - Authors: Kirill Antonov, Teus Tukker, Tiago Botari, Thomas Bäck, Anna V. Kononova, Niki van Stein - Venue: Computational Optics 2024 - Date: 2024-06-17 - Page: https://www.nikivanstein.nl/publication/2024-06-17-Quality-diversity-driven-robust-evolutionary-optimization-of-optical-designs - DOI: https://doi.org/10.1117/12.3017498 Developing optical systems, particularly those consisting of spherical lenses, is relevant for various applications such as lithographic scanners and metrology equipment. The design process of an optical system typically involves the optimization of specific objectives to ensure the best performance. As a common example of such an objective, we consider the problem of determining the lens curvatures that result in a sufficiently small root mean square (RMS) spot size. Optimization algorithms are commonly employed to solve this problem by heuristically eliminating sub-optimal optical designs. This class of algorithms includes the damped least squares (DLS) widely applied in commercial software and advanced methods like Saddle Point Construction. However, within a restricted computational budget, these optimizers are limited in exploring potentially promising novel solutions since they heavily rely on the initial specific designs that must conform to complex or unknown requirements. In this work, we address the considered problem with a modified Hill-Valley Evolutionary Algorithm (HillVallEA), which proved itself as one of the best state-of-the-art metaheuristics for multimodal black-box optimization. We demonstrate that our algorithm locates a diverse set of high-quality optical designs with four lenses in a single run even when initialized with random starting curvatures. This is the first result in this domain when an optimization algorithm that does not take specific optical properties into account can still generate relevant and high-performing optical systems. Furthermore, we show the benefits of the proposed methodology for the diversity of the obtained set of solutions, while maintaining a solution of the same quality as the one found by the most prominent algorithm in the domain. We provide analyses of the obtained solutions according to: 1) tolerance to the alignment of lenses, 2) susceptibility to small variations of lens curvatures. ## A Survey of Meta-features Used for Automated Selection of Algorithms for Black-box Single-objective Continuous Optimization - Authors: Gjorgjina Cenikj, Ana Nikolikj, Gašper Petelin, Niki van Stein, Carola Doerr, Tome Eftimov - Venue: arXiv preprint arXiv:2406.06629 - Date: 2024-06-10 - Page: https://www.nikivanstein.nl/publication/2024-06-10-A-Survey-of-Meta-features-Used-for-Automated-Selection-of-Algorithms-for-Black-box-Single-objective-Continuous-Optimization - arXiv: https://arxiv.org/abs/2406.06629 - PDF: https://arxiv.org/pdf/2406.06629 This survey examines key advancements in designing features to represent optimization problem instances, algorithm instances, and their interactions within the context of single-objective continuous black-box optimization. These features support machine learning tasks such as algorithm selection, algorithm configuration, and problem classification, and they are also used to evaluate the complementarity of benchmark problem sets. We provide a comprehensive overview of problem landscape features, algorithm features, high-level problem-algorithm interaction features, and trajectory features, including the latest works from the past five years. We also point out limitations of the current state-of-the-art and suggest directions for future research. ## Generating cheap representative functions for expensive automotive crashworthiness optimization - Authors: Fu Xing Long, Niki van Stein, Moritz Frenzel, Peter Krause, Markus Gitterle, Thomas Bäck - Venue: ACM Transactions on Evolutionary Learning and Optimization - Date: 2024-06-08 - Page: https://www.nikivanstein.nl/publication/2024-06-08-Generating-cheap-representative-functions-for-expensive-automotive-crashworthiness-optimization - DOI: https://doi.org/10.1145/3646554 - PDF: https://dl.acm.org/doi/pdf/10.1145/3646554 Solving real-world engineering optimization problems, such as automotive crashworthiness optimization, is extremely challenging, because the problem characteristics are oftentimes not well understood. Furthermore, typical hyperparameter optimization (HPO) approaches that require a large function evaluation budget are computationally hindered, if the function evaluation is expensive, for example, requires finite element (FE) simulation runs. In this article, we propose an approach to characterize real-world expensive black-box optimization problems using the exploratory landscape analysis (ELA). Based on these landscape characteristics, we can identify test functions that are fast-to-evaluate and representative for HPO purposes. Focusing on 20 problem instances from automotive crashworthiness optimization, our results reveal that these 20 crashworthiness problems exhibit landscape features different from classical optimization benchmark test suites, such as the widely-used black-box optimization benchmarking (BBOB) problem set. In fact, these 20 problem instances belong to problem classes that are distinct from the BBOB test functions based on the clustering results. Further analysis indicates that, as far as the ELA features concern, they are most similar to problem classes of tree-based test functions. By analyzing the performance of two optimization algorithms with different hyperparameters, namely the covariance matrix adaptation evolutionary strategy (CMA-ES) and Bayesian optimization (BO), we show that the tree-based test functions are indeed representative in terms of predicting the algorithm performances. Following this, such scalable and fast-to-evaluate tree-based test functions have promising potential for automated design of an optimization algorithm for specific real-world problem classes. ## Parallel multi-objective optimization for expensive and inexpensive objectives and constraints - Authors: Roy de Winter, Bas Milatz, Julian Blank, Niki van Stein, Thomas Bäck, Kalyanmoy Deb - Venue: Swarm and Evolutionary Computation - Date: 2024-04-01 - Page: https://www.nikivanstein.nl/publication/2024-04-01-Parallel-multi-objective-optimization-for-expensive-and-inexpensive-objectives-and-constraints - DOI: https://doi.org/10.1016/j.swevo.2024.101508 - PDF: https://ars.els-cdn.com/content/image/1-s2.0-S2210650224000415-ga1_lrg.jpg Expensive objectives and constraints are key characteristics of real-world multi-objective optimization problems. In practice, they often occur jointly with inexpensive objectives and constraints. This paper presents the Inexpensive Objectives and Constraints Self-Adapting Multi-Objective Constraint Optimization algorithm that uses Radial Basis function Approximations (IOC-SAMO-COBRA) for such problems. This is motivated by the recently proposed Inexpensive Constraint Surrogate-Assisted Non-dominated Sorting Genetic Algorithm II (IC-SA-NSGA-II). These algorithms and their counterparts that do not explicitly differentiate between expensive and inexpensive objectives and constraints are compared on 22 widely used test functions. The IOC-SAMO-COBRA algorithm finds significantly better (identical/worse) Pareto fronts in at least 78% (6%/16%) of all test problems compared to IC-SA-NSGA-II measured with both the hypervolume and Inverted Generational Distance+ performance metric. The empirical cumulative distribution functions confirm this advantage for both algorithm variants that exploit the inexpensive constraints. In addition, the proposed method is compared against state-of-the-art practices on a real-world cargo vessel design problem. On this 17-dimensional two-objective practical problem, the proposed IOC-SAMO-COBRA outperforms SA-NSGA-II as well. From an algorithmic perspective, the comparison identifies specific strengths of both approaches and indicates how they should be hybridized to combine their best components. ## Shapelet-based Model-agnostic Counterfactual Local Explanations for Time Series Classification - Authors: Qi Huang, Wei Chen, Thomas Bäck, Niki van Stein - Venue: Preprint on arXiv preprint arXiv:2402.01343 - Date: 2024-02-02 - Page: https://www.nikivanstein.nl/publication/2024-02-02-Shapelet-based-Model-agnostic-Counterfactual-Local-Explanations-for-Time-Series-Classification - arXiv: https://arxiv.org/abs/2402.01343 - PDF: https://arxiv.org/pdf/2402.01343 In this work, we propose a model-agnostic instance-based post-hoc explainability method for time series classification. The proposed algorithm, namely Time-CF, leverages shapelets and TimeGAN to provide counterfactual explanations for arbitrary time series classifiers. We validate the proposed method on several real-world univariate time series classification tasks from the UCR Time Series Archive. The results indicate that the counterfactual instances generated by Time-CF when compared to state-of-the-art methods, demonstrate better performance in terms of four explainability metrics: closeness, sensibility, plausibility, and sparsity. ## Adaptive model pruning in federated learning through loss exploration - Authors: Christian Interno, Elena Raponi, Niki van Stein, Thomas Bäck, Markus Olhofer, Yaochu Jin, Barbara Hammer - Venue: 2nd Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@ ICML 2024) - Date: 2024-01-01 - Page: https://www.nikivanstein.nl/publication/2024-01-01-Adaptive-model-pruning-in-federated-learning-through-loss-exploration ## The Opaque Nature of Intelligence and the Pursuit of Explainable AI - Authors: Sarah Thomson, Niki van Stein, Daan van den Berg, Cees van Leeuwen - Venue: ECTA 2023 - Date: 2023-10-26 - Page: https://www.nikivanstein.nl/publication/2023-10-26-The-Opaque-Nature-of-Intelligence-and-the-Pursuit-of-Explainable-AI - DOI: https://doi.org/10.5220/0012249500003595 In this work We consider and discuss the problems which come with trying to explain human and machine intelligence.How explainable artificial intelligence research is being carried out, the pitfalls and limitations of current approaches and the bigger question of whether we need explanations for trusting inherently complex and large intelligent systems, whether artificial or not. ## Clustering-based Domain-Incremental Learning - Authors: Christiaan Lamers, René Vidal, Nabil Belbachir, Niki van Stein, Thomas Bäck, Paris Giampouras - Venue: IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) - Date: 2023-09-21 - Page: https://www.nikivanstein.nl/publication/2023-09-21-Clustering-based-Domain-Incremental-Learning - DOI: https://doi.org/10.1109/iccvw60793.2023.00363 - arXiv: https://arxiv.org/abs/2309.12078 - PDF: https://arxiv.org/pdf/2309.12078 We consider the problem of learning multiple tasks in a continual learning setting in which data from different tasks is presented to the learner in a streaming fashion. A key challenge in this setting is the so-called "catastrophic forgetting problem", in which the performance of the learner in an "old task" decreases when subsequently trained on a "new task". Existing continual learning methods, such as Averaged Gradient Episodic Memory (A-GEM) and Orthogonal Gradient Descent (OGD), address catastrophic forgetting by minimizing the loss for the current task without increasing the loss for previous tasks. However, these methods assume the learner knows when the task changes, which is unrealistic in practice. In this paper, we alleviate the need to provide the algorithm with information about task changes by using an online clustering-based approach on a dynamically updated finite pool of samples or gradients. We thereby successfully counteract catastrophic forgetting in one of the hardest settings, namely: domain-incremental learning, a setting for which the problem was previously unsolved. We showcase the benefits of our approach by applying these ideas to projection-based methods, such as A-GEM and OGD, which lead to task-agnostic versions of them. Experiments on real datasets demonstrate the effectiveness of the proposed strategy and its promising performance compared to state-of-the-art methods. ## Curing ill-Conditionality via Representation-Agnostic Distance-Driven Perturbations - Authors: Kirill Antonov, Anna V. Kononova, Thomas Bäck, Niki van Stein - Venue: Proceedings of the 17th ACM/SIGEVO Conference on Foundations of Genetic Algorithms - Date: 2023-08-30 - Page: https://www.nikivanstein.nl/publication/2023-08-30-Curing-ill-Conditionality-via-Representation-Agnostic-Distance-Driven-Perturbations - DOI: https://doi.org/10.1145/3594805.3607138 - PDF: https://dl.acm.org/doi/pdf/10.1145/3594805.3607138 The objective value of an ill-conditioned function may significantly change with a minor shift of the argument in the search space. Ill-conditioned functions do not have at all or exhibit very few hints towards better solutions and, thus, they are usually difficult to optimize with randomized search heuristics. However, problems that emerge in practical applications are likely to be formulated as ill-conditioned functions, as often Euclidean metric is used to measure distance in the search space. At the same time, it may be possible to use domain-specific knowledge to define such a metric in the search space so that the function stops being ill-conditioned. We consider finite search spaces and propose two mutation operators that leverage such metric to optimize the function more efficiently. The first operator assumes prior knowledge about the distance, the second operator uses the distance as a black box. Those operators apply an estimation of distribution algorithm to find the best mutant according to the defined function, which employs the given metric. For pseudo-Boolean and integer optimization problems, we experimentally show that both mutation operators speed up the search on most of the functions when applied in considered evolutionary algorithms and random local search. Moreover, those operators can be applied in any randomized search heuristic which uses perturbations. However, our mutation operators increase wall-clock time and so are helpful in practice when distance is (much) cheaper to compute than the real objective function. ## New solutions to Cooke triplet problem via analysis of attraction basins - Authors: Kirill Antonov, Tiago Botari, Teus Tukker, Thomas Bäck, Niki van Stein, Anna V. Kononova - Venue: Digital Optical Technologies 2023 - Date: 2023-08-07 - Page: https://www.nikivanstein.nl/publication/2023-08-07-New-solutions-to-Cooke-triplet-problem-via-analysis-of-attraction-basins - DOI: https://doi.org/10.1117/12.2675836 Optical systems in lithography machines play a significant role in their performance and, therefore, need to be optimized for specific applications. Artificial Intelligence (AI) and, in particular, metaheuristics are utilized in optimization algorithms for finding a diverse set of feasible and high-performing designs. The diversity requirement of the produced solutions is enforced to allow taking into account additional constraints that are difficult to formalize. In this work, we analyse the space of solutions previously produced by a niching evolutionary algorithm for the Cooke Triplet optical system and propose an approximation of the manifold where all high-performing designs exist. First, we show the existence of high-performing optical designs that are structurally different from the 21 previously known theoretical solutions. In order to do this, we develop a general computationally efficient methodology to create a partition of known high-quality points and their (accidentally found) improvements to their corresponding attraction basins, in the case when neither the exact number of landscape attractors nor their locations are known. We construct a manifold estimation which contains high-performing solutions by fitting a Gaussian Process-based classifier which predicts if an arbitrary design is close to high-performing. The proposed approach shows that AI-assisted optimization is beneficial, and it can be used to extend the capabilities of lithographic scanners and metrology equipment. Furthermore, it opens the possibility of studying other industrial applications. ## Representation-agnostic distance-driven perturbation for optimizing ill-conditioned problems - Authors: Kirill Antonov, Anna V. Kononova, Thomas Bäck, Niki van Stein - Venue: Preprint on arXiv preprint arXiv:2306.02985 - Date: 2023-06-05 - Page: https://www.nikivanstein.nl/publication/2023-06-05-Representation-agnostic-distance-driven-perturbation-for-optimizing-ill-conditioned-problems - arXiv: https://arxiv.org/abs/2306.02985 - PDF: https://arxiv.org/pdf/2306.02985 Locality is a crucial property for efficiently optimising black-box problems with randomized search heuristics. However, in practical applications, it is not likely to always find such a genotype encoding of candidate solutions that this property is upheld with respect to the Hamming distance. At the same time, it may be possible to use domain-specific knowledge to define a metric with locality property. We propose two mutation operators to solve such optimization problems more efficiently using the metric. The first operator assumes prior knowledge about the distance, the second operator uses the distance as a black box. Those operators apply an estimation of distribution algorithm to find the best mutant according to the defined in the paper function, which employs the given distance. For pseudo-boolean and integer optimization problems, we experimentally show that both mutation operators speed up the search on most of the functions when applied in considered evolutionary algorithms and random local search. Moreover, those operators can be applied in any randomized search heuristic which uses perturbations. However, our mutation operators increase wall-clock time and so are helpful in practice when distance is (much) cheaper to compute than the real objective function. ## Evolutionary Algorithms for Parameter Optimization—Thirty Years Later - Authors: Thomas Bäck, Anna V. Kononova, Niki van Stein, Hao Wang, Kirill A. Antonov, Roman T. Kalkreuth, Jacob de Nobel, Diederick Vermetten, Roy de Winter, Furong Ye - Venue: Evolutionary Computation - Date: 2023-06-01 - Page: https://www.nikivanstein.nl/publication/2023-06-01-Evolutionary-Algorithms-for-Parameter-Optimization-Thirty-Years-Later - DOI: https://doi.org/10.1162/evco_a_00325 - PDF: https://direct.mit.edu/evco/article-pdf/31/2/81/2133989/evco_a_00325.pdf Thirty years, 1993–2023, is a huge time frame in science. We address some major developments in the field of evolutionary algorithms, with applications in parameter optimization, over these 30 years. These include the covariance matrix adaptation evolution strategy and some fast-growing fields such as multimodal optimization, surrogate-assisted optimization, multiobjective optimization, and automated algorithm design. Moreover, we also discuss particle swarm optimization and differential evolution, which did not exist 30 years ago, either. One of the key arguments made in the paper is that we need fewer algorithms, not more, which, however, is the current trend through continuously claiming paradigms from nature that are suggested to be useful as new optimization algorithms. Moreover, we argue that we need proper benchmarking procedures to sort out whether a newly proposed algorithm is useful or not. We also briefly discuss automated algorithm design approaches, including configurable algorithm design frameworks, as the proposed next step toward designing optimization algorithms automatically, rather than by hand. ## Application of Functional Kernel Hypothesis Testing for Channel Selection in Time Series Classification - Authors: Qi Huang, Thomas Bäck, Niki van Stein - Venue: Conference on Algebraic Informatics - Date: 2023-06-01 - Page: https://www.nikivanstein.nl/publication/2023-06-01-Application-of-Functional-Kernel-Hypothesis-Testing-for-Channel-Selection-in-Time-Series-Classification - DOI: https://doi.org/10.1109/cai54212.2023.00111 - PDF: https://scholarlypublications.universiteitleiden.nl/access/item%3A3762455/view Multi-variate time series classification tasks are prevalent in various real-world engineering domains, including but not limited to activity recognition and anomaly detection. However, due to the abundance of sensors available, selecting the appropriate channels for successful classification can be a daunting task. In this study, we propose to use a two-sample hypothesis test, to determine the relevance of channels in time series classification tasks. We illustrate the industrial usecase that motivated this algorithm and validate our approach on open-source benchmarks. The proposed method has the potential to address the challenge of channel selection in multi-variate time series classification tasks and can significantly impact various real-world engineering applications. ## AI for Expensive Optimization Problems in Industry - Authors: Niki van Stein, Roy de Winter, Thomas Bäck, Anna V. Kononova - Venue: Conference on Algebraic Informatics - Date: 2023-06-01 - Page: https://www.nikivanstein.nl/publication/2023-06-01-AI-for-Expensive-Optimization-Problems-in-Industry - DOI: https://doi.org/10.1109/cai54212.2023.00113 - PDF: https://scholarlypublications.universiteitleiden.nl/download/1887/3673410/1?col=81522199340002711 The optimization of real-world engineering problems can be a challenging task, owing to the limited understanding of problem characteristics and the high cost of evaluating objectives and constraints in terms of computing time or licenses. This study proposes an AI-assisted optimization pipeline that addresses these challenges by using proxy functions in order to select and optimize the optimization algorithm and its hyper-parameters, thereby significantly accelerating the optimization process on the real (expensive) problem. These proxy functions are inexpensive to evaluate and are selected to exhibit similar landscape characteristics as the original problem. To obtain such proxy functions, we adopt an approach, which involves computing Exploratory Landscape Analysis (ELA) features to characterize the problem’s landscape. The ELA features are then used to identify an artificial function that replicates the original problem’s properties, which can then be employed as a low-cost proxy function for the hyper-parameter optimization of our pipeline. Several real-world industrial applications are discussed as use-case of our proposed approach. ## Challenges of ELA-guided Function Evolution using Genetic Programming - Authors: Fu Xing Long, Diederick Vermetten, Anna Kononova, Roman Kalkreuth, Kaifeng Yang, Thomas Bäck, Niki van Stein - Venue: Preprint on arXiv preprint arXiv:2305.15245 - Date: 2023-05-24 - Page: https://www.nikivanstein.nl/publication/2023-05-24-Challenges-of-ELA-guided-Function-Evolution-using-Genetic-Programming - DOI: https://doi.org/10.5220/0012206200003595 - arXiv: https://arxiv.org/abs/2305.15245 - PDF: https://arxiv.org/pdf/2305.15245 Within the optimization community, the question of how to generate new optimization problems has been gaining traction in recent years. Within topics such as instance space analysis (ISA), the generation of new problems can provide new benchmarks which are not yet explored in existing research. Beyond that, this function generation can also be exploited for solving complex real-world optimization problems. By generating functions with similar properties to the target problem, we can create a robust test set for algorithm selection and configuration. However, the generation of functions with specific target properties remains challenging. While features exist to capture low-level landscape properties, they might not always capture the intended high-level features. We show that a genetic programming (GP) approach guided by these exploratory landscape analysis (ELA) properties is not always able to find satisfying functions. Our results suggest that careful considerations of the weighting of landscape properties, as well as the distance measure used, might be required to evolve functions that are sufficiently representative to the target landscape. ## BBOB Instance Analysis: Landscape Properties and Algorithm Performance Across Problem Instances - Authors: Fu Xing Long, Diederick Vermetten, Niki van Stein, Anna V. Kononova - Venue: International Conference on the Applications of Evolutionary Computation (Part of EvoStar) - Date: 2023-04-09 - Page: https://www.nikivanstein.nl/publication/2023-04-09-BBOB-Instance-Analysis-Landscape-Properties-and-Algorithm-Performance-Across-Problem-Instances - DOI: https://doi.org/10.1007/978-3-031-30229-9_25 - arXiv: https://arxiv.org/abs/2211.16318 - PDF: https://arxiv.org/pdf/2211.16318 Benchmarking is a key aspect of research into optimization algorithms, and as such the way in which the most popular benchmark suites are designed implicitly guides some parts of algorithm design. One of these suites is the black-box optimization benchmarking (BBOB) suite of 24 single-objective noiseless functions, which has been a standard for over a decade. Within this problem suite, different instances of a single problem can be created, which is beneficial for testing the stability and invariance of algorithms under transformations. In this paper, we investigate the BBOB instance creation protocol by considering a set of 500 instances for each BBOB problem. Using exploratory landscape analysis, we show that the distribution of landscape features across BBOB instances is highly diverse for a large set of problems. In addition, we run a set of eight algorithms across these 500 instances, and investigate for which cases statistically significant differences in performance occur. We argue that, while the transformations applied in BBOB instances do indeed seem to preserve the high-level properties of the functions, their difference in practice should not be overlooked, particularly when treating the problems as box-constrained instead of unconstrained. ## Deep-BIAS: Detecting Structural Bias using Explainable AI - Authors: Niki van Stein, Diederick Vermetten, Fabio Caraffini, Anna V. Kononova - Venue: Preprint on arXiv preprint arXiv:2304.01869 - Date: 2023-04-04 - Page: https://www.nikivanstein.nl/publication/2023-04-04-Deep-BIAS-Detecting-Structural-Bias-using-Explainable-AI - DOI: https://doi.org/10.1145/3583133.3590551 - arXiv: https://arxiv.org/abs/2304.01869 - PDF: https://arxiv.org/pdf/2304.01869 Evaluating the performance of heuristic optimisation algorithms is essential to determine how well they perform under various conditions. Recently, the BIAS toolbox was introduced as a behaviour benchmark to detect structural bias (SB) in search algorithms. The toolbox can be used to identify biases in existing algorithms, as well as to test for bias in newly developed algorithms. In this article, we introduce a novel and explainable deep-learning expansion of the BIAS toolbox, called Deep-BIAS. Where the original toolbox uses 39 statistical tests and a Random Forest model to predict the existence and type of SB, the Deep-BIAS method uses a trained deep-learning model to immediately detect the strength and type of SB based on the raw performance distributions. Through a series of experiments with a variety of structurally biased scenarios, we demonstrate the effectiveness of Deep-BIAS. We also present the results of using the toolbox on 336 state-of-the-art optimisation algorithms, which showed the presence of various types of structural bias, particularly towards the centre of the objective space or exhibiting discretisation behaviour. The Deep-BIAS method outperforms the BIAS toolbox both in detecting bias and for classifying the type of SB. Furthermore, explanations can be derived using XAI techniques. ## Evaluation of deep unsupervised anomaly detection methods with a data-centric approach for on-line inspection - Authors: Alexander Zeiser, Bekir Özcan, Niki van Stein, Thomas Bäck - Venue: Computers in Industry - Date: 2023-04-01 - Page: https://www.nikivanstein.nl/publication/2023-04-01-Evaluation-of-deep-unsupervised-anomaly-detection-methods-with-a-data-centric-approach-for-on-line-inspection - DOI: https://doi.org/10.1016/j.compind.2023.103852 - PDF: https://www.sciencedirect.com/science/article/pii/S0166361523000027 ## DoE2Vec: Deep-learning Based Features for Exploratory Landscape Analysis - Authors: Niki van Stein, Fu Xing Long, Moritz Frenzel, Peter Krause, Markus Gitterle, Thomas Bäck - Venue: Preprint on arXiv preprint arXiv:2304.01219 - Date: 2023-03-31 - Page: https://www.nikivanstein.nl/publication/2023-03-31-DoE2Vec-Deep-learning-Based-Features-for-Exploratory-Landscape-Analysis - DOI: https://doi.org/10.1145/3583133.3590609 - arXiv: https://arxiv.org/abs/2304.01219 - PDF: https://arxiv.org/pdf/2304.01219 We propose DoE2Vec, a variational autoencoder (VAE)-based methodology to learn optimization landscape characteristics for downstream meta-learning tasks, e.g., automated selection of optimization algorithms. Principally, using large training data sets generated with a random function generator, DoE2Vec self-learns an informative latent representation for any design of experiments (DoE). Unlike the classical exploratory landscape analysis (ELA) method, our approach does not require any feature engineering and is easily applicable for high dimensional search spaces. For validation, we inspect the quality of latent reconstructions and analyze the latent representations using different experiments. The latent representations not only show promising potentials in identifying similar (cheap-to-evaluate) surrogate functions, but also can significantly boost performances when being used complementary to the classical ELA features in classification tasks. ## A data-centric approach to anomaly detection in layer-based additive manufacturing - Authors: Alexander Zeiser, Bekir Özcan, Christoph Kracke, Niki van Stein, Thomas Bäck - Venue: at-Automatisierungstechnik - Date: 2023-01-13 - Page: https://www.nikivanstein.nl/publication/2023-01-13-A-data-centric-approach-to-anomaly-detection-in-layer-based-additive-manufacturing - DOI: https://doi.org/10.1515/auto-2022-0104 - PDF: https://scholarlypublications.universiteitleiden.nl/download/1887/3732015/1?col=81522199340002711 Anomaly detection describes methods of finding abnormal states, instances or data points that differ from a normal value space. Industrial processes are a domain where predicitve models are needed for finding anomalous data instances for quality enhancement. A main challenge, however, is absence of labels in this environment. This paper contributes to a data-centric way of approaching artificial intelligence in industrial production. With a use case from additive manufacturing for automotive components we present a deep-learning-based image processing pipeline. We integrate the concept of domain randomisation and synthetic data in the loop that shows promising results for bridging advances in deep learning and its application to real-world, industrial production processes. ## Multi-surrogate Assisted Efficient Global Optimization for Discrete Problems - Authors: Qi Huang, Roy De Winter, Niki van Stein, Thomas Bäck, Anna V. Kononova - Venue: Preprint on arXiv preprint arXiv:2212.06438 - Date: 2022-12-13 - Page: https://www.nikivanstein.nl/publication/2022-12-13-Multi-surrogate-Assisted-Efficient-Global-Optimization-for-Discrete-Problems - DOI: https://doi.org/10.1109/ssci51031.2022.10022132 - arXiv: https://arxiv.org/abs/2212.06438 - PDF: https://arxiv.org/pdf/2212.06438 Decades of progress in simulation-based surrogate-assisted optimization and unprecedented growth in computational power have enabled researchers and practitioners to optimize previously intractable complex engineering problems. This paper investigates the possible benefit of a concurrent utilization of multiple simulation-based surrogate models to solve complex discrete optimization problems. To fulfill this, the so-called Self-Adaptive Multi-surrogate Assisted Efficient Global Optimization algorithm (SAMA-DiEGO), which features a two-stage online model management strategy, is proposed and further benchmarked on fifteen binary-encoded combinatorial and fifteen ordinal problems against several state-of-the-art non-surrogate or single surrogate assisted optimization algorithms. Our findings indicate that SAMA-DiEGO can rapidly converge to better solutions on a majority of the test problems, which shows the feasibility and advantage of using multiple surrogate models in optimizing discrete problems. ## GSAreport: Easy to Use Global Sensitivity Reporting - Authors: Niki van Stein, Elena Raponi - Venue: Journal of Open Source Software - Date: 2022-10-14 - Page: https://www.nikivanstein.nl/publication/2022-10-14-GSAreport-Easy-to-Use-Global-Sensitivity-Reporting - DOI: https://doi.org/10.21105/joss.04721 ## A comparison of global sensitivity analysis methods for explainable AI with an application in genomic prediction - Authors: Niki van Stein, Elena Raponi, Zahra Sadeghi, Niek Bouman, Roeland C. H. J. Van Ham, Thomas Bäck - Venue: IEEE Access - Date: 2022-08-29 - Page: https://www.nikivanstein.nl/publication/2022-08-29-A-comparison-of-global-sensitivity-analysis-methods-for-explainable-AI-with-an-application-in-genomic-prediction - DOI: https://doi.org/10.1109/access.2022.3210175 - PDF: https://ieeexplore.ieee.org/ielx7/6287639/6514899/09903639.pdf Explainable Artificial Intelligence (XAI) is an increasingly important field of research required to bring AI to the next level in real-world applications. Global sensitivity analysis methods play an important role in XAI, as they can provide an understanding of which (groups of) parameters have high influence in the predictions of machine learning models and the output of simulators and real-world processes. In this paper, we conduct a survey into global sensitivity methods in an XAI context and present both a qualitative and a quantitative analysis of these methods under different conditions. In addition to the overview and comparison, we propose an open source application, GSAreport, that allows you to easily generate extensive reports using a carefully selected set of global sensitivity analysis methods depending on the number of dimensions and samples, to gain a deep understanding of the role of each feature for a given model or data set.We finally present the methods discussed in a complex real-world application of genomic prediction and draw conclusions about when to use which GSA methods. ## BIAS: a toolbox for benchmarking structural bias in the continuous domain - Authors: Diederick Vermetten, Niki van Stein, Fabio Caraffini, Leandro L. Minku, Anna V. Kononova - Venue: IEEE Transactions on Evolutionary Computation - Date: 2022-07-13 - Page: https://www.nikivanstein.nl/publication/2022-07-13-BIAS-a-toolbox-for-benchmarking-structural-bias-in-the-continuous-domain - DOI: https://doi.org/10.1109/tevc.2022.3189848 Benchmarking heuristic algorithms is vital to understand under which conditions and on what kind of problems certain algorithms perform well. Most benchmarks are performance based, to test algorithm performance under a wide set of conditions. There is also resource- and behavior-based benchmarks to test the resource consumption and the behavior of algorithms. In this article, we propose a novel behavior-based benchmark toolbox: BIAS (Bias in algorithms, structural). This toolbox can detect structural bias (SB) per dimension and across dimension-based on 39 statistical tests. Moreover, it predicts the type of SB using a random forest model. BIAS can be used to better understand and improve existing algorithms (removing bias) as well as to test novel algorithms for SB in an early phase of development. Experiments with a large set of generated SB scenarios show that BIAS was successful in identifying bias. In addition, we also provide the results of BIAS on 432 existing state-of-the-art optimization algorithms showing that different kinds of SB are present in these algorithms, mostly toward the center of the objective space or showing discretization behavior. The proposed toolbox is made available open-source and recommendations are provided for the sample size and hyper-parameters to be used when applying the toolbox on other algorithms. ## Using structural bias to analyse the behaviour of modular CMA-ES - Authors: Diederick Vermetten, Fabio Caraffini, Niki van Stein, Anna V. Kononova - Venue: Proceedings of the Genetic and Evolutionary Computation Conference Companion - Date: 2022-07-09 - Page: https://www.nikivanstein.nl/publication/2022-07-09-Using-structural-bias-to-analyse-the-behaviour-of-modular-CMA-ES - DOI: https://doi.org/10.1145/3520304.3534035 - PDF: https://dl.acm.org/doi/pdf/10.1145/3520304.3534035 The Covariance Matrix Adaptation Evolution Strategy (CMA-ES) is a commonly used iterative optimisation heuristic for optimising black-box functions. CMA-ES comes in many flavours with different configuration settings. In this work, we investigate whether CMA-ES suffers from structural bias and which modules and parameters affect the strength and type of structural bias. Structural bias occurs when an algorithm or a component of the algorithm biases the search towards a specific direction in the search space irrespective of the objective function. In addition to this investigation, we propose a method to assess the relationship between structural bias and the performance of configurations with different types of bias on the BBOB suite of benchmark functions. Surprisingly for such a popular algorithm, 90.3% of the 1 620 CMA-ES configurations were found to have Structural Bias. Some interesting patterns between module settings and bias types are presented and further insights are discussed. ## Multi-point acquisition function for constraint parallel efficient multi-objective optimization - Authors: Roy de Winter, Niki van Stein, Thomas Bäck - Venue: Proceedings of the Genetic and Evolutionary Computation Conference - Date: 2022-07-08 - Page: https://www.nikivanstein.nl/publication/2022-07-08-Multi-point-acquisition-function-for-constraint-parallel-efficient-multi-objective-optimization - DOI: https://doi.org/10.1145/3512290.3528696 - PDF: https://dl.acm.org/doi/pdf/10.1145/3512290.3528696 Bayesian optimization is often used to optimize expensive black box optimization problems with long simulation times. Typically Bayesian optimization algorithms propose one solution per iteration. The downside of this strategy is the sub-optimal use of available computing power. To efficiently use the available computing power (or a number of licenses etc.) we introduce a multi-point acquisition function for parallel efficient multi-objective optimization algorithms. The multi-point acquisition function is based on the hypervolume contribution of multiple solutions simultaneously, leading to well spread solutions along the Pareto frontier. By combining this acquisition function with a constraint handling technique, multiple feasible solutions can be proposed and evaluated in parallel every iteration. The hypervolume and feasibility of the solutions can easily be estimated by using multiple cheap radial basis functions as surrogates with different configurations. The acquisition function can be used with different population sizes and even for one shot optimization. The strength and generalizability of the new acquisition function is demonstrated by optimizing a set of black box constraint multi-objective problem instances. The experiments show a huge time saving factor by using our novel multi-point acquisition function, while only marginally worsening the hypervolume after the same number of function evaluations. ## Learning the characteristics of engineering optimization problems with applications in automotive crash - Authors: Fu Xing Long, Niki van Stein, Moritz Frenzel, Peter Krause, Markus Gitterle, Thomas Bäck - Venue: Proceedings of the Genetic and Evolutionary Computation Conference - Date: 2022-07-08 - Page: https://www.nikivanstein.nl/publication/2022-07-08-Learning-the-characteristics-of-engineering-optimization-problems-with-applications-in-automotive-crash - DOI: https://doi.org/10.1145/3512290.3528712 - PDF: https://dl.acm.org/doi/pdf/10.1145/3512290.3528712 Oftentimes the characteristics of real-world engineering optimization problems are not well understood. In this paper, we introduce an approach for characterizing highly nonlinear and Finite Element (FE) simulation-based engineering optimization problems, focusing on ten representative problem instances from automotive crashworthiness optimization. By computing characteristic Exploratory Landscape Analysis (ELA) features, we show that these ten crashworthiness problem instances exhibit landscape features different from classical optimization benchmark test suites, such as the widely-used Black-Box Optimization Benchmarking (BBOB) problem set. Using clustering approaches, we demonstrate that these ten problem instances are clearly distinct from the BBOB test functions. Further analysis of the crashworthiness problem instances reveal that, as far as ELA concerns, they are most similar to a class of artificially generated functions. We identify such artificially generated functions and propose to use them as scalable and fast-to-evaluate representatives of the real-world problems. Such artificially generated functions could be used for the automated design of an optimization algorithm for specific real-world problem classes. ## An end-to-end pipeline for uncertainty quantification and remaining useful life estimation: An application on aircraft engines - Authors: Marios Kefalas, Niki van Stein, Mitra Baratchi, Asteris Apostolidis, Thomas Bäck - Venue: PHM Society European Conference - Date: 2022-06-29 - Page: https://www.nikivanstein.nl/publication/2022-06-29-An-end-to-end-pipeline-for-uncertainty-quantification-and-remaining-useful-life-estimation-An-application-on-aircraft-engines - DOI: https://doi.org/10.36001/phme.2022.v7i1.3317 - PDF: https://papers.phmsociety.org/index.php/phme/article/download/3317/1929 Estimating the remaining useful life (RUL) of an asset lies at the heart of prognostics and health management (PHM) of many operations-critical industries such as aviation. Modern methods of RUL estimation adopt techniques from deep learning (DL). However, most of these contemporary techniques deliver only single-point estimates for the RUL without reporting on the confidence of the prediction. This practice usually provides overly confident predictions that can have severe consequences in operational disruptions or even safety. To address this issue, we propose a technique for uncertainty quantification (UQ) based on Bayesian deep learning (BDL). The hyperparameters of the framework are tuned using a novel bi-objective Bayesian optimization method with objectives the predictive performance and predictive uncertainty. The method also integrates the data pre-processing steps into the hyperparameter optimization (HPO) stage, models the RUL as a Weibull distribution, and returns the survival curves of the monitored assets to allow informed decision-making. We validate this method on the widely used C-MAPSS dataset against a single-objective HPO baseline that aggregates the two objectives through the harmonic mean (HM). We demonstrate the existence of trade-offs between the predictive performance and the predictive uncertainty and observe that the bi-objective HPO returns a larger number of hyperparameter configurations compared to the single-objective baseline. Furthermore, we see that with the proposed approach, it is possible to configure models for RUL estimation that exhibit better or comparable performance to the single-objective baseline when validated on the test sets. ## Explainable artificial intelligence for exhaust gas temperature of turbofan engines - Authors: Marios Kefalas, Juan de Santiago Rojo, Asteris Apostolidis, Dirk van den Herik, Niki van Stein, Thomas Bäck - Venue: Journal of Aerospace Information Systems - Date: 2022-06-01 - Page: https://www.nikivanstein.nl/publication/2022-06-01-Explainable-artificial-intelligence-for-exhaust-gas-temperature-of-turbofan-engines - DOI: https://doi.org/10.2514/1.i011058 - arXiv: https://arxiv.org/abs/2203.13108 - PDF: https://arxiv.org/pdf/2203.13108 Data-driven modeling is an imperative tool in various industrial applications, including many applications in the sectors of aeronautics and commercial aviation. These models are in charge of providing key insights, such as which parameters are important on a specific measured outcome or which parameter values we should expect to observe given a set of input parameters. At the same time, however, these models rely heavily on assumptions (e.g., stationarity) or are "black box" (e.g., deep neural networks), meaning that they lack interpretability of their internal working and can be viewed only in terms of their inputs and outputs. An interpretable alternative to the "black box" models and with considerably less assumptions is symbolic regression (SR). SR searches for the optimal model structure while simultaneously optimizing the model's parameters without relying on an a-priori model structure. In this work, we apply SR on real-life exhaust gas temperature (EGT) data, collected at high frequencies through the entire flight, in order to uncover meaningful algebraic relationships between the EGT and other measurable engine parameters. The experimental results exhibit promising model accuracy, as well as explainability returning an absolute difference of 3°C compared to the ground truth and demonstrating consistency from an engineering perspective. ## Constrained multi-objective optimization with a limited budget of function evaluations - Authors: Roy de Winter, Philip Bronkhorst, Niki van Stein, Thomas Bäck - Venue: Memetic Computing - Date: 2022-04-08 - Page: https://www.nikivanstein.nl/publication/2022-04-08-Constrained-multi-objective-optimization-with-a-limited-budget-of-function-evaluations - DOI: https://doi.org/10.1007/s12293-022-00363-y - PDF: https://link.springer.com/content/pdf/10.1007/s12293-022-00363-y.pdf This paper proposes the Self-Adaptive algorithm for Multi-Objective Constrained Optimization by using Radial Basis Function Approximations, SAMO-COBRA. This algorithm automatically determines the best Radial Basis Function-fit as surrogates for the objectives as well as the constraints, to find new feasible Pareto-optimal solutions. SAMO-COBRA is compared to a wide set of other state-of-the-art algorithms (IC-SA-NSGA-II, SA-NSGA-II, NSGA-II, NSGA-III, CEGO, SMES-RBF) on 18 constrained multi-objective problems. In the first experiment, SAMO-COBRA outperforms the other algorithms in terms of achieved Hypervolume (HV) after being given a fixed small evaluation budget on the majority of test functions. In the second experiment, SAMO-COBRA outperforms the majority of competitors in terms of required function evaluations to achieve $$95\%$$ 95 % of the maximum achievable Hypervolume. In addition to academic test functions, SAMO-COBRA has been applied on a real-world ship design optimization problem with three objectives, two complex constraints, and five decision variables. ## Optimally weighted ensembles for efficient multi-objective optimization - Authors: Gideon Hanse, Roy de Winter, Niki van Stein, Thomas Bäck - Venue: International Conference on Machine Learning, Optimization, and Data Science. Springer - Date: 2022-02-02 - Page: https://www.nikivanstein.nl/publication/2022-02-02-Optimally-weighted-ensembles-for-efficient-multi-objective-optimization - DOI: https://doi.org/10.1007/978-3-030-95467-3_12 - PDF: https://scholarlypublications.universiteitleiden.nl/download/1887/3279362/1?col=81522199450002711 ## Analysis of structural bias in differential evolution configurations - Authors: Diederick Vermetten, Niki van Stein, Anna V. Kononova, Fabio Caraffini - Venue: Differential Evolution: From Theory to Practice - Date: 2022-01-25 - Page: https://www.nikivanstein.nl/publication/2022-01-25-Analysis-of-structural-bias-in-differential-evolution-configurations - DOI: https://doi.org/10.1007/978-981-16-8082-3_1 ## Using Machine Learning to Detect Rotational Symmetries from Reflectional Symmetries in 2D Images - Authors: Koen Ponse, Anna V. Kononova, Maria Loleyt, Niki van Stein - Venue: 2021 IEEE Symposium Series on Computational Intelligence (SSCI) - Date: 2021-12-05 - Page: https://www.nikivanstein.nl/publication/2021-12-05-Using-Machine-Learning-to-Detect-Rotational-Symmetries-from-Reflectional-Symmetries-in-2D-Images - DOI: https://doi.org/10.1109/ssci50451.2021.9660120 - arXiv: https://arxiv.org/abs/2201.06594 - PDF: https://arxiv.org/pdf/2201.06594 Automated symmetry detection is still a difficult task in 2021. However, it has applications in computer vision, and it also plays an important part in understanding art. This paper focuses on aiding the latter by comparing different state-of-the-art automated symmetry detection algorithms. For one of such algorithms aimed at reflectional symmetries, we propose post-processing improvements to find localised symmetries in images, improve the selection of detected symmetries and identify another symmetry type (rotational). In order to detect rotational symmetries, we contribute a machine learning model which detects rotational symmetries based on provided reflection symmetry axis pairs. We demonstrate and analyze the performance of the extended algorithm to detect localised symmetries and the machine learning model to classify rotational symmetries. ## Exploiting generative models for performance predictions of 3D car designs - Authors: Sneha Saha, Thiago Rios, Leandro L. Minku, Niki van Stein, Patricia Wollstadt, Xin Yao, Thomas Bäck, Bernhard Sendhoff, Stefan Menzel - Venue: 2021 IEEE Symposium Series on Computational Intelligence (SSCI) - Date: 2021-12-05 - Page: https://www.nikivanstein.nl/publication/2021-12-05-Exploiting-generative-models-for-performance-predictions-of-3D-car-designs - DOI: https://doi.org/10.1109/ssci50451.2021.9660034 - PDF: https://scholarlypublications.universiteitleiden.nl/download/1887/3277029/1?col=81522199450002711 In automotive digital development, engineers utilize multiple virtual prototyping tools to design and assess the performance of 3D shapes. However, accurate performance simulations are computationally expensive and time-consuming, which may be prohibitive for design optimization tasks. To address this challenge, we envision a 3D design assistance system for design exploration with performance assessment in the automotive domain. Recent advances in deep learning methods for learning geometric data are a promising step towards realizing such systems. Deep learning-based (variational) autoencoder models have been used for learning and compressing 3D data allowing engineers to generate low-dimensional representations of 3D designs. Finding representations in a data-driven fashion results in representations that are agnostic to downstream tasks performed on these representations and are believed to capture relevant design features. In this paper, we evaluate whether such data-driven representations contain relevant information about the input data and whether representations are meaningful in performance prediction tasks for the input data. We use machine learning-based surrogate models to predict the performances of car shapes based on the low-dimensional representation learned by 3D point cloud (variational) autoencoders. Furthermore, we exploit the stochastic nature of the representation learned by variational autoencoders to augment the training data for our surrogate models, since the limited amount of data is usually a challenge for surrogate modeling in engineering. We demonstrate that augmenting training with generated shapes improves prediction accuracy. In sum, we find that geometric deep learning approaches offer powerful tools to support the engineering design process. ## Point2FFD: learning shape representations of simulation-ready 3D models for engineering design optimization - Authors: Thiago Rios, Niki van Stein, Thomas Bäck, Bernhard Sendhoff, Stefan Menzel - Venue: 2021 International Conference on 3D Vision (3DV) - Date: 2021-12-01 - Page: https://www.nikivanstein.nl/publication/2021-12-01-Point2FFD-learning-shape-representations-of-simulation-ready-3D-models-for-engineering-design-optimization - DOI: https://doi.org/10.1109/3dv53792.2021.00110 - PDF: https://scholarlypublications.universiteitleiden.nl/download/1887/3280044/1?col=81522199340002711 Methods for learning on 3D point clouds became ubiquitous due to the popularization of 3D scanning technology and advances of machine learning techniques. Among these methods, point-based deep neural networks have been utilized to explore 3D designs in optimization tasks. However, engineering computer simulations require high-quality meshed models, which are challenging to automatically generate from unordered point clouds. In this work, we propose Point2FFD: A novel deep neural network for learning compact geometric representations and generating simulation-ready meshed models. Built upon an autoencoder architecture, Point2FFD learns to compress 3D point clouds into a latent design space, from which the network generates 3D polygonal meshes by selecting and deforming simulation-ready mesh templates. Through benchmark experiments, we show that our proposed network achieves comparable shape-generative performance than existing state-of-the-art point-based generative models. In real world-inspired vehicle aerodynamic optimizations, we demonstrate that Point2FFD generates simulation-ready meshes of realistic car shapes and leads to better optimized designs than the benchmarked networks. ## Emergence of structural bias in differential evolution - Authors: Niki van Stein, Fabio Caraffini, Anna V. Kononova - Venue: Proceedings of the Genetic and Evolutionary Computation Conference Companion - Date: 2021-07-07 - Page: https://www.nikivanstein.nl/publication/2021-07-07-Emergence-of-structural-bias-in-differential-evolution - DOI: https://doi.org/10.1145/3449726.3463223 - arXiv: https://arxiv.org/abs/2105.04693 - PDF: https://arxiv.org/pdf/2105.04693 Heuristic optimisation algorithms are in high demand due to the overwhelming amount of complex optimisation problems that need to be solved. The complexity of these problems is well beyond the boundaries of applicability of exact optimisation algorithms and therefore require modern heuristics to find feasible solutions quickly. These heuristics and their effects are almost always evaluated and explained by particular problem instances. In previous works, it has been shown that many such algorithms show structural bias, by either being attracted to a certain region of the search space or by consistently avoiding regions of the search space, on a special test function designed to ensure uniform 'exploration' of the domain. In this paper, we analyse the emergence of such structural bias for Differential Evolution (DE) configurations and, specifically, the effect of different mutation, crossover and correction strategies. We also analyse the emergence of the structural bias during the run-time of each algorithm. We conclude with recommendations of which configurations should be avoided in order to run DE unbiased. ## Exploiting local geometric features in vehicle design optimization with 3D point cloud autoencoders - Authors: Thiago Rios, Niki van Stein, Patricia Wollstadt, Thomas Bäck, Bernhard Sendhoff, Stefan Menzel - Venue: 2021 IEEE Congress on Evolutionary Computation (CEC) - Date: 2021-06-28 - Page: https://www.nikivanstein.nl/publication/2021-06-28-Exploiting-local-geometric-features-in-vehicle-design-optimization-with-3D-point-cloud-autoencoders - DOI: https://doi.org/10.1109/cec45853.2021.9504746 - PDF: https://scholarlypublications.universiteitleiden.nl/download/1887/3277026/1?col=81522199340002711 Methods for learning and compressing high-dimensional data allow designers to generate novel and low-dimensional design representations for shape optimization problems. By using compact design spaces, global optimization algorithms require less function evaluations to characterize the problem landscape. Furthermore, data-driven representations are often domain-agnostic and independent of the user expertise, and thus potentially capture more relevant design features than a human designer would suggest. However, more factors than the dimensionality play a role in the efficiency of design representations. In this paper, we perform a comparative analysis of design representations for 3D shape optimization problems obtained with principal component analysis, kernel-principal component analysis and a 3D point cloud autoencoder, which we apply on a benchmark data set of computer aided engineering car models. We evaluate the shape-generative capabilities of these methods and show that we can modify the geometries more locally with the autoencoder than with the remaining methods. In a vehicle aerodynamic optimization framework, we verify that this property of the autoencoder representation improves the optimization performance by enabling potentially complementary degrees of freedom for the optimizer. With our study, we provide insights on the qualitative properties and quantifiable measures on the efficiency of deep neural networks as shape generative models for engineering optimization problems, as well as analyses of geometric representations for engineering optimization with evolutionary algorithms. ## Multitask Shape Optimization Using a 3-D Point Cloud Autoencoder as Unified Representation - Authors: Thiago Rios, Niki van Stein, Thomas Bäck, Bernhard Sendhoff, Stefan Menzel - Venue: IEEE Transactions on Evolutionary Computation - Date: 2021-06-03 - Page: https://www.nikivanstein.nl/publication/2021-06-03-Multitask-Shape-Optimization-Using-a-3-D-Point-Cloud-Autoencoder-as-Unified-Representation - DOI: https://doi.org/10.1109/tevc.2021.3086308 - PDF: https://ieeexplore.ieee.org/ielx7/4235/9745105/09446541.pdf The choice of design representations, as of search operators, is central to the performance of evolutionary optimization algorithms, in particular, for multitask problems. The multitask approach pushes further the parallelization aspect of these algorithms by solving simultaneously multiple optimization tasks using a single population. During the search, the operators implicitly transfer knowledge between solutions to the offspring, taking advantage of potential synergies between problems to drive the solutions to optimality. Nevertheless, in order to operate on the individuals, the design space of each task has to be mapped to a common search space, which is challenging in engineering cases without clear semantic overlap between parameters. Here, we apply a 3-D point cloud autoencoder to map the representations from the Cartesian to a unified design representation: the latent space of the autoencoder. The transfer of latent space features between design representations allows the reconstruction of shapes with interpolated characteristics and maintenance of common parts, which potentially improves the performance of the designs in one or more tasks during the optimization. Compared to traditional representations for shape optimization, such as free-form deformation, the latent representation enables more representative design modifications, while keeping the baseline characteristics of the learned classes of objects. We demonstrate the efficiency of our approach in an optimization scenario where we minimize the aerodynamic drag of two different car shapes with common underbodies for cost-efficient vehicle platform design. ## Requirements towards optimizing analytics in industrial processes - Authors: Alexander Zeiser, Niki van Stein, Thomas Bäck - Venue: Procedia Computer Science - Date: 2021-05-18 - Page: https://www.nikivanstein.nl/publication/2021-05-18-Requirements-towards-optimizing-analytics-in-industrial-processes - DOI: https://doi.org/10.1016/j.procs.2021.03.074 - PDF: https://www.sciencedirect.com/science/article/pii/S1877050921007079/pdf Modern production systems are composed of complex manufacturing processes with highly technology specific cause-effect relationships. Developments in sensor technology and computational science allow for data-driven decision making that facilitate effcient and objective production management. However, process data may only be beneficial if it is enriched with meta information and process expertise, reduced to relevant information and modelling results interpreted correctly. The importance of data integration in the heterogeneous industrial environment rises at the same momentum as new metrology techniques are deployed. In this paper, we focus on optimizing analytics, containing data-driven decision making for predictive quality and maintenance. We summarize key requirements for data analytics and machine learning application in industrial processes. With a use case from automotive component manufacturing we characterize industrial production, categorize process data and put requirements in context to a real-world example. ## Samo-cobra: A fast surrogate assisted constrained multi-objective optimization algorithm - Authors: Roy de Winter, Niki van Stein, Thomas Bäck - Venue: Evolutionary Multi-Criterion Optimization: 11th International Conference, EMO 2021, Shenzhen, China, March 28--31, 2021, Proceedings 11 - Date: 2021-03-24 - Page: https://www.nikivanstein.nl/publication/2021-03-24-Samo-cobra-A-fast-surrogate-assisted-constrained-multi-objective-optimization-algorithm - DOI: https://doi.org/10.1007/978-3-030-72062-9_22 - PDF: https://scholarlypublications.universiteitleiden.nl/download/1887/3158667/1?col=81522199450002711 ## Ship design performance and cost optimization with machine learning - Authors: R Winter, Niki van Stein, THW Bäck, V Bertram - Venue: COMPIT'21 - Date: 2021-01-01 - Page: https://www.nikivanstein.nl/publication/2021-01-01-Ship-design-performance-and-cost-optimization-with-machine-learning ## Neural network design: learning from neural architecture search - Authors: Niki van Stein, Hao Wang, Thomas Bäck - Venue: 2020 IEEE Symposium Series on Computational Intelligence (SSCI) - Date: 2020-12-01 - Page: https://www.nikivanstein.nl/publication/2020-12-01-Neural-network-design-learning-from-neural-architecture-search - DOI: https://doi.org/10.1109/ssci47803.2020.9308394 - arXiv: https://arxiv.org/abs/2011.00521 - PDF: https://arxiv.org/pdf/2011.00521 Neural Architecture Search (NAS) aims to optimize deep neural networks' architecture for better accuracy or smaller computational cost and has recently gained more research interests. Despite various successful approaches proposed to solve the NAS task, the landscape of it, along with its properties, are rarely investigated. In this paper, we argue for the necessity of studying the landscape property thereof and propose to use the so-called Exploratory Landscape Analysis (ELA) techniques for this goal. Taking a broad set of designs of the deep convolutional network, we conduct extensive experimentation to obtain their performance. Based on our analysis of the experimental results, we observed high similarities between well-performing architecture designs, which is then used to significantly narrow the search space to improve the efficiency of any NAS algorithm. Moreover, we extract the ELA features over the NAS landscapes on three common image classification data sets, MNIST, Fashion, and CIFAR-10, which shows that the NAS landscape can be distinguished for those three data sets. Also, when comparing to the ELA features of the well-known Black-Box Optimization Benchmarking (BBOB) problem set, we found out that the NAS landscapes surprisingly form a new problem class on its own, which can be separated from all $24$ BBOB problems. Given this interesting observation, we, therefore, state the importance of further investigation on selecting an efficient optimizer for the NAS landscape as well as the necessity of augmenting the current benchmark problem set. ## Back to meshes: Optimal simulation-ready mesh prototypes for autoencoder-based 3D car point clouds - Authors: Thiago Rios, Jiawen Kong, Niki van Stein, Thomas Bäck, Patricia Wollstadt, Bernhard Sendhoff, Stefan Menzel - Venue: 2020 IEEE Symposium Series on Computational Intelligence (SSCI) - Date: 2020-12-01 - Page: https://www.nikivanstein.nl/publication/2020-12-01-Back-to-meshes-Optimal-simulation-ready-mesh-prototypes-for-autoencoder-based-3D-car-point-clouds - DOI: https://doi.org/10.1109/ssci47803.2020.9308400 - PDF: https://scholarlypublications.universiteitleiden.nl/download/1887/3766136/1?col=81522199340002711 Point cloud autoencoders were recently introduced as powerful models for data compression. They learn a lowdimensional set of variables that are suitable as design parameters for shape generation and optimization problems. In engineering tasks, 3D point clouds are often derived from fine polygon meshes, which are the most suitable representations for physics simulation, e.g., computational fluid dynamics (CFD). Yet, the reconstruction of high-quality meshes from autoencoder-based point clouds is challenging, often requiring supervised and manual work, which is prohibitive during the optimization. Target shape matching optimization using existing mesh prototypes overcomes the difficulties of recovering shape information from the point coordinates. However, for autoencoders trained on data sets comprising shapes with high degree of dissimilarity, there is not a single mesh prototype that can fit any autoencoder-based point cloud, and the selection of a set of prototypes is nontrivial. In the present paper we propose a method for optimizing a selection of prototypical meshes to match the maximum number of shapes in the autoencoder output space as possible, which is achieved by linking the advantages of the latent space representation of an autoencoder and the state-of-the-art free form deformation (FFD) method. Furthermore, we approached the balance between costs (number of mesh prototypes) and number of covered shapes by varying the number of prototypes and the dimensionality of the autoencoder latent space, showing that higher-dimensional latent spaces encode finer geometric changes, requiring more sophisticated FFD setups. ## A tailored NSGA-III for multi-objective flexible job shop scheduling - Authors: Yali Wang, Niki van Stein, Thomas Bäck, Michael Emmerich - Venue: 2020 IEEE Symposium Series on Computational Intelligence (SSCI) - Date: 2020-12-01 - Page: https://www.nikivanstein.nl/publication/2020-12-01-A-tailored-NSGA-III-for-multi-objective-flexible-job-shop-scheduling - DOI: https://doi.org/10.1109/ssci47803.2020.9308373 - PDF: https://scholarlypublications.universiteitleiden.nl/download/1887/3753471/1?col=81522199340002711 A customized multi-objective evolutionary algorithm (MOEA) is proposed for the flexible job shop scheduling problem (FJSP) with three objectives: makespan, total workload, critical workload. In general, the algorithm can be integrated with any standard MOEA. In this paper, it has been combined with NSGA-III to solve the state-of-the-art benchmark FJSPs, whereas an off-the-shelf implementation of NSGA-III is not capable of solving them. Most importantly, we use the various algorithm adaptations to enhance the performance of our algorithm. To be specific, it uses smart initialization approaches to enrich the first-generation population, and proposes new crossover operator to create a better diversity on the Pareto front approximation. The MIP-EGO configurator is adopted to automatically tune the mutation probabilities, which are important hyper-parameters of the algorithm. Furthermore, different local search strategies are employed to explore the neighborhood for better solutions. The experimental results from the combination of these techniques show the good performance as compared to classical evolutionary scheduling algorithms and it requires less computing budget. Even some previously unknown non-dominated solutions for the BRdata benchmark problems could be discovered. ## Improving NSGA-III for flexible job shop scheduling using automatic configuration, smart initialization and local search - Authors: Yali Wang, Niki van Stein, Thomas Bäck, Michael Emmerich - Venue: Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion - Date: 2020-07-08 - Page: https://www.nikivanstein.nl/publication/2020-07-08-Improving-NSGA-III-for-flexible-job-shop-scheduling-using-automatic-configuration-smart-initialization-and-local-search - DOI: https://doi.org/10.1145/3377929.3389924 - PDF: https://scholarlypublications.universiteitleiden.nl/download/1887/3618523/1?col=81522199340002711 This paper provides a short summary of a novel algorithm tailored towards multi-objective flexible job shop scheduling problems (FJSP). The result shows that for challenging real-world problems in combinatorial optimization, off-the-shelf implementations of multi-objective optimization evolutionary algorithms (MOEAs) might not work, but by using various adaptations, these methods can be tailored to provide excellent results. This is demonstrated for a state of the art MOEA, that is NSGA-III, and the following adaptations: (1) initialization approaches to enrich the first-generation population, (2) various crossover operators to create a better diversity of offspring, (3) parameter tuning, to determine the optimal mutation probabilities, using the MIP-EGO configurator, (4) local search strategies to explore the neighborhood for better solutions. Using these measures, NSGA-III has been enabled to solve benchmark multi-objective FJSPs and experimental results show excellent performance. ## Feature visualization for 3D point cloud autoencoders - Authors: Thiago Rios, Niki van Stein, Stefan Menzel, Thomas Bäck, Bernhard Sendhoff, Patricia Wollstadt - Venue: 2020 International Joint Conference on Neural Networks (IJCNN) - Date: 2020-07-01 - Page: https://www.nikivanstein.nl/publication/2020-07-01-Feature-visualization-for-3D-point-cloud-autoencoders - DOI: https://doi.org/10.1109/ijcnn48605.2020.9207326 - PDF: https://scholarlypublications.universiteitleiden.nl/download/1887/4092794/1?col=81522199340002711 In order to reduce the dimensionality of 3D point cloud representations, autoencoder architectures generate increasingly abstract, compressed features of the input data. Visualizing these features is central to understanding the learning process, however, while successful visualization techniques exist for neural networks applied to computer vision tasks, similar methods for geometric, especially non-Euclidean, input data are currently lacking. Hence, we propose a first-of-kind method to project the features learned by point cloud autoencoders into a 3D-space augmented with color maps. Our proposal explores the properties of 1D-convolutions, used in state-of-the art point cloud autoencoder architectures to handle the input data, which leads to an intuitive interpretation of the visualized features. Furthermore, we tackle the search for relevant co-activations in the feature space by clustering the input data in the latent space, where we explore the correspondence between network features and geometric characteristics of typical shapes of the clusters. We tested our approach with experiments on a benchmark data set, and with three different configurations of a point cloud autoencoder, where we show that the features learned by the autoencoder correlate with the occupancy of the input space by the training data. ## Scalability of learning tasks on 3D CAE models using point cloud autoencoders - Authors: Thiago Rios, Patricia Wollstadt, Niki van Stein, Thomas Bäck, Zhao Xu, Bernhard Sendhoff, Stefan Menzel - Venue: 2019 IEEE Symposium Series on Computational Intelligence (SSCI) - Date: 2019-12-01 - Page: https://www.nikivanstein.nl/publication/2019-12-01-Scalability-of-learning-tasks-on-3D-CAE-models-using-point-cloud-autoencoders - DOI: https://doi.org/10.1109/ssci44817.2019.9002982 - PDF: https://scholarlypublications.universiteitleiden.nl/download/1887/135538/1?col=81522199450002711 Geometric Deep Learning (GDL) methods have recently gained interest as powerful, high-dimensional models for approaching various geometry processing tasks. However, training deep neural network models on geometric input requires considerable computational effort. Even more so, if one considers typical problem sizes found in application domains such as engineering tasks, where geometric data are often orders of magnitude larger than the inputs currently considered in GDL literature. Hence, an assessment of the scalability of the training task is necessary, where model and data set parameters can be mapped to the computational demand during training. The present paper therefore studies the effects of data set size and the number of free model parameters on the computational effort of training a Point Cloud Autoencoder (PC-AE). We further review pre-processing techniques to obtain efficient representations of high-dimensional inputs to the PC-AE and investigate the effects of these techniques on the information abstracted by the trained model. We perform these experiments on synthetic geometric data inspired by engineering applications using computing hardware with particularly recent graphics processing units (GPUs) with high memory specifications. The present study thus provides a comprehensive evaluation of how to scale geometric deep learning architectures to high-dimensional inputs to allow for an application of state-of-the-art deep learning methods in real-world tasks. ## On the efficiency of a point cloud autoencoder as a geometric representation for shape optimization - Authors: Thiago Rios, Bernhard Sendhoff, Stefan Menzel, Thomas Bäck, Niki van Stein - Venue: 2019 IEEE Symposium Series on Computational Intelligence (SSCI) - Date: 2019-12-01 - Page: https://www.nikivanstein.nl/publication/2019-12-01-On-the-efficiency-of-a-point-cloud-autoencoder-as-a-geometric-representation-for-shape-optimization - DOI: https://doi.org/10.1109/ssci44817.2019.9003161 - PDF: https://scholarlypublications.universiteitleiden.nl/download/1887/135537/1?col=81522199450002711 A crucial step for optimizing a system is to formulate the objective function, and part of it concerns the selection of the design parameters. One of the major goals is to achieve a fair trade-off between exploring feasible solutions in the design space and maintaining admissible computational effort. In order to achieve such balance in optimization problems with Computer Aided Engineering (CAE) models, the conventional constructive geometric representations are substituted by deformation methods, e.g. free form deformation, where the position of a few control points might be capable of handling large scale shape modifications. In light of the recent developments in the field of geometric deep learning, autoencoders have risen as a promising alternative for efficiently condensing high-dimensional models into compact representations. In this paper, we present a novel perspective on geometric deep learning models by exploring the applicability of the latent space of a point cloud autoencoder in shape optimization problems with evolutionary algorithms. Focusing on engineering applications, a target shape matching optimization is used as a surrogate to the computationally expensive CAE simulations required in engineering optimizations. Through the quality assessment of the solutions achieved in the optimization and further aspects, such as shape feasibility, point cloud autoencoders showed to be consistent and suitable geometric representations for such problems, adding a new perspective on the approaches for handling high-dimensional models to optimization tasks. ## A new approach towards the combined algorithm selection and hyper-parameter optimization problem - Authors: Xin Guo, Niki van Stein, Thomas Bäck - Venue: 2019 IEEE Symposium Series on Computational Intelligence (SSCI) - Date: 2019-12-01 - Page: https://www.nikivanstein.nl/publication/2019-12-01-A-new-approach-towards-the-combined-algorithm-selection-and-hyper-parameter-optimization-problem - DOI: https://doi.org/10.1109/ssci44817.2019.9003174 Machine learning algorithms often have many hyper-parameters that can be tuned to improve empirical performance. However, manually exploring the complex search spaces is tedious and cannot guarantee to find satisfactory outcomes. Recently, the Efficient Global Optimization (EGO) algorithm for solving the hyper-parameter optimization problem have shown substantial improvements. However, these algorithms cannot easily be expanded to include the selection of an algorithm. Automatically determining and optimizing an algorithm is known as the combined algorithm selection and hyper-parameter optimization (CASH) problem. In this paper, a novel mixed integer efficient global optimization algorithm and its variants are proposed to solve the CASH problem efficiently. The proposed algorithm is compared with a wide set of state-of-the-art algorithms in both a Black Box Optimization and a CASH problem setting. For the CASH problem setup, seven machine learning algorithms are optimized for a large set of classification t asks. Results show that the proposed algorithms outperform on the black box optimization task and can also outperform alternative state-of-the-art approaches on the CASH problem for specific instances. ## Cluster-based Kriging approximation algorithms for complexity reduction - Authors: Niki van Stein, Hao Wang, Wojtek Kowalczyk, Michael Emmerich, Thomas Bäck - Venue: Applied Intelligence - Date: 2019-09-09 - Page: https://www.nikivanstein.nl/publication/2019-09-09-Cluster-based-Kriging-approximation-algorithms-for-complexity-reduction - DOI: https://doi.org/10.1007/s10489-019-01549-7 - arXiv: https://arxiv.org/abs/1702.01313 - PDF: https://arxiv.org/pdf/1702.01313 Kriging or Gaussian Process Regression is applied in many fields as a non-linear regression model as well as a surrogate model in the field of evolutionary computation. However, the computational and space complexity of Kriging, that is cubic and quadratic in the number of data points respectively, becomes a major bottleneck with more and more data available nowadays. In this paper, we propose a general methodology for the complexity reduction, called cluster Kriging, where the whole data set is partitioned into smaller clusters and multiple Kriging models are built on top of them. In addition, four Kriging approximation algorithms are proposed as candidate algorithms within the new framework. Each of these algorithms can be applied to much larger data sets while maintaining the advantages and power of Kriging. The proposed algorithms are explained in detail and compared empirically against a broad set of existing state-of-the-art Kriging approximation methods on a well-defined testing framework. According to the empirical study, the proposed algorithms consistently outperform the existing algorithms. Moreover, some practical suggestions are provided for using the proposed algorithms. ## Automatic configuration of deep neural networks with parallel efficient global optimization - Authors: Niki van Stein, Hao Wang, Thomas Bäck - Venue: 2019 International Joint Conference on Neural Networks (IJCNN) - Date: 2019-07-01 - Page: https://www.nikivanstein.nl/publication/2019-07-01-Automatic-configuration-of-deep-neural-networks-with-parallel-efficient-global-optimization - DOI: https://doi.org/10.1109/ijcnn.2019.8851720 Designing the architecture for an artificial neural network is a cumbersome task because of the numerous parameters to configure, including activation functions, layer types, and hyper-parameters. With the large number of parameters for most networks nowadays, it is intractable to find a good configuration for a given task by hand. In this paper the Mixed Integer Parallel Efficient Global Optimization (MIP-EGO) algorithm is proposed to automatically configure convolutional neural network architectures. It is shown that on several image classification tasks this approach is able to find competitive network architectures in terms of prediction accuracy, compared to the best hand-crafted ones in literature, when using only a fraction of the number of training epochs. Moreover, instead of the standard sequential evaluation in EGO, several candidate architectures are proposed and evaluated in parallel, which reduces the execution overhead significantly and leads to an efficient automation for deep neural network design. ## Designing ships using constrained multi-objective efficient global optimization - Authors: Roy de Winter, Niki van Stein, Matthys Dijkman, Thomas Bäck - Venue: Machine Learning, Optimization, and Data Science: 4th International Conference, LOD 2018, Volterra, Italy, September 13-16, 2018, Revised Selected Papers 4 - Date: 2019-02-14 - Page: https://www.nikivanstein.nl/publication/2019-02-14-Designing-ships-using-constrained-multi-objective-efficient-global-optimization - DOI: https://doi.org/10.1007/978-3-030-13709-0_16 ## A novel uncertainty quantification method for efficient global optimization - Authors: Niki van Stein, Hao Wang, Wojtek Kowalczyk, Thomas Bäck - Venue: Information Processing and Management of Uncertainty in Knowledge-Based Systems. Applications: 17th International Conference, IPMU 2018, C'adiz, Spain, June 11-15, 2018, Proceedings, Part III 17 - Date: 2018-05-18 - Page: https://www.nikivanstein.nl/publication/2018-05-18-A-novel-uncertainty-quantification-method-for-efficient-global-optimization - DOI: https://doi.org/10.1007/978-3-319-91479-4_40 ## A new acquisition function for Bayesian optimization based on the moment-generating function - Authors: Hao Wang, Niki van Stein, Michael Emmerich, Thomas Bäck - Venue: 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC) - Date: 2017-10-01 - Page: https://www.nikivanstein.nl/publication/2017-10-01-A-new-acquisition-function-for-Bayesian-optimization-based-on-the-moment-generating-function - DOI: https://doi.org/10.1109/smc.2017.8122656 Bayesian Optimization or Efficient Global Optimization (EGO) is a global search strategy that is designed for expensive black-box functions. In this algorithm, a statistical model (usually the Gaussian process model) is constructed on some initial data samples. The global optimum is approached by iteratively maximizing a so-called acquisition function, that balances the exploration and exploitation effect of the search. The performance of such an algorithm is largely affected by the choice of the acquisition function. Inspired by the usage of higher moments from the Gaussian process model, it is proposed to construct a novel acquisition function based on the moment-generating function (MGF) of the improvement, which is the stochastic gain over the current best fitness value by sampling at an unknown point. This MGF-based acquisition function takes all the higher moments into account and introduces an additional real-valued parameter to control the trade-off between exploration and exploitation. The motivation, rationale and closed-form expression of the proposed function are discussed in detail. In addition, we also illustrate its advantage over other acquisition functions, especially the so-called generalized expected improvement. ## A multi-method simulation of a high-frequency bus line - Authors: Thierry van der Spek, Niki van Stein, Marcel van der Holst, Thomas Bäck - Venue: 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC) - Date: 2017-10-01 - Page: https://www.nikivanstein.nl/publication/2017-10-01-A-multi-method-simulation-of-a-high-frequency-bus-line - DOI: https://doi.org/10.1109/itsc.2017.8317770 A methodology to model high-frequency bus lines is proposed and a realistic event-simulation model for such a line in the Netherlands is presented. This simulation model helps policy makers to predict changes that have to be made to bus routes and planned travel times before problems occur. With this model, different passenger growth scenarios can be easily evaluated. The model is validated using different key performance indicators, showing that under some model assumptions, it can realistically simulate real-life scenarios. The simulations workings are illustrated by a case study of passenger growth. ## Time complexity reduction in efficient global optimization using cluster kriging - Authors: Hao Wang, Niki van Stein, Michael Emmerich, Thomas Bäck - Venue: Proceedings of the Genetic and Evolutionary Computation Conference - Date: 2017-07-01 - Page: https://www.nikivanstein.nl/publication/2017-07-01-Time-complexity-reduction-in-efficient-global-optimization-using-cluster-kriging - DOI: https://doi.org/10.1145/3071178.3071321 Efficient Global Optimization (EGO) is an effective method to optimize expensive black-box functions and utilizes Kriging models (or Gaussian process regression) trained on a relatively small design data set. In real-world applications, such as experimental optimization, where a large data set is available, the EGO algorithm becomes computationally infeasible due to the time and space complexity of Kriging. Recently, the so-called Cluster Kriging methods have been proposed to reduce such complexities for the big data, where data sets are clustered and Kriging models are built on each cluster. Furthermore, Kriging models are combined in an optimal way for the prediction. In addition, we analyze the Cluster Kriging landscape to adopt the existing infill-criteria, e.g., the expected improvement. The approach is tested on selected global optimization problems. It is shown by the empirical studies that this approach significantly reduces the CPU time of the EGO algorithm while maintaining the convergence rate of the algorithm. ## Algorithm configuration data mining for cma evolution strategies - Authors: Sander van Rijn, Hao Wang, Niki van Stein, Thomas Bäck - Venue: Proceedings of the Genetic and Evolutionary Computation Conference - Date: 2017-07-01 - Page: https://www.nikivanstein.nl/publication/2017-07-01-Algorithm-configuration-data-mining-for-cma-evolution-strategies - DOI: https://doi.org/10.1145/3071178.3071205 In the past years, quite a number of algorithmic extensions of the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) have been proposed. These extensions define a large algorithm design space, but relatively little is known about the performance of most of these variations and the interaction between them. ## Towards data driven process control in manufacturing car body parts - Authors: Niki van Stein, Matthijs Van Leeuwen, Hao Wang, Stephan Purr, Sebastian Kreissl, Josef Meinhardt, Thomas Bäck - Venue: 2016 International Conference on Computational Science and Computational Intelligence (CSCI) - Date: 2016-12-01 - Page: https://www.nikivanstein.nl/publication/2016-12-01-Towards-data-driven-process-control-in-manufacturing-car-body-parts - DOI: https://doi.org/10.1109/csci.2016.0093 The manufacturing process of car body parts is a complex industrial process where many machine parameters and material measurements are involved in establishing the quality of the final product. Data driven models have shown great advantages in helping decision makers to optimize this kind of complex processes where good physical models are hard to build. In this paper a framework for on-line process monitoring and predictive modeling is proposed to optimize a car body part production process. Anomaly detection plays an important role in this framework as it can provide an early alert for operators on the production line using a complex set of machine parameters and material properties. In this paper an anomaly detection algorithm, Gloss, that is successfully implemented as the first module in the process, is introduced. Gloss finds local outliers in high dimensional mixed data-sets using a relative density measure that takes the global neighborhood into account while searching for outliers in subspaces of the data. An overview of the application and implementation of the algorithm in the car body part press shop is presented. ## Local subspace-based outlier detection using global neighbourhoods - Authors: Niki van Stein, Matthijs van Leeuwen, Thomas Bäck - Venue: 2016 IEEE International Conference on Big Data (Big Data) - Date: 2016-12-01 - Page: https://www.nikivanstein.nl/publication/2016-12-01-Local-subspace-based-outlier-detection-using-global-neighbourhoods - DOI: https://doi.org/10.1109/bigdata.2016.7840717 - arXiv: https://arxiv.org/abs/1611.00183 - PDF: https://arxiv.org/pdf/1611.00183 Outlier detection in high-dimensional data is a challenging yet important task, as it has applications in, e.g., fraud detection and quality control. State-of-the-art density-based algorithms perform well because they 1) take the local neighbourhoods of data points into account and 2) consider feature subspaces. In highly complex and high-dimensional data, however, existing methods are likely to overlook important outliers because they do not explicitly take into account that the data is often a mixture distribution of multiple components. We therefore introduce GLOSS, an algorithm that performs local subspace outlier detection using global neighbourhoods. Experiments on synthetic data demonstrate that GLOSS more accurately detects local outliers in mixed data than its competitors. Moreover, experiments on real-world data show that our approach identifies relevant outliers overlooked by existing methods, confirming that one should keep an eye on the global perspective even when doing local outlier detection. ## A framework for evaluating meta-models for simulation-based optimisation - Authors: Pepijn van Heiningen, Niki van Stein, Thomas Bäck - Venue: 2016 IEEE Symposium Series on Computational Intelligence (SSCI) - Date: 2016-12-01 - Page: https://www.nikivanstein.nl/publication/2016-12-01-A-framework-for-evaluating-meta-models-for-simulation-based-optimisation - DOI: https://doi.org/10.1109/ssci.2016.7850207 In order to evaluate complex and computationally expensive experiments, data-driven meta-models are used to replace costly experiments and approximate the real experiments' outcome. In this study, an evaluation framework is proposed for measuring the performance of these models. Explored is how the performance is related to the difference between the benchmark optimum and the model optimum, and a novel method to measure the performance of this `optima-fit' is proposed. The evaluation framework is presented using an experimental setup of four meta-modeling techniques (Decision Tree, Random Forests, Support Vector Regression and Kriging), which are systematically compared to each other. The techniques are fitted to eleven benchmarks, with dimensionality ranging from 2 to 32. The meta-models are trained with varying input sample sizes and sampling strategies. In addition, the relations between the performance to the various sampling strategies and sizes on these benchmarks are explored. Our novel designed metric can provide additional insight in the performance of a specific range of meta-models. ## Fuzzy clustering for optimally weighted cluster kriging - Authors: Niki van Stein, Hao Wang, Wojtek Kowalczyk, Michael Emmerich, Thomas Bäck - Venue: 2016 IEEE international conference on fuzzy systems (FUZZ-IEEE) - Date: 2016-07-01 - Page: https://www.nikivanstein.nl/publication/2016-07-01-Fuzzy-clustering-for-optimally-weighted-cluster-kriging - DOI: https://doi.org/10.1109/fuzz-ieee.2016.7737789 Kriging or Gaussian Process Regression has been successfully applied in many fields. One of the major bottlenecks of Kriging is the complexity in both processing time (cubic) and memory (quadratic) in the number of data points. To overcome these limitations, a variety of approximation algorithms have been proposed. One of these approximation algorithms is Optimally Weighted Cluster Kriging (OWCK). In this paper, OWCK is extended and enhanced by the use of fuzzy clustering methods in order to increase the accuracy. Several options are proposed and evaluated against both the original OWCK and a variety of other Kriging approximation algorithms. ## Analysis and visualization of missing value patterns - Authors: Niki van Stein, Wojtek Kowalczyk, Thomas Bäck - Venue: Information Processing and Management of Uncertainty in Knowledge-Based Systems: 16th International Conference, IPMU 2016, Eindhoven, The Netherlands, June 20-24, 2016, Proceedings, Part II 16 - Date: 2016-06-11 - Page: https://www.nikivanstein.nl/publication/2016-06-11-Analysis-and-visualization-of-missing-value-patterns - DOI: https://doi.org/10.1007/978-3-319-40581-0_16 ## An incremental algorithm for repairing training sets with missing values - Authors: Niki van Stein, Wojtek Kowalczyk - Venue: Information Processing and Management of Uncertainty in Knowledge-Based Systems: 16th International Conference, IPMU 2016, Eindhoven, The Netherlands, June 20-24, 2016, Proceedings, Part II 16 - Date: 2016-06-11 - Page: https://www.nikivanstein.nl/publication/2016-06-11-An-incremental-algorithm-for-repairing-training-sets-with-missing-values - DOI: https://doi.org/10.1007/978-3-319-40581-0_15 ## Optimally weighted cluster kriging for big data regression - Authors: Niki van Stein, Hao Wang, Wojtek Kowalczyk, Thomas Bäck, Michael Emmerich - Venue: Advances in Intelligent Data Analysis XIV: 14th International Symposium, IDA 2015, Saint Etienne. France, October 22-24, 2015. Proceedings - Date: 2015-11-22 - Page: https://www.nikivanstein.nl/publication/2015-11-22-Optimally-weighted-cluster-kriging-for-big-data-regression - DOI: https://doi.org/10.1007/978-3-319-24465-5_27 ## Fitness landscape analysis of nk landscapes and vehicle routing problems by expanded barrier trees - Authors: Niki van Stein, Michael Emmerich, Zhiwei Yang - Venue: EVOLVE-A Bridge between Probability, Set Oriented Numerics, and Evolutionary Computation IV: International Conference held at Leiden University, July 10-13, 2013 - Date: 2013-06-12 - Page: https://www.nikivanstein.nl/publication/2013-06-12-Fitness-landscape-analysis-of-nk-landscapes-and-vehicle-routing-problems-by-expanded-barrier-trees - DOI: https://doi.org/10.1007/978-3-319-01128-8_6