Article Sidebar

Download
PDF
Statistic
Read Counter : 0 Download : 0

Main Article Content

Abstract

More regulations require manufacturers to enhance material circularity and reduce their environmental footprint simultaneously; however, it remains unclear whether these two objectives align, and if so, in which domain. A discrete event simulation model for a remanufacturing plant operating multiple production lines with environmental performance measures at the process level is defined as a multiobjective Markov decision process, which is then solved using a preference-conditioned actor–critic technique, employing feasibility masking for the precedence and capacity constraints and adaptive sampling between the surrogate model and the full-scale simulator. Two life cycle accounting methods have been used: Boundary A (cut-off) and Boundary B (avoided burden). The environmental burden reveals three regimes, synergistic, trade-off, and inverted, characterized by transition points in secondary-material input levels at 0.38 and 0.61 for Boundary A and at 0.47 and 0.78 for Boundary B; sweeps with fixed ratios validate these transition points. One conditionally generated network accounts for 87.4% of the non-dominated solutions of eleven scalarized agents using 9.1% of the simulation budget allocated to them, and the switching strategy reduces the number of simulation calls needed to reach the reference hypervolume by 27.7%. A screening based on the unit-cost criterion limits the joint synergistic and economically viable domain to input levels ranging from 0.21 to 0.38. All parameters are synthetic values from published ranges for a single facility, so the boundaries describe a mechanism rather than plant-specific forecasts.

Keywords

Circular manufacturing Discrete-event simulation Multi-objective reinforcement learning Resource circularity Environmental performance

Article Details

How to Cite
Li, T., & Indiran, L. (2026). Multi-objective reinforcement-learning simulation for co-optimizing resource circularity and environmental performance in circular manufacturing. Future Technology, 5(4), 376–389. Retrieved from https://fupubco.com/futech/article/view/1206
Download Citation
Endnote/Zotero/Mendeley (RIS)
BibTeX
Bookmark and Share

References

  1. European Parliament and Council of the European Union, “Regulation (EU) 2023/1542 of the European Parliament and of the Council of 12 July 2023 concerning batteries and waste batteries, amending Directive 2008/98/EC and Regulation (EU) 2019/1020 and repealing Directive 2006/66/EC,” Official Journal of the European Union, vol. L 191, pp. 1–117, 28 Jul. 2023. [Online]. Available: http://data.europa.eu/eli/reg/2023/1542/oj
  2. L. Li, F. Dababneh and J. Zhao, "Cost-effective supply chain for electric vehicle battery remanufacturing," Applied Energy, vol. 226, pp. 277–286, 2018. DOI: https://doi.org/10.1016/j.apenergy.2018.05.115
  3. H. Liao, Q. Zhang, N. Shen and L. Li, "Stochastic analysis of quality uncertainty and optimal acquisition strategies for engine remanufacturing," Journal of Cleaner Production, vol. 261, Art. no. 121088, 2020. DOI: https://doi.org/10.1016/j.jclepro.2020.121088
  4. L. Rigamonti and E. Mancini, "Life cycle assessment and circularity indicators," The International Journal of Life Cycle Assessment, vol. 26, no. 10, pp. 1937–1942, 2021. DOI: https://doi.org/10.1007/s11367-021-01966-2
  5. W. Zhang, Y. Zheng and R. Ahmad, "The integrated process planning and scheduling of flexible job-shop-type remanufacturing systems using improved artificial bee colony algorithm," Journal of Intelligent Manufacturing, vol. 34, no. 7, pp. 2963–2988, 2023. DOI: https://doi.org/10.1007/s10845-022-01969-2
  6. M. I. Rizova, T. C. Wong and W. Ijomah, "A systematic review of decision-making in remanufacturing," Computers & Industrial Engineering, vol. 147, Art. no. 106681, 2020. DOI: https://doi.org/10.1016/j.cie.2020.106681
  7. M. Panzer and B. Bender, "Deep reinforcement learning in production systems: A systematic literature review," International Journal of Production Research, vol. 60, no. 13, pp. 4316–4341, 2022. DOI: https://doi.org/10.1080/00207543.2021.1973138
  8. Y. Li, W. Gu, M. Yuan and Y. Tang, "Real-time data-driven dynamic scheduling for flexible job shop with insufficient transportation resources using hybrid deep Q network," Robotics and Computer-Integrated Manufacturing, vol. 74, Art. no. 102283, 2022. DOI: https://doi.org/10.1016/j.rcim.2021.102283
  9. M. Cutler, T. J. Walsh and J. P. How, "Real-world reinforcement learning via multifidelity simulators," IEEE Transactions on Robotics, vol. 31, no. 3, pp. 655–671, 2015. DOI: https://doi.org/10.1109/TRO.2015.2419431
  10. G. Lonca, R. Muggéo, H. Imbeault-Tétreault, S. Bernard and M. Margni, "Does material circularity rhyme with environmental efficiency? Case studies on used tires," Journal of Cleaner Production, vol. 183, pp. 424–435, 2018. DOI: https://doi.org/10.1016/j.jclepro.2018.02.108
  11. C. G. Castro, A. H. Trevisan, D. C. A. Pigosso and J. Mascarenhas, "The rebound effect of circular economy: Definitions, mechanisms and a research agenda," Journal of Cleaner Production, vol. 345, Art. no. 131136, 2022. DOI: https://doi.org/10.1016/j.jclepro.2022.131136
  12. D. Font Vivanco, J. Freire-González, R. Galvin, T. Santarius, H. J. Walnum, T. Makov et al., "Rebound effect and sustainability science: A review," Journal of Industrial Ecology, vol. 26, no. 4, pp. 1543–1563, 2022. DOI: https://doi.org/10.1111/jiec.13295
  13. X. Guo, M. Zhou, A. Abusorrah, F. Alsokhiry and K. Sedraoui, "Disassembly sequence planning: A survey," IEEE/CAA Journal of Automatica Sinica, vol. 8, no. 7, pp. 1308–1324, 2021. DOI: https://doi.org/10.1109/JAS.2020.1003515
  14. X. Guo, Z. Zhang, L. Qi, S. Liu, Y. Tang and Z. Zhao, "Stochastic hybrid discrete grey wolf optimizer for multi-objective disassembly sequencing and line balancing planning in disassembling multiple products," IEEE Transactions on Automation Science and Engineering, vol. 19, no. 3, pp. 1744–1756, 2022. DOI: https://doi.org/10.1109/TASE.2021.3133601
  15. Y. Fu, Z. Zhang, P. Liang, G. Tian and C. Zhang, "Integrated remanufacturing scheduling of disassembly, reprocessing and reassembly considering energy efficiency and stochasticity through group teaching optimization and simulation approaches," Engineering Optimization, vol. 56, no. 12, pp. 2018–2039, 2024. DOI: https://doi.org/10.1080/0305215X.2023.2296538
  16. J. Zhao, S. Peng, T. Li, S. Lv, M. Li and H. Zhang, "Energy-aware fuzzy job-shop scheduling for engine remanufacturing at the multi-machine level," Frontiers of Mechanical Engineering, vol. 14, no. 4, pp. 474–488, 2019. DOI: https://doi.org/10.1007/s11465-019-0560-z
  17. H. Lu, D. Herman and Y. Yu, "Multi-objective reinforcement learning: Convexity, stationarity and Pareto optimality," in Proc. 11th International Conference on Learning Representations (ICLR), 2023. [Online]. Available: https://openreview.net/forum?id=TjEzIsyEsQ6
  18. T. Kegyes, Z. Süle and J. Abonyi, "Disassembly line optimization with reinforcement learning," Central European Journal of Operations Research, vol. 32, no. 4, pp. 1115–1142, 2024. DOI: https://doi.org/10.1007/s10100-024-00906-3
  19. C. F. Hayes, R. Rădulescu, E. Bargiacchi, J. Källström, M. Macfarlane, M. Reymond et al., "A practical guide to multi-objective reinforcement learning and planning," Autonomous Agents and Multi-Agent Systems, vol. 36, no. 1, Art. no. 26, 2022. DOI: https://doi.org/10.1007/s10458-022-09552-y
  20. F. Felten, E.-G. Talbi and G. Danoy, "Multi-objective reinforcement learning based on decomposition: A taxonomy and framework," Journal of Artificial Intelligence Research, vol. 79, pp. 679–723, 2024. DOI: https://doi.org/10.1613/jair.1.15702
  21. T. Basaklar, S. Gumussoy and U. Y. Ogras, "PD-MORL: Preference-driven multi-objective reinforcement learning algorithm," in Proc. 11th International Conference on Learning Representations (ICLR), 2023. [Online]. Available: https://openreview.net/forum?id=zS9sRyaPFlJ
  22. M. Reymond, C. F. Hayes, D. Steckelmacher, D. M. Roijers and A. Nowé, "Actor-critic multi-objective reinforcement learning for non-linear utility functions," Autonomous Agents and Multi-Agent Systems, vol. 37, no. 2, Art. no. 23, 2023. DOI: https://doi.org/10.1007/s10458-023-09604-x
  23. S. Lassoued, L. S. Baheti, N. Weiß-Borkowski, S. Lier and A. Schwung, "Flexible Manufacturing Systems intralogistics: Dynamic optimization of AGVs and tool sharing using Colored-Timed Petri Nets and actor–critic RL with actions masking," Journal of Manufacturing Systems, vol. 82, pp. 405–419, 2025. DOI: https://doi.org/10.1016/j.jmsy.2025.06.017
  24. M. Stappert, B. Lutz, N. Goby and D. Neumann, "Integrating human knowledge through action masking in reinforcement learning for operations research," Computers & Industrial Engineering, vol. 222, Art. no. 112340, 2026. DOI: https://doi.org/10.1016/j.cie.2026.112340
  25. W. Song, X. Chen, Q. Li and Z. Cao, "Flexible job-shop scheduling via graph neural network and deep reinforcement learning," IEEE Transactions on Industrial Informatics, vol. 19, no. 2, pp. 1600–1610, 2023. DOI: https://doi.org/10.1109/TII.2022.3189725
  26. A. Seipolt, R. Buschermöhle, V. Haag, W. Hasselbring, M. Höfinghoff, M. Schumacher et al., "Reinforcement learning and digital twin-driven optimization of production scheduling with the digital model playground," Discover Internet of Things, vol. 4, no. 1, Art. no. 34, 2024. DOI: https://doi.org/10.1007/s43926-024-00087-0
  27. S. Khairy and P. Balaprakash, "Multi-fidelity reinforcement learning with control variates," Neurocomputing, vol. 597, Art. no. 127963, 2024. DOI: https://doi.org/10.1016/j.neucom.2024.127963
  28. F. Paschko, A. Krini, M. Kemke and S. Knorn, "Reinforcement learning approach for decision making in disassembly processes," Journal of Intelligent Manufacturing, vol. 37, no. 5, pp. 1829–1855, 2026. DOI: https://doi.org/10.1007/s10845-025-02622-4
  29. B. Löfgren and A.-M. Tillman, "Relating manufacturing system configuration to life-cycle environmental performance: discrete-event simulation supplemented with LCA," Journal of Cleaner Production, vol. 19, no. 17–18, pp. 2015–2024, 2011. DOI: https://doi.org/10.1016/j.jclepro.2011.07.014
  30. G. Wernet, C. Bauer, B. Steubing, J. Reinhard, E. Moreno-Ruiz and B. Weidema, "The ecoinvent database version 3 (part I): overview and methodology," The International Journal of Life Cycle Assessment, vol. 21, no. 9, pp. 1218–1230, 2016. DOI: https://doi.org/10.1007/s11367-016-1087-8
  31. Ellen MacArthur Foundation and Granta Design, "Circularity Indicators: An Approach to Measuring Circularity – Methodology," Ellen MacArthur Foundation, Cowes, UK, 2015 (adapted 2019). [Online]. Available: https://ellenmacarthurfoundation.org/material-circularity-indicator
  32. R. Yang, X. Sun and K. Narasimhan, "A generalized algorithm for multi-objective reinforcement learning and policy adaptation," in Advances in Neural Information Processing Systems 32 (NeurIPS), 2019. [Online]. Available: https://proceedings.neurips.cc/paper/2019/hash/4a46fbfca3f1465a27b210f4bdfe6ab3-Abstract.html
  33. J. Blank and K. Deb, "Pymoo: Multi-objective optimization in Python," IEEE Access, vol. 8, pp. 89497–89509, 2020. DOI: https://doi.org/10.1109/ACCESS.2020.2990567
  34. B. H. Lowe, A. Genovese, D. Font Vivanco and T. Zink, "Revisiting circular economy rebound: Market dynamics, policy implications, and future research directions," Journal of Industrial Ecology, vol. 29, no. 6, pp. 1936–1945, 2025. DOI: https://doi.org/10.1111/jiec.70121
  35. A. Esteso, D. Peidro, J. Mula and M. Díaz-Madroñero, "Reinforcement learning applied to production planning and control," International Journal of Production Research, vol. 61, no. 16, pp. 5772–5789, 2023. DOI: https://doi.org/10.1080/00207543.2022.2104180
  36. L. N. Alegre, A. L. C. Bazzan, D. M. Roijers, A. Nowé and B. C. da Silva, "Sample-efficient multi-objective learning via generalized policy improvement prioritization," in Proc. 22nd International Conference on Autonomous Agents and Multiagent Systems (AAMAS), London, UK, 2023, pp. 2003–2012. [Online]. Available: https://arxiv.org/abs/2301.07784
  37. M. Wurster, M. Michel, M. C. May, A. Kuhnle, N. Stricker and G. Lanza, "Modelling and condition-based control of a flexible and hybrid disassembly system with manual and autonomous workstations using reinforcement learning," Journal of Intelligent Manufacturing, vol. 33, no. 2, pp. 575–591, 2022. DOI: https://doi.org/10.1007/s10845-021-01863-3
  38. C. Hu, Z. Wang, J. Liu, J. Wen, B. Mao and X. Yao, "Constrained reinforcement learning for dynamic material handling," in Proc. 2023 International Joint Conference on Neural Networks (IJCNN), 2023, pp. 1–9. DOI: https://doi.org/10.1109/IJCNN54540.2023.10191999
  39. R. Stolz, H. Krasowski, J. Thumm, M. Eichelbeck, P. Gassert and M. Althoff, "Excluding the irrelevant: Focusing reinforcement learning through continuous action masking," in Advances in Neural Information Processing Systems (NeurIPS), vol. 37, pp. 95067–95094, 2024. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2024/hash/acf4a08f67724e9d2de34099f57a9c25-Abstract.html
  40. Z. Wang, X. Li, L. Sun, H. Zhang, H. Liu and J. Wang, "Learning state-specific action masks for reinforcement learning," Algorithms, vol. 17, no. 2, Art. no. 60, 2024. DOI: https://doi.org/10.3390/a17020060
  41. European Commission, "Commission Recommendation (EU) 2021/2279 of 15 December 2021 on the use of the Environmental Footprint methods to measure and communicate the life cycle environmental performance of products and organisations," Official Journal of the European Union, L 471, p. 1, 30 Dec. 2021. [Online]. Available: https://eur-lex.europa.eu/eli/reco/2021/2279/oj
  42. Si Z, Ali D A, Rosli R B, Bhaumik A, Ghosh A. Omni-channel retail marketing effect evaluation framework integrating big data and artificial intelligence. Edelweiss Applied Science and Technology, 2025, 9(3): 568-583. DOI: https://doi.org/10.55214/25768484.v9i3.5255
  43. Si Z, Ali D A, Rosli R B. AI-driven risk management decision framework and value creation in chemical enterprises. BEM Quarterly, 2025, 1(1): 1-32. Available: https://system.wisacad-pub.com/index.php/bem/article/view/195