1. Proceedings
  2. I3M
  3. 2020
  4. EMSS
  5. 10.46354/i3m.2020.emss.002

A simulation of autonomous robot movement directed by reinforcement learning

  • a Andrew Greasley 
  • a Aston University, Aston St, Birmingham B4 7ET, United Kingdom
Cite as
Greasley A. (2020). A simulation of autonomous robot movement directed by reinforcement learning. Proceedings of the 32nd European Modeling & Simulation Symposium (EMSS 2020), pp. 10-15. DOI: https://doi.org/10.46354/i3m.2020.emss.002

Abstract

As companies embrace Industry 4.0 and embed intelligent robots and other intelligent facilities in their factories, decision making can be derived from machine learning algorithms and so if we are to simulate these systems we need to model these algorithms too. This article presents a discrete-event simulation (DES) that incorporates the use of a reinforcement learning (RL) algorithm which determines an approximate best route for robots in a factory moving from one physical location to another whilst avoiding collisions with fixed barriers. The study shows how the object oriented and graphical facilities of an industry ready commercial off-the-shelf (COTS) DES software package enables an RL capability without the need to use program code or require an interface with external RL software. Thus the article aims to contribute to the methodology of simulation practitioners who wish to implement AI techniques as a supplement to their input modelling approaches.

References

  1. Benotsmane, R., Kovács, G., Dudás, L.: Economic, Social Impacts and Operation of Smart Factories in Industry 4.0: Focusing on Simulation and Artificial Intelligence of Collaborating Robots, Social Sciences, 8(143), (2019). 
  2. Bergmann, S., Feldkamp, N., & Strassburger, S.: Emulation of control strategies through machine learning in manufacturing simulations, Journal of Simulation, 11(1), 38-50, (2017).
  3. Brailsford, S.: Theoretical comparison of discreteevent simulation and system dynamics. In S. Brailsford, L. Churilov, & B. Dangerfield (Eds.), Discrete-Event Simulation and System Dynamics
    for Management Decision Making, John Wiley and Sons Ltd, Chichester (2014).
  4. Chewu, C.C.E. and Kumar V.M.: Autonomous navigation of a mobile robot in dynamic indoor environments using SLAM and reinforcement learning, IOP Conf. Series: Materials Science and Engineering, 402, 012022, (2018).
  5. Creighton, D.C. and Nahavandi, S.: Optimising discrete event simulation models using a reinforcement learning agent, Proceedings of the 2002 Winter Simulation Conference, pp. 1945-1950, (2002).
  6. Elbattah, M. and Molloy, O.: Analytics Using Machine Learning-Guided Simulations with Application to Healthcare Scenarios IN Analytics and Knowledge Management (S. Hawamdeh and H.C. Chang (eds.)), (2018).
  7. Greasley, A.: Simulating Business Processes for Descriptive, Predictive and Prescriptive Analytics, DeGruyter Press (2019).
  8. Harrell, C. and Tumay, K.: Simulation Made Easy: A Manager’s Guide, Industrial Engineering and
    Management Press, Norcross, USA (1995).
  9. He, Y., Stecke, K.E., Smith, M.L. 2016. “Robot and machine scheduling with state-dependent part input sequencing in flexible manufacturing systems”, International Journal of Production
    Research, Vol. 54, pp. 6736-6746.
  10. Hosokawa, S., Kato, J., Nakomo, K.: A reward allocation method for reinforcement learning in stabilizing control tasks, Artif Life Robotics 19, 109-114, (2014).
  11. Khare, A., Motwani, R., Akash, S., Patil, J, Kala, R.: (2018) Learning the goal seeking behaviour for
    mobile robots, 3rd Asia-Pacific Conference on Intelligent Robot Systems, IEEE, pp. 56-60, (2018).
  12. Klass, A., Laroque, C., Fischer, M., Dangelmaier, W.: Simulation aided, knowledge based routing for AGVs in a distribution warehouse, Proceedings of the 2011 Winter Simulation Conference, IEEE, pp. 1668-1679, (2011).
  13. Kumar, U.D.: Business Analytics: The Science of DataDriven Decision Making, New Del-hi: Wiley, (2017).
  14. Law, A.M.: Simulation Modeling and Analysis, 5th Edition, New York: McGraw-Hill Edu-cation,
    (2015).
  15. North, M.J. and Macal, C.M.: Managing Business Complexity: Discovering Strategic Solutions with Agent-based Modeling and Simulation, Oxford University Press (2007).
  16. Ono, Y. and Ishigami, G.: Routing problem of multiple mobile robots with human workers for pickup and dispatch tasks in warehouse, Proceedings of the 2019 IEEE/SICE International Symposium of System Integration, IEEE, pp. 176-181, (2019).
  17. Posada, J., Toro, C., Barandiaran, I., Oyarzun, D., Stricker, D., de Amicis, R., Pinto, E. B., Eisert, P.,
    Dollner, J., Vallarino, I.: Visual Computing as a Key Enabling Technology for Industrie 4.0 and
    Industrial Internet, IEEE Computer Graphics & Applications, 35(2), 26-40, (2015).
  18. Robinson, S.: Simulation: The practice of model development and use, Second Edition, Palgrave
    Macmillan (2014).
  19. Rüsmann, M., Lorenz, M., Gerbert, P., Waldner, M., Justus, J., Engel, P. Harnisch, M.: In-dustry 4.0: The Future of Productivity and Growth in Manufacturing Industries, The Boston Consulting Group, Boston (2015).
  20. Sartoretti, G., Kerr, J., Shi, Y., Wagner, G., Kumar, T.K.S., Koenig, C., Choset, H.: PRIMAL: Pathfinding via Reinforcement Learning and Imitation MultiAgent Learning, IEEE Robotics and Automation Letters, 4(3), 2378-2385, (2019).
  21. Seifert, R.W., Kay, M.G., Wilson, J.R.: Evaluation of AGV routeing strategies using hierarchical simulation, International Journal of Production Research, 36(7), 1961-1976, (1998).
  22. Smith, J.S., Sturrock, D.T., Kelton, W.D.: Simio and Simulation: Modeling, Analysis, Applications, 5th Edition, Simio LLC, (2018).
  23. Sutton, R.S. and Barto, A.G.: Reinforcement Learning: An Introduction, Second Edition, The MIT Press (2018).
  24. Truong, X.T., Ngo, T.D.: Toward socially aware robot navigation in dynamic and crowded environments: A proactive social motion model, IEEE Transactions on Automation Science and Engineering, 14(4), 1743-1760, (2017).
  25. Waschneck, B., Reichstaller, A., Belzner, L., Altenmüller, T., Bauernhansl, T., Knapp, A., Kyek,
    A.: Optimization of global production scheduling with deep reinforcement learning, Procedia CIRP 72, 1264-1269, (2018).
  26. Watkins, C.J.C.H. and Dayan, P.: Q-learning, Machine Learning, 8(3-4), 279-292, (1992).
  27. Zaayman, G. and Innamorato, A.: The application of Simio scheduling in Industry 4.0, Proceedings of the 2017 Winter Simulation Conference, pp 4425-4434, (2017).