Finding Features of Actions Efficiently Synchronized with Dishwashing Robot


  • Kosuke Nishio a:1:{s:5:"en_US";s:22:"Ritsumeikan University";}
  • Fumiko Harada
  • Hiromitsu Shimakawa



Human–robot Interaction, Human Behavior, Dishwashing Robot, Wearable sensors


In this study, we propose a method for extracting the characteristics of body motions that contribute to reducing the takt time in a cooperative task between a dishwashing robot and a human operator. The proposed method collects the takt time and motion data from novice operators until they become experienced using an inexpensive acceleration sensor. The operation data is classified into experienced and novice periods using the variance value of the takt time. In addition, the Hidden Markov Model is generated to classify the motion data into multiple motion phases. The motion features of the operator are extracted for each phase from the generated model. The proposed method finds the motion features whose difference between the experienced and novice periods are similar to the takt time transition.  It uses them as important variables. We verified the effectiveness of the proposed method by conducting experiments that simulate actual work at a restaurant. The Hidden Markov Model classified the operation phases into three categories with the AUC of 0.9. In all samples, we were able to extract the motion characteristics of the experienced operators. This study showed the potential to improve the speed of novice's progress by the extracted motion characteristics to improve education guidelines and to show operators how they should physically move.


[1]. Scott R. Baker, et al. "How does household spending respond to an epidemic? Consumption during the 2020 COVID-19 pandemic." The Review of Asset Pricing Studies 10.4 (2020): 834-862.
[2]. Masaki Takahashi, et al. "Developing a mobile robot for transport applications in the hospital domain." Robotics and Autonomous Systems 58.7 (2010): 889-899.
[3]. Ashutosh Saxena, et al. "Robotic grasping of novel objects using vision." The International Journal of Robotics Research 27.2 (2008): 157-173.
[4]. Isobel Voysey, et al. "Autonomous dishwasher loading from cluttered trays using pre‐trained deep neural networks." Engineering Reports (2020): e12321.
[5]. Rüdiger Dillmann. "Teaching and learning of robot tasks via observation of human performance." Robotics and Autonomous Systems 47.2-3 (2004): 109-116.
[6]. Aneeq Zia, et al. "Video and accelerometer-based motion analysis for automated surgical skills assessment." International Journal of Computer Assisted Radiology and Surgery 13.3 (2018): 443-455.
[7]. Yufan Wang, et al. "Volleyball skill assessment using a single wearable micro inertial measurement unit at wrist." IEEE Access 6 (2018): 13758-13765.
[8]. Terry Taewoong Um, et al. "Exercise motion classification from large-scale wearable sensor data using convolutional neural networks." 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2017.
[9]. Yongqiang Huang, and Yu Sun. "Accurate robotic pouring for serving drinks." arXiv preprint arXiv:1906.12264 (2019).
[10]. Mehdi Khamassi, et al. "Active exploration and parameterized reinforcement learning applied to a simulated human-robot interaction task." 2017 First IEEE International Conference on Robotic Computing (IRC). IEEE, 2017.
[11]. Youngjoon Choi, et al. "Service robots in hotels: understanding the service quality perceptions of human-robot interaction." Journal of Hospitality Marketing and Management 29.6 (2020): 613-635.
[12]. Juan P. Vasconez, et al. "Human-robot interaction in agriculture: A survey and current challenges." Biosystems Engineering 179 (2019): 35-48.
[13]. Leonard E. Baum, and Ted Petrie. "Statistical inference for probabilistic functions of finite state Markov chains." The Annals of Mathematical Statistics 37.6 (1966): 1554-1563.
[14]. Wataru Takano, et al. "Recognition of human driving behaviors based on stochastic symbolization of time series signal." 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2008.
[15]. Aditi Roy, et al. "An HMM-based behavior modeling approach for continuous mobile authentication." 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2014.
[16]. Mono Wireless Inc.. (2020.10.5)




How to Cite

Nishio, K., Harada, F., & Shimakawa, H. (2021). Finding Features of Actions Efficiently Synchronized with Dishwashing Robot. Advances in Social Sciences Research Journal, 8(2), 206–224.