End-to-end Human-Object Interaction Network with Multi-modal Prior

This is a research project in COMP8536 Advanced Topics in Deep Learning for Computer Vision, ANU.

For Human Object Interaction (HOI) Detection task, One essential perception how to locate the human position, and combine the human object instances with a specific interaction more effectively. However, the current one-stage HOI Detection model based on transformer doesn’t give much prior information on HOI locations locations and interaction classes.

We proposed a novel HOI grouping block to group the human and objects into a HOI instance in a more efficient way. To enhance the prediction of HOI action, we fuse the pre-trained verb vectors for all actions in the dataset as linguistic query to help model predicting the interaction type. Finally, we come up with a new two dimensional positional encoding based on two-dimensional fourier series for enhancing the priori location encoding of HOI instances.

Results: We acheievd best performance in class. And I got 88 (High Distinction) in this course.

To see our report here