skip to main content
Language:
Search Limited to: Search Limited to: Resource type Show Results with: Show Results with: Search type Index

A Crossmodal Approach to Multimodal Fusion in Video Hyperlinking

IEEE multimedia, 2018-04, Vol.25 (2), p.11-23 [Peer Reviewed Journal]

Distributed under a Creative Commons Attribution 4.0 International License ;ISSN: 1070-986X ;EISSN: 1941-0166 ;DOI: 10.1109/MMUL.2018.023121161

Digital Resources/Online E-Resources

Citations Cited by
  • Title:
    A Crossmodal Approach to Multimodal Fusion in Video Hyperlinking
  • Author: Vukotić, Vedran ; Raymond, Christian ; Gravier, Guillaume
  • Subjects: Computer Science ; Computer Vision and Pattern Recognition ; Information Retrieval ; Multimedia ; Neural and Evolutionary Computing
  • Is Part Of: IEEE multimedia, 2018-04, Vol.25 (2), p.11-23
  • Description: With the recent resurgence of neural networks and the proliferation of massive amounts of unlabeled data, unsupervised learning algorithms became very popular for organizing and retrieving large video collections in a task defined as video hyperlinking. Information stored as videos typically contain two modalities, namely an audio and a visual one, that are used conjointly in multimodal systems by undergoing fusion. Multimodal autoencoders have been long used for performing multimodal fusion. In this work, we start by evaluating different initial, single-modal representations for automatic speech transcripts and for video keyframes. We progress to evaluating different autoencoding methods of performing multimodal fusion in an offline setup. The best performing setup is then evaluated in a live setup at TRECVID's 2016 video hyperlinking task. As in offline evaluations, we show that focusing on crossmodal translations as a way of performing multimodal fusion yields improved multimodal representations and that our simple system, trained in an unsupervised manner, with no external information information, defines the new state of the art in a live video hyperlinking setup. We conclude by performing an analysis on data gathered after the live evaluations at TRECVID 2016 and express our thoughts on the overall performance of our proposed system.
  • Publisher: Institute of Electrical and Electronics Engineers
  • Language: English
  • Identifier: ISSN: 1070-986X
    EISSN: 1941-0166
    DOI: 10.1109/MMUL.2018.023121161
  • Source: Hyper Article en Ligne (HAL) (Open Access)
Back to results list
INSPIRE LIBRARY - TON DUC THANG UNIVERSITY (84-028) 37 755 057 Feedback
19 Nguyen Huu Tho St. Dist.7, HCM thuvien@tdtu.edu.vn

Copyright © 2017 INSPIRE LIBRARY - TON DUC THANG UNIVERSITY

Searching Remote Databases, Please Wait