The Internet is full of opinion hidden under tons of irrelevant and unstructured data. From micro-blogging platforms like Twitter to video repositories such as YouTube the users express sentiments about products, brands, institutions and governments. This project aims to address opinion analysis from a holistic point of view, where text and structured data (i.e. Linked Data from DBPedia, Freebase) together with images and videos are considered in order to derive a more precise opinion assessment.



MADM presents at the press event covering Googles engagement with the DFKI

The DFKI announced that Google became a shareholder of the DFKI. The MADM group presented their Eye Tracking technology and Visual Sentiment Analysis (SentiBank) with Deep Convolutional Neural Networks (CNN), which was names as one of the key technologies by Google. In this context, MADM was also covered in an TechCrunch article.


"Financial Data Science" a discipline combining Data Mining and Financial Statistics

A new discipline emerges - “Financial Data Science” - which combines approaches from data mining and financial statistics leveraging large-scale data processing, machine learning, statistics, and financial modeling.

Nowadays the Internet, as a major platform for communication and information exchange, provides a rich repository of people’s opinion and sentiment about a vast spectrum of topics. Such knowledge is embedded in multiple facets, such as comments, tags, browsing actions, as well as shared media objects. The analysis of such information either in the area of opinion mining, affective computing or sentiment analysis plays an important role in behavior sciences, which aims to understand and predict human decision making and enables applications such as brand monitoring, stock market prediction, or political voting forecasts. (read more)


"Multimedia Opinion Mining for Social Good" presented at UN PRI Event

In the context of the BMBF sponsored project “Multimedia Opinion Mining”, Dr. Damian Borth, gave a guest lecture at the ICMA Center of the Henley Business School about how tools developed at MADM can be used for social good. The lecture presented novel way of mining social multimedia to gain insights about critical issues around environmental, social, and governance issues.


Dr. Damian Borth part of assessment committee for the Investment Innovation Benchmark (IIB)

Dr. Damian Borth served as a member of the assessment committee at the Investment Innovation Benchmark (IIB) summit in London. Damian was appointed to the committee due to his background in machine learning and large-scale data processing and will serve as an expert to evaluate FinTech submissions. The assessment committee consists of member from academia and some of the largest pension fund worldwide.


"YFCC100M: The New Data in Multimedia Research" got accepted at CACM

Our paper describing the 100 million images and videos Flickr dataset: “YFCC100M: The New Data in Multimedia Research” got accepted at “Communication of the ACM”. The paper describes the up-to-date largest dataset in the computer vision and multimedia research community. This work was a collaboration between Yahoo Inc., ICSI Berkeley, and the Lawrence Livermore National Laboratory. A pre-print is available on


Interview with Hannoversche Allgemeine Zeitung about Computer Vision Research

Dr. Damian Borth gave an Interview to Hannoversche Allgemeine Zeitung about the success of deep learning in computer vision research, recent progress to recognize sentiment and emotions in visual content, and how artificial intelligence (AI) can be used for social good.


Dr. Damian Borth moderates a Technology Panel at Bloomberg, New York, USA

With the advent of FinTech, disciplines like machine learning, natural language processing, and large-scale data mining are moving into the finance and investment industry. To discuss the potential of FinTech, Dr. Damian Borth was invited to moderate a Technology Panel at FinTech Bloomberg event. The panel was including CEOs and CTOs of leading startups in the area such as: eRevalue, TruValue and Bloomberg Labs.


Dr. Damian Borth speaks at RI Europe about Financial Data Science, London, UK

Dr. Damian Borth speaks about Big Data and Environmental, Social and Governance research and introduces together with Prof. Andreas Hoepner the concept of “Financial Data Science” at the RI Europe - a major conference in the responsible investment community. Financial Data Science embraces the processing of large datasets with machine learning methods and welcomes approaches such as opinion mining from public available data on the web.


Call for paper for Multimedia COMMONS Workshop is out

The CfP to the Multimedia COMMONS workshop (MMCommons) is out! We invite the community to attend MMCommons and explore the possibilities for novel research, future data challenges, and new benchmarks offered by this large-scale open dataset (YFCC100m dataset). The MMCommons will be held at the ACM Multimedia 2015, Brisbane, Australia around October 26-30.


Dr. Damian Borth appointed new head of MADM group

By April 2015, Damian Borth became the new head of MADM. Before this position Damian was a postdoctoral research fellow at the International Computer Science Institute (ICSI) with Dr. Gerald Friedland and at UC Berkeley with Prof. Trevor Darrell where he was involved in various project sponsored by DARPA, IARPA, and the Lawrence Livermore National Laboratory. His research focuses concept detection from social multimedia including trending topic detection, visual sentiment analysis, and multimedia opinion mining.


Invited talk about "Data Science and Opinion Mining" at the London School of Economics, UK

The project Multimedia Opinion Mining (MOM) was presented in the context of Data Science during an UN PRI event at the London School of Economics. Dr. Damian Borth introduced data science and opinion mining and outlined how it can be used for social good. The event was later interrupted by an evacuation due to the Holborn Fire in London.


Dr. Damian Borth gives an invited talk at the University of Reading, UK

Dr. Damian Borth speaks about “Mining Insights from Public Data” at an event of the University of Reading and outlines concepts and goals of the Multimedia Opinion Mining (MOM) project to the audience.


Dr. Damian Borth invited to BMBF dinner with Dr. Georg Schütte in Berkeley, USA

During a visit of the Bundesministerium für Forschung und Entwicklung (BMBF) in the Bay Area, the BMBF invited DAAD scholarship researcher from ICSI to join an exclusive dinner with Deputy Secretary Dr. Georg Schütte. Dr. Damian Borth introduces the BMBF funded project Multimedia Opinion Mining (MOM) to the delegation during the dinner.


BMBF funded project Multimedia Opinion Mining (MOM) starts

Multimedia Opinion Mining (MOM), a 2.5 years long project funded by the BMBF, aims to address the challenge of opinion mining in social multimedia content. The main goals of MOM are: (1) analyze social media data to detect and track trending topics. (2) understand multimedia content with respect to its sentiment and opinion. (3) discover individual persons or groups able to influence propagation of topics in social networks. And finally, (4) forecasting the progression of identified topics into the near future. The project will be coordinated within MADM and lead by Dr. Damian Borth.



This project aims to address the challenge of opinion mining of multimedia content from the web. This comprises a large-scale multi-modal analysis of social media streams and their underlying network dynamics considering different media channels such as Twitter, YouTube, Flickr, Google, and Wikipedia. The main contribution of MOM is four-fold as illustrated below:

1. Trending Topics

social media data is analyzed utilizing information from structured data sources to detect and track trending topics.

2. Sentiment Analysis

large amounts of multimedia content are analyzed with respect to its sentiment and opinion. This follows a holistic approach i.e. the analysis of a rich set of different modalities such as textual, visual, and tempo-visual content.

3. Social Network Analysis

this information is further enriched by network analysis to grasp structural diffusion as well as global and local impact of information in social networks.

4. Forecasting

the possibility of forecasting the progression of identified opinion topics into the near future is investigated.

Research Activities

The MOM project defines the following research activities and work packages.


  • AudioSentibank: Large-scale Semantic Ontology of Acoustic Concepts for Audio Content Analysis

    Sebastian Sager, Damian Borth, Benjamin Elizalde, Christian Schulze, Bhiksha Raj, Ian Lane, Andreas Dengel
    arXiv preprint arXiv:1607.03766, 2016


  • Multimedia COMMONS Workshop 2016 (MMCommons’16) - Datasets, Evaluation, and Reproducibility

    Bart Thomee, Damian Borth, Julia Bernd
    ACM Multimedia Workshops, 2016


  • Generating Captions with Emotions using Concept and Syntax Transition Networks

    Philipp Blandfort, Tushar Karayil, Damian Borth, Andreas Dengel
    ACM Multimedia, Grand Challenge, 2016 (accepted)

    [challenge website]

  • Contextual Enrichment of Remote-Sensed Events with Social Media Streams

    Benjamin Bischke, Damian Borth, Christian Schulze, Andreas Dengel
    ACM Multimedia, Grand Challenge, 2016 (accepted)


  • Which languages do people speak on Flickr? A Language and Geo-Location Study of the YFCC100m Dataset

    Alireza Koochali, Sebastian Kalkowski, Damian Borth, Christan Schulze, Andreas Dengel
    ACM Multimeda Community-Organized Multimodal Mining: Opportunities for Novel Solutions (MMCOMMONS) Workshop, 2016 (accepted)

    [YFCC100m Browser]

  • Introducing Concept And Syntax Transition Networks for Image Captioning

    Philipp Blandfort, Tushar Karayil, Damian Borth, Andreas Dengel
    ACM Int. Conf. on Multimedia Retrieval (ICMR), 2016
  • An Evolutionary Algorithm to Learn SPARQL Queries: Finding Patterns for Human Associations in DBpedia

    Jörn Hees, Rouven Bauer, Joachim Folz, Damian Borth, Andreas Dengel
    arXiv preprint arXiv:1607.07249, 2016


  • Trending Topic Aggregation by News-based Context Modeling

    Sebastian Fuchs, Damian Borth and Adrian Ulges
    KI, 2016 (accepted)
  • YFCC100m: The New Data in Multimedia Research

    Bart Thomee, David Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Dough Poland, Damian Borth, Li-Jia Li
    Communication of the ACM, 2016, (YFCC100m Dataset Paper)

    [paper] [watch video] [YFCC100m Browser]

  • Kickstarting the Commons: The YFCC100M and the YLI Corpora

    Julia Bernd, Damian Borth, Carmen Carrano, Jaeyoung Choi, Benjamin Elizalde, Gerald Friedland, Luke Gottlieb, Karl Ni, Roger Pearce, Doug Poland, Khalid Ashraf, David A. Shamma, and Bart Thomee
    ACM Multimeda Community-Organized Multimodal Mining: Opportunities for Novel Solutions (MMCOMMONS) Workshop, 2015
  • Real-time Analysis and Visualization of the YFCC100m Dataset

    Sebastian Kalkowski, Christian Schulze, Andreas Dengel, Damian Borth
    ACM Multimeda Community-Organized Multimodal Mining: Opportunities for Novel Solutions (MMCOMMONS) Workshop, 2015

    [YFCC100m Browser]

  • What makes a Beautiful Landscape beautiful: Adjective Noun Pairs Attention by Eye-Tracking and Gaze Analysis

    Mohammad Al-Naser, Seyyed Saleh Mozaffari Chanijani, Syed Saqib Bukhari, Damian Borth, Andreas Dengel
    ACM Workshop on Affect and Sentiment in Multimedia (ASM), 2015
  • Aesthetic Photo Enhancement using Machine Learning and Case-Based Reasoning

    Joachim Folz, Christian Schulze, Damian Borth, Andreas Dengel
    ACM Workshop on Affect and Sentiment in Multimedia (ASM), 2015
  • The New Data and New Challenges in Multimedia Research

    Bart Thomee, David Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Dough Poland, Damian Borth, Li-Jia Li
    arXiv, preprint arXiv:1503.01817, March, 2015


  • Large-Scale Deep Learning on the YFCC100M Dataset

    Karl Ni, Roger Pearce, Kofi Boakye, Brian Van Essen, Damian Borth, Barry Chen, Eric Wang
    arXiv, preprint arXiv:1502.03409, February, 2015



The following people contributed to the project (past and current).

Prof. Dr. Prof. h.c. Andreas Dengel


    Dr. Damian Borth

    Project Lead

      Dr. Christian Schulze


        Jörn Hees


          Sebastian Palacio


            Joachim Folz

            Junior Researcher

              Philipp Blandfort

              PhD student

                Tushar Karayil

                PhD student

                  Prakriti Bhardwaj


                    Benjamin Bischke


                      Peter Burkert


                        Tobias Hamann


                          Sebastian Kalkowski


                            Alireza Koochali


                              Jalpa Punjabi


                                Ahsan Naeem


                                  Sebastian Säger


                                    Rajesh Chidambaram


                                      Jessica Chwalek


                                        Mihai Fieraru


                                          Aman Gautam


                                            Monit Shah Singh


                                              Kshitij Kumar



                                                • Dr. Damian Borth
                                                • Trippstaedter Str. 122
                                                  67663 Kaiserslautern
                                                • +49 (631) 20575-4184