With the Yahoo Flickr Creative Commons 100 Million (YFCC100m) dataset, a great novel dataset was introduced to the computer vision and multimedia research community. To maximize the benefit for the research community and utilize its potential, this dataset has to be made accessible by tools allowing to search for target concepts within the dataset and mechanism to browse images and videos of the dataset.

Every day real-world event happen. These events are reflected in social media and can be detection as trending topics within multiple streams. This demon crawls and archives trending topics from Google, Twitter and Wikipedia and visualizes them to inform us about the current Zeitgeist.

We present a novel approach towards sentiment analysis based on the semantic understanding of visual content. To research this goal, we automatically construct a large-scale ontology of semantic concepts.

DeepSentiBank allows user to upload images to assign Adjective Noun Pairs (ANP) concepts to them. This allows for a much more effective tagging than current state-of-the-art systems. Furthermore, the deep learning system is demonstrating the capability to segment ANP within the image.

Arrow
Arrow
Slider

INTRODUCTION


The Internet is full of opinion hidden under tons of irrelevant and unstructured data. From micro-blogging platforms like Twitter to video repositories such as YouTube the users express sentiments about products, brands, institutions and governments. This project aims to address opinion analysis from a holistic point of view, where text and structured data (i.e. Linked Data from DBPedia, Freebase) together with images and videos are considered in order to derive a more precise opinion assessment.


NEWS

06.11.2015:
1st Financial Data Science Association Conference in Zurich



Having the goal to improve the synergies between data science and financial markets more than 30 international participants joined the first Financial Data Science Association Conference. Beside several plenary sessions and panel discussions, Dr. Damian Borth spoke introduced the principles for Financial Data Science and discussed usage of Artificial Intelligence for Good.

06.10.2015:
MADM presents at the press event covering Googles engagement with the DFKI



The DFKI announced that Google became a shareholder of the DFKI. The MADM group presented their Eye Tracking technology and Visual Sentiment Analysis (SentiBank) with Deep Convolutional Neural Networks (CNN), which was names as one of the key technologies by Google. In this context, MADM was also covered in an TechCrunch article.

15.09.2015:
"Financial Data Science" a discipline combining Data Mining and Financial Statistics



A new discipline emerges - “Financial Data Science” - which combines approaches from data mining and financial statistics leveraging large-scale data processing, machine learning, statistics, and financial modeling.

Nowadays the Internet, as a major platform for communication and information exchange, provides a rich repository of people’s opinion and sentiment about a vast spectrum of topics. Such knowledge is embedded in multiple facets, such as comments, tags, browsing actions, as well as shared media objects. The analysis of such information either in the area of opinion mining, affective computing or sentiment analysis plays an important role in behavior sciences, which aims to understand and predict human decision making and enables applications such as brand monitoring, stock market prediction, or political voting forecasts. (read more)

07.09.2015:
"Multimedia Opinion Mining for Social Good" presented at UN PRI Event



In the context of the BMBF sponsored project “Multimedia Opinion Mining”, Dr. Damian Borth, gave a guest lecture at the ICMA Center of the Henley Business School about how tools developed at MADM can be used for social good. The lecture presented novel way of mining social multimedia to gain insights about critical issues around environmental, social, and governance issues.

04.09.2015:
Dr. Damian Borth part of assessment committee for the Investment Innovation Benchmark (IIB)



Dr. Damian Borth served as a member of the assessment committee at the Investment Innovation Benchmark (IIB) summit in London. Damian was appointed to the committee due to his background in machine learning and large-scale data processing and will serve as an expert to evaluate FinTech submissions. The assessment committee consists of member from academia and some of the largest pension fund worldwide.

03.08.2015:
"YFCC100M: The New Data in Multimedia Research" got accepted at CACM



Our paper describing the 100 million images and videos Flickr dataset: “YFCC100M: The New Data in Multimedia Research” got accepted at “Communication of the ACM”. The paper describes the up-to-date largest dataset in the computer vision and multimedia research community. This work was a collaboration between Yahoo Inc., ICSI Berkeley, and the Lawrence Livermore National Laboratory. A pre-print is available on Arxiv.org.

26.07.2015:
Interview with Hannoversche Allgemeine Zeitung about Computer Vision Research



Dr. Damian Borth gave an Interview to Hannoversche Allgemeine Zeitung about the success of deep learning in computer vision research, recent progress to recognize sentiment and emotions in visual content, and how artificial intelligence (AI) can be used for social good.

25.6.2015:
Dr. Damian Borth moderates a Technology Panel at Bloomberg, New York, USA



With the advent of FinTech, disciplines like machine learning, natural language processing, and large-scale data mining are moving into the finance and investment industry. To discuss the potential of FinTech, Dr. Damian Borth was invited to moderate a Technology Panel at FinTech Bloomberg event. The panel was including CEOs and CTOs of leading startups in the area such as: eRevalue, TruValue and Bloomberg Labs.

02.06.2015:
Dr. Damian Borth speaks at RI Europe about Financial Data Science, London, UK



Dr. Damian Borth speaks about Big Data and Environmental, Social and Governance research and introduces together with Prof. Andreas Hoepner the concept of “Financial Data Science” at the RI Europe - a major conference in the responsible investment community. Financial Data Science embraces the processing of large datasets with machine learning methods and welcomes approaches such as opinion mining from public available data on the web.

14.04.2015:
Call for paper for Multimedia COMMONS Workshop is out



The CfP to the Multimedia COMMONS workshop (MMCommons) is out! We invite the community to attend MMCommons and explore the possibilities for novel research, future data challenges, and new benchmarks offered by this large-scale open dataset (YFCC100m dataset). The MMCommons will be held at the ACM Multimedia 2015, Brisbane, Australia around October 26-30.

01.04.2015:
Dr. Damian Borth appointed new head of MADM group



By April 2015, Damian Borth became the new head of MADM. Before this position Damian was a postdoctoral research fellow at the International Computer Science Institute (ICSI) with Dr. Gerald Friedland and at UC Berkeley with Prof. Trevor Darrell where he was involved in various project sponsored by DARPA, IARPA, and the Lawrence Livermore National Laboratory. His research focuses concept detection from social multimedia including trending topic detection, visual sentiment analysis, and multimedia opinion mining.

01.04.2015:
Invited talk about "Data Science and Opinion Mining" at the London School of Economics, UK



The project Multimedia Opinion Mining (MOM) was presented in the context of Data Science during an UN PRI event at the London School of Economics. Dr. Damian Borth introduced data science and opinion mining and outlined how it can be used for social good. The event was later interrupted by an evacuation due to the Holborn Fire in London.

31.03.2015:
Dr. Damian Borth gives an invited talk at the University of Reading, UK



Dr. Damian Borth speaks about “Mining Insights from Public Data” at an event of the University of Reading and outlines concepts and goals of the Multimedia Opinion Mining (MOM) project to the audience.

18.03.2015:
Dr. Damian Borth invited to BMBF dinner with Dr. Georg Schütte in Berkeley, USA



During a visit of the Bundesministerium für Forschung und Entwicklung (BMBF) in the Bay Area, the BMBF invited DAAD scholarship researcher from ICSI to join an exclusive dinner with Deputy Secretary Dr. Georg Schütte. Dr. Damian Borth introduces the BMBF funded project Multimedia Opinion Mining (MOM) to the delegation during the dinner.

01.03.2015:
BMBF funded project Multimedia Opinion Mining (MOM) starts



Multimedia Opinion Mining (MOM), a 2.5 years long project funded by the BMBF, aims to address the challenge of opinion mining in social multimedia content. The main goals of MOM are: (1) analyze social media data to detect and track trending topics. (2) understand multimedia content with respect to its sentiment and opinion. (3) discover individual persons or groups able to influence propagation of topics in social networks. And finally, (4) forecasting the progression of identified topics into the near future. The project will be coordinated within MADM and lead by Dr. Damian Borth.


DEMOS








VISION

This project aims to address the challenge of opinion mining of multimedia content from the web. This comprises a large-scale multi-modal analysis of social media streams and their underlying network dynamics considering different media channels such as Twitter, YouTube, Flickr, Google, and Wikipedia. The main contribution of MOM is four-fold as illustrated below:


1. Trending Topics

social media data is analyzed utilizing information from structured data sources to detect and track trending topics.

2. Sentiment Analysis

large amounts of multimedia content are analyzed with respect to its sentiment and opinion. This follows a holistic approach i.e. the analysis of a rich set of different modalities such as textual, visual, and tempo-visual content.

3. Social Network Analysis

this information is further enriched by network analysis to grasp structural diffusion as well as global and local impact of information in social networks.

4. Forecasting

the possibility of forecasting the progression of identified opinion topics into the near future is investigated.


PUBLICATIONS


  • AudioSentibank: Large-scale Semantic Ontology of Acoustic Concepts for Audio Content Analysis
    Sebastian Sager, Damian Borth, Benjamin Elizalde, Christian Schulze, Bhiksha Raj, Ian Lane, Andreas Dengel
    arXiv preprint arXiv:1607.03766, 2016

    [paper]

  • Multimedia COMMONS Workshop 2016 (MMCommons’16) - Datasets, Evaluation, and Reproducibility
    Bart Thomee, Damian Borth, Julia Bernd
    ACM Multimedia Workshops, 2016

    [website]

  • Generating Captions with Emotions using Concept and Syntax Transition Networks
    Philipp Blandfort, Tushar Karayil, Damian Borth, Andreas Dengel
    ACM Multimedia, Grand Challenge, 2016 (accepted)

    [challenge website]

  • Contextual Enrichment of Remote-Sensed Events with Social Media Streams
    Benjamin Bischke, Damian Borth, Christian Schulze, Andreas Dengel
    ACM Multimedia, Grand Challenge, 2016 (accepted)

    [website]

  • Which languages do people speak on Flickr? A Language and Geo-Location Study of the YFCC100m Dataset
    Alireza Koochali, Sebastian Kalkowski, Damian Borth, Christan Schulze, Andreas Dengel
    ACM Multimeda Community-Organized Multimodal Mining: Opportunities for Novel Solutions (MMCOMMONS) Workshop, 2016 (accepted)

    [YFCC100m Browser]

  • Introducing Concept And Syntax Transition Networks for Image Captioning
    Philipp Blandfort, Tushar Karayil, Damian Borth, Andreas Dengel
    ACM Int. Conf. on Multimedia Retrieval (ICMR), 2016

  • An Evolutionary Algorithm to Learn SPARQL Queries: Finding Patterns for Human Associations in DBpedia
    Jörn Hees, Rouven Bauer, Joachim Folz, Damian Borth, Andreas Dengel
    arXiv preprint arXiv:1607.07249, 2016

    [paper]

  • Trending Topic Aggregation by News-based Context Modeling
    Sebastian Fuchs, Damian Borth and Adrian Ulges
    KI, 2016 (accepted)

  • YFCC100m: The New Data in Multimedia Research
    Bart Thomee, David Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Dough Poland, Damian Borth, Li-Jia Li
    Communication of the ACM, 2016, (YFCC100m Dataset Paper)

    [paper] [watch video] [YFCC100m Browser]

  • Kickstarting the Commons: The YFCC100M and the YLI Corpora
    Julia Bernd, Damian Borth, Carmen Carrano, Jaeyoung Choi, Benjamin Elizalde, Gerald Friedland, Luke Gottlieb, Karl Ni, Roger Pearce, Doug Poland, Khalid Ashraf, David A. Shamma, and Bart Thomee
    ACM Multimeda Community-Organized Multimodal Mining: Opportunities for Novel Solutions (MMCOMMONS) Workshop, 2015

  • Real-time Analysis and Visualization of the YFCC100m Dataset
    Sebastian Kalkowski, Christian Schulze, Andreas Dengel, Damian Borth
    ACM Multimeda Community-Organized Multimodal Mining: Opportunities for Novel Solutions (MMCOMMONS) Workshop, 2015

    [YFCC100m Browser]

  • What makes a Beautiful Landscape beautiful: Adjective Noun Pairs Attention by Eye-Tracking and Gaze Analysis
    Mohammad Al-Naser, Seyyed Saleh Mozaffari Chanijani, Syed Saqib Bukhari, Damian Borth, Andreas Dengel
    ACM Workshop on Affect and Sentiment in Multimedia (ASM), 2015

  • Aesthetic Photo Enhancement using Machine Learning and Case-Based Reasoning
    Joachim Folz, Christian Schulze, Damian Borth, Andreas Dengel
    ACM Workshop on Affect and Sentiment in Multimedia (ASM), 2015

  • The New Data and New Challenges in Multimedia Research
    Bart Thomee, David Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Dough Poland, Damian Borth, Li-Jia Li
    arXiv, preprint arXiv:1503.01817, March, 2015

    [preprint]

  • Large-Scale Deep Learning on the YFCC100M Dataset
    Karl Ni, Roger Pearce, Kofi Boakye, Brian Van Essen, Damian Borth, Barry Chen, Eric Wang
    arXiv, preprint arXiv:1502.03409, February, 2015

    [paper]


PEOPLE


The following people contribute to the project.

Prof. Dr. Prof. h.c. Andreas Dengel

Head of Department

    Dr. Damian Borth

    Project Lead

      Dr. Christian Schulze

      Researcher

        Jörn Hees

        Researcher

          Sebastian Palacio

          Researcher

            Joachim Folz

            Junior Researcher

              Philipp Blandfort

              PhD student

                Tushar Karayil

                PhD student

                  Benjamin Bischke

                  Master

                    Sebastian Kalkowski

                    Master

                      Alireza Koochali

                      Master

                        Jalpa Punjabi

                        Master

                          Ahsan Naeem

                          Master

                            Jessica Chwalek

                            HiWi

                              Ayushman Dash

                              HiWi

                                Aman Gautam

                                HiWi

                                  Monit Shah Singh

                                  HiWi

                                    The following people are former member of the project.

                                    Prakriti Bhardwaj

                                    Master

                                      Peter Burkert

                                      Master

                                        Tobias Hamann

                                        Master

                                          Sebastian Säger

                                          Master

                                            Rajesh Chidambaram

                                            HiWi

                                              Mihai Fieraru

                                              HiWi

                                                Kshitij Kumar

                                                Internship


                                                  CONTACT










                                                  • Dr. Damian Borth
                                                    • Trippstaedter Str. 122
                                                      67663 Kaiserslautern
                                                      • +49 (631) 20575-4184
                                                        • damian.borth@dfki.de


                                                        German Research Center for Articial Intelligence
                                                        [Internal login]