(PDF) Personality-Driven Social Multimedia Content Recommendation

Personality-Driven Social Multimedia Content Recommendation Qi Yang Sergey Nikolenko

ITMO Univerity Steklov Institute of Mathematics at St. Petersburg Saint Petersburg, Russia Saint Petersburg, Russia Alfred Huang Aleksandr Farseev

Multimedia Research Lab, SoMin.ai ITMO Univerity Singapore Saint Petersburg, Russia ABSTRACT digital ad strategy recommendations, which when deployed are able Social media marketing plays a vital role in promoting brand and to improve digital advertising efficiency by over 420% as compared product values to wide audiences. In order to boost their adver- to the original human-guided approach. tising revenues, global media buying platforms such as Facebook Ads constantly reduce the reach of branded organic posts, pushing CCS CONCEPTS brands to spend more on paid media ads. In order to run organic • Information systems → Learning to rank; Multimedia and and paid social media marketing efficiently, it is necessary to un- multimodal retrieval; Computational advertising. derstand the audience, tailoring the content to fit their interests and online behaviours, which is impossible to do manually at a KEYWORDS large scale. At the same time, various personality type categoriza- tion schemes such as the Myers-Briggs Personality Type indicator User Profiling, Multimedia Retrieval, Machine Learning, Recom- make it possible to reveal the dependencies between personality mender System, Deep Learning traits and user content preferences on a wider scale by categoriz- ACM Reference Format: ing audience behaviours in a unified and structured manner. Still, Qi Yang, Sergey Nikolenko, Alfred Huang, and Aleksandr Farseev. 2022. McKinsey-style manual categorization is a very labour-intensive Personality-Driven Social Multimedia Content Recommendation. In Pro- task that is probably impractical in a real-world scenario, so auto- ceedings of the 30th ACM International Conference on Multimedia (MM ’22), mated incorporation of audience behaviour and personality mining October 10–14, 2022, Lisboa, Portugal. ACM, New York, NY, USA, 10 pages. into industrial applications is necessary. This problem is yet to be https://doi.org/10.1145/3503161.3548769 studied in depth by the research community, while the level of impact of different personality traits on content recommendation accuracy has not been widely utilised and comprehensively evalu- 1 INTRODUCTION ated so far. Even worse, there is no dataset available for the research Over the past decade, social networks have become an integral part community to serve as a benchmark and drive further research in of our lives. Hundreds of millions, if not billions, of people, use so- this direction. The present study is one of the first attempts to cial networks on a daily basis, reading their feeds, communicating bridge this important industrial gap, contributing not just a novel with friends, watching videos... and watching ads. The growing personality-driven content recommendation approach and dataset, use of social media has led to a corresponding exponential growth but also facilitating a real-world ready solution which is scalable in social media advertising, with ads on Facebook and other social and sufficiently accurate to be applied in real-world settings. Specif- media playing an increasingly important role in product promotion ically, in this work we investigate the impact of human personality and customer human making. At the same time, another impor- traits on the content recommendation model by applying a novel tant recent trend to note is a decisive decrease of the so-called [50] personality-driven multi-view content recommender system called audience reach, which is the number of people who can see one’s Personality Content Marketing Recommender Engine, or PersiC. Our post on social media with no paid digital advertising (also known experimental results and real-world case study demonstrate not as Facebook posts boosting) involved. It looks like the users are be- just PersiC’s ability to perform efficient human personality-driven coming increasingly harder to “trick” into engaging with the digital multi-view content recommendation, but also allow for actionable content and spreading the word about the ads they see organically, while Facebook itself reduces the organic ad impressions artificially Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed leading brands to scale on digital via paid ads. for profit or commercial advantage and that copies bear this notice and the full citation Following this global trend, brands and marketing agencies pro- on the first page. Copyrights for components of this work owned by others than ACM mote products on social media via paid advertising campaigns on must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a ad distribution engines such as Meta Ads (formerly Facebook Ads). fee. Request permissions from

. A typical campaign consists of multiple ads grouped together, and MM ’22, October 10–14, 2022, Lisboa, Portugal even in the simplest case an ad can contain both text and image © 2022 Association for Computing Machinery. ACM ISBN 978-1-4503-9203-7/22/10. . . $15.00 data. Moreover, modern advertisers and agencies, like Somin.ai, are https://doi.org/10.1145/3503161.3548769 often constrained by particular ad targeting capabilities aimed at 7290 MM ’22, October 10–14, 2022, Lisboa, Portugal Qi Yang, Sergey Nikolenko, Alfred Huang, and Aleksandr Farseev psychology and used for personality profiling in, e.g., career plan- ning and market research. However, until now there has not been much research devoted to the incorporation of personality traits into industry-applicable actionable content recommendations, and we are making one of the first large-scale attempts to solve the problem at a new level of adaptability and performance. Encouraged by the industrial needs and research gap described above, we pose the following research questions: RQ1 is it possible to predict personality traits from data available to recommender systems in a social network setting? RQ2 can we improve content recommendations by incorporating personality traits predicted from social media data? RQ3 what is the difference in content consumption preferences between users exhibiting different personality traits and if such insights could add value to digital advertising routines? To answer our research questions, in this study we make one of the first attempts to perform personality-driven content rec- ommendation, that is sufficiently accurate to be applied in real- world settings. Particularly, we investigate the impact of human personality on the content recommendation model performance. Our experimental results and real-world case study demonstrate not just PersiC’s ability to perform efficient human personality-driven multi-view content recommendation, but also allow for gaining actionable digital ad strategy insights, which are able to drive 420% Figure 1: Four sample brand posts from BMW, BOOHOO, digital advertising performance boost as compared to traditional Dior, and KrispyCreme on Twitter; note the different content human-based approaches1 . The paper is organized as follows: in styles brands use to promote their products or events. Section 2 we give an overview of previous research on personality traits and recommender systems, Section 3 introduces the datasets used to learn a personality trait model and give recommendations, Section 4 presents our approach, the PersiC framework for rec- ommendation with personality trait features, Section 5 presents a capturing the desired social network audience base (e.g., casual fit- comprehensive experimental evaluation, and Section 6 concludes ness practitioners), which, in turn, is expected to be associated with the paper. higher ad conversion rates (e.g., more likely to purchase sports- related products). Figure 1 shows four sample brand posts that 2 RELATED WORK illustrate how brands use different content styles to promote their This work presents a new recommender system with personal- products. ity features so the nearest prior art is related to recommendation In order to find a viable solution to this important problem for systems. Classical collaborative filtering, i.e., completion of the the content marketing industry, many recommender systems have user-item feedback matrix, is mostly based on matrix factorization been proposed in both academia and industry. In addition to classi- (MF) [4, 39, 52]; MF-based approaches still present a competitive cal collaborative filtering, modern recommender systems are able to baseline, and many variations and applications have been developed leverage multi-view item content (in our case, text and images), user over the years [60], but there are several viable alternatives even demographics, temporal criteria, and many more (see Section 2). In for the standard matrix completion problem. In particular, we note this work, we are making another step towards personalized con- factorization machines [58] that combine SVMs and matrix factoriza- tent recommendation by proposing to leverage, arguably, the most tion and are able to train on implicit feedback. Most popular modern influential human decision-making component: our personality. approaches utilize user and item embeddings, vectors in a space that Specifically, we use personality traits, generalized labels proposed should represent the similarity between users and items that would in psychology to classify mental activity patterns related to acquir- be useful for recommendations, and the mapping to the embedding ing information, making decisions, and generally dealing with life; space does not have to be linear as in classical MF. In particular, it stands to reason that such labels can be utilized to understand autoencoder-based methods replace direct learning of the user fea- content preferences and tailor ads to a specific person. ture matrix with learning a function that maps user feedback to user In this work, we have adopted a widely-utilized personality cat- embeddings; this field began with shallow autoencoders [61, 76] egorization scheme called the Myers–Briggs Type Indicator (MBTI), and then moved to variational autoencoders that allow training which breaks down human personality into 16 types, one for ev- deeper and more complex models [22, 36, 44, 47, 51, 63]. Another ery composition of four binary components (traits): Extroversion approach to collaborative filtering is based on graph convolutional vs. Introversion, Sensing vs. iNtuition, Thinking vs. Feeling, and Judging vs. Perceiving. MBTI categorization is widely accepted in 1 Special thanks to Somin.ai for providing the environment of experiment. 7291 Personality-Driven Social Multimedia Content Recommendation MM ’22, October 10–14, 2022, Lisboa, Portugal networks, including NFCF [74] and LightGCN [31]; training such by using personality-labeled word categories [35, 65], while the networks is computationally intensive but they demonstrate im- works [2, 67] instead utilized pretrained GloVe embeddings [56] and pressive performance, and recent approaches such as GF-CF [62] were the first to report the results of machine learning-driven uni- and UltraGCN [49] improved both performance and computational modal personality inference. Image-based personality recognition efficiency. As collaborative filtering baselines, in this work we use methods can utilize, e.g., correlating specific image features with basic MF, Neural Collaborative Filtering (NeuCF) that uses deep neu- personality traits [37, 57], localization of attention on images a user ral networks to learn user and item embeddings [32], and Bilateral liked [84], joint learning of personality types and emotions from Variational Autoencoder for Collaborative Filtering (BiVAECF) that facial images [82], and more. Finally, there are user behavior-based learns embeddings via variational autoencoders [70]. personality extraction methods based, e.g., on Facebook likes [66]. We propose a model that takes into account personality traits, Modern social network data, however, is multi-source, multi- so we need recommender systems able to incorporate external in- view, and multimodal, combining text, images, videos, and other formation. Such recommender systems usually employ a content data sources for a single person. There exist several studies that analysis module that extracts item representations from unstruc- have approached user profiling from a multimodal data perspective. tured information such as text and images and a user profiling For example, Farseev et al. [21] proposed a multimodal ensemble module that extracts user representations from features such as model for the demographic profiling problem from multimodal demographics (or, in our case, personality traits). Most modern data, a work that was later extended to leverage sensor data and approaches are actually hybrid recommender systems [9, 19] able multi-source multi-task learning for wellness profiling Farseev and to leverage both content/demographics and the user-item matrix; Chua [17, 18]. In Buraya et al. [7], relationship status between so- additional information can include external knowledge bases [11], cial network users was predicted by applying classical machine content-related features such as tags or keywords [64], dynamically learning techniques on early-fused data from Twitter, Instagram, chosen expert users whose opinions should be trusted more [42], Facebook, and Foursquare, achieving a significant 17% increase in and so on. In embedding-based approaches, additional user and performance compared to unimodal learning. Going further, Tsai item features are used to inform either the neural network produc- et al. [71] proposed a factorization method to model the intra-modal ing these embeddings or the neural network that performs recom- and inter-modal relationships within multimodal data inputs, which mendations based on them; such approaches include the widely proved to be important for the incorporation of multimodal data used DeepCoNN [83], YouTube recommendations [10], embeddings into user profiling, while Buraya et al. [6] instead leveraged the based on topic models [72], and more. As the primary baseline for temporal component of the multimodal data, being the first to apply this work, we selected the Personalized Content Discovery (PCD) deep learning methods for multi-view personality profiling. While model [29] because it was tailored to a similar problem of con- multimodal data has already been tackled in these works, all of them tent discovery for brands but also note several other works that still lacked multi-source cross-social network data processing [16], extend recommender systems with extra features and extra data which limits their applicability in the majority of real-world sce- modalities [8, 20, 28, 34, 45, 46, 68, 69, 73, 77, 81]. narios. Therefore, in this work, we base our personality profiling Finally, we note prior art in the research on personality traits. on the PERS framework [79] that is able to learn from multi-view There have been several research directions that model human data for personality profiling by efficiently leveraging highly varied personality traits with different statistical approaches. First, the Big data from diverse social multimedia sources. Five, or NEO-PI model proposed in [13] was based on the assump- Recommender systems have previously used personality traits in tion that human personality is reflected in their written language, several different ways [12]. Most of them use automatic personality so statistical analysis of the latter can inform us of the former. In- profiling since questionnaire data can hardly be assumed to be avail- spired by this idea, the LIWC word categorization scheme [54] later able in a real-world setting. Classical approaches use personality provided a numerical connection between personality traits and features to define a similarity score (proximity function) between written language utilization patterns; relations between the Big users and use it (perhaps together with standard CF proximity) Five and MBTI personality types are well known [27]. for recommendations [3, 26, 75] or add personality features to ma- Automatic personality profiling started in the early 2000s, with trix factorization models in a way similar to SVD++ [15, 24, 25]. cross-disciplinary research using machine learning techniques for Approaches based on deep learning have only recently begun to automatic human personality inference based on data collected incorporate personality features, and so far these approaches have from questionnaires and personality tests [1, 48]. Early studies, not used standard personality types but rather inferred their own however, were conducted on relatively small datasets and did not personality feature vectors [23, 33, 43]. In this work, we propose make use of the huge data provided by social networks. This prob- a hybrid approach that uses predicted MBTI personality type to lem was acknowledged and partially mitigated in the MyPersonality inform a deep-learning-based recommender system. project [40] that was able to provide the first large-scale personality- labeled dataset that includes user-generated data from Facebook. This dataset soon became the basis for larger-scale studies in social 3 DATASET media personality profiling research [30, 41, 66]. These studies made Our main contribution in this work is to infer a user’s personality a big leap in the field, but most of them were still focused on a single with multi-modal data from their timeline and recommend suitable data source such as, e.g., Facebook, or a single data modality, e.g., text. content based on the user’s content preferences guided by their In particular, the Linguistic Inquiry and Word Count (LIWC) works personality traits. Therefore, to benchmark our model on real-world are mostly focused on text processing and predicting personality data, we need a large-scale dataset of user interaction with historical 7292 MM ’22, October 10–14, 2022, Lisboa, Portugal Qi Yang, Sergey Nikolenko, Alfred Huang, and Aleksandr Farseev social media content of various brands; it would also be preferable to cover several different industries in order to minimize industry bias. There exist several datasets which consist of posts from social media networks and personality traits. For example, the myPersonality dataset [40] had been widely used in research but is not available anymore, and we know of no other large-scale efforts that would be suitable for our task. Thus, we need to build our own dataset; in this section, we outline our data gathering and preprocessing methodology. 3.1 Data acquisition We choose Twitter as our main data source since it is one of the most open social media outlets, known to concentrate on the users’ self-expression rather than their identity and capture more about public personality intended for the broader public because of its high engagement rate. Inspired by [29], we composed a list of official Twitter accounts of various brands and collected all of their historical posts. We then collected a list of users who liked the posts and the corresponding users’ posts and liked posts in their timeline. As for image content of timeline data and brand posts, we selected image previews for video posts and chose the first image for multiple-image posts. Finally, we filtered out brands with less than 100 posts, kept the most recent 100 posts for every brand, and filtered out users who had only one interaction with a brand’s post. The resulting dataset statistics are shown in Table 1. Figure 2: Illustration of the PersiC framework. 3.2 Data representation For all models utilized in this work, we extracted both textual and 4.1 Problem setting visual features from the ads as follows. We denote users by U = {𝑢 1, 𝑢 2, . . . , 𝑢𝑖 , . . . , 𝑢𝑛 } and brand posts For textual features: by P = {𝑝 1, 𝑝 2, . . . , 𝑝𝑖 , . . . , 𝑝𝑚 }. In this notation, the model’s goal • for each of Brand posts, we extracted the tf-idf features is to learn a scoring function 𝑓 to recommend content for users for every post to form the document-term matrix and then such that for a post 𝑝𝑥 that a user 𝑢 likes and a post 𝑝 𝑦 that user applied latent semantic analysis (LSA) [14] to reduce the 𝑢 did not interact with we would have 𝑓 (𝑢, 𝑝𝑥 ) > 𝑓 (𝑢, 𝑝 𝑦 ). In this textual feature dimension to 100; work, given a group of brand posts, we aim to learn a ranking • for each user, we concatenated their timeline data into the model 𝑓 to rank the content that the user has not interacted with corresponding user-specific “documents” and then extracted to indicate which posts have a higher chance that the user will like the tf-idf features and the sentiment feature by LIWC lexi- them according to the user’s content preferences. con; For visual features, we have chosen to represent visual data in 4.2 Proposed Method terms of emotional and sentiment-related concepts. To do that, we The structure of the PersiC framework is illustrated in Figure 2. It used a pretrained visual concept detector model SentiBank [5] to ex- has two primary components: post representation and user repre- tract visual features; the model outputs a distribution of 2089 visual sentation learning. sentiment concepts such as BeautifulNight, HappyFace, or Clas- 4.2.1 User Representation Learning. The first component is de- sicDesign. For each brand post, we extracted the visual sentiment signed to map the users into a common latent space. The most concept distribution, and to represent a user’s visual preferences common approach to learn user representations is through the we extracted the concept distribution of every image in the user’s one-hot representation of features [78, 80] or a fixed size latent timeline and averaged across the concepts, getting a distribution of vector [29]. However, in our case personality traits do not provide user preferences with respect to the concepts. enough information to distinguish users: with the default approach users who share very similar personality traits would have very 4 PERSIC FRAMEWORK high similarity and we would fail to learn fine-grained user repre- This section is dedicated to presenting our Personality-Driven Con- sentations and fail to recommend relevant content. For this reason, tent Recommendation (PersiC) framework for the problem of content PersiC learns a fine-grained user representation by utilizing the recommendation for users. We first present the problem setting, rich multi-modal data from the user’s timeline in a social media then describe the framework itself, and finally report details of the platform, using personality traits as additional features to inform optimization method we used to train the model. this representation. 7293 Personality-Driven Social Multimedia Content Recommendation MM ’22, October 10–14, 2022, Lisboa, Portugal Table 1: Dataset statistics. Item Brands Brands Posts Interaction Users User Posts User Images Sparsity Quantity 48 4800 330545 41901 6547342 1407775 99.835% Following the data representation method described in Section 3.2, As a result, we learn 𝑓 : 𝑈 × 𝑃 → R by minimizing we represent the timeline data of a user 𝑢𝑖 as a collection of features: ∑︁ 𝐿(𝐷) = ℓ (𝑢, 𝑝 pos, 𝑝 neg ), d𝑖 = {𝑇𝑝𝑖 , 𝐼𝑝𝑖 ,𝑇𝑙𝑖 , 𝐼𝑙𝑖 , 𝐿𝑝𝑖 , 𝐿𝑙𝑖 }, 𝑢,𝑝 pos ,𝑝 neg ∈𝐷 where 𝑇 denotes text features, 𝐼 represents the average value of the where triples (𝑢, 𝑝 pos, 𝑝 neg ) are sampled as above, and image concepts distribution, and 𝐿 denotes the LIWC feature [55]; ℓ (𝑢, 𝑝 pos, 𝑝 neg ) = log 𝜎 𝑦ˆ (𝑢, 𝑝 pos ) − 𝑦ˆ (𝑢, 𝑝 neg ) − 𝜆𝜃 ∥ Θ ∥ 2, the subscript 𝑝 represents features extracted from the user’s time- line and 𝑙 represents features extracted from the user’s historical where 𝑦ˆ (𝑢, 𝑝 pos ) and 𝑦ˆ (𝑢, 𝑝 neg ) are scores predicted by our model favorite posts. for user 𝑢 and items 𝑝 pos and 𝑝 neg respectively, and Θ represents To obtain a representation of personality traits, we used the the model parameters. PERS model proposed in [79]. This model predicts user personality We trained the model with Adam optimizer for 30 epochs, using traits from multi-modal inputs (collected user posts), producing mini-batches of 64 user-post pairs shuffled at the beginning of every four classifiers for each of the four MBTI trait pairs (E-I, S-N, T-F, epoch. We also utilized dropout with rate 0.3. and J-P). We use the features from the penultimate layer of the PERS model, with 3 features for each classifier, so in total we obtain 5 EXPERIMENTAL EVALUATION a vector pers𝑖 ∈ R12 . Using this output, we represent the 𝑖th user In this section, we report on our experimental study conducted to with evaluate the performance of PersiC. We begin by describing the 𝑢𝑖 = Concat 𝜓 (d𝑖 ), pers𝑖 , experimental setup, then provide the results of various tests to illus- where trate the efficiency of our proposed model, including comparisons 𝜓 (d𝑖 ) = Linear (Concat(d𝑖 )) with baselines and an ablation study to illustrate the impact of personality traits on performance. Finally, we present qualitative is a linear perceptron with ReLU activation that fuses the previously case studies on different personality traits. extracted features d𝑖 ; as a result, we get a user latent representation as u𝑖 ∈ R512 . 5.1 Quality metrics 4.2.2 Post Representation. The second component of PersiC is ob- In order to evaluate the impact of human personality traits on taining post representations. Pure collaborative filtering methods content recommendation, we have used the following standard would represent posts as one-hot encoded items and would not uti- metrics: lize any information from the post; our approach follows the latent • Area Under Curve (AUC): AUC computes the area under the representation paradigm (see Section 2 for an overview): PersiC ROC (Receiver Operating Characteristic) curve for classifi- learns from the post’s multi-modal data and projects the post in a cation problems; larger AUC is better; common latent space with the users, where recommendations will • Normalized Discounted Cumulative Gain (nDCG): nDCG mea- be made. To achieve this goal, similar to the user representation, sures the quality of ranking models based on evaluating a we denote the 𝑗th collection of extracted textual and visual features ranked list of 𝑛 top results predicted by the model as follows: from a brand’s post by e 𝑗 = {𝑇𝑏 𝑗 , 𝐼𝑏 𝑗 }, where 𝑇 denotes text fea- 𝑛 tures, 𝐼 represents the average value of image concepts distribution, ∑︁ 𝑟𝑖 DCG𝑛 DCG𝑛 = , nDCG𝑛 = , and subscript 𝑏 denotes features extracted from brand posts. We 𝑖=1 log 2 (𝑖 + 1) IDCG 𝑛 then add one fully connected layer 𝛾 with ReLU activations to fuse where 𝑟𝑖 is the ground truth relevance of the 𝑖th item on the textual and image features, and obtain the final representation the list, and IDCG is the ideal DCG, i.e., DCG that would be as 𝑝 𝑗 = 𝛾 Concat(e 𝑗 ) . obtained if the results were ranked according to their actual relevance, so nDCG𝑛 is a number between 0 and 1; higher 4.3 Loss function and optimization nDCG is better; Our dataset 𝐷 := {(𝑢, 𝑝)} consists of user-post pairs that show • 𝐹 1 -measure: 𝐹 1 -measure is defined as the harmonic average a post liked by a given user. Inspired by [29], we utilize pairwise 2·precision·recall of precision and recall, 𝐹 1 = precision+recall ; higher precision learning, sampling a fixed number of negative samples for each pair. and recall are better, so higher 𝐹 1 -measure is also better. Based on the assumption that relevant content, i.e., a liked post 𝑝 pos should have a higher score than irrelevant content 𝑝 neg for a user In this work, we truncated the nDCG and 𝐹 1 -measure ranked 𝑢, we adopted the Bayesian personalized ranking (BPR) loss func- lists of results at 10 and 50 respectively. tion [59] for our model. Thus, for learning we use (𝑢, 𝑝 pos, 𝑝 neg ) triples; to construct them, we take a pair (𝑢, 𝑝 pos ) ∈ 𝐷 and uni- 5.2 Baselines formly sample 20 negative posts 𝑝 neg (posts that user 𝑢 did not like) To the best of our knowledge, there are no preexisting models devel- for each positive pair. oped specifically for personality-driven content recommendation 7294 MM ’22, October 10–14, 2022, Lisboa, Portugal Qi Yang, Sergey Nikolenko, Alfred Huang, and Aleksandr Farseev based on brand posts on social media platforms. Thus, we com- where 𝑞(p𝑖 | 𝑋𝑖∗ ) = N (p𝑖 | 𝝁 (𝑋𝑖∗, 𝝍), 𝝈 (𝑋𝑖∗, 𝝍)) and 𝑞(q 𝑗 | pare our approach with several recommender system baselines (see 𝑋 ∗𝑗 ) = N (q 𝑗 | 𝝁 (𝑋 ∗𝑗 , 𝝓), 𝝈 (𝑋 ∗𝑗 , 𝝓)) are Gaussians whose Section 2 for a general survey). parameters are functions of the corresponding rows and columns of 𝑋 with additional variational parameters 𝝍 and 𝝓; 5.2.1 Factorization Machines (FM). Since we are in the implicit in BiVAE-CF, these functions are defined by neural networks feedback setting, for a classical low-rank factorization approach to with parameters 𝝍 and 𝝓, specifically multilayer perceptrons; collaborative filtering we chose factorization machines that show • variational parameters 𝝍 and 𝝓 are optimized by minimizing good performance in this setting [58]. For each user 𝑖 and each item the variational lower bound 𝑗, degree two FM models their possible interaction with a vector ∑︁ x ∈ R𝑛 that contains one-hot representations of 𝑖 and 𝑗 and any 𝐿VAE = E𝑞 log 𝑝 (𝑥𝑖 𝑗 | p𝑖 , q 𝑗 ) − additional features. Then the model learns to predict the target 𝑖,𝑗 variable 𝑦 (“click”/“no click” in our case) as ∑︁ ∑︁ − KL(𝑞(p𝑖 | 𝑋𝑖∗ )∥𝑝 (p𝑖 )) − KL(𝑞(q 𝑗 | 𝑋 ∗𝑗 ) ∥𝑝 (q 𝑗 )) 𝑛 ∑︁ ∑︁ 𝑑 𝑖 𝑗 𝑦ˆFM (x) = 𝑤 0 + w⊤ x + (v𝑘⊤ v𝑙 )𝑤𝑘𝑙 , 𝑘=1 𝑙=𝑘+1 via the reparametrization trick [38] and alternating optimiza- tion with respect to user and item variational parameters where w and𝑊 are weights of the model (in particular, the matrix𝑊 (we refer to [70] for details). represents weights of interactions between features), and v𝑘 ∈ R𝑑 are feature embeddings, so the user and item embeddings p𝑖 ∈ R𝑑 We consider BiVAE-CF to be a strong state-of-the-art baseline for and q 𝑗 ∈ R𝑑 in FM are v𝑘 for the corresponding components of x; collaborative filtering in our setting. see [58] for details about optimization in FM. 5.2.4 Personalized Content Discovery (PCD). Introduced in [29], 5.2.2 Neural Collaborative Filtering (NeuCF). This is a popular PCD is the model which is nearest to ours in the overall setting and approach to recommender systems that generalizes matrix factor- data used; in particular, it is able to leverage multi-view data and ization to nonlinear mappings by replacing the inner product with make full use of the dataset we have collected. PCD is also inspired a neural architecture that can learn a more expressive function of by matrix factorization but designed to learn latent representations data to produce user and item features. We use the neural matrix for brands and posts on social networks. It proceeds as follows: factorization model from [32] designed for implicit feedback: one- • one-hot brand id representation and brand associations mod- hot representations of each user 𝑖 and item 𝑗 are used as input for eled as a matrix of association vectors 𝐴 ∈ R𝑛×𝑘 are com- Í two different embeddings. Matrix factorization user and item vec- bined into a brand representation vector x𝑏 = 𝑛𝑖=1 𝐴𝑖∗ ◦ w𝑏 , tors p𝑖MF ∈ R𝑑 and qMF 𝑑 MF MF 𝑗 ∈ R are combined into 𝝓 𝑖 𝑗 = p𝑖 ◦ q 𝑗 MF where w𝑏 are importance weights for brand 𝑏, so a brand is with componentwise multiplication ◦; this represented as a weighted combination of associations; could lead to the gen- eralized matrix factorization 𝑦ˆGMF = ℎ w⊤ (p𝑖MF ◦ qMF • a post with an image is processed first via a pretrained feature 𝑗 ) with a extractor network and then with additional two linear layers weight wector w and activation function ℎ, so for w = 1 and ℎ = id with leaky ReLU activations to produce a post feature vector GMF degenerates into regular matrix factorization. The multilayer x𝑝 for every post 𝑝; perceptron user and item vectors p𝑖MLP ∈ R𝑑 and qMLP ∈ R𝑑 go 𝑗 • finally, the network parameters are trained with a pairwise through a neural architecture (several MLP layers) to obtain 𝝓𝑖MLP 𝑗 , ranking loss and then the last layer uses the concatenation of 𝝓𝑖MF MLP to 𝑗 and 𝝓 𝑖 𝑗 ∑︁ 𝐿PCD = max 0, 𝑓 (𝑏, 𝑝 pos ) − 𝑓 (𝑏, 𝑝 pos ) + 𝜂 + 𝛼 |w𝑏 | + 𝛽 ∥𝜽 ∥ 2, ˆ see [32] for details. predict 𝑦; 𝑏 5.2.3 Bilateral Variational Autoencoder for Collaborative Filtering x𝑏⊤ x𝑝 (BiVAE-CF). This model, presented in [70] learns embeddings via where 𝑓 (𝑏, 𝑝) = ∥ is the normalized scalar product, ∥x𝑏 ∥ ∥x𝑝 variational autoencoders. This is a generative model that uses the and the 𝐿1 regularizer encourages sparsity in the attention user-item matrix 𝑋 to learn latent representations p𝑖 ∈ R𝑑 and weights w𝑏 . q 𝑗 ∈ R𝑑 as follows: As a result, PCD learns latent brand and post representations to- • standard Gaussian priors: 𝑝 (p𝑖 ) = N (0, 1), 𝑝 (q 𝑗 ) = N (0, 1); gether with a fine-grained structure of brand associations. To apply • conditional on p𝑖 and q 𝑗 , the observations in 𝑋 are drawn PCD to recommendations with implicit feedback, we replace brands from a distribution from the univariate exponential family with users, and the rest of the model is unchanged. 𝑝 (𝑥𝑖 𝑗 | p𝑖 , q 𝑗 ) = ℎ(𝑥𝑖 𝑗 )𝑒 𝜂 (p𝑖 ,q 𝑗 ,𝜔)𝑥𝑖 𝑗 −𝑎 (𝜂 (p𝑖 ,q 𝑗 ,𝜔)) ; 5.3 Evaluation results in our case, we used the Bernoulli distribution since the In this section, we evaluate the performance of PersiC against other matrix 𝑋 is binary (likes); baselines. We split the dataset into training and test subsets in the • the (untractable) posterior 𝑝 (𝑃, 𝑄 | 𝑋 ) in this model is ap- 80 : 20 ratio, stratified by the number of each user’s posts. For a fair proximated with a tractable distribution comparison, we have reproduced all the baselines with the settings Ö ! suggested in the original papers and optimized on the training set, ©Ö then evaluated the performance with the test set separately. Results 𝑞(𝑃, 𝑄 | 𝑋 ) = 𝑞(p𝑖 | 𝑋𝑖∗ ) ­ 𝑞(q 𝑗 | 𝑋 ∗𝑗 ) ® , ª 𝑖 « 𝑗 ¬ are listed in the Table 2. 7295 Personality-Driven Social Multimedia Content Recommendation MM ’22, October 10–14, 2022, Lisboa, Portugal Table 2: Experimental evaluation: performance of PersiC and Table 3: Ablation study of various feature combinations for baseline models. user representations. AUC nDCG10 nDCG50 F110 F150 AUC nDCG10 nDCG50 F110 F150 MF 0.801 0.011 0.027 0 0.006 One-hot 0.765 0.042 0.079 0.013 0.023 NeuCF 0.807 0.061 0.105 0.039 0.065 Posts 0.889 0.087 0.113 0.045 0.089 BiVAECF 0.852 0.075 0.115 0.045 0.048 Likes 0.791 0.077 0.095 0.031 0.076 PCD 0.881 0.082 0.121 0.048 0.091 Posts+Likes 0.891 0.087 0.113 0.045 0.091 PersiC 0.905 0.092 0.125 0.052 0.095 Posts+Pers 0.897 0.089 0.123 0.047 0.091 Posts+Likes+Pers 0.905 0.092 0.125 0.052 0.095 The table clearly shows that PersiC performs best against other baselines in terms of all considered metrics. The baseline models themselves also perform more or less as expected: the simplest MF model is the worst, neural collaborative filtering in the form of NeuCF and BiVAE-CF performed better than MF, but, naturally, they are still inferior when compared to the content-based approaches of PCD and PersiC. Note the very low performance of MF that indicates that the recommendation problem we consider is quite hard. These findings indicate that in this sparse multi-view scenario, straightforward collaborative filtering is insufficient, and additional information in the form of rich multi-modal data from both users and items can significantly improve the performance. Moreover, the proposed PersiC model has significantly improved results compared to a different commonly used content-based ap- proach, the PCD model, across all metrics and especially for the nDCG metric where PersiC performed 12% better. We attribute this finding to two reasons: first, in the method of learning the user representation PCD only maps a user id into a fixed sized vector in pre-allocated latent space, while PersiC learns a fine-grained user representation by leveraging rich multi-modal data in the user’s timeline and records of their favorite posts, and second and most important, PersiC makes use of the personality traits inferred from the user’s social media platform activities. This gives a positive an- swer to our RQ1: indeed, even automatically inferred personality traits can significantly improve the performance of downstream Figure 3: Most correlated concepts for each personality trait recommendation models. Overall, Table 2 shows that PersiC is from the Brand’s timeline data. able to adapt to the multi-view sparse environment by learning fine-grained multimodal post and user representations and provide recommendations superior to other approaches. posts. This can be explained by the high noise in users’ historical favorite posts and may be impacted by the algorithm of content 5.4 Ablation study and the influence of recommendation in the social media platform. Finally, comparing personality traits the performance of “Posts+Likes+Pers” with “Posts+Likes” and We have conducted an ablation study to evaluate the importance “Posts+Pers” with “Posts”, we see significant performance boosts of various features for the user representation and their influence from adding user personality features. This finding answers pos- on the final results in terms of recommendation quality metrics. itively our RQ2: introducing personality features into a content Table 3 shows the results of this ablation study. recommendation system has been able to substantially improve First, obviously, the simple one-hot encoding obtains the worst personalized content recommendation performance. results since it does not utilize additional user information at all. Learning user representations only from historical post data achieved 5.5 Qualitative results and case study significantly better scores. Comparing to the performance of PCD in To further evaluate the performance of PersiC, we have conducted Table 2, we also see that the performance is improved by leveraging a case study with real-world data collected from the audience of a the user’s historical data. sports brand (hereinafter called Brand). Brand is a major Decathlon Table 3 shows that posts are more useful than likes for the quality Competitor in one of the European markets. The main goal of the of downstream recommendations; note also that the performance Brand is to increase the engagement rate and return on advertising generally improves only a little compared to the one-hot encoding spend (ROAS) by designing better content and choosing proper if we learn the user representation by the user’s historical favorite psychographics. We have collected historical posts from Brand’s 7296 MM ’22, October 10–14, 2022, Lisboa, Portugal Qi Yang, Sergey Nikolenko, Alfred Huang, and Aleksandr Farseev Based on these psychographic insights, with the help of Somin.ai platform, we have automatically created an advertising campaign for the Brand on Meta (a.k.a. Facebook Ads) that leverages pre- dicted personality traits and other user interests [7]. The above has helped to achieve more than 4.2x cost reduction (for cost per result) compared to the traditional approach adopted by the Brand’s agency. This growth in conversion rate has further led to 6.11x more app installs (the advertising budget was increased proportionally) while bringing more insights into the audience’s psychographic behavioural traits. To be precise, we were able to correlate back the best ad units and the corresponding user personality traits, which brought another insight for the Brand: it turned out that purchasing decisions of the majority of the Brand’s e-commerce customers are shifted towards long-term implications of owning a product rather than particular product features or its price. As a result, Brand has modified the Figure 4: Most highly correlated images for each personality content of their reach and awareness ads that were previously fo- trait from the Brand’s timeline data. cused on feature- and price-centric content and achieved further improvement of their digital advertising campaign strategy. New fresh audiences visiting the Brand’s website were receiving a consis- and a potential competitor’s timelines on Twitter and Instagram. tent message on the “last mile” of their purchasing journey, just as We then further harvested timeline data from the audiences that they did at the stage of familiarizing themselves with the product. had interacted with Brand’s posts. Next, as described in Section 4, we extracted visual concepts from Brand’s and competitor’s time- 6 CONCLUSION lines. Finally, we extracted audience-specific concepts and inferred In this work, we have presented PersiC, a novel personality-driven psychographic attributes from the audiences’ timelines. multi-view content recommender system, which is driven by per- We begin with a qualitative picture of PersiC’s results. Figure 3 sonality traits inferred from user activities on social media. PersiC shows image concepts mined by SentiBank that have turned out to is able to capture fine-grained user representations by extracting be most highly correlated with the 16 MBTI personality categories multi-view features from the user’s posts to provide personalized based on images from Brand’s timeline, and Fig. 4 shows several recommendations. Moreover, we have shown a case study with a sample images for each personality trait. Note that the distributions very successful real-life advertising campaign (improving ROAS by of concepts are very different across personality traits; in particular, > 4x and app installs by > 6x) guided by insights learned by PersiC. there are several strong positive correlations: HotLegs, SexyGirls, Finally, we are also publishing our multi-view large-scale content YoungCouple, AwesomeTatto, and YoungFriends for the Extrovert recommendation dataset for further research in this exciting direc- trait and Text Block, Beautiful Night, Nice Guy, Awesome Design, tion. and LonelyGuy with the Introvert trait. This observation conforms For further work, we note that although the PersiC framework well with the theory that extroverts are more active and enjoy improves recommendation performance according to user content social interaction while introverts are thought-oriented and enjoy preferences, there is an important potential issue that can pollute spending time alone [53]. We also note similarities and differences the results: user activities on their timeline are impacted by content in concept distributions between traits. For example, Introvert and recommendation algorithms adopted by social media platforms Judging are significantly positively correlated to similar concepts such as Twitter or Facebook. Therefore, we see the further work’s such as text and design, reflecting the fact that both traits are backbone direction in studying what is the impact of recommenda- thought-oriented and prefer structured, firm decisions [53]. We tion algorithms from social media platforms on user activities and can also verify this observation on Fig. 4: Introvert and Judging how it can be accounted for in recommender systems. traits share similar visual content preferences of numbers, deals, and similar shapes. Fig. 3 can yield many such observations: e.g., SexyGirls are most important for Extrovert and Perceiving traits, 7 ACKNOWLEDGEMENT while NiceGuy is best suited for Introvert and Thinking traits. This work was funded by the Russian Science Foundation grant №. Overall, we have found that the distributions of concepts are 22-11-00135 https://rscf.ru/en/project/22-11-00135/. reasonable and conform well to the assumptions behind this study. In this practical case study for Brand, we have gone beyond REFERENCES just making recommendations with PersiC into qualitative content [1] Shlomo Argamon, Sushant Dhawle, Moshe Koppel, and James W. Pennebaker. 2005. Lexical Predictors Of Personality Type. In Proceedings of the Joint Annual generation insights. We have extracted from PersiC the learned Meeting of the Interface and the Classification Society of North America. mappings and correlations between user content preferences with [2] Pierre-Hadrien Arnoux, Anbang Xu, Neil Boyette, Jalal Mahmud, Rama Akkiraju, Brand’s post concepts and the common concepts for each person- and Vibha Sinha. 2017. 25 Tweets to Know You: A New Model to Predict Per- sonality with Social Media. Proceedings of the International AAAI Conference on ality trait (cf. Figure 3). Then, we have studied insights from the Web and Social Media 11, 1 (May 2017). https://ojs.aaai.org/index.php/ICWSM/ concepts (cf. Figure 4) to guide Brand’s content design. article/view/14963 7297 Personality-Driven Social Multimedia Content Recommendation MM ’22, October 10–14, 2022, Lisboa, Portugal [3] Nana Yaw Asabere, Amevi Acakpovi, and Mathias Bennet Michael. 2018. Im- arXiv:abs/2002.01726 proving Socially-Aware Recommendation Accuracy Through Personality. IEEE [24] He Feng and Xueming Qian. 2013. Recommendation via User’s Personality and Transactions on Affective Computing 9, 3 (2018), 351–361. https://doi.org/10.1109/ Social Contextual. In Proceedings of the 22nd ACM International Conference on TAFFC.2017.2695605 Information & Knowledge Management (San Francisco, California, USA) (CIKM [4] Robert Bell, Yehuda Koren, and Chris Volinsky. 2007. Modeling relationships at ’13). Association for Computing Machinery, New York, NY, USA, 1521–1524. multiple scales to improve accuracy of large recommender systems. In Proceedings https://doi.org/10.1145/2505515.2507834 of the 13th ACM SIGKDD international conference on Knowledge discovery and [25] Ignacio Fernández-Tobías, Matthias Braunhofer, Mehdi Elahi, Francesco Ricci, data mining. 95–104. and Iván Cantador. 2016. Alleviating the New User Problem in Collaborative [5] Damian Borth, Rongrong Ji, Tao Chen, Thomas Breuel, and Shih-Fu Chang. 2013. Filtering by Exploiting Personality Information. User Modeling and User-Adapted Large-Scale Visual Sentiment Ontology and Detectors Using Adjective Noun Interaction 26, 2–3 (jun 2016), 221–255. https://doi.org/10.1007/s11257-016-9172- Pairs. In Proceedings of the 21st ACM International Conference on Multimedia z (Barcelona, Spain) (MM ’13). Association for Computing Machinery, New York, [26] Ignacio Fernández-Tobías and Iván Cantador. 2014. Personality-Aware Collabo- NY, USA, 223–232. https://doi.org/10.1145/2502081.2502282 rative Filtering: An Empirical Study in Multiple Domains with Facebook Data. [6] Kseniya Buraya, Aleksandr Farseev, and Andrey Filchenkov. 2018. Multi-view In E-Commerce and Web Technologies, Martin Hepp and Yigal Hoffner (Eds.). personality profiling based on longitudinal data. In International Conference of Springer International Publishing, Cham, 125–137. the Cross-Language Evaluation Forum for European Languages. Springer, 15–27. [27] Adrian Furnham. 1996. The big five versus the big four: the relationship be- [7] Kseniya Buraya, Aleksandr Farseev, Andrey Filchenkov, and Tat-Seng Chua. 2017. tween the Myers-Briggs Type Indicator (MBTI) and NEO-PI five factor model Towards user personality profiling from multiple social networks. In Proceedings of personality. Personality and Individual Differences 21, 2 (1996), 303–307. of the AAAI Conference on Artificial Intelligence, Vol. 31. https://doi.org/10.1016/0191-8869(96)00033-5 [8] Yingying Cai, Xiali Li, Licheng Wu, and Qian Li. 2022. Knowledge-Graph-Aware [28] Chen Gao, Yong Li, Fuli Feng, Xiangning Chen, Kai Zhao, Xiangnan He, and Recommendation in Movie Domain. In Proceedings of 2021 Chinese Intelligent Depeng Jin. 2021. Cross-Domain Recommendation with Bridge-Item Embeddings. Automation Conference, Zhidong Deng (Ed.). Springer Singapore, Singapore, 211– ACM Trans. Knowl. Discov. Data 16, 1, Article 2 (jul 2021), 23 pages. https: 218. //doi.org/10.1145/3447683 [9] Erion Çano and Maurizio Morisio. 2019. Hybrid Recommender Systems: A [29] Francesco Gelli, Tiberio Uricchio, Xiangnan He, Alberto Del Bimbo, and Tat-Seng Systematic Literature Review. CoRR abs/1901.03888 (2019). arXiv:1901.03888 Chua. 2018. Beyond the Product: Discovering Image Posts for Brands in Social http://arxiv.org/abs/1901.03888 Media. In Proceedings of the 26th ACM International Conference on Multimedia [10] Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep Neural Networks (Seoul, Republic of Korea) (MM ’18). Association for Computing Machinery, New for YouTube Recommendations. In Proceedings of the 10th ACM Conference on York, NY, USA, 465–473. https://doi.org/10.1145/3240508.3240689 Recommender Systems (Boston, Massachusetts, USA) (RecSys ’16). Association for [30] Matej Gjurković and Jan Snajder. 2018. Reddit: A Gold Mine for Personality Computing Machinery, New York, NY, USA, 191–198. https://doi.org/10.1145/ Prediction. In Proceedings of the Second Workshop on Computational Modeling 2959100.2959190 of People’s Opinions, Personality, and Emotions in Social Media. Association for [11] Yong Deng, Zhonghai Wu, Cong Tang, Huayou Si, Hu Xiong, and Zhong Chen. Computational Linguistics, New Orleans, Louisiana, USA, 87–97. https://doi. 2010. A Hybrid Movie Recommender Based on Ontology and Neural Networks. org/10.18653/v1/W18-1112 In Proceedings of the 2010 IEEE/ACM Int’l Conference on Green Computing and [31] Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Communications & Int’l Conference on Cyber, Physical and Social Computing Wang. 2020. LightGCN: Simplifying and powering graph convolution network for (GREENCOM-CPSCOM ’10). IEEE Computer Society, USA, 846–851. https://doi. recommendation. In Proceedings of the 43rd International ACM SIGIR conference org/10.1109/GreenCom-CPSCom.2010.144 on research and development in Information Retrieval. 639–648. [12] Sahraoui Dhelim, Nyothiri Aung, Mohammed Amine Bouras, Huansheng Ning, [32] Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng and Erik Cambria. 2021. A Survey on Personality-Aware Recommendation Chua. 2017. Neural Collaborative Filtering. In Proceedings of the 26th International Systems. CoRR abs/2101.12153 (2021). arXiv:2101.12153 https://arxiv.org/abs/ Conference on World Wide Web (Perth, Australia) (WWW ’17). International World 2101.12153 Wide Web Conferences Steering Committee, Republic and Canton of Geneva, [13] J M Digman. 1990. Personality Structure: Emergence of the Five-Factor Model. An- CHE, 173–182. https://doi.org/10.1145/3038912.3052569 nual Review of Psychology 41, 1 (1990), 417–440. https://doi.org/10.1146/annurev. [33] Zhenwei He, Lei Zhang, and Fangyi Liu. 2020. DiscoStyle: Multi-level Logistic ps.41.020190.002221 arXiv:https://doi.org/10.1146/annurev.ps.41.020190.002221 Ranking for Personalized Image Style Preference Inference. Int. J. Autom. Comput. [14] Susan T. Dumais. 2004. Latent semantic analysis. Annual Review of Information Sci- 17, 5 (2020), 637–651. https://doi.org/10.1007/s11633-020-1244-1 ence and Technology 38, 1 (2004), 188–230. https://doi.org/10.1002/aris.1440380105 [34] Zhankui He, Handong Zhao, Zhe Lin, Zhaowen Wang, Ajinkya Kale, and Julian arXiv:https://asistdl.onlinelibrary.wiley.com/doi/pdf/10.1002/aris.1440380105 Mcauley. 2021. Locker: Locally Constrained Self-Attentive Sequential Recommen- [15] Mehdi Elahi, Matthias Braunhofer, Francesco Ricci, and Marko Tkalcic. 2013. dation. Association for Computing Machinery, New York, NY, USA, 3088–3092. Personality-Based Active Learning for Collaborative Filtering Recommender https://doi.org/10.1145/3459637.3482136 Systems. In AI*IA 2013: Advances in Artificial Intelligence, Matteo Baldoni, Cristina [35] Thomas Holtgraves. 2011. Text messaging, personality, and the social context. Baroglio, Guido Boella, and Roberto Micalizio (Eds.). Springer International Journal of Research in Personality 45, 1 (2011), 92–99. https://doi.org/10.1016/j. Publishing, Cham, 360–371. jrp.2010.11.015 [16] Aleksandr Farseev. 2017. 360° user profile learning from multiple social networks for [36] Daeryong Kim and Bongwon Suh. 2019. Enhancing VAEs for collaborative wellness and urban mobility applications. Ph. D. Dissertation. National University filtering: flexible priors & gating mechanisms. In Proceedings of the 13th ACM of Singapore (Singapore). Conference on Recommender Systems. 403–407. [17] Aleksandr Farseev and Tat-Seng Chua. 2017. Tweet can be Fit: Integrating [37] Yunhwan Kim and Jang Hyun Kim. 2018. Using computer vision techniques Data from Wearable Sensors and Multiple Social Networks for Wellness Profile on Instagram to link users’ personalities and genders to the features of their Learning. (2017). photos: An exploratory study. Information Processing & Management 54, 6 (2018), [18] Aleksandr Farseev and Tat-Seng Chua. 2017. TweetFit: Fusing Multiple Social 1101–1114. https://doi.org/10.1016/j.ipm.2018.07.005 Media and Sensor Data for Wellness Profile Learning. In Proceedings of the Thirty- [38] Diederik P. Kingma and Max Welling. 2014. Auto-Encoding Variational Bayes. In First AAAI Conference on Artificial Intelligence. AAAI. 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, [19] Aleksandr Farseev, Denis Kotkov, Alexander Semenov, Jari Veijalainen, and Tat- Canada, April 14-16, 2014, Conference Track Proceedings, Yoshua Bengio and Yann Seng Chua. 2015. Cross-Social Network Collaborative Recommendation. In LeCun (Eds.). http://arxiv.org/abs/1312.6114 Proceedings of the ACM International Conference on Web Science (WebSci). ACM. [39] Yehuda Koren and Robert M. Bell. 2011. Advances in Collaborative Filter- [20] Aleksandr Farseev, Kirill Lepikhin, Hendrik Schwartz, Eu Khoon Ang, and Kenny ing. In Recommender Systems Handbook, Francesco Ricci, Lior Rokach, Bracha Powar. 2018. SoMin. ai: Social multimedia influencer discovery marketplace. In Shapira, and Paul B. Kantor (Eds.). Springer, 145–186. http://dblp.uni-trier.de/ Proceedings of the 26th ACM international conference on Multimedia. 1234–1236. db/reference/rsh/rsh2011.html#KorenB11 [21] Aleksandr Farseev, Liqiang Nie, Mohammad Akbari, and Tat-Seng Chua. 2015. [40] Michal Kosinski, Sandra Matz, Samuel Gosling, Vesselin Popov, and David Still- Harvesting multiple sources for user profile learning: a big data study. In Proceed- well. 2015. Facebook as a Research Tool for the Social Sciences. The American ings of the 5th ACM on International Conference on Multimedia Retrieval. ACM, psychologist 70 (09 2015), 543–556. https://doi.org/10.1037/a0039210 235–242. [41] K. N. P. Kumar and M. L. Gavrilova. 2019. Personality Traits Classification on [22] Aleksandr Farseev, Ivan Samborskii, Andrey Filchenkov, and Tat-Seng Chua. Twitter. In 2019 16th IEEE International Conference on Advanced Video and Signal 2017. Cross-domain recommendation via clustering on multi-layer graphs. In Based Surveillance (AVSS). 1–8. https://doi.org/10.1109/AVSS.2019.8909839 Proceedings of the 40th International ACM SIGIR Conference on Research and [42] Kibeom Lee and Kyogu Lee. 2014. Using Dynamically Promoted Experts for Development in Information Retrieval. ACM, 195–204. Music Recommendation. IEEE Transactions on Multimedia 16, 5 (2014), 1201–1210. [23] Aleksnadr Farseev, Qi Yang, Andrey Filchenkov, Kirill Lepikhin, Yu-Yi Chu- https://doi.org/10.1109/TMM.2014.2311012 Farseeva, and Daron-Benjamin Loo. 2020. SoMin.ai: Personality-Driven Con- [43] Leida Li, Hancheng Zhu, Sicheng Zhao, Guiguang Ding, Hongyan Jiang, and tent Generation Platform. (2020). https://doi.org/10.1145/3437963.3441714 Allen Tan. 2019. Personality Driven Multi-task Learning for Image Aesthetic 7298 MM ’22, October 10–14, 2022, Lisboa, Portugal Qi Yang, Sergey Nikolenko, Alfred Huang, and Aleksandr Farseev Assessment. In 2019 IEEE International Conference on Multimedia and Expo (ICME). The 2nd International Conference on Computer Science and Computational 430–435. https://doi.org/10.1109/ICME.2019.00081 Intelligence (ICCSCI 2017). [44] Dawen Liang, Rahul G Krishnan, Matthew D Hoffman, and Tony Jebara. 2018. [68] Md Mehrab Tanjim, Congzhe Su, Ethan Benjamin, Diane Hu, Liangjie Hong, and Variational autoencoders for collaborative filtering. In Proceedings of the 2018 Julian McAuley. 2020. Attentive Sequential Models of Latent Intent for Next Item world wide web conference. 689–698. Recommendation. Association for Computing Machinery, New York, NY, USA, [45] L. Liao, R. Takanobu, Y. Ma, X. Yang, M. Huang, and T. Chua. 2022. Topic- 2528–2534. https://doi.org/10.1145/3366423.3380002 Guided Conversational Recommender in Multiple Domains. IEEE Transactions [69] Zhulin Tao, Xiaohao Liu, Yewei Xia, Xiang Wang, Lifang Yang, Xianglin Huang, on Knowledge & Data Engineering 34, 05 (may 2022), 2485–2496. https://doi.org/ and Tat-Seng Chua. 2022. Self-supervised Learning for Multimedia Recommen- 10.1109/TKDE.2020.3008563 dation. IEEE Transactions on Multimedia (2022), 1–10. https://doi.org/10.1109/ [46] Han Liu, Yinwei Wei, Jianhua Yin, and Liqiang Nie. 2022. HS-GCN: Hamming TMM.2022.3187556 Spatial Graph Convolutional Networks for Recommendation. IEEE Transactions [70] Quoc-Tuan Truong, Aghiles Salah, and Hady W. Lauw. 2021. Bilateral Varia- on Knowledge and Data Engineering (2022). tional Autoencoder for Collaborative Filtering. In Proceedings of the 14th ACM [47] Sam Lobel, Chunyuan Li, Jianfeng Gao, and Lawrence Carin. 2019. RaCT: Toward International Conference on Web Search and Data Mining (Virtual Event, Israel) Amortized Ranking-Critical Training For Collaborative Filtering. In International (WSDM ’21). Association for Computing Machinery, New York, NY, USA, 292–300. Conference on Learning Representations. https://doi.org/10.1145/3437963.3441759 [48] François Mairesse, Marilyn A. Walker, Matthias R. Mehl, and Roger K. Moore. [71] Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency, and 2007. Using Linguistic Cues for the Automatic Recognition of Personality in Ruslan Salakhutdinov. 2019. Learning Factorized Multimodal Representations. In Conversation and Text. J. Artif. Int. Res. 30, 1 (Nov. 2007), 457–500. 7th International Conference on Learning Representations, ICLR 2019, New Orleans, [49] Kelong Mao, Jieming Zhu, Xi Xiao, Biao Lu, Zhaowei Wang, and Xiuqiang He. LA, USA, May 6-9, 2019. OpenReview.net. https://openreview.net/forum?id= 2021. UltraGCN: Ultra Simplification of Graph Convolutional Networks for rygqqsA9KX Recommendation. In Proceedings of the 30th ACM International Conference on [72] Hao Wang, Naiyan Wang, and Dit-Yan Yeung. 2014. Collaborative Deep Learning Information & Knowledge Management. 1253–1262. for Recommender Systems. CoRR abs/1409.2944 (2014). arXiv:1409.2944 http: [50] Michelle Martin. 2022. 39 Facebook Stats That Matter to Marketers in 2022. Re- //arxiv.org/abs/1409.2944 trieved May 2, 2022 from https://blog.hootsuite.com/facebook-statistics/ [73] Xin Wang, Hong Chen, and Wenwu Zhu. 2021. Multimodal Disentangled Repre- [51] Leyla Mirvakhabova, Evgeny Frolov, Valentin Khrulkov, Ivan Oseledets, and sentation for Recommendation. In 2021 IEEE International Conference on Multi- Alexander Tuzhilin. 2020. Performance of hyperbolic geometry models on top-N media and Expo (ICME). 1–6. https://doi.org/10.1109/ICME51207.2021.9428193 recommendation tasks. In Fourteenth ACM Conference on Recommender Systems. [74] Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. 2019. 527–532. Neural graph collaborative filtering. In Proceedings of the 42nd international ACM [52] Andriy Mnih and Russ R Salakhutdinov. 2007. Probabilistic matrix factorization. SIGIR conference on Research and development in Information Retrieval. 165–174. Advances in neural information processing systems 20 (2007). [75] Wen Wu. 2017. Implicit Acquisition of User Personality for Augmenting Rec- [53] I.B. Myers. 1998. MBTI Manual: A Guide to the Development and Use of the ommender Systems. In Proceedings of the 22nd International Conference on Intel- Myers-Briggs Type Indicator. Consulting Psychologists Press. ligent User Interfaces Companion (Limassol, Cyprus) (IUI ’17 Companion). As- [54] J. Pennebaker and L. King. 1999. Linguistic styles: language use as an individual sociation for Computing Machinery, New York, NY, USA, 201–204. https: difference. Journal of personality and social psychology 77 6 (1999), 1296–312. //doi.org/10.1145/3030024.3038287 [55] James W. Pennebaker, Martha E. Francis, and Roger J. Booth. 2001. Linguistic [76] Yao Wu, Christopher DuBois, Alice X Zheng, and Martin Ester. 2016. Collabora- Inquiry and Word Count. Lawerence Erlbaum Associates, Mahwah, NJ. tive denoising auto-encoders for top-n recommender systems. In Proceedings of [56] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: the ninth ACM international conference on web search and data mining. 153–162. Global Vectors for Word Representation. In Empirical Methods in Natural Lan- [77] Yuxia Wu, Lizi Liao, Gangyi Zhang, Wenqiang Lei, Guoshuai Zhao, Xueming Qian, guage Processing (EMNLP). 1532–1543. http://www.aclweb.org/anthology/D14- and Tat-Seng Chua. 2022. State Graph Reasoning for Multimodal Conversational 1162 Recommendation. IEEE Transactions on Multimedia (2022), 1–1. https://doi.org/ [57] Yang Qi, Farseev Aleksandr, and Filchenkov Andrey. 2020. I Know Where You Are 10.1109/TMM.2022.3155900 Coming From: On the Impact of Social Media Sources on AI Model Performance [78] Shahpar Yakhchi, Amin Beheshti, Seyed-Mohssen Ghafari, Mehmet A. Orgun, and (Student Abstract). Proceedings of the AAAI Conference on Artificial Intelligence Guanfeng Liu. 2020. Towards a Deep Attention-Based Sequential Recommender 34, 10 (Apr. 2020), 13971–13972. https://doi.org/10.1609/aaai.v34i10.7258 System. IEEE Access 8 (2020), 178073–178084. https://doi.org/10.1109/ACCESS. [58] Steffen Rendle. 2010. Factorization Machines. In 2010 IEEE International Confer- 2020.3004656 ence on Data Mining. 995–1000. https://doi.org/10.1109/ICDM.2010.127 [79] Qi Yang, Aleksandr Farseev, and Andrey Filchenkov. 2021. Two-Faced Humans [59] Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. on Twitter and Facebook: Harvesting Social Multimedia for Human Personality 2009. BPR: Bayesian Personalized Ranking from Implicit Feedback. In Proceedings Profiling. In ICDAR@ICMR 2021: Proceedings of the 2021 Workshop on Intelligent of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence (Montreal, Cross-Data Analysis and Retrieval, Taipei, Taiwan, 21 August 2021, Minh-Son Quebec, Canada) (UAI ’09). AUAI Press, Arlington, Virginia, USA, 452–461. Dao, Duc-Tien Dang-Nguyen, and Michael Riegler (Eds.). ACM, 39–47. https: [60] Francesco Ricci, Lior Rokach, and Bracha Shapira (Eds.). 2022. Recommender //doi.org/10.1145/3463944.3469270 Systems Handbook. Springer, 3rd edition. [80] Haochao Ying, Fuzhen Zhuang, Fuzheng Zhang, Yanchi Liu, Guandong Xu, [61] Suvash Sedhain, Aditya Krishna Menon, Scott Sanner, and Lexing Xie. 2015. Xing Xie, Hui Xiong, and Jian Wu. 2018. Sequential Recommender System Autorec: Autoencoders meet collaborative filtering. In Proceedings of the 24th based on Hierarchical Attention Networks. In Proceedings of the Twenty-Seventh international conference on World Wide Web. 111–112. International Joint Conference on Artificial Intelligence, IJCAI-18. International [62] Yifei Shen, Yongji Wu, Yao Zhang, Caihua Shan, Jun Zhang, B Khaled Letaief, Joint Conferences on Artificial Intelligence Organization, 3926–3932. https: and Dongsheng Li. 2021. How Powerful is Graph Convolution for Recommenda- //doi.org/10.24963/ijcai.2018/546 tion?. In Proceedings of the 30th ACM International Conference on Information & [81] Fuzheng Zhang, Nicholas Jing Yuan, Defu Lian, Xing Xie, and Wei-Ying Ma. Knowledge Management. 1619–1629. 2016. Collaborative Knowledge Base Embedding for Recommender Systems. [63] Ilya Shenbin, Anton Alekseev, Elena Tutubalina, Valentin Malykh, and Sergey I In Proceedings of the 22nd ACM SIGKDD International Conference on Knowl- Nikolenko. 2020. Recvae: A new variational autoencoder for top-n recommenda- edge Discovery and Data Mining (San Francisco, California, USA) (KDD ’16). tions with implicit feedback. In Proceedings of the 13th International Conference Association for Computing Machinery, New York, NY, USA, 353–362. https: on Web Search and Data Mining. 528–536. //doi.org/10.1145/2939672.2939673 [64] Ana Stanescu, Swapnil Nagar, and Doina Caragea. 2013. A Hybrid Recommender [82] L. Zhang, S. Peng, and S. Winkler. 2022. PersEmoN: A Deep Network for Joint System: User Profiling from Keywords and Ratings. In 2013 IEEE/WIC/ACM Analysis of Apparent Personality, Emotion and Their Relationship. IEEE Transac- International Joint Conferences on Web Intelligence (WI) and Intelligent Agent tions on Affective Computing 13, 01 (jan 2022), 298–305. https://doi.org/10.1109/ Technologies (IAT), Vol. 1. 73–80. https://doi.org/10.1109/WI-IAT.2013.11 TAFFC.2019.2951656 [65] C. Sumner, A. Byers, R. Boochever, and G. J. Park. 2012. Predicting Dark Triad [83] Lei Zheng, Vahid Noroozi, and Philip S. Yu. 2017. Joint Deep Modeling of Users Personality Traits from Twitter Usage and a Linguistic Analysis of Tweets. In and Items Using Reviews for Recommendation. In Proceedings of the Tenth ACM 2012 11th International Conference on Machine Learning and Applications, Vol. 2. International Conference on Web Search and Data Mining (Cambridge, United 386–393. https://doi.org/10.1109/ICMLA.2012.218 Kingdom) (WSDM ’17). Association for Computing Machinery, New York, NY, [66] Michael M. Tadesse, Hongfei Lin, Bo Xu, and Liang Yang. 2018. Personality USA, 425–434. https://doi.org/10.1145/3018661.3018665 Predictions Based on User Behavior on the Facebook Social Media Platform. IEEE [84] Hancheng Zhu, Leida Li, Hongyan Jiang, and Allen Tan. 2020. Inferring Person- Access 6 (2018), 61959–61969. https://doi.org/10.1109/ACCESS.2018.2876502 ality Traits from Attentive Regions of User Liked Images Via Weakly Supervised [67] Tommy Tandera, Hendro, Derwin Suhartono, Rini Wongso, and Yen Lina Prasetio. Dual Convolutional Network. Neural Process. Lett. 51, 3 (jun 2020), 2105–2121. 2017. Personality Prediction System from Facebook Users. Procedia Computer https://doi.org/10.1007/s11063-019-09987-7 Science 116 (2017), 604–611. https://doi.org/10.1016/j.procs.2017.10.016 Discovery and innovation of computer science technology in artificial intelligence era: 7299