-
Latent Performance Profiling of Large Language Models
Authors:
Tanmoy Chakraborty,
Ayan Sengupta,
Suparna Bhattacharya,
Partha Pratim Chakrabarti,
Amlan Chakrabarti,
Supratik Chakraborty,
Partha Pratim Das,
Lipika Dey,
Richa Singh,
Mayank Vatsa
Abstract:
Large language models (LLMs) frequently achieve impressive scores on standardized benchmarks, yet accuracy alone offers a limited view of their capabilities. Evaluating open-source LLMs through leaderboards faces persistent issues like data contamination, narrow task scope, and weak alignment with real-world reliability. Benchmark-based evaluations such as MMLU PRO, BBH, or IFEval primarily captur…
▽ More
Large language models (LLMs) frequently achieve impressive scores on standardized benchmarks, yet accuracy alone offers a limited view of their capabilities. Evaluating open-source LLMs through leaderboards faces persistent issues like data contamination, narrow task scope, and weak alignment with real-world reliability. Benchmark-based evaluations such as MMLU PRO, BBH, or IFEval primarily capture what a model outputs on fixed test sets, not how it processes information, calibrates uncertainty, or structures internal knowledge. In this article, we advocate for a shift from benchmark-centric evaluation toward a complementary, state-centered intrinsic assessment of LLMs. To this end, we introduce Latent Performance Profiling (LPP) -- a framework that derives task-agnostic diagnostics from hidden activations and output distributions. LPP defines a set of scalar metrics on a model's latent representations and dynamics, revealing scale-independent traits that enable interpretable comparisons and uncover hidden vulnerabilities. Unlike static accuracy scores, LPP provides stable, architecture-sensitive signatures across models of similar size. With extensive empirical analyses across eight LLMs, spanning a size range of 0.5B-14B, we demonstrate that models with similar benchmark scores can exhibit contrasting latent profiles, such as differences in entropy or adaptability. Guided by these insights, we design synthetic probes for uncertainty and symbolic reasoning that align with intrinsic metrics while decoupling from leaderboard bias. We recommend that reporting LPP alongside benchmarks provides a deeper, interpretable understanding of model behavior, enabling more reliable model selection, safety assessment, and evaluation beyond surface-level accuracy.
△ Less
Submitted 29 May, 2026; v1 submitted 28 May, 2026;
originally announced May 2026.
-
WTKO-CNN: Deep Learning Reveals Sequence Motifs Distinguishing Wild-Type and Knockout ATAC-seq Peaks
Authors:
Lopamudra Dey
Abstract:
Chromatin regulators can alter transcriptional programs by modifying the accessibility of regulatory DNA elements. Understanding how regulatory sequences differ between wild-type (WT) and knockout (KO) conditions is crucial for deciphering transcriptional control. Here, we applied a convolutional neural network, \textbf{WTKO-CNN} with an attention mechanism to classify DNA sequences as WT or KO, a…
▽ More
Chromatin regulators can alter transcriptional programs by modifying the accessibility of regulatory DNA elements. Understanding how regulatory sequences differ between wild-type (WT) and knockout (KO) conditions is crucial for deciphering transcriptional control. Here, we applied a convolutional neural network, \textbf{WTKO-CNN} with an attention mechanism to classify DNA sequences as WT or KO, achieving high predictive performance. To interpret the model, we generated saliency maps to identify nucleotide positions most influential for the classification decision. From these high-saliency regions, we extracted and clustered k-mers, enabling de novo motif discovery. Sequence logos and consensus motifs derived from the CNN filters revealed biologically meaningful patterns, which are further validated using MEME, TOMTOM, and HOMER against known transcription factor binding sites. Our analysis identified motifs associated with transcription factor families that discriminate WT from KO sequences, demonstrating that CNN-guided saliency mapping is a powerful approach for uncovering functional sequence features.
△ Less
Submitted 21 May, 2026;
originally announced May 2026.
-
What's Not on the Plate? Rethinking Food Computing through Indigenous Indian Datasets
Authors:
Pamir Gogoi,
Neha Joshi,
Ayushi Pandey,
Deepthi Sudharsan,
Saransh Kumar Gupta,
Lipika Dey,
Partha Pratim Das,
Kalika Bali,
Vivek Seshadri
Abstract:
This paper presents a multimodal dataset of 1,000 indigenous recipes from remote regions of India, collected through a participatory model involving first-time digital workers from rural areas. The project covers ten endangered language communities in six states. Documented using a dedicated mobile app, the data set includes text, images, and audio, capturing traditional food practices along with…
▽ More
This paper presents a multimodal dataset of 1,000 indigenous recipes from remote regions of India, collected through a participatory model involving first-time digital workers from rural areas. The project covers ten endangered language communities in six states. Documented using a dedicated mobile app, the data set includes text, images, and audio, capturing traditional food practices along with their ecological and cultural contexts. This initiative addresses gaps in food computing, such as the lack of culturally inclusive, multimodal, and community-authored data. By documenting food as it is practiced rather than prescribed, this work advances inclusive, ethical, and scalable approaches to AI-driven food systems and opens new directions in cultural AI, public health, and sustainable agriculture.
△ Less
Submitted 19 September, 2025;
originally announced September 2025.
-
Towards an Action-Centric Ontology for Cooking Procedures Using Temporal Graphs
Authors:
Aarush Kumbhakern,
Saransh Kumar Gupta,
Lipika Dey,
Partha Pratim Das
Abstract:
Formalizing cooking procedures remains a challenging task due to their inherent complexity and ambiguity. We introduce an extensible domain-specific language for representing recipes as directed action graphs, capturing processes, transfers, environments, concurrency, and compositional structure. Our approach enables precise, modular modeling of complex culinary workflows. Initial manual evaluatio…
▽ More
Formalizing cooking procedures remains a challenging task due to their inherent complexity and ambiguity. We introduce an extensible domain-specific language for representing recipes as directed action graphs, capturing processes, transfers, environments, concurrency, and compositional structure. Our approach enables precise, modular modeling of complex culinary workflows. Initial manual evaluation on a full English breakfast recipe demonstrates the DSL's expressiveness and suitability for future automated recipe analysis and execution. This work represents initial steps towards an action-centric ontology for cooking, using temporal graphs to enable structured machine understanding, precise interpretation, and scalable automation of culinary processes - both in home kitchens and professional culinary settings.
△ Less
Submitted 4 September, 2025;
originally announced September 2025.
-
Extending FKG.in: Towards a Food Claim Traceability Network
Authors:
Saransh Kumar Gupta,
Rizwan Gulzar Mir,
Lipika Dey,
Partha Pratim Das,
Anirban Sen,
Ramesh Jain
Abstract:
The global food landscape is rife with scientific, cultural, and commercial claims about what foods are, what they do, what they should not do, or should not do. These range from rigorously studied health benefits (probiotics improve gut health) and misrepresentations (soaked almonds make one smarter) to vague promises (superfoods boost immunity) and culturally rooted beliefs (cold foods cause cou…
▽ More
The global food landscape is rife with scientific, cultural, and commercial claims about what foods are, what they do, what they should not do, or should not do. These range from rigorously studied health benefits (probiotics improve gut health) and misrepresentations (soaked almonds make one smarter) to vague promises (superfoods boost immunity) and culturally rooted beliefs (cold foods cause coughs). Despite their widespread influence, the infrastructure for tracing, verifying, and contextualizing these claims remains fragmented and underdeveloped. In this paper, we propose a Food Claim-Traceability Network (FCN) as an extension of FKG[.]in, a knowledge graph of Indian food that we have been incrementally building. We also present the ontology design and the semi-automated knowledge curation workflow that we used to develop a proof of concept of FKG[.]in-FCN using Reddit data and Large Language Models. FCN integrates curated data inputs, structured schemas, and provenance-aware pipelines for food-related claim extraction and validation. While directly linked to the Indian food knowledge graph as an application, our methodology remains application-agnostic and adaptable to other geographic, culinary, or regulatory settings. By modeling food claims and their traceability in a structured, verifiable, and explainable way, we aim to contribute to more transparent and accountable food knowledge ecosystems, supporting researchers, policymakers, and most importantly, everyday consumers in navigating a world saturated with dietary assertions.
△ Less
Submitted 4 September, 2025; v1 submitted 22 August, 2025;
originally announced August 2025.
-
Enhancing FKG.in: automating Indian food composition analysis
Authors:
Saransh Kumar Gupta,
Lipika Dey,
Partha Pratim Das,
Geeta Trilok-Kumar,
Ramesh Jain
Abstract:
This paper presents a novel approach to compute food composition data for Indian recipes using a knowledge graph for Indian food (FKG[.]in) and LLMs. The primary focus is to provide a broad overview of an automated food composition analysis workflow and describe its core functionalities: nutrition data aggregation, food composition analysis, and LLM-augmented information resolution. This workflow…
▽ More
This paper presents a novel approach to compute food composition data for Indian recipes using a knowledge graph for Indian food (FKG[.]in) and LLMs. The primary focus is to provide a broad overview of an automated food composition analysis workflow and describe its core functionalities: nutrition data aggregation, food composition analysis, and LLM-augmented information resolution. This workflow aims to complement FKG[.]in and iteratively supplement food composition data from verified knowledge bases. Additionally, this paper highlights the challenges of representing Indian food and accessing food composition data digitally. It also reviews three key sources of food composition data: the Indian Food Composition Tables, the Indian Nutrient Databank, and the Nutritionix API. Furthermore, it briefly outlines how users can interact with the workflow to obtain diet-based health recommendations and detailed food composition information for numerous recipes. We then explore the complex challenges of analyzing Indian recipe information across dimensions such as structure, multilingualism, and uncertainty as well as present our ongoing work on LLM-based solutions to address these issues. The methods proposed in this workshop paper for AI-driven knowledge curation and information resolution are application-agnostic, generalizable, and replicable for any domain.
△ Less
Submitted 4 September, 2025; v1 submitted 6 December, 2024;
originally announced December 2024.
-
Building FKG.in: a Knowledge Graph for Indian Food
Authors:
Saransh Kumar Gupta,
Lipika Dey,
Partha Pratim Das,
Ramesh Jain
Abstract:
This paper presents an ontology design along with knowledge engineering, and multilingual semantic reasoning techniques to build an automated system for assimilating culinary information for Indian food in the form of a knowledge graph. The main focus is on designing intelligent methods to derive ontology designs and capture all-encompassing knowledge about food, recipes, ingredients, cooking char…
▽ More
This paper presents an ontology design along with knowledge engineering, and multilingual semantic reasoning techniques to build an automated system for assimilating culinary information for Indian food in the form of a knowledge graph. The main focus is on designing intelligent methods to derive ontology designs and capture all-encompassing knowledge about food, recipes, ingredients, cooking characteristics, and most importantly, nutrition, at scale. We present our ongoing work in this workshop paper, describe in some detail the relevant challenges in curating knowledge of Indian food, and propose our high-level ontology design. We also present a novel workflow that uses AI, LLM, and language technology to curate information from recipe blog sites in the public domain to build knowledge graphs for Indian food. The methods for knowledge curation proposed in this paper are generic and can be replicated for any domain. The design is application-agnostic and can be used for AI-driven smart analysis, building recommendation systems for Personalized Digital Health, and complementing the knowledge graph for Indian food with contextual information such as user information, food biochemistry, geographic information, agricultural information, etc.
△ Less
Submitted 1 September, 2024;
originally announced September 2024.
-
Trustworthy and Responsible AI for Human-Centric Autonomous Decision-Making Systems
Authors:
Farzaneh Dehghani,
Mahsa Dibaji,
Fahim Anzum,
Lily Dey,
Alican Basdemir,
Sayeh Bayat,
Jean-Christophe Boucher,
Steve Drew,
Sarah Elaine Eaton,
Richard Frayne,
Gouri Ginde,
Ashley Harris,
Yani Ioannou,
Catherine Lebel,
John Lysack,
Leslie Salgado Arzuaga,
Emma Stanley,
Roberto Souza,
Ronnie de Souza Santos,
Lana Wells,
Tyler Williamson,
Matthias Wilms,
Zaman Wahid,
Mark Ungrin,
Marina Gavrilova
, et al. (1 additional authors not shown)
Abstract:
Artificial Intelligence (AI) has paved the way for revolutionary decision-making processes, which if harnessed appropriately, can contribute to advancements in various sectors, from healthcare to economics. However, its black box nature presents significant ethical challenges related to bias and transparency. AI applications are hugely impacted by biases, presenting inconsistent and unreliable fin…
▽ More
Artificial Intelligence (AI) has paved the way for revolutionary decision-making processes, which if harnessed appropriately, can contribute to advancements in various sectors, from healthcare to economics. However, its black box nature presents significant ethical challenges related to bias and transparency. AI applications are hugely impacted by biases, presenting inconsistent and unreliable findings, leading to significant costs and consequences, highlighting and perpetuating inequalities and unequal access to resources. Hence, developing safe, reliable, ethical, and Trustworthy AI systems is essential.
Our team of researchers working with Trustworthy and Responsible AI, part of the Transdisciplinary Scholarship Initiative within the University of Calgary, conducts research on Trustworthy and Responsible AI, including fairness, bias mitigation, reproducibility, generalization, interpretability, and authenticity. In this paper, we review and discuss the intricacies of AI biases, definitions, methods of detection and mitigation, and metrics for evaluating bias. We also discuss open challenges with regard to the trustworthiness and widespread application of AI across diverse domains of human-centric decision making, as well as guidelines to foster Responsible and Trustworthy AI models.
△ Less
Submitted 2 September, 2024; v1 submitted 28 August, 2024;
originally announced August 2024.
-
Mapping Patient Trajectories: Understanding and Visualizing Sepsis Prognostic Pathways from Patients Clinical Narratives
Authors:
Sudeshna Jana,
Tirthankar Dasgupta,
Lipika Dey
Abstract:
In recent years, healthcare professionals are increasingly emphasizing on personalized and evidence-based patient care through the exploration of prognostic pathways. To study this, structured clinical variables from Electronic Health Records (EHRs) data have traditionally been employed by many researchers. Presently, Natural Language Processing models have received great attention in clinical res…
▽ More
In recent years, healthcare professionals are increasingly emphasizing on personalized and evidence-based patient care through the exploration of prognostic pathways. To study this, structured clinical variables from Electronic Health Records (EHRs) data have traditionally been employed by many researchers. Presently, Natural Language Processing models have received great attention in clinical research which expanded the possibilities of using clinical narratives. In this paper, we propose a systematic methodology for developing sepsis prognostic pathways derived from clinical notes, focusing on diverse patient subgroups identified by exploring comorbidities associated with sepsis and generating explanations of these subgroups using SHAP. The extracted prognostic pathways of these subgroups provide valuable insights into the dynamic trajectories of sepsis severity over time. Visualizing these pathways sheds light on the likelihood and direction of disease progression across various contexts and reveals patterns and pivotal factors or biomarkers influencing the transition between sepsis stages, whether toward deterioration or improvement. This empowers healthcare providers to implement more personalized and effective healthcare strategies for individual patients.
△ Less
Submitted 20 July, 2024;
originally announced July 2024.
-
Generating insights about financial asks from Reddit posts and user interactions
Authors:
Sachin Thukral,
Suyash Sangwan,
Vipul Chauhan,
Arnab Chatterjee,
Lipika Dey
Abstract:
As an increasingly large number of people turn to platforms like Reddit, YouTube, Twitter, Instagram, etc. for financial advice, generating insights about the content generated and interactions taking place within these platforms have become a key research question. This study proposes content and interaction analysis techniques for a large repository created from social media content, where peopl…
▽ More
As an increasingly large number of people turn to platforms like Reddit, YouTube, Twitter, Instagram, etc. for financial advice, generating insights about the content generated and interactions taking place within these platforms have become a key research question. This study proposes content and interaction analysis techniques for a large repository created from social media content, where people interactions are centered around financial information exchange. We propose methods for content analysis that can generate human-interpretable insights using topic-centered clustering and multi-document abstractive summarization. We share details of insights generated from our experiments with a large repository of data gathered from subreddit for personal finance. We have also explored the use of ChatGPT and Vicuna for generating responses to queries and compared them with human responses. The methods proposed in this work are generic and applicable to all large social media platforms.
△ Less
Submitted 12 March, 2024; v1 submitted 7 March, 2024;
originally announced March 2024.
-
Understanding how social discussion platforms like Reddit are influencing financial behavior
Authors:
Sachin Thukral,
Suyash Sangwan,
Arnab Chatterjee,
Lipika Dey,
Aaditya Agrawal,
Pramit Kumar Chandra,
Animesh Mukherjee
Abstract:
This study proposes content and interaction analysis techniques for a large repository created from social media content. Though we have presented our study for a large platform dedicated to discussions around financial topics, the proposed methods are generic and applicable to all platforms. Along with an extension of topic extraction method using Latent Dirichlet Allocation, we propose a few mea…
▽ More
This study proposes content and interaction analysis techniques for a large repository created from social media content. Though we have presented our study for a large platform dedicated to discussions around financial topics, the proposed methods are generic and applicable to all platforms. Along with an extension of topic extraction method using Latent Dirichlet Allocation, we propose a few measures to assess user participation, influence and topic affinities specifically. Our study also maps user-generated content to components of behavioral finance. While these types of information are usually gathered through surveys, it is obvious that large scale data analysis from social media can reveal many potentially unknown or rare insights. Characterising users based on their platform behavior to provide critical insights about how communities are formed and trust is established in these platforms using graphical analysis is also studied.
△ Less
Submitted 12 March, 2024; v1 submitted 7 March, 2024;
originally announced March 2024.
-
Detection and Classification of Novel Attacks and Anomaly in IoT Network using Rule based Deep Learning Model
Authors:
Sanjay Chakraborty,
Saroj Kumar Pandey,
Saikat Maity,
Lopamudra Dey
Abstract:
Attackers are now using sophisticated techniques, like polymorphism, to change the attack pattern for each new attack. Thus, the detection of novel attacks has become the biggest challenge for cyber experts and researchers. Recently, anomaly and hybrid approaches are used for the detection of network attacks. Detecting novel attacks, on the other hand, is a key enabler for a wide range of IoT appl…
▽ More
Attackers are now using sophisticated techniques, like polymorphism, to change the attack pattern for each new attack. Thus, the detection of novel attacks has become the biggest challenge for cyber experts and researchers. Recently, anomaly and hybrid approaches are used for the detection of network attacks. Detecting novel attacks, on the other hand, is a key enabler for a wide range of IoT applications. Novel attacks can easily evade existing signature-based detection methods and are extremely difficult to detect, even going undetected for years. Existing machine learning models have also failed to detect the attack and have a high rate of false positives. In this paper, a rule-based deep neural network technique has been proposed as a framework for addressing the problem of detecting novel attacks. The designed framework significantly improves respective benchmark results, including the CICIDS 2017 dataset. The experimental results show that the proposed model keeps a good balance between attack detection, untruthful positive rates, and untruthful negative rates. For novel attacks, the model has an accuracy of more than 99%. During the automatic interaction between network-devices (IoT), security and privacy are the primary obstacles. Our proposed method can handle these obstacles efficiently and finally identify, and classify the different levels of threats.
△ Less
Submitted 29 July, 2023;
originally announced August 2023.
-
Identifying pandemic-related stress factors from social-media posts -- effects on students and young-adults
Authors:
Sachin Thukral,
Suyash Sangwan,
Arnab Chatterjee,
Lipika Dey
Abstract:
The COVID-19 pandemic has thrown natural life out of gear across the globe. Strict measures are deployed to curb the spread of the virus that is causing it, and the most effective of them have been social isolation. This has led to wide-spread gloom and depression across society but more so among the young and the elderly. There are currently more than 200 million college students in 186 countries…
▽ More
The COVID-19 pandemic has thrown natural life out of gear across the globe. Strict measures are deployed to curb the spread of the virus that is causing it, and the most effective of them have been social isolation. This has led to wide-spread gloom and depression across society but more so among the young and the elderly. There are currently more than 200 million college students in 186 countries worldwide, affected due to the pandemic. The mode of education has changed suddenly, with the rapid adaptation of e-learning, whereby teaching is undertaken remotely and on digital platforms. This study presents insights gathered from social media posts that were posted by students and young adults during the COVID times. Using statistical and NLP techniques, we analyzed the behavioral issues reported by users themselves in their posts in depression-related communities on Reddit. We present methodologies to systematically analyze content using linguistic techniques to find out the stress-inducing factors. Online education, losing jobs, isolation from friends, and abusive families emerge as key stress factors.
△ Less
Submitted 1 December, 2020;
originally announced December 2020.
-
Identification, Tracking and Impact: Understanding the trade secret of catchphrases
Authors:
Jagriti Jalal,
Mayank Singh,
Arindam Pal,
Lipika Dey,
Animesh Mukherjee
Abstract:
Understanding the topical evolution in industrial innovation is a challenging problem. With the advancement in the digital repositories in the form of patent documents, it is becoming increasingly more feasible to understand the innovation secrets -- "catchphrases" of organizations. However, searching and understanding this enormous textual information is a natural bottleneck. In this paper, we pr…
▽ More
Understanding the topical evolution in industrial innovation is a challenging problem. With the advancement in the digital repositories in the form of patent documents, it is becoming increasingly more feasible to understand the innovation secrets -- "catchphrases" of organizations. However, searching and understanding this enormous textual information is a natural bottleneck. In this paper, we propose an unsupervised method for the extraction of catchphrases from the abstracts of patents granted by the U.S. Patent and Trademark Office over the years. Our proposed system achieves substantial improvement, both in terms of precision and recall, against state-of-the-art techniques. As a second objective, we conduct an extensive empirical study to understand the temporal evolution of the catchphrases across various organizations. We also show how the overall innovation evolution in the form of introduction of newer catchphrases in an organization's patents correlates with the future citations received by the patents filed by that organization. Our code and data sets will be placed in the public domain soon.
△ Less
Submitted 20 July, 2020;
originally announced July 2020.
-
Innovation and Revenue: Deep Diving into the Temporal Rank-shifts of Fortune 500 Companies
Authors:
Mayank Singh,
Arindam Pal,
Lipika Dey,
Animesh Mukherjee
Abstract:
Research and innovation is important agenda for any company to remain competitive in the market. The relationship between innovation and revenue is a key metric for companies to decide on the amount to be invested for future research. Two important parameters to evaluate innovation are the quantity and quality of scientific papers and patents. Our work studies the relationship between innovation a…
▽ More
Research and innovation is important agenda for any company to remain competitive in the market. The relationship between innovation and revenue is a key metric for companies to decide on the amount to be invested for future research. Two important parameters to evaluate innovation are the quantity and quality of scientific papers and patents. Our work studies the relationship between innovation and patenting activities for several Fortune 500 companies over a period of time. We perform a comprehensive study of the patent citation dataset available in the Reed Technology Index collected from the US Patent Office. We observe several interesting relations between parameters like the number of (i) patent applications, (ii) patent grants, (iii) patent citations and Fortune 500 ranks of companies. We also study the trends of these parameters varying over the years and derive causal explanations for these with qualitative and intuitive reasoning. To facilitate reproducible research, we make all the processed patent dataset publicly available at https://github.com/mayank4490/Innovation-and-revenue.
△ Less
Submitted 20 April, 2020;
originally announced April 2020.
-
Characterizing behavioral trends in a community driven discussion platform
Authors:
Sachin Thukral,
Arnab Chatterjee,
Hardik Meisheri,
Tushar Kataria,
Aman Agarwal,
Ishan Verma,
Lipika Dey
Abstract:
This article presents a systematic analysis of the patterns of behavior of individuals as well as groups observed in community-driven platforms for discussion like Reddit, where users usually exchange information and viewpoints on their topics of interest. We perform a statistical analysis of the behavior of posts and model the users' interactions around them. A platform like Reddit which has grow…
▽ More
This article presents a systematic analysis of the patterns of behavior of individuals as well as groups observed in community-driven platforms for discussion like Reddit, where users usually exchange information and viewpoints on their topics of interest. We perform a statistical analysis of the behavior of posts and model the users' interactions around them. A platform like Reddit which has grown exponentially, starting from a very small community to one of the largest social networks, with its large user base and popularity harboring a variety of behavior of users in terms of their activity. Our work provides interesting insights about a huge number of inactive posts which fail to attract attention despite their authors exhibiting Cyborg-like behavior to attract attention. We also observe short-lived yet extremely active posts emulate a phenomenon like Mayfly Buzz. A method is presented, to study the activity around posts which are highly active, to determine the presence of Limelight hogging activity. We also present a systematic analysis to study the presence of controversies in posts. We analyzed data from two periods of one-year duration but separated by few years in time, to understand how social media has evolved through the years.
△ Less
Submitted 7 November, 2019;
originally announced November 2019.
-
Analyzing behavioral trends in community driven discussion platforms like Reddit
Authors:
Sachin Thukral,
Hardik Meisheri,
Tushar Kataria,
Aman Agarwal,
Ishan Verma,
Arnab Chatterjee,
Lipika Dey
Abstract:
The aim of this paper is to present methods to systematically analyze individual and group behavioral patterns observed in community driven discussion platforms like Reddit where users exchange information and views on various topics of current interest. We conduct this study by analyzing the statistical behavior of posts and modeling user interactions around them. We have chosen Reddit as an exam…
▽ More
The aim of this paper is to present methods to systematically analyze individual and group behavioral patterns observed in community driven discussion platforms like Reddit where users exchange information and views on various topics of current interest. We conduct this study by analyzing the statistical behavior of posts and modeling user interactions around them. We have chosen Reddit as an example, since it has grown exponentially from a small community to one of the biggest social network platforms in the recent times. Due to its large user base and popularity, a variety of behavior is present among users in terms of their activity. Our study provides interesting insights about a large number of inactive posts which fail to gather attention despite their authors exhibiting Cyborg-like behavior to draw attention. We also present interesting insights about short-lived but extremely active posts emulating a phenomenon like Mayfly Buzz. Further, we present methods to find the nature of activity around highly active posts to determine the presence of Limelight hogging activity, if any. We analyzed over $2$ million posts and more than $7$ million user responses to them during entire 2008 and over $63$ million posts and over $608$ million user responses to them from August 2014 to July 2015 amounting to two one-year periods, in order to understand how social media space has evolved over the years.
△ Less
Submitted 19 September, 2018;
originally announced September 2018.
-
A Machine Learning Approach to Quantitative Prosopography
Authors:
Aayushee Gupta,
Haimonti Dutta,
Srikanta Bedathur,
Lipika Dey
Abstract:
Prosopography is an investigation of the common characteristics of a group of people in history, by a collective study of their lives. It involves a study of biographies to solve historical problems. If such biographies are unavailable, surviving documents and secondary biographical data are used. Quantitative prosopography involves analysis of information from a wide variety of sources about "ord…
▽ More
Prosopography is an investigation of the common characteristics of a group of people in history, by a collective study of their lives. It involves a study of biographies to solve historical problems. If such biographies are unavailable, surviving documents and secondary biographical data are used. Quantitative prosopography involves analysis of information from a wide variety of sources about "ordinary people". In this paper, we present a machine learning framework for automatically designing a people gazetteer which forms the basis of quantitative prosopographical research. The gazetteer is learnt from the noisy text of newspapers using a Named Entity Recognizer (NER). It is capable of identifying influential people from it by making use of a custom designed Influential Person Index (IPI). Our corpus comprises of 14020 articles from a local newspaper, "The Sun", published from New York in 1896. Some influential people identified by our algorithm include Captain Donald Hankey (an English soldier), Dame Nellie Melba (an Australian operatic soprano), Hugh Allan (a Canadian shipping magnate) and Sir Hugh John McDonald (the first Prime Minister of Canada).
△ Less
Submitted 30 January, 2018;
originally announced January 2018.
-
Multi-Document Summarization using Distributed Bag-of-Words Model
Authors:
Kaustubh Mani,
Ishan Verma,
Hardik Meisheri,
Lipika Dey
Abstract:
As the number of documents on the web is growing exponentially, multi-document summarization is becoming more and more important since it can provide the main ideas in a document set in short time. In this paper, we present an unsupervised centroid-based document-level reconstruction framework using distributed bag of words model. Specifically, our approach selects summary sentences in order to mi…
▽ More
As the number of documents on the web is growing exponentially, multi-document summarization is becoming more and more important since it can provide the main ideas in a document set in short time. In this paper, we present an unsupervised centroid-based document-level reconstruction framework using distributed bag of words model. Specifically, our approach selects summary sentences in order to minimize the reconstruction error between the summary and the documents. We apply sentence selection and beam search, to further improve the performance of our model. Experimental results on two different datasets show significant performance gains compared with the state-of-the-art baselines.
△ Less
Submitted 11 June, 2018; v1 submitted 7 October, 2017;
originally announced October 2017.
-
Sentiment Analysis of Review Datasets Using Naive Bayes and K-NN Classifier
Authors:
Lopamudra Dey,
Sanjay Chakraborty,
Anuraag Biswas,
Beepa Bose,
Sweta Tiwari
Abstract:
The advent of Web 2.0 has led to an increase in the amount of sentimental content available in the Web. Such content is often found in social media web sites in the form of movie or product reviews, user comments, testimonials, messages in discussion forums etc. Timely discovery of the sentimental or opinionated web content has a number of advantages, the most important of all being monetization.…
▽ More
The advent of Web 2.0 has led to an increase in the amount of sentimental content available in the Web. Such content is often found in social media web sites in the form of movie or product reviews, user comments, testimonials, messages in discussion forums etc. Timely discovery of the sentimental or opinionated web content has a number of advantages, the most important of all being monetization. Understanding of the sentiments of human masses towards different entities and products enables better services for contextual advertisements, recommendation systems and analysis of market trends. The focus of our project is sentiment focussed web crawling framework to facilitate the quick discovery of sentimental contents of movie reviews and hotel reviews and analysis of the same. We use statistical methods to capture elements of subjective style and the sentence polarity. The paper elaborately discusses two supervised machine learning algorithms: K-Nearest Neighbour(K-NN) and Naive Bayes and compares their overall accuracy, precisions as well as recall values. It was seen that in case of movie reviews Naive Bayes gave far better results than K-NN but for hotel reviews these algorithms gave lesser, almost same accuracies.
△ Less
Submitted 31 October, 2016;
originally announced October 2016.
-
Weather forecasting using Convex hull & K-Means Techniques An Approach
Authors:
Ratul Dey Sanjay Chakraborty Lopamudra Dey
Abstract:
Data mining is a popular concept of mined necessary data from a large set of data. Data mining using clustering is a powerful way to analyze data and gives prediction. In this paper non structural time series data is used to forecast daily average temperature, humidity and overall weather conditions of Kolkata city. The air pollution data have been taken from West Bengal Pollution Control Board to…
▽ More
Data mining is a popular concept of mined necessary data from a large set of data. Data mining using clustering is a powerful way to analyze data and gives prediction. In this paper non structural time series data is used to forecast daily average temperature, humidity and overall weather conditions of Kolkata city. The air pollution data have been taken from West Bengal Pollution Control Board to build the original dataset on which the prediction approach of this paper is studied and applied. This paper describes a new technique to predict the weather conditions using convex hull which gives structural data and then apply incremental K-means to define the appropriate clusters. It splits the total database into four separate databases with respect to different weather conditions. In the final step, the result will be calculated on the basis of priority based protocol which is defined based on some mathematical deduction.
△ Less
Submitted 26 January, 2015;
originally announced January 2015.
-
Canonical PSO Based k-Means Clustering Approach for Real Datasets
Authors:
Lopamudra Dey,
Sanjay Chakraborty
Abstract:
"Clustering" the significance and application of this technique is spread over various fields. Clustering is an unsupervised process in data mining, that is why the proper evaluation of the results and measuring the compactness and separability of the clusters are important issues.The procedure of evaluating the results of a clustering algorithm is known as cluster validity measure. Different type…
▽ More
"Clustering" the significance and application of this technique is spread over various fields. Clustering is an unsupervised process in data mining, that is why the proper evaluation of the results and measuring the compactness and separability of the clusters are important issues.The procedure of evaluating the results of a clustering algorithm is known as cluster validity measure. Different types of indexes are used to solve different types of problems and indices selection depends on the kind of available data.This paper first proposes Canonical PSO based K-means clustering algorithm and also analyses some important clustering indices (intercluster, intracluster) and then evaluates the effects of those indices on real-time air pollution database,wholesale customer, wine, and vehicle datasets using typical K-means, Canonical PSO based K-means, simple PSO based K-means,DBSCAN, and Hierarchical clustering algorithms.This paper also describes the nature of the clusters and finally compares the performances of these clustering algorithms according to the validity assessment. It also defines which algorithm will be more desirable among all these algorithms to make proper compact clusters on this particular real life datasets. It actually deals with the behaviour of these clustering algorithms with respect to validation indexes and represents their results of evaluation in terms of mathematical and graphical forms.
△ Less
Submitted 26 November, 2014;
originally announced November 2014.
-
Weather Forecasting using Incremental K-means Clustering
Authors:
Sanjay Chakraborty,
N. K. Nagwani,
Lopamudra Dey
Abstract:
Clustering is a powerful tool which has been used in several forecasting works, such as time series forecasting, real time storm detection, flood forecasting and so on. In this paper, a generic methodology for weather forecasting is proposed by the help of incremental K-means clustering algorithm. Weather forecasting plays an important role in day to day applications.Weather forecasting of this pa…
▽ More
Clustering is a powerful tool which has been used in several forecasting works, such as time series forecasting, real time storm detection, flood forecasting and so on. In this paper, a generic methodology for weather forecasting is proposed by the help of incremental K-means clustering algorithm. Weather forecasting plays an important role in day to day applications.Weather forecasting of this paper is done based on the incremental air pollution database of west Bengal in the years of 2009 and 2010. This paper generally uses typical K-means clustering on the main air pollution database and a list of weather category will be developed based on the maximum mean values of the clusters.Now when the new data are coming, the incremental K-means is used to group those data into those clusters whose weather category has been already defined. Thus it builds up a strategy to predict the weather of the upcoming data of the upcoming days. This forecasting database is totally based on the weather of west Bengal and this forecasting methodology is developed to mitigating the impacts of air pollutions and launch focused modeling computations for prediction and forecasts of weather events. Here accuracy of this approach is also measured.
△ Less
Submitted 18 June, 2014;
originally announced June 2014.
-
Performance Comparison of Incremental K-means and Incremental DBSCAN Algorithms
Authors:
Sanjay Chakraborty,
N. K. Nagwani,
Lopamudra Dey
Abstract:
Incremental K-means and DBSCAN are two very important and popular clustering techniques for today's large dynamic databases (Data warehouses, WWW and so on) where data are changed at random fashion. The performance of the incremental K-means and the incremental DBSCAN are different with each other based on their time analysis characteristics. Both algorithms are efficient compare to their existing…
▽ More
Incremental K-means and DBSCAN are two very important and popular clustering techniques for today's large dynamic databases (Data warehouses, WWW and so on) where data are changed at random fashion. The performance of the incremental K-means and the incremental DBSCAN are different with each other based on their time analysis characteristics. Both algorithms are efficient compare to their existing algorithms with respect to time, cost and effort. In this paper, the performance evaluation of incremental DBSCAN clustering algorithm is implemented and most importantly it is compared with the performance of incremental K-means clustering algorithm and it also explains the characteristics of these two algorithms based on the changes of the data in the database. This paper also explains some logical differences between these two most popular clustering algorithms. This paper uses an air pollution database as original database on which the experiment is performed.
△ Less
Submitted 18 June, 2014;
originally announced June 2014.