+
+ + Computation and Language 53 +
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ + ☆ MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct +
+
+
+
+
+
+
+
+ Run Luo, Haonan Zhang, Longze Chen, Ting-En Lin, Xiong Liu, Yuchuan Wu, Min Yang, Minzheng Wang, Pengpeng Zeng, Lianli Gao, Heng Tao Shen, Yunshui Li, Xiaobo Xia, Fei Huang, Jingkuan Song, Yongbin Li
+
+
+ The development of Multimodal Large Language Models (MLLMs) has seen
+significant advancements. However, the quantity and quality of multimodal
+instruction data have emerged as significant bottlenecks in their progress.
+Manually creating multimodal instruction data is both time-consuming and
+inefficient, posing challenges in producing instructions of high complexity.
+Moreover, distilling instruction data from black-box commercial models (e.g.,
+GPT-4o, GPT-4V) often results in simplistic instruction data, which constrains
+performance to that of these models. The challenge of curating diverse and
+complex instruction data remains substantial. We propose MMEvol, a novel
+multimodal instruction data evolution framework that combines fine-grained
+perception evolution, cognitive reasoning evolution, and interaction evolution.
+This iterative approach breaks through data quality bottlenecks to generate a
+complex and diverse image-text instruction dataset, thereby empowering MLLMs
+with enhanced capabilities. Beginning with an initial set of instructions,
+SEED-163K, we utilize MMEvol to systematically broadens the diversity of
+instruction types, integrates reasoning steps to enhance cognitive
+capabilities, and extracts detailed information from images to improve visual
+understanding and robustness. To comprehensively evaluate the effectiveness of
+our data, we train LLaVA-NeXT using the evolved data and conduct experiments
+across 13 vision-language tasks. Compared to the baseline trained with seed
+data, our approach achieves an average accuracy improvement of 3.1 points and
+reaches state-of-the-art (SOTA) performance on 9 of these tasks.
+
+
+
+
+ + ☆ Improving Pretraining Data Using Perplexity Correlations +
+ +
+ Quality pretraining data is often seen as the key to high-performance
+language models. However, progress in understanding pretraining data has been
+slow due to the costly pretraining runs required for data selection
+experiments. We present a framework that avoids these costs and selects
+high-quality pretraining data without any LLM training of our own. Our work is
+based on a simple observation: LLM losses on many pretraining texts are
+correlated with downstream benchmark performance, and selecting
+high-correlation documents is an effective pretraining data selection method.
+We build a new statistical framework for data selection centered around
+estimates of perplexity-benchmark correlations and perform data selection using
+a sample of 90 LLMs taken from the Open LLM Leaderboard on texts from tens of
+thousands of web domains. In controlled pretraining experiments at the 160M
+parameter scale on 8 benchmarks, our approach outperforms DSIR on every
+benchmark, while matching the best data selector found in DataComp-LM, a
+hand-engineered bigram classifier.
+
+
+
+
+ + ☆ Benchmarking Chinese Knowledge Rectification in Large Language Models +
+ +
+ While Large Language Models (LLMs) exhibit remarkable generative
+capabilities, they are not without flaws, particularly in the form of
+hallucinations. This issue is even more pronounced when LLMs are applied to
+specific languages and domains. For example, LLMs may generate nonsense
+information when handling Chinese ancient poetry, proverbs, or idioms, owing to
+the lack of specific knowledge. To this end, this paper introduces a benchmark
+for rectifying Chinese knowledge in LLMs via knowledge editing. Specifically,
+we introduce a new Chinese dataset, CKnowEdit, by collecting seven type of
+knowledge from various sources, including classical texts, idioms, and content
+from Baidu Tieba Ruozhiba, thereby accounting for the unique polyphony,
+antithesis, and logical constructs inherent in the Chinese language. Through
+the analysis of this dataset, we uncover the challenges faced by current LLMs
+in mastering Chinese. Furthermore, our evaluation of state-of-the-art knowledge
+editing techniques on this dataset unveil the substantial scope for advancement
+in the rectification of Chinese knowledge. Code and dataset are available at
+https://github.com/zjunlp/EasyEdit.
+
+
+
+ comment: Ongoing work; code and dataset are available at
+ https://github.com/zjunlp/EasyEdit
+
+
+
+ + ☆ PDAF: A Phonetic Debiasing Attention Framework For Speaker Verification +
+ +
+ Speaker verification systems are crucial for authenticating identity through
+voice. Traditionally, these systems focus on comparing feature vectors,
+overlooking the speech's content. However, this paper challenges this by
+highlighting the importance of phonetic dominance, a measure of the frequency
+or duration of phonemes, as a crucial cue in speaker verification. A novel
+Phoneme Debiasing Attention Framework (PDAF) is introduced, integrating with
+existing attention frameworks to mitigate biases caused by phonetic dominance.
+PDAF adjusts the weighting for each phoneme and influences feature extraction,
+allowing for a more nuanced analysis of speech. This approach paves the way for
+more accurate and reliable identity authentication through voice. Furthermore,
+by employing various weighting strategies, we evaluate the influence of
+phonetic features on the efficacy of the speaker verification system.
+
+
+
+ comment: Accepted to SLT
+
+
+
+ + ☆ Evidence from fMRI Supports a Two-Phase Abstraction Process in Language + Models NeurIPS +
+ +
+ Research has repeatedly demonstrated that intermediate hidden states
+extracted from large language models are able to predict measured brain
+response to natural language stimuli. Yet, very little is known about the
+representation properties that enable this high prediction performance. Why is
+it the intermediate layers, and not the output layers, that are most capable
+for this unique and highly general transfer task? In this work, we show that
+evidence from language encoding models in fMRI supports the existence of a
+two-phase abstraction process within LLMs. We use manifold learning methods to
+show that this abstraction process naturally arises over the course of training
+a language model and that the first "composition" phase of this abstraction
+process is compressed into fewer layers as training continues. Finally, we
+demonstrate a strong correspondence between layerwise encoding performance and
+the intrinsic dimensionality of representations from LLMs. We give initial
+evidence that this correspondence primarily derives from the inherent
+compositionality of LLMs and not their next-word prediction properties.
+
+
+
+ comment: Equal contribution from both authors. Submitted to NeurIPS NeuroAI
+ workshop 2024
+
+
+
+ + ☆ Towards Democratizing Multilingual Large Language Models For Medicine + Through A Two-Stage Instruction Fine-tuning Approach +
+ +
+ Open-source, multilingual medical large language models (LLMs) have the
+potential to serve linguistically diverse populations across different regions.
+Adapting generic LLMs for healthcare often requires continual pretraining, but
+this approach is computationally expensive and sometimes impractical.
+Instruction fine-tuning on a specific task may not always guarantee optimal
+performance due to the lack of broader domain knowledge that the model needs to
+understand and reason effectively in diverse scenarios. To address these
+challenges, we introduce two multilingual instruction fine-tuning datasets,
+MMed-IFT and MMed-IFT-MC, containing over 200k high-quality medical samples in
+six languages. We propose a two-stage training paradigm: the first stage
+injects general medical knowledge using MMed-IFT, while the second stage
+fine-tunes task-specific multiple-choice questions with MMed-IFT-MC. Our method
+achieves competitive results on both English and multilingual benchmarks,
+striking a balance between computational efficiency and performance. We plan to
+make our dataset and model weights public at
+\url{https://github.com/SpassMed/Med-Llama3} in the future.
+
+
+
+ comment: Technical Report v1, work in progress
+
+
+
+ + ☆ Referring Expression Generation in Visually Grounded Dialogue with + Discourse-aware Comprehension Guiding +
+ +
+ We propose an approach to referring expression generation (REG) in visually
+grounded dialogue that is meant to produce referring expressions (REs) that are
+both discriminative and discourse-appropriate. Our method constitutes a
+two-stage process. First, we model REG as a text- and image-conditioned
+next-token prediction task. REs are autoregressively generated based on their
+preceding linguistic context and a visual representation of the referent.
+Second, we propose the use of discourse-aware comprehension guiding as part of
+a generate-and-rerank strategy through which candidate REs generated with our
+REG model are reranked based on their discourse-dependent discriminatory power.
+Results from our human evaluation indicate that our proposed two-stage approach
+is effective in producing discriminative REs, with higher performance in terms
+of text-image retrieval accuracy for reranked REs compared to those generated
+using greedy decoding.
+
+
+
+ comment: Accepted for publication at INLG 2024
+
+
+
+ + ☆ RegNLP in Action: Facilitating Compliance Through Automated Information + Retrieval and Answer Generation +
+ +
+ Regulatory documents, issued by governmental regulatory bodies, establish
+rules, guidelines, and standards that organizations must adhere to for legal
+compliance. These documents, characterized by their length, complexity and
+frequent updates, are challenging to interpret, requiring significant
+allocation of time and expertise on the part of organizations to ensure ongoing
+compliance.Regulatory Natural Language Processing (RegNLP) is a
+multidisciplinary subfield aimed at simplifying access to and interpretation of
+regulatory rules and obligations. We define an Automated Question-Passage
+Generation task for RegNLP, create the ObliQA dataset containing 27,869
+questions derived from the Abu Dhabi Global Markets (ADGM) financial regulation
+document collection, design a baseline Regulatory Information Retrieval and
+Answer Generation system, and evaluate it with RePASs, a novel evaluation
+metric that tests whether generated answers accurately capture all relevant
+obligations and avoid contradictions.
+
+
+
+
+ + ☆ Evaluation of real-time transcriptions using end-to-end ASR models +
+ +
+ Automatic Speech Recognition (ASR) or Speech-to-text (STT) has greatly
+evolved in the last few years. Traditional architectures based on pipelines
+have been replaced by joint end-to-end (E2E) architectures that simplify and
+streamline the model training process. In addition, new AI training methods,
+such as weak-supervised learning have reduced the need for high-quality audio
+datasets for model training. However, despite all these advancements, little to
+no research has been done on real-time transcription. In real-time scenarios,
+the audio is not pre-recorded, and the input audio must be fragmented to be
+processed by the ASR systems. To achieve real-time requirements, these
+fragments must be as short as possible to reduce latency. However, audio cannot
+be split at any point as dividing an utterance into two separate fragments will
+generate an incorrect transcription. Also, shorter fragments provide less
+context for the ASR model. For this reason, it is necessary to design and test
+different splitting algorithms to optimize the quality and delay of the
+resulting transcription. In this paper, three audio splitting algorithms are
+evaluated with different ASR models to determine their impact on both the
+quality of the transcription and the end-to-end delay. The algorithms are
+fragmentation at fixed intervals, voice activity detection (VAD), and
+fragmentation with feedback. The results are compared to the performance of the
+same model, without audio fragmentation, to determine the effects of this
+division. The results show that VAD fragmentation provides the best quality
+with the highest delay, whereas fragmentation at fixed intervals provides the
+lowest quality and the lowest delay. The newly proposed feedback algorithm
+exchanges a 2-4% increase in WER for a reduction of 1.5-2s delay, respectively,
+to the VAD splitting.
+
+
+
+ comment: 15 pages, 4 figures
+
+
+
+ + ☆ Revisiting English Winogender Schemas for Consistency, Coverage, and + Grammatical Case +
+
+
+
+
+
+
+
+ Vagrant Gautam, Julius Steuer, Eileen Bingert, Ray Johns, Anne Lauscher, Dietrich Klakow
+
+
+ While measuring bias and robustness in coreference resolution are important
+goals, such measurements are only as good as the tools we use to measure them
+with. Winogender schemas (Rudinger et al., 2018) are an influential dataset
+proposed to evaluate gender bias in coreference resolution, but a closer look
+at the data reveals issues with the instances that compromise their use for
+reliable evaluation, including treating different grammatical cases of pronouns
+in the same way, violations of template constraints, and typographical errors.
+We identify these issues and fix them, contributing a new dataset: Winogender
+2.0. Our changes affect performance with state-of-the-art supervised
+coreference resolution systems as well as all model sizes of the language model
+FLAN-T5, with F1 dropping on average 0.1 points. We also propose a new method
+to evaluate pronominal bias in coreference resolution that goes beyond the
+binary. With this method and our new dataset which is balanced for grammatical
+case, we empirically demonstrate that bias characteristics vary not just across
+pronoun sets, but also across surface forms of those sets.
+
+
+
+
+ + ☆ Longer is (Not Necessarily) Stronger: Punctuated Long-Sequence Training + for Enhanced Speech Recognition and Translation +
+
+
+
+
+
+
+
+ Nithin Rao Koluguri, Travis Bartley, Hainan Xu, Oleksii Hrinchuk, Jagadeesh Balam, Boris Ginsburg, Georg Kucsko
+
+
+ This paper presents a new method for training sequence-to-sequence models for
+speech recognition and translation tasks. Instead of the traditional approach
+of training models on short segments containing only lowercase or partial
+punctuation and capitalization (PnC) sentences, we propose training on longer
+utterances that include complete sentences with proper punctuation and
+capitalization. We achieve this by using the FastConformer architecture which
+allows training 1 Billion parameter models with sequences up to 60 seconds long
+with full attention. However, while training with PnC enhances the overall
+performance, we observed that accuracy plateaus when training on sequences
+longer than 40 seconds across various evaluation settings. Our proposed method
+significantly improves punctuation and capitalization accuracy, showing a 25%
+relative word error rate (WER) improvement on the Earnings-21 and Earnings-22
+benchmarks. Additionally, training on longer audio segments increases the
+overall model accuracy across speech recognition and translation benchmarks.
+The model weights and training code are open-sourced though NVIDIA NeMo.
+
+
+
+ comment: Accepted at SLT 2024
+
+
+
+ + ☆ ExDDI: Explaining Drug-Drug Interaction Predictions with Natural + Language +
+ +
+ Predicting unknown drug-drug interactions (DDIs) is crucial for improving
+medication safety. Previous efforts in DDI prediction have typically focused on
+binary classification or predicting DDI categories, with the absence of
+explanatory insights that could enhance trust in these predictions. In this
+work, we propose to generate natural language explanations for DDI predictions,
+enabling the model to reveal the underlying pharmacodynamics and
+pharmacokinetics mechanisms simultaneously as making the prediction. To do
+this, we have collected DDI explanations from DDInter and DrugBank and
+developed various models for extensive experiments and analysis. Our models can
+provide accurate explanations for unknown DDIs between known drugs. This paper
+contributes new tools to the field of DDI prediction and lays a solid
+foundation for further research on generating explanations for DDI predictions.
+
+
+
+ comment: 17 pages, 4 figures
+
+
+
+ + ☆ MemoRAG: Moving towards Next-Gen RAG Via Memory-Inspired Knowledge + Discovery +
+ +
+ Retrieval-Augmented Generation (RAG) leverages retrieval tools to access
+external databases, thereby enhancing the generation quality of large language
+models (LLMs) through optimized context. However, the existing retrieval
+methods are constrained inherently, as they can only perform relevance matching
+between explicitly stated queries and well-formed knowledge, but unable to
+handle tasks involving ambiguous information needs or unstructured knowledge.
+Consequently, existing RAG systems are primarily effective for straightforward
+question-answering tasks. In this work, we propose \textbf{MemoRAG}, a novel
+retrieval-augmented generation paradigm empowered by long-term memory. MemoRAG
+adopts a dual-system architecture. On the one hand, it employs a \textit{light
+but long-range} LLM to form the global memory of database. Once a task is
+presented, it generates draft answers, cluing the retrieval tools to locate
+useful information within the database. On the other hand, it leverages an
+\textit{expensive but expressive} LLM, which generates the ultimate answer
+based on the retrieved information. Building on this general framework, we
+further optimize MemoRAG's performance by enhancing its cluing mechanism and
+memorization capacity. In our experiment, MemoRAG achieves superior performance
+across a variety of evaluation tasks, including both complex ones where
+conventional RAG fails and straightforward ones where RAG is commonly applied.
+
+
+
+ comment: Codes and models are in https://github.com/qhjqhj00/MemoRAG
+
+
+
+ + ☆ Spatially-Aware Speaker for Vision-and-Language Navigation Instruction + Generation +
+ +
+ Embodied AI aims to develop robots that can \textit{understand} and execute
+human language instructions, as well as communicate in natural languages. On
+this front, we study the task of generating highly detailed navigational
+instructions for the embodied robots to follow. Although recent studies have
+demonstrated significant leaps in the generation of step-by-step instructions
+from sequences of images, the generated instructions lack variety in terms of
+their referral to objects and landmarks. Existing speaker models learn
+strategies to evade the evaluation metrics and obtain higher scores even for
+low-quality sentences. In this work, we propose SAS (Spatially-Aware Speaker),
+an instruction generator or \textit{Speaker} model that utilises both
+structural and semantic knowledge of the environment to produce richer
+instructions. For training, we employ a reward learning method in an
+adversarial setting to avoid systematic bias introduced by language evaluation
+metrics. Empirically, our method outperforms existing instruction generation
+models, evaluated using standard metrics. Our code is available at
+\url{https://github.com/gmuraleekrishna/SAS}.
+
+
+
+
+ + ☆ SciAgents: Automating scientific discovery through multi-agent + intelligent graph reasoning +
+ +
+ A key challenge in artificial intelligence is the creation of systems capable
+of autonomously advancing scientific understanding by exploring novel domains,
+identifying complex patterns, and uncovering previously unseen connections in
+vast scientific data. In this work, we present SciAgents, an approach that
+leverages three core concepts: (1) the use of large-scale ontological knowledge
+graphs to organize and interconnect diverse scientific concepts, (2) a suite of
+large language models (LLMs) and data retrieval tools, and (3) multi-agent
+systems with in-situ learning capabilities. Applied to biologically inspired
+materials, SciAgents reveals hidden interdisciplinary relationships that were
+previously considered unrelated, achieving a scale, precision, and exploratory
+power that surpasses traditional human-driven research methods. The framework
+autonomously generates and refines research hypotheses, elucidating underlying
+mechanisms, design principles, and unexpected material properties. By
+integrating these capabilities in a modular fashion, the intelligent system
+yields material discoveries, critique and improve existing hypotheses, retrieve
+up-to-date data about existing research, and highlights their strengths and
+limitations. Our case studies demonstrate scalable capabilities to combine
+generative AI, ontological representations, and multi-agent modeling,
+harnessing a `swarm of intelligence' similar to biological systems. This
+provides new avenues for materials discovery and accelerates the development of
+advanced materials by unlocking Nature's design principles.
+
+
+
+
+ + ☆ QiBERT -- Classifying Online Conversations Messages with BERT as a + Feature +
+ +
+ Recent developments in online communication and their usage in everyday life
+have caused an explosion in the amount of a new genre of text data, short text.
+Thus, the need to classify this type of text based on its content has a
+significant implication in many areas. Online debates are no exception, once
+these provide access to information about opinions, positions and preferences
+of its users. This paper aims to use data obtained from online social
+conversations in Portuguese schools (short text) to observe behavioural trends
+and to see if students remain engaged in the discussion when stimulated. This
+project used the state of the art (SoA) Machine Learning (ML) algorithms and
+methods, through BERT based models to classify if utterances are in or out of
+the debate subject. Using SBERT embeddings as a feature, with supervised
+learning, the proposed model achieved results above 0.95 average accuracy for
+classifying online messages. Such improvements can help social scientists
+better understand human communication, behaviour, discussion and persuasion.
+
+
+
+
+ + ☆ Harmonic Reasoning in Large Language Models +
+ +
+ Large Language Models (LLMs) are becoming very popular and are used for many
+different purposes, including creative tasks in the arts. However, these models
+sometimes have trouble with specific reasoning tasks, especially those that
+involve logical thinking and counting. This paper looks at how well LLMs
+understand and reason when dealing with musical tasks like figuring out notes
+from intervals and identifying chords and scales. We tested GPT-3.5 and GPT-4o
+to see how they handle these tasks. Our results show that while LLMs do well
+with note intervals, they struggle with more complicated tasks like recognizing
+chords and scales. This points out clear limits in current LLM abilities and
+shows where we need to make them better, which could help improve how they
+think and work in both artistic and other complex areas. We also provide an
+automatically generated benchmark data set for the described tasks.
+
+
+
+
+ + ☆ Elsevier Arena: Human Evaluation of Chemistry/Biology/Health + Foundational Large Language Models +
+
+
+
+
+
+
+
+ Camilo Thorne, Christian Druckenbrodt, Kinga Szarkowska, Deepika Goyal, Pranita Marajan, Vijay Somanath, Corey Harper, Mao Yan, Tony Scerri
+
+
+ The quality and capabilities of large language models cannot be currently
+fully assessed with automated, benchmark evaluations. Instead, human
+evaluations that expand on traditional qualitative techniques from natural
+language generation literature are required. One recent best-practice consists
+in using A/B-testing frameworks, which capture preferences of human evaluators
+for specific models. In this paper we describe a human evaluation experiment
+focused on the biomedical domain (health, biology, chemistry/pharmacology)
+carried out at Elsevier. In it a large but not massive (8.8B parameter)
+decoder-only foundational transformer trained on a relatively small (135B
+tokens) but highly curated collection of Elsevier datasets is compared to
+OpenAI's GPT-3.5-turbo and Meta's foundational 7B parameter Llama 2 model
+against multiple criteria. Results indicate -- even if IRR scores were
+generally low -- a preference towards GPT-3.5-turbo, and hence towards models
+that possess conversational abilities, are very large and were trained on very
+large datasets. But at the same time, indicate that for less massive models
+training on smaller but well-curated training sets can potentially give rise to
+viable alternatives in the biomedical domain.
+
+
+
+ comment: 11 pages, 5 tables, 6 figures
+
+
+
+ + ☆ Representational Analysis of Binding in Large Language Models +
+ +
+ Entity tracking is essential for complex reasoning. To perform in-context
+entity tracking, language models (LMs) must bind an entity to its attribute
+(e.g., bind a container to its content) to recall attribute for a given entity.
+For example, given a context mentioning ``The coffee is in Box Z, the stone is
+in Box M, the map is in Box H'', to infer ``Box Z contains the coffee'' later,
+LMs must bind ``Box Z'' to ``coffee''. To explain the binding behaviour of LMs,
+Feng and Steinhardt (2023) introduce a Binding ID mechanism and state that LMs
+use a abstract concept called Binding ID (BI) to internally mark
+entity-attribute pairs. However, they have not directly captured the BI
+determinant information from entity activations. In this work, we provide a
+novel view of the Binding ID mechanism by localizing the prototype of BI
+information. Specifically, we discover that there exists a low-rank subspace in
+the hidden state (or activation) of LMs, that primarily encodes the order of
+entity and attribute and which is used as the prototype of BI to causally
+determine the binding. To identify this subspace, we choose principle component
+analysis as our first attempt and it is empirically proven to be effective.
+Moreover, we also discover that when editing representations along directions
+in the subspace, LMs tend to bind a given entity to other attributes
+accordingly. For example, by patching activations along the BI encoding
+direction we can make the LM to infer ``Box Z contains the stone'' and ``Box Z
+contains the map''.
+
+
+
+
+ + ☆ STLM Engineering Report: Dropout +
+ +
+ In this work we explore the relevance of dropout for modern language models,
+particularly in the context of models on the scale of <100M parameters. We
+explore it's relevance firstly in the regime of improving the sample efficiency
+of models given small, high quality datasets, and secondly in the regime of
+improving the quality of its fit on larger datasets where models may underfit.
+We find that concordant with conventional wisdom, dropout remains effective in
+the overfitting scenario, and that furthermore it may have some relevance for
+improving the fit of models even in the case of excess data, as suggested by
+previous research. In the process we find that the existing explanation for the
+mechanism behind this performance gain is not applicable in the case of
+language modelling.
+
+
+
+ comment: 6 pages, 3 figures, For code base see
+ https://github.com/LeonGuertler/SuperTinyLanguageModels
+
+
+
+ + ☆ NLLB-E5: A Scalable Multilingual Retrieval Model +
+ +
+ Despite significant progress in multilingual information retrieval, the lack
+of models capable of effectively supporting multiple languages, particularly
+low-resource like Indic languages, remains a critical challenge. This paper
+presents NLLB-E5: A Scalable Multilingual Retrieval Model. NLLB-E5 leverages
+the in-built multilingual capabilities in the NLLB encoder for translation
+tasks. It proposes a distillation approach from multilingual retriever E5 to
+provide a zero-shot retrieval approach handling multiple languages, including
+all major Indic languages, without requiring multilingual training data. We
+evaluate the model on a comprehensive suite of existing benchmarks, including
+Hindi-BEIR, highlighting its robust performance across diverse languages and
+tasks. Our findings uncover task and domain-specific challenges, providing
+valuable insights into the retrieval performance, especially for low-resource
+languages. NLLB-E5 addresses the urgent need for an inclusive, scalable, and
+language-agnostic text retrieval model, advancing the field of multilingual
+information access and promoting digital inclusivity for millions of users
+globally.
+
+
+
+
+ + ☆ Towards Building a Robust Knowledge Intensive Question Answering Model + with Large Language Models NLPCC-2024 +
+ +
+ The development of LLMs has greatly enhanced the intelligence and fluency of
+question answering, while the emergence of retrieval enhancement has enabled
+models to better utilize external information. However, the presence of noise
+and errors in retrieved information poses challenges to the robustness of LLMs.
+In this work, to evaluate the model's performance under multiple interferences,
+we first construct a dataset based on machine reading comprehension datasets
+simulating various scenarios, including critical information absence, noise,
+and conflicts. To address the issue of model accuracy decline caused by noisy
+external information, we propose a data augmentation-based fine-tuning method
+to enhance LLM's robustness against noise. Additionally, contrastive learning
+approach is utilized to preserve the model's discrimination capability of
+external information. We have conducted experiments on both existing LLMs and
+our approach, the results are evaluated by GPT-4, which indicates that our
+proposed methods improve model robustness while strengthening the model's
+discrimination capability.
+
+
+
+ comment: This paper has been accepted by NLPCC-2024
+
+
+
+ + ☆ Application Specific Compression of Deep Learning Models KDD +
+ +
+ Large Deep Learning models are compressed and deployed for specific
+applications. However, current Deep Learning model compression methods do not
+utilize the information about the target application. As a result, the
+compressed models are application agnostic. Our goal is to customize the model
+compression process to create a compressed model that will perform better for
+the target application. Our method, Application Specific Compression (ASC),
+identifies and prunes components of the large Deep Learning model that are
+redundant specifically for the given target application. The intuition of our
+work is to prune the parts of the network that do not contribute significantly
+to updating the data representation for the given application. We have
+experimented with the BERT family of models for three applications: Extractive
+QA, Natural Language Inference, and Paraphrase Identification. We observe that
+customized compressed models created using ASC method perform better than
+existing model compression methods and off-the-shelf compressed models.
+
+
+
+ comment: Accepted in the Proceedings of the 8th Joint International Conference
+ on Data Science & Management of Data (12th ACM IKDD CODS and 30th COMAD) for
+ the Short Research Paper track, 5 pages
+
+
+
+ + ☆ Diagnostic Reasoning in Natural Language: Computational Model and + Application +
+
+
+
+
+
+
+
+ Nils Dycke, Matej Zečević, Ilia Kuznetsov, Beatrix Suess, Kristian Kersting, Iryna Gurevych
+
+
+ Diagnostic reasoning is a key component of expert work in many domains. It is
+a hard, time-consuming activity that requires expertise, and AI research has
+investigated the ways automated systems can support this process. Yet, due to
+the complexity of natural language, the applications of AI for diagnostic
+reasoning to language-related tasks are lacking. To close this gap, we
+investigate diagnostic abductive reasoning (DAR) in the context of
+language-grounded tasks (NL-DAR). We propose a novel modeling framework for
+NL-DAR based on Pearl's structural causal models and instantiate it in a
+comprehensive study of scientific paper assessment in the biomedical domain. We
+use the resulting dataset to investigate the human decision-making process in
+NL-DAR and determine the potential of LLMs to support structured
+decision-making over text. Our framework, open resources and tools lay the
+groundwork for the empirical study of collaborative diagnostic reasoning in the
+age of LLMs, in the scholarly domain and beyond.
+
+
+
+
+ + ☆ IndicVoices-R: Unlocking a Massive Multilingual Multi-speaker Speech + Corpus for Scaling Indian TTS +
+
+
+
+
+
+
+
+ Ashwin Sankar, Srija Anand, Praveen Srinivasa Varadhan, Sherry Thomas, Mehak Singal, Shridhar Kumar, Deovrat Mehendale, Aditi Krishana, Giri Raju, Mitesh Khapra
+
+
+ Recent advancements in text-to-speech (TTS) synthesis show that large-scale
+models trained with extensive web data produce highly natural-sounding output.
+However, such data is scarce for Indian languages due to the lack of
+high-quality, manually subtitled data on platforms like LibriVox or YouTube. To
+address this gap, we enhance existing large-scale ASR datasets containing
+natural conversations collected in low-quality environments to generate
+high-quality TTS training data. Our pipeline leverages the cross-lingual
+generalization of denoising and speech enhancement models trained on English
+and applied to Indian languages. This results in IndicVoices-R (IV-R), the
+largest multilingual Indian TTS dataset derived from an ASR dataset, with 1,704
+hours of high-quality speech from 10,496 speakers across 22 Indian languages.
+IV-R matches the quality of gold-standard TTS datasets like LJSpeech, LibriTTS,
+and IndicTTS. We also introduce the IV-R Benchmark, the first to assess
+zero-shot, few-shot, and many-shot speaker generalization capabilities of TTS
+models on Indian voices, ensuring diversity in age, gender, and style. We
+demonstrate that fine-tuning an English pre-trained model on a combined dataset
+of high-quality IndicTTS and our IV-R dataset results in better zero-shot
+speaker generalization compared to fine-tuning on the IndicTTS dataset alone.
+Further, our evaluation reveals limited zero-shot generalization for Indian
+voices in TTS models trained on prior datasets, which we improve by fine-tuning
+the model on our data containing diverse set of speakers across language
+families. We open-source all data and code, releasing the first TTS model for
+all 22 official Indian languages.
+
+
+
+
+ + ☆ Mpox Narrative on Instagram: A Labeled Multilingual Dataset of Instagram + Posts on Mpox for Sentiment, Hate Speech, and Anxiety Analysis +
+ +
+ The world is currently experiencing an outbreak of mpox, which has been
+declared a Public Health Emergency of International Concern by WHO. No prior
+work related to social media mining has focused on the development of a dataset
+of Instagram posts about the mpox outbreak. The work presented in this paper
+aims to address this research gap and makes two scientific contributions to
+this field. First, it presents a multilingual dataset of 60,127 Instagram posts
+about mpox, published between July 23, 2022, and September 5, 2024. The
+dataset, available at https://dx.doi.org/10.21227/7fvc-y093, contains Instagram
+posts about mpox in 52 languages. For each of these posts, the Post ID, Post
+Description, Date of publication, language, and translated version of the post
+(translation to English was performed using the Google Translate API) are
+presented as separate attributes in the dataset. After developing this dataset,
+sentiment analysis, hate speech detection, and anxiety or stress detection were
+performed. This process included classifying each post into (i) one of the
+sentiment classes, i.e., fear, surprise, joy, sadness, anger, disgust, or
+neutral, (ii) hate or not hate, and (iii) anxiety/stress detected or no
+anxiety/stress detected. These results are presented as separate attributes in
+the dataset. Second, this paper presents the results of performing sentiment
+analysis, hate speech analysis, and anxiety or stress analysis. The variation
+of the sentiment classes - fear, surprise, joy, sadness, anger, disgust, and
+neutral were observed to be 27.95%, 2.57%, 8.69%, 5.94%, 2.69%, 1.53%, and
+50.64%, respectively. In terms of hate speech detection, 95.75% of the posts
+did not contain hate and the remaining 4.25% of the posts contained hate.
+Finally, 72.05% of the posts did not indicate any anxiety/stress, and the
+remaining 27.95% of the posts represented some form of anxiety/stress.
+
+
+
+
+ + ☆ Seek and Solve Reasoning for Table Question Answering +
+ +
+ Table-based Question Answering (TQA) involves answering questions based on
+tabular data. The complexity of table structures and question logic makes this
+task difficult even for Large Language Models (LLMs). This paper improves TQA
+performance by leveraging LLMs' reasoning capabilities. Inspired by how humans
+solve TQA tasks, we propose a Seek-and-Solve pipeline that instructs the LLM to
+first seek relevant information and then answer questions. The two stages are
+integrated at the reasoning level, and their Chain of Thought (CoT) paths are
+integrated into a coherent Seek-and-Solve CoT (SS-CoT). Furthermore, we present
+a compact single-stage TQA-solving prompt distilled from the pipeline.
+Experiments demonstrate that under In-Context Learning settings, using samples
+with SS-CoT paths as demonstrations, the TQA-solving prompt can effectively
+guide the LLM to solve complex TQA tasks, resulting in improved performance and
+reliability. Our results highlight the importance of properly eliciting LLMs'
+reasoning capabilities in solving complex TQA tasks.
+
+
+
+
+ + ☆ On the Relationship between Truth and Political Bias in Language Models +
+
+
+
+
+
+
+
+ Suyash Fulay, William Brannon, Shrestha Mohanty, Cassandra Overney, Elinor Poole-Dayan, Deb Roy, Jad Kabbara
+
+
+ Language model alignment research often attempts to ensure that models are
+not only helpful and harmless, but also truthful and unbiased. However,
+optimizing these objectives simultaneously can obscure how improving one aspect
+might impact the others. In this work, we focus on analyzing the relationship
+between two concepts essential in both language model alignment and political
+science: \textit{truthfulness} and \textit{political bias}. We train reward
+models on various popular truthfulness datasets and subsequently evaluate their
+political bias. Our findings reveal that optimizing reward models for
+truthfulness on these datasets tends to result in a left-leaning political
+bias. We also find that existing open-source reward models (i.e. those trained
+on standard human preference datasets) already show a similar bias and that the
+bias is larger for larger models. These results raise important questions about
+both the datasets used to represent truthfulness and what language models
+capture about the relationship between truth and politics.
+
+
+
+
+ + ☆ RexUniNLU: Recursive Method with Explicit Schema Instructor for + Universal NLU +
+
+
+
+
+
+
+
+ Chengyuan Liu, Shihang Wang, Fubang Zhao, Kun Kuang, Yangyang Kang, Weiming Lu, Changlong Sun, Fei Wu
+
+
+ Information Extraction (IE) and Text Classification (CLS) serve as the
+fundamental pillars of NLU, with both disciplines relying on analyzing input
+sequences to categorize outputs into pre-established schemas. However, there is
+no existing encoder-based model that can unify IE and CLS tasks from this
+perspective. To fully explore the foundation shared within NLU tasks, we have
+proposed a Recursive Method with Explicit Schema Instructor for Universal NLU.
+Specifically, we firstly redefine the true universal information extraction
+(UIE) with a formal formulation that covers almost all extraction schemas,
+including quadruples and quintuples which remain unsolved for previous UIE
+models. Then, we expands the formulation to all CLS and multi-modal NLU tasks.
+Based on that, we introduce RexUniNLU, an universal NLU solution that employs
+explicit schema constraints for IE and CLS, which encompasses all IE and CLS
+tasks and prevent incorrect connections between schema and input sequence. To
+avoid interference between different schemas, we reset the position ids and
+attention mask matrices. Extensive experiments are conducted on IE, CLS in both
+English and Chinese, and multi-modality, revealing the effectiveness and
+superiority. Our codes are publicly released.
+
+
+
+ comment: arXiv admin note: substantial text overlap with arXiv:2304.14770
+
+
+
+ + ☆ UPCS: Unbiased Persona Construction for Dialogue Generation +
+ +
+ Narrative systems, such as dialogue and storytelling systems, often utilize
+persona profiles to enhance personalized interactions. Existing persona
+profiles frequently exhibit biases, posing risks to system integrity and
+fairness. To address this, we introduce the UPCS framework, which categorizes
+character descriptions into eight dimensions, including bias mitigation
+strategies. Experimental results demonstrate UPCS's superiority in accuracy,
+diversity, bias elimination, and user satisfaction, marking a significant
+advancement in persona construction for reliable narrative systems.
+
+
+
+
+ + ♻ ☆ Using Natural Language Explanations to Rescale Human Judgments +
+ +
+ The rise of large language models (LLMs) has brought a critical need for
+high-quality human-labeled data, particularly for processes like human feedback
+and evaluation. A common practice is to label data via consensus annotation
+over human judgments. However, annotators' judgments for subjective tasks can
+differ in many ways: they may reflect different qualitative judgments about an
+example, and they may be mapped to a labeling scheme in different ways. We show
+that these nuances can be captured by natural language explanations, and
+propose a method to rescale ordinal annotations and explanations using LLMs.
+Specifically, we feed annotators' Likert ratings and corresponding explanations
+into an LLM and prompt it to produce a numeric score anchored in a scoring
+rubric. These scores should reflect the annotators' underlying assessments of
+the example. The rubric can be designed or modified after annotation, and
+include distinctions that may not have been known when the original error
+taxonomy was devised. We explore our technique in the context of rating system
+outputs for a document-grounded question answering task, where LLMs achieve
+near-human performance. Our method rescales the raw judgments without impacting
+agreement and brings the scores closer to human judgments grounded in the same
+scoring rubric.
+
+
+
+ comment: Data available at
+ https://github.com/ManyaWadhwa/explanation_based_rescaling
+
+
+
+ + ♻ ☆ Bi-Directional Transformers vs. word2vec: Discovering Vulnerabilities in + Lifted Compiled Code +
+ +
+ Detecting vulnerabilities within compiled binaries is challenging due to lost
+high-level code structures and other factors such as architectural
+dependencies, compilers, and optimization options. To address these obstacles,
+this research explores vulnerability detection using natural language
+processing (NLP) embedding techniques with word2vec, BERT, and RoBERTa to learn
+semantics from intermediate representation (LLVM IR) code. Long short-term
+memory (LSTM) neural networks were trained on embeddings from encoders created
+using approximately 48k LLVM functions from the Juliet dataset. This study is
+pioneering in its comparison of word2vec models with multiple bidirectional
+transformer (BERT, RoBERTa) embeddings built using LLVM code to train neural
+networks to detect vulnerabilities in compiled binaries. word2vec Skip-Gram
+models achieved 92% validation accuracy in detecting vulnerabilities,
+outperforming word2vec Continuous Bag of Words (CBOW), BERT, and RoBERTa. This
+suggests that complex contextual embeddings may not provide advantages over
+simpler word2vec models for this task when a limited number (e.g. 48K) of data
+samples are used to train the bidirectional transformer-based models. The
+comparative results provide novel insights into selecting optimal embeddings
+for learning compiler-independent semantic code representations to advance
+machine learning detection of vulnerabilities in compiled binaries.
+
+
+
+ comment: Updated with improvements"
+
+
+
+ + ♻ ☆ Instruct-SkillMix: A Powerful Pipeline for LLM Instruction Tuning +
+ +
+ We introduce Instruct-SkillMix, an automated approach for creating diverse,
+high quality SFT data. The Instruct-SkillMix pipeline involves two stages, each
+leveraging an existing powerful LLM: (1) Skill extraction: uses the LLM to
+extract core "skills" for instruction-following, either from existing datasets,
+or by directly prompting the model; (2) Data generation: uses the powerful LLM
+to generate (instruction, response) data that exhibit a randomly chosen pair of
+these skills. Here, the use of random skill combinations promotes diversity and
+difficulty.
+ Vanilla SFT (i.e., no PPO, DPO, or RL methods) on data generated from
+Instruct-SkillMix leads to strong gains on instruction following benchmarks
+such as AlpacaEval 2.0, MT-Bench, and WildBench. With just $4$K examples,
+LLaMA-3-8B-Base achieves 42.76% length-controlled win rate on AlpacaEval 2.0.
+To our knowledge, this achieves state-of-the-art performance among all models
+that have only undergone SFT (no RL methods) and competes with proprietary
+models such as Claude 3 Opus and LLaMA-3.1-405B-Instruct.
+ Ablation studies also suggest plausible reasons for why creating open
+instruction-tuning datasets via naive crowd-sourcing has proved difficult.
+Introducing low quality answers ("shirkers") in $20\%$ of Instruct-SkillMix
+examples causes performance to plummet, sometimes catastrophically.
+ The Instruct-SkillMix pipeline is flexible and is adaptable to other
+settings.
+
+
+
+
+ + ♻ ☆ Balancing Rigor and Utility: Mitigating Cognitive Biases in Large + Language Models for Multiple-Choice Questions +
+ +
+ This paper examines the role of cognitive biases in the decision-making
+processes of large language models (LLMs), challenging the conventional goal of
+eliminating all biases. We show that certain cognitive biases when properly
+balanced, can enhance decision-making efficiency through rational deviations
+and heuristic shortcuts. By introducing heuristic moderation and an abstention
+option, which allows LLMs to withhold responses when uncertain, we reduce error
+rates, improve decision accuracy, and optimize decision rates. Using the
+Balance Rigor and Utility (BRU) dataset, developed through expert
+collaboration, our findings demonstrate that targeted inspection of cognitive
+biases aligns LLM decisions more closely with human reasoning, enhancing
+reliability and suggesting strategies for future improvements. This approach
+offers a novel way to leverage cognitive biases to improve the practical
+utility of LLMs across various applications.
+
+
+
+ comment: This article is currently under review. All data will be open on
+ GitHub once the review is complete.
+ https://github.com/limanwang/Balancing-Rigor-and-Utility
+
+
+
+ + ♻ ☆ X-InstructBLIP: A Framework for aligning X-Modal instruction-aware + representations to LLMs and Emergent Cross-modal Reasoning +
+
+
+
+
+
+
+
+ Artemis Panagopoulou, Le Xue, Ning Yu, Junnan Li, Dongxu Li, Shafiq Joty, Ran Xu, Silvio Savarese, Caiming Xiong, Juan Carlos Niebles
+
+
+ Recent research has achieved significant advancements in visual reasoning
+tasks through learning image-to-language projections and leveraging the
+impressive reasoning abilities of Large Language Models (LLMs). This paper
+introduces an efficient and effective framework that integrates multiple
+modalities (images, 3D, audio and video) to a frozen LLM and demonstrates an
+emergent ability for cross-modal reasoning (2+ modality inputs). Our approach
+explores two distinct projection mechanisms: Q-Formers and Linear Projections
+(LPs). Through extensive experimentation across all four modalities on 16
+benchmarks, we explore both methods and assess their adaptability in integrated
+and separate cross-modal reasoning. The Q-Former projection demonstrates
+superior performance in single modality scenarios and adaptability in joint
+versus discriminative reasoning involving two or more modalities. However, it
+exhibits lower generalization capabilities than linear projection in contexts
+where task-modality data are limited. To enable this framework, we devise a
+scalable pipeline that automatically generates high-quality, instruction-tuning
+datasets from readily available captioning data across different modalities,
+and contribute 24K QA data for audio and 250K QA data for 3D. To facilitate
+further research in cross-modal reasoning, we introduce the DisCRn
+(Discriminative Cross-modal Reasoning) benchmark comprising 9K audio-video QA
+samples and 28K image-3D QA samples that require the model to reason
+discriminatively across disparate input modalities.
+
+
+
+
+ + ♻ ☆ PanoSent: A Panoptic Sextuple Extraction Benchmark for Multimodal + Conversational Aspect-based Sentiment Analysis ACM MM 2024 +
+
+
+
+
+
+
+
+ Meng Luo, Hao Fei, Bobo Li, Shengqiong Wu, Qian Liu, Soujanya Poria, Erik Cambria, Mong-Li Lee, Wynne Hsu
+
+
+ While existing Aspect-based Sentiment Analysis (ABSA) has received extensive
+effort and advancement, there are still gaps in defining a more holistic
+research target seamlessly integrating multimodality, conversation context,
+fine-granularity, and also covering the changing sentiment dynamics as well as
+cognitive causal rationales. This paper bridges the gaps by introducing a
+multimodal conversational ABSA, where two novel subtasks are proposed: 1)
+Panoptic Sentiment Sextuple Extraction, panoramically recognizing holder,
+target, aspect, opinion, sentiment, rationale from multi-turn multi-party
+multimodal dialogue. 2) Sentiment Flipping Analysis, detecting the dynamic
+sentiment transformation throughout the conversation with the causal reasons.
+To benchmark the tasks, we construct PanoSent, a dataset annotated both
+manually and automatically, featuring high quality, large scale, multimodality,
+multilingualism, multi-scenarios, and covering both implicit and explicit
+sentiment elements. To effectively address the tasks, we devise a novel
+Chain-of-Sentiment reasoning framework, together with a novel multimodal large
+language model (namely Sentica) and a paraphrase-based verification mechanism.
+Extensive evaluations demonstrate the superiority of our methods over strong
+baselines, validating the efficacy of all our proposed methods. The work is
+expected to open up a new era for the ABSA community, and thus all our codes
+and data are open at https://PanoSent.github.io/
+
+
+
+ comment: Accepted by ACM MM 2024 (Oral)
+
+
+
+ + ♻ ☆ LoQT: Low-Rank Adapters for Quantized Pre-Training +
+
+
+
+
+
+
+
+ Sebastian Loeschcke, Mads Toftrup, Michael J. Kastoryano, Serge Belongie, Vésteinn Snæbjarnarson
+
+
+ Training of large neural networks requires significant computational
+resources. Despite advances using low-rank adapters and quantization,
+pretraining of models such as LLMs on consumer hardware has not been possible
+without model sharding, offloading during training, or per-layer gradient
+updates. To address these limitations, we propose LoQT, a method for
+efficiently training quantized models. LoQT uses gradient-based tensor
+factorization to initialize low-rank trainable weight matrices that are
+periodically merged into quantized full-rank weight matrices. Our approach is
+suitable for both pretraining and fine-tuning of models, which we demonstrate
+experimentally for language modeling and downstream task adaptation. We find
+that LoQT enables efficient training of models up to 7B parameters on a
+consumer-grade 24GB GPU. We also demonstrate the feasibility of training a 13B
+parameter model using per-layer gradient updates on the same hardware.
+
+
+
+
+ + ♻ ☆ An Empirical Study on Information Extraction using Large Language Models +
+ +
+ Human-like large language models (LLMs), especially the most powerful and
+popular ones in OpenAI's GPT family, have proven to be very helpful for many
+natural language processing (NLP) related tasks. Therefore, various attempts
+have been made to apply LLMs to information extraction (IE), which is a
+fundamental NLP task that involves extracting information from unstructured
+plain text. To demonstrate the latest representative progress in LLMs'
+information extraction ability, we assess the information extraction ability of
+GPT-4 (the latest version of GPT at the time of writing this paper) from four
+perspectives: Performance, Evaluation Criteria, Robustness, and Error Types.
+Our results suggest a visible performance gap between GPT-4 and
+state-of-the-art (SOTA) IE methods. To alleviate this problem, considering the
+LLMs' human-like characteristics, we propose and analyze the effects of a
+series of simple prompt-based methods, which can be generalized to other LLMs
+and NLP tasks. Rich experiments show our methods' effectiveness and some of
+their remaining issues in improving GPT-4's information extraction ability.
+
+
+
+ comment: Need to submit this paper as the replacement of arXiv:2305.14450
+
+
+
+ + ♻ ☆ Improving Factuality in Large Language Models via Decoding-Time + Hallucinatory and Truthful Comparators +
+
+
+
+
+
+
+
+ Dingkang Yang, Dongling Xiao, Jinjie Wei, Mingcheng Li, Zhaoyu Chen, Ke Li, Lihua Zhang
+
+
+ Despite their remarkable capabilities, Large Language Models (LLMs) are prone
+to generate responses that contradict verifiable facts, i.e., unfaithful
+hallucination content. Existing efforts generally focus on optimizing model
+parameters or editing semantic representations, which compromise the internal
+factual knowledge of target LLMs. In addition, hallucinations typically exhibit
+multifaceted patterns in downstream tasks, limiting the model's holistic
+performance across tasks. In this paper, we propose a Comparator-driven
+Decoding-Time (CDT) framework to alleviate the response hallucination. Firstly,
+we construct hallucinatory and truthful comparators with multi-task fine-tuning
+samples. In this case, we present an instruction prototype-guided mixture of
+experts strategy to enhance the ability of the corresponding comparators to
+capture different hallucination or truthfulness patterns in distinct task
+instructions. CDT constrains next-token predictions to factuality-robust
+distributions by contrasting the logit differences between the target LLMs and
+these comparators. Systematic experiments on multiple downstream tasks show
+that our framework can significantly improve the model performance and response
+factuality.
+
+
+
+ comment: Hallucination Mitigation in LLMs
+
+
+
+ + ♻ ☆ CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding + Benchmark +
+
+
+
+
+
+
+
+ Ge Zhang, Xinrun Du, Bei Chen, Yiming Liang, Tongxu Luo, Tianyu Zheng, Kang Zhu, Yuyang Cheng, Chunpu Xu, Shuyue Guo, Haoran Zhang, Xingwei Qu, Junjie Wang, Ruibin Yuan, Yizhi Li, Zekun Wang, Yudong Liu, Yu-Hsuan Tsai, Fengji Zhang, Chenghua Lin, Wenhao Huang, Jie Fu
+
+
+ As the capabilities of large multimodal models (LMMs) continue to advance,
+evaluating the performance of LMMs emerges as an increasing need. Additionally,
+there is an even larger gap in evaluating the advanced knowledge and reasoning
+abilities of LMMs in non-English contexts such as Chinese. We introduce CMMMU,
+a new Chinese Massive Multi-discipline Multimodal Understanding benchmark
+designed to evaluate LMMs on tasks demanding college-level subject knowledge
+and deliberate reasoning in a Chinese context. CMMMU is inspired by and
+strictly follows the annotation and analysis pattern of MMMU. CMMMU includes
+12k manually collected multimodal questions from college exams, quizzes, and
+textbooks, covering six core disciplines: Art & Design, Business, Science,
+Health & Medicine, Humanities & Social Science, and Tech & Engineering, like
+its companion, MMMU. These questions span 30 subjects and comprise 39 highly
+heterogeneous image types, such as charts, diagrams, maps, tables, music
+sheets, and chemical structures. CMMMU focuses on complex perception and
+reasoning with domain-specific knowledge in the Chinese context. We evaluate 11
+open-source LLMs and one proprietary GPT-4V(ision). Even GPT-4V only achieves
+accuracies of 42%, indicating a large space for improvement. CMMMU will boost
+the community to build the next-generation LMMs towards expert artificial
+intelligence and promote the democratization of LMMs by providing diverse
+language contexts.
+
+
+
+
+ + ♻ ☆ AnyMatch -- Efficient Zero-Shot Entity Matching with a Small Language + Model +
+ +
+ Entity matching (EM) is the problem of determining whether two records refer
+to same real-world entity, which is crucial in data integration, e.g., for
+product catalogs or address databases. A major drawback of many EM approaches
+is their dependence on labelled examples. We thus focus on the challenging
+setting of zero-shot entity matching where no labelled examples are available
+for an unseen target dataset. Recently, large language models (LLMs) have shown
+promising results for zero-shot EM, but their low throughput and high
+deployment cost limit their applicability and scalability.
+ We revisit the zero-shot EM problem with AnyMatch, a small language model
+fine-tuned in a transfer learning setup. We propose several novel data
+selection techniques to generate fine-tuning data for our model, e.g., by
+selecting difficult pairs to match via an AutoML filter, by generating
+additional attribute-level examples, and by controlling label imbalance in the
+data.
+ We conduct an extensive evaluation of the prediction quality and deployment
+cost of our model, in a comparison to thirteen baselines on nine benchmark
+datasets. We find that AnyMatch provides competitive prediction quality despite
+its small parameter size: it achieves the second-highest F1 score overall, and
+outperforms several other approaches that employ models with hundreds of
+billions of parameters. Furthermore, our approach exhibits major cost benefits:
+the average prediction quality of AnyMatch is within 4.4% of the
+state-of-the-art method MatchGPT with the proprietary trillion-parameter model
+GPT-4, yet AnyMatch requires four orders of magnitude less parameters and
+incurs a 3,899 times lower inference cost (in dollars per 1,000 tokens).
+
+
+
+ comment: 12 pages excluding references, 3 figures, and 5 tables
+
+
+
+ + ♻ ☆ RAGLAB: A Modular and Research-Oriented Unified Framework for + Retrieval-Augmented Generation +
+
+
+
+
+
+
+
+ Xuanwang Zhang, Yunze Song, Yidong Wang, Shuyun Tang, Xinfeng Li, Zhengran Zeng, Zhen Wu, Wei Ye, Wenyuan Xu, Yue Zhang, Xinyu Dai, Shikun Zhang, Qingsong Wen
+
+
+ Large Language Models (LLMs) demonstrate human-level capabilities in
+dialogue, reasoning, and knowledge retention. However, even the most advanced
+LLMs face challenges such as hallucinations and real-time updating of their
+knowledge. Current research addresses this bottleneck by equipping LLMs with
+external knowledge, a technique known as Retrieval Augmented Generation (RAG).
+However, two key issues constrained the development of RAG. First, there is a
+growing lack of comprehensive and fair comparisons between novel RAG
+algorithms. Second, open-source tools such as LlamaIndex and LangChain employ
+high-level abstractions, which results in a lack of transparency and limits the
+ability to develop novel algorithms and evaluation metrics. To close this gap,
+we introduce RAGLAB, a modular and research-oriented open-source library.
+RAGLAB reproduces 6 existing algorithms and provides a comprehensive ecosystem
+for investigating RAG algorithms. Leveraging RAGLAB, we conduct a fair
+comparison of 6 RAG algorithms across 10 benchmarks. With RAGLAB, researchers
+can efficiently compare the performance of various algorithms and develop novel
+algorithms.
+
+
+
+ comment: 6 pages, 3 figures
+
+
+
+ + ♻ ☆ WildVis: Open Source Visualizer for Million-Scale Chat Logs in the Wild +
+ +
+ The increasing availability of real-world conversation data offers exciting
+opportunities for researchers to study user-chatbot interactions. However, the
+sheer volume of this data makes manually examining individual conversations
+impractical. To overcome this challenge, we introduce WildVis, an interactive
+tool that enables fast, versatile, and large-scale conversation analysis.
+WildVis provides search and visualization capabilities in the text and
+embedding spaces based on a list of criteria. To manage million-scale datasets,
+we implemented optimizations including search index construction, embedding
+precomputation and compression, and caching to ensure responsive user
+interactions within seconds. We demonstrate WildVis' utility through three case
+studies: facilitating chatbot misuse research, visualizing and comparing topic
+distributions across datasets, and characterizing user-specific conversation
+patterns. WildVis is open-source and designed to be extendable, supporting
+additional datasets and customized search and visualization functionalities.
+
+
+
+
+ + ♻ ☆ Towards a Unified View of Preference Learning for Large Language Models: + A Survey +
+
+
+
+
+
+
+
+ Bofei Gao, Feifan Song, Yibo Miao, Zefan Cai, Zhe Yang, Liang Chen, Helan Hu, Runxin Xu, Qingxiu Dong, Ce Zheng, Wen Xiao, Ge Zhang, Daoguang Zan, Keming Lu, Bowen Yu, Dayiheng Liu, Zeyu Cui, Jian Yang, Lei Sha, Houfeng Wang, Zhifang Sui, Peiyi Wang, Tianyu Liu, Baobao Chang
+
+
+ Large Language Models (LLMs) exhibit remarkably powerful capabilities. One of
+the crucial factors to achieve success is aligning the LLM's output with human
+preferences. This alignment process often requires only a small amount of data
+to efficiently enhance the LLM's performance. While effective, research in this
+area spans multiple domains, and the methods involved are relatively complex to
+understand. The relationships between different methods have been
+under-explored, limiting the development of the preference alignment. In light
+of this, we break down the existing popular alignment strategies into different
+components and provide a unified framework to study the current alignment
+strategies, thereby establishing connections among them. In this survey, we
+decompose all the strategies in preference learning into four components:
+model, data, feedback, and algorithm. This unified view offers an in-depth
+understanding of existing alignment algorithms and also opens up possibilities
+to synergize the strengths of different strategies. Furthermore, we present
+detailed working examples of prevalent existing algorithms to facilitate a
+comprehensive understanding for the readers. Finally, based on our unified
+perspective, we explore the challenges and future research directions for
+aligning large language models with human preferences.
+
+
+
+ comment: 23 pages, 6 figures
+
+
+
+ + ♻ ☆ YOLO-Stutter: End-to-end Region-Wise Speech Dysfluency Detection +
+
+
+
+
+
+
+
+ Xuanru Zhou, Anshul Kashyap, Steve Li, Ayati Sharma, Brittany Morin, David Baquirin, Jet Vonk, Zoe Ezzes, Zachary Miller, Maria Luisa Gorno Tempini, Jiachen Lian, Gopala Krishna Anumanchipalli
+
+
+ Dysfluent speech detection is the bottleneck for disordered speech analysis
+and spoken language learning. Current state-of-the-art models are governed by
+rule-based systems which lack efficiency and robustness, and are sensitive to
+template design. In this paper, we propose YOLO-Stutter: a first end-to-end
+method that detects dysfluencies in a time-accurate manner. YOLO-Stutter takes
+imperfect speech-text alignment as input, followed by a spatial feature
+aggregator, and a temporal dependency extractor to perform region-wise boundary
+and class predictions. We also introduce two dysfluency corpus, VCTK-Stutter
+and VCTK-TTS, that simulate natural spoken dysfluencies including repetition,
+block, missing, replacement, and prolongation. Our end-to-end method achieves
+state-of-the-art performance with a minimum number of trainable parameters for
+on both simulated data and real aphasia speech. Code and datasets are
+open-sourced at https://github.com/rorizzz/YOLO-Stutter
+
+
+
+ comment: Interspeech 2024
+
+
+
+ + ♻ ☆ Natural Language Processing RELIES on Linguistics +
+ +
+ Large Language Models (LLMs) have become capable of generating highly fluent
+text in certain languages, without modules specially designed to capture
+grammar or semantic coherence. What does this mean for the future of linguistic
+expertise in NLP? We highlight several aspects in which NLP (still) relies on
+linguistics, or where linguistic thinking can illuminate new directions. We
+argue our case around the acronym RELIES that encapsulates six major facets
+where linguistics contributes to NLP: Resources, Evaluation, Low-resource
+settings, Interpretability, Explanation, and the Study of language. This list
+is not exhaustive, nor is linguistics the main point of reference for every
+effort under these themes; but at a macro level, these facets highlight the
+enduring importance of studying machine systems vis-\`a-vis systems of human
+language.
+
+
+
+
+ + ♻ ☆ CL4KGE: A Curriculum Learning Method for Knowledge Graph Embedding +
+ +
+ Knowledge graph embedding (KGE) constitutes a foundational task, directed
+towards learning representations for entities and relations within knowledge
+graphs (KGs), with the objective of crafting representations comprehensive
+enough to approximate the logical and symbolic interconnections among entities.
+In this paper, we define a metric Z-counts to measure the difficulty of
+training each triple ($<$head entity, relation, tail entity$>$) in KGs with
+theoretical analysis. Based on this metric, we propose \textbf{CL4KGE}, an
+efficient \textbf{C}urriculum \textbf{L}earning based training strategy for
+\textbf{KGE}. This method includes a difficulty measurer and a training
+scheduler that aids in the training of KGE models. Our approach possesses the
+flexibility to act as a plugin within a wide range of KGE models, with the
+added advantage of adaptability to the majority of KGs in existence. The
+proposed method has been evaluated on popular KGE models, and the results
+demonstrate that it enhances the state-of-the-art methods. The use of Z-counts
+as a metric has enabled the identification of challenging triples in KGs, which
+helps in devising effective training strategies.
+
+
+
+ comment: 16 pages, 3 figures
+
+
+
+ + ♻ ☆ ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG + Capabilities +
+
+
+
+
+
+
+
+ Peng Xu, Wei Ping, Xianchao Wu, Chejian Xu, Zihan Liu, Mohammad Shoeybi, Bryan Catanzaro
+
+
+ In this work, we introduce ChatQA 2, an Llama 3.0-based model with a 128K
+context window, designed to bridge the gap between open-source LLMs and leading
+proprietary models (e.g., GPT-4-Turbo) in long-context understanding and
+retrieval-augmented generation (RAG) capabilities. These two capabilities are
+essential for LLMs to process large volumes of information that cannot fit into
+a single prompt and are complementary to each other, depending on the
+downstream tasks and computational budgets. We present a detailed continued
+training recipe to extend the context window of Llama3-70B-base from 8K to 128K
+tokens, along with a three-stage instruction tuning process to enhance the
+model's instruction-following, RAG performance, and long-context understanding
+capabilities. Our results demonstrate that the Llama3-ChatQA-2-70B model
+outperforms most existing state-of-the-art models, including
+GPT-4-Turbo-2024-04-09, Qwen2-72B-Instruct, and Llama3.1-70B-Instruct, on
+ultra-long tasks beyond 100K tokens, as well as on the RAG benchmark using only
+a 4K context window, showing the strong long context capability across varying
+sequence lengths. We further provide extensive comparisons between direct
+long-context and RAG solutions using the same state-of-the-art long-context
+LLMs. Interestingly, we find that the performance of strong long-context LLMs
+using RAG improves when retrieving a larger number of chunks. With a large set
+of top-k chunks, RAG consistently outperforms direct long-context solution
+using the same state-of-the-art long-context models (e.g., Llama3-ChatQA-2-70B
+and Qwen2-72B-Instruct) on both 32K benchmarks and real-world 128K tasks. To
+advance research in this field, we open-sourced the model weights, training
+data, and the evaluation setup for the for the community:
+https://chatqa2-project.github.io/
+
+
+
+ comment: v2: major update with significantly improved results
+
+
+
+ + ♻ ☆ A Survey on Employing Large Language Models for Text-to-SQL Tasks +
+ +
+ The increasing volume of data stored in relational databases has led to the
+need for efficient querying and utilization of this data in various sectors.
+However, writing SQL queries requires specialized knowledge, which poses a
+challenge for non-professional users trying to access and query databases.
+Text-to-SQL parsing solves this issue by converting natural language queries
+into SQL queries, thus making database access more accessible for non-expert
+users. To take advantage of the recent developments in Large Language Models
+(LLMs), a range of new methods have emerged, with a primary focus on prompt
+engineering and fine-tuning. This survey provides a comprehensive overview of
+LLMs in text-to-SQL tasks, discussing benchmark datasets, prompt engineering,
+fine-tuning methods, and future research directions. We hope this review will
+enable readers to gain a broader understanding of the recent advances in this
+field and offer some insights into its future trajectory.
+
+
+
+
+ + ♻ ☆ Advancing Aspect-Based Sentiment Analysis through Deep Learning Models +
+ +
+ Aspect-based sentiment analysis predicts sentiment polarity with fine
+granularity. While graph convolutional networks (GCNs) are widely utilized for
+sentimental feature extraction, their naive application for syntactic feature
+extraction can compromise information preservation. This study introduces an
+innovative edge-enhanced GCN, named SentiSys, to navigate the syntactic graph
+while preserving intact feature information, leading to enhanced performance.
+Specifically,we first integrate a bidirectional long short-term memory
+(Bi-LSTM) network and a self-attention-based transformer. This combination
+facilitates effective text encoding, preventing the loss of information and
+predicting long dependency text. A bidirectional GCN (Bi-GCN) with message
+passing is then employed to encode relationships between entities.
+Additionally, unnecessary information is filtered out using an aspect-specific
+masking technique. To validate the effectiveness of our proposed model, we
+conduct extensive evaluation experiments on four benchmark datasets. The
+experimental results demonstrate enhanced performance in aspect-based sentiment
+analysis with the use of SentiSys.
+
+
+
+ comment: This paper has already been accepted by the 20th International
+ Conference on Advanced Data Mining and Applications (ADMA2024)
+
+
+
+ + ♻ ☆ Disentangling Length from Quality in Direct Preference Optimization +
+ +
+ Reinforcement Learning from Human Feedback (RLHF) has been a crucial
+component in the recent success of Large Language Models. However, RLHF is know
+to exploit biases in human preferences, such as verbosity. A well-formatted and
+eloquent answer is often more highly rated by users, even when it is less
+helpful and objective. A number of approaches have been developed to control
+those biases in the classical RLHF literature, but the problem remains
+relatively under-explored for Direct Alignment Algorithms such as Direct
+Preference Optimization (DPO). Unlike classical RLHF, DPO does not train a
+separate reward model or use reinforcement learning directly, so previous
+approaches developed to control verbosity cannot be directly applied to this
+setting. Our work makes several contributions. For the first time, we study the
+length problem in the DPO setting, showing significant exploitation in DPO and
+linking it to out-of-distribution bootstrapping. We then develop a principled
+but simple regularization strategy that prevents length exploitation, while
+still maintaining improvements in model quality. We demonstrate these effects
+across datasets on summarization and dialogue, where we achieve up to 20\%
+improvement in win rates when controlling for length, despite the GPT4 judge's
+well-known verbosity bias.
+
+
+
+
+ + ♻ ☆ MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop + Questions EMNLP 2023 +
+ +
+ The information stored in large language models (LLMs) falls out of date
+quickly, and retraining from scratch is often not an option. This has recently
+given rise to a range of techniques for injecting new facts through updating
+model weights. Current evaluation paradigms are extremely limited, mainly
+validating the recall of edited facts, but changing one fact should cause
+rippling changes to the model's related beliefs. If we edit the UK Prime
+Minister to now be Rishi Sunak, then we should get a different answer to Who is
+married to the British Prime Minister? In this work, we present a benchmark,
+MQuAKE (Multi-hop Question Answering for Knowledge Editing), comprising
+multi-hop questions that assess whether edited models correctly answer
+questions where the answer should change as an entailed consequence of edited
+facts. While we find that current knowledge-editing approaches can recall
+edited facts accurately, they fail catastrophically on the constructed
+multi-hop questions. We thus propose a simple memory-based approach, MeLLo,
+which stores all edited facts externally while prompting the language model
+iteratively to generate answers that are consistent with the edited facts.
+While MQuAKE remains challenging, we show that MeLLo scales well with LLMs
+(e.g., OpenAI GPT-3.5-turbo) and outperforms previous model editors by a large
+margin.
+
+
+
+ comment: EMNLP 2023. Our code and datasets are available at
+ https://github.com/princeton-nlp/MQuAKE
+
+
+
+ + ♻ ☆ Shared Latent Space by Both Languages in Non-Autoregressive Neural + Machine Translation +
+ +
+ Non-autoregressive neural machine translation (NAT) offers substantial
+translation speed up compared to autoregressive neural machine translation (AT)
+at the cost of translation quality. Latent variable modeling has emerged as a
+promising approach to bridge this quality gap, particularly for addressing the
+chronic multimodality problem in NAT. In the previous works that used latent
+variable modeling, they added an auxiliary model to estimate the posterior
+distribution of the latent variable conditioned on the source and target
+sentences. However, it causes several disadvantages, such as redundant
+information extraction in the latent variable, increasing the number of
+parameters, and a tendency to ignore some information from the inputs. In this
+paper, we propose a novel latent variable modeling that integrates a dual
+reconstruction perspective and an advanced hierarchical latent modeling with a
+shared intermediate latent space across languages. This latent variable
+modeling hypothetically alleviates or prevents the above disadvantages. In our
+experiment results, we present comprehensive demonstrations that our proposed
+approach infers superior latent variables which lead better translation
+quality. Finally, in the benchmark translation tasks, such as WMT, we
+demonstrate that our proposed method significantly improves translation quality
+compared to previous NAT baselines including the state-of-the-art NAT model.
+
+
+