llm_misc.md

LLM Misc

• LLM Misc
  • Survey
  • Interpretability
  • Unlearning
  • Personalization
  • Forecasting
  • Detection
  • Generalization
  • Editing
  • Calibration
  • Tokenization
  • Visualization
  • Privacy
  • Society
  • Tutorials
  • Toolkits
  • Misc Talks
  • Misc

Survey

• Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges, ieee access, 2024, arxiv, pdf, cication: -1

  Minghao Shao, Abdul Basit, Ramesh Karri, ..., Muhammad Shafique

• A Survey on Human-Centric LLMs, arXiv, 2411.14491, arxiv, pdf, cication: -1

  Jing Yi Wang, Nicholas Sukiennik, Tong Li, ..., Fengli Xu, Yong Li

• Multilingual Large Language Models: A Systematic Survey, arXiv, 2411.11072, arxiv, pdf, cication: -1

  Shaolin Zhu, Supryadi, Shaoyang Xu, ..., António Branco, Deyi Xiong · (Awesome-Multilingual-LLMs-Papers - tjunlp-lab)

• awesome-discrete-diffusion-models - kuleshov-group

• Survey of Cultural Awareness in Language Models: Text and Beyond, arXiv, 2411.00860, arxiv, pdf, cication: -1

  Siddhesh Pawar, Junyeong Park, Jiho Jin, ..., Alice Oh, Isabelle Augenstein

• LLM-based Optimization of Compound AI Systems: A Survey, arXiv, 2410.16392, arxiv, pdf, cication: -1

  Matthieu Lin, Jenny Sheng, Andrew Zhao, ..., Gao Huang, Yong-Jin Liu

  · (LLM-based-Optimization-of-Compound-AI-Systems - linyuhongg)

• Ecosystem Graphs: The Social Footprint of Foundation Models, arXiv, 2303.15772, arxiv, pdf, cication: 26

  Rishi Bommasani, Dilara Soylu, Thomas I. Liao, ..., Kathleen A. Creel, Percy Liang · (crfm.stanford)

Interpretability

• Announcing Open-Source SAEs for Llama 3.3 70B and Llama 3.1 8B

  · (𝕏)

• The Dark Matter of AI [Mechanistic Interpretability] 🎬

• Goodfire Ember: Scaling Interpretability for Frontier Model Alignment

  · (𝕏)

• Understanding Transformer reasoning capabilities via graph algorithms

• Towards scientific discovery with dictionary learning: Extracting biological concepts from microscopy foundation models, arXiv, 2412.16247, arxiv, pdf, cication: -1

  Konstantin Donhauser, Kristina Ulicna, Gemma Elyse Moran, ..., Cian Eastwood, Jason Hartford

• Refusal Tokens: A Simple Way to Calibrate Refusals in Large Language Models, arXiv, 2412.06748, arxiv, pdf, cication: -1

  Neel Jain, Aditya Shrivastava, Chenyang Zhu, ..., Micah Goldblum, Tom Goldstein

• Emergence of Abstractions: Concept Encoding and Decoding Mechanism for In-Context Learning in Transformers, arXiv, 2412.12276, arxiv, pdf, cication: -1

  Seungwook Han, Jinyeop Song, Jeff Gore, ..., Pulkit Agrawal

• Frame Representation Hypothesis: Multi-Token LLM Interpretability and Concept-Guided Text Generation, arXiv, 2412.07334, arxiv, pdf, cication: -1

  Pedro H. V. Valois, Lincon S. Souza, Erica K. Shimomoto, ..., Kazuhiro Fukui · (frame-representation-hypothesis.git - phvv-me)

• 🌟 Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capability, arXiv, 2411.19943, arxiv, pdf, cication: -1

  Zicheng Lin, Tian Liang, Jiahao Xu, ..., Yujiu Yang, Zhaopeng Tu

• Large Multi-modal Models Can Interpret Features in Large Multi-modal Models, arXiv, 2411.14982, arxiv, pdf, cication: -1

  Kaichen Zhang, Yifei Shen, Bo Li, ..., Ziwei Liu

• LLMs Do Not Think Step-by-step In Implicit Reasoning, arXiv, 2411.15862, arxiv, pdf, cication: -1

  Yijiong Yu

  · (𝕏)

• Large Multi-modal Models Can Interpret Features in Large Multi-modal Models, arXiv, 2411.14982, arxiv, pdf, cication: -1

  Kaichen Zhang, Yifei Shen, Bo Li, ..., Ziwei Liu · (multimodal-sae - EvolvingLMMs-Lab)

• Do Large Language Models Perform Latent Multi-Hop Reasoning without Exploiting Shortcuts?, arXiv, 2411.16679, arxiv, pdf, cication: -1

  Sohee Yang, Nora Kassner, Elena Gribovskaya, ..., Sebastian Riedel, Mor Geva · (𝕏)

• Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models, arXiv, 2411.12580, arxiv, pdf, cication: -1

  Laura Ruis, Maximilian Mozes, Juhan Bae, ..., Edward Grefenstette, Max Bartolo

• Dario Amodei: Anthropic CEO on Claude, AGI & the Future of AI & Humanity | Lex Fridman Podcast #452 🎬

• The Rate Distortion Dance of Sparse Autoencoders

  · (𝕏)

• The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities, arXiv, 2411.04986, arxiv, pdf, cication: -1

  Zhaofeng Wu, Xinyan Velocity Yu, Dani Yogatama, ..., Jiasen Lu, Yoon Kim

• Interpretable Language Modeling via Induction-head Ngram Models, arXiv, 2411.00066, arxiv, pdf, cication: -1

  Eunji Kim, Sriya Mantena, Weiwei Yang, ..., Sungroh Yoon, Jianfeng Gao · (induction-gram - ejkim47)

• Analyzing The Language of Visual Tokens, arXiv, 2411.05001, arxiv, pdf, cication: -1

  David M. Chan, Rodolfo Corona, Joonyong Park, ..., Yutong Bai, Trevor Darrell

• Physics in Next-token Prediction, arXiv, 2411.00660, arxiv, pdf, cication: -1

  Hongjun An, Yiliang Song, Xuelong Li · (youtube)

• Mixture of Parrots: Experts improve memorization more than reasoning, arXiv, 2410.19034, arxiv, pdf, cication: -1

  Samy Jelassi, Clara Mohri, David Brandfonbrener, ..., Sham M. Kakade, Eran Malach

• On Memorization of Large Language Models in Logical Reasoning, arXiv, 2410.23123, arxiv, pdf, cication: -1

  Chulin Xie, Yangsibo Huang, Chiyuan Zhang, ..., Badih Ghazi, Ravi Kumar · (mem-kk-logic - AlphaPav) · (memkklogic.github)

• What Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A Gradient Perspective, arXiv, 2410.23743, arxiv, pdf, cication: -1

  Ming Li, Yanhong Li, Tianyi Zhou · (Layer_Gradient - MingLiiii) · (aimodels)

• Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics, arXiv, 2410.21272, arxiv, pdf, cication: -1

  Yaniv Nikankin, Anja Reusch, Aaron Mueller, ..., Yonatan Belinkov · (x)

• Large Language Models Reflect the Ideology of their Creators, arXiv, 2410.18417, arxiv, pdf, cication: -1

  Maarten Buyl, Alexander Rogiers, Sander Noels, ..., Jefrey Lijffijt, Tijl De Bie

• Evaluating feature steering: A case study in mitigating social biases

  · (x)

• Introducing Transluce

• 🎬 ICML 2024 Tutorial: Physics of Language Models

• Physics of Language Models

  · (bilibili)

Unlearning

• Does your LLM truly unlearn? An embarrassingly simple approach to recover unlearned knowledge, arXiv, 2410.16454, arxiv, pdf, cication: -1

  Zhiwei Zhang, Fali Wang, Xiaomin Li, ..., Wenpeng Yin, Suhang Wang · (𝕏) · (t)

• 🌟 CLEAR: Character Unlearning in Textual and Visual Modalities, arXiv, 2410.18057, arxiv, pdf, cication: -1

  Alexey Dontsov, Dmitrii Korzh, Alexey Zhavoronkin, ..., Ivan Oseledets, Elena Tutubalina · (huggingface) · (multimodal_unlearning - somvy)

Personalization

• Personalization of Large Language Models: A Survey, arXiv, 2411.00027, arxiv, pdf, cication: -1

  Zhehao Zhang, Ryan A. Rossi, Branislav Kveton, ..., Nesreen Ahmed, Yu Wang

Forecasting

Detection

• LLMmap: Fingerprinting For Large Language Models, arXiv, 2407.15847, arxiv, pdf, cication: -1

  Dario Pasquini, Evgenios M. Kornaropoulos, Giuseppe Ateniese

• Robust Watermarking Using Generative Priors Against Image Editing: From Benchmarking to Advances, arXiv, 2410.18775, arxiv, pdf, cication: -1

  Shilin Lu, Zihan Zhou, Jiayou Lu, ..., Yuanzhi Zhu, Adams Wai-Kin Kong

  · (VINE - Shilin-LU)

• Introducing SynthID Text 🤗

• A Watermark for Black-Box Language Models, arXiv, 2410.02099, arxiv, pdf, cication: -1

  Dara Bahri, John Wieting, Dana Alon, ..., Donald Metzler

• Are AI Detectors Good Enough? A Survey on Quality of Datasets With Machine-Generated Texts, arXiv, 2410.14677, arxiv, pdf, cication: -1

  German Gritsai, Anastasia Voznyuk, Andrey Grabovoy, ..., Yury Chekhovich

Generalization

Editing

• uncensored version of Qwen/QwQ-32B-Preview created with abliteration 🤗

  · (remove-refusals-with-transformers - Sumandora)

• Can Knowledge Editing Really Correct Hallucinations?, arXiv, 2410.16251, arxiv, pdf, cication: -1

  Baixiang Huang, Canyu Chen, Xiongxiao Xu, ..., Ali Payani, Kai Shu

  · (llm-editing.github)

Calibration

Tokenization

• Byte Latent Transformer: Patches Scale Better Than Tokens

• added a few lines of code to the base Llama 3 tokenizer and got a free boost in arithmetic performance 𝕏

• Counting Ability of Large Language Models and Impact of Tokenization, arXiv, 2410.19730, arxiv, pdf, cication: -1

  Xiang Zhang, Juntai Cao, Chenyu You

Visualization

• data-formulator - microsoft

  Create Rich Visualizations with AI

Privacy

• PAPILLON: PrivAcy Preservation from Internet-based and Local Language MOdel ENsembles, arXiv, 2410.17127, arxiv, pdf, cication: -1

  Li Siyan, Vethavikashini Chithrra Raghuram, Omar Khattab, ..., Julia Hirschberg, Zhou Yu · (PAPILLON - siyan-sylvia-li)

Society

Tutorials

• smol-course - huggingface
• Understanding LLMs from Scratch Using Middle School Math
• An attempt to reconstruct Ilya Sutskever's 2020 AI reading list

Toolkits

Misc Talks

Misc

• Speech and Language Processing(3rd ed. draft)
• CIS 7000 - Large Language Models
• WHAT LLM PROVIDER?