Video

How can I be part of this program?

This video can be a terrific example of overfitting 😁😆

Pleass my youtube compleat now

So amazing 😊

Yydhkxhxhx

Đìh

Please upload slides and perhaps sync it to the audio if you can. Can’t believe we are in age of AI and humans are already losing it

like this

Not liking the current layout. It is not possible to make up a lot of detail (you can see a large room w people in it but nothing of value is discernible) in the small image, and placing it side by side w the presented material essentially makes that harder to see as well, resulting in a need for more screen estate or get used to watching things in half the size that is customary. How about instructing presenters to leave a small box area in their presentations and paste the room in that?

Please support The End and Prevention of Homelessness USA 2030!

If y’all could make sure you actually have clean audio in future that’d be great. The banging is super super distracting and brings down the quality of the whole thing.

🎯 Key Takeaways for quick navigation: 00:12 *🎓 Introduction of Sasha Rush by David Parks* - Introduction of Sasha Rush, a professor at Cornell University and a contributor to Hugging Face, now part of Google, known for his work in AI and large language models. - Sasha's background in computer science, his contributions to AI research, and his commitment to open-source initiatives. 04:28 *🗣️ Overview of Large Language Models (LLMs)* - Large language models (LLMs) are discussed in terms of their significance, complexity, and implications. - LLMs are described as extremely useful, expensive, important, and sometimes perceived as intimidating, but also capable of providing remarkable outputs with consistency and creativity. - The talk aims to provide insights into LLMs and address questions regarding their functioning, reasoning, and impact on various domains. 07:38 *📊 Dividing LLM Understanding into Five Formulas* - Sasha introduces the concept of understanding LLMs through five formulas: perplexity, attention, gem, chinchilla, and rasp. - Each formula represents a different aspect of LLMs, such as generation, memory, efficiency, scaling, and reasoning, providing a comprehensive view of their functionalities. - These formulas aim to simplify complex concepts and enable a deeper understanding of LLMs for both technical and non-technical audiences. 11:33 *🔍 Understanding Perplexity in Language Models* - Perplexity in language models is explained, focusing on the probabilistic model of documents and the probability distribution of word sequences. - The Markov assumption and the representation of Theta as categorical distributions are discussed as historical aspects of language models. - Modern advancements in LLMs are highlighted, indicating departures from past assumptions and embracing more complex neural network architectures. - Language models' historical evolution is discussed, from early Markov models to modern neural network-based LLMs. - Changes in assumptions and approaches, such as fixed-length history and categorical representations, are contrasted with contemporary methods emphasizing neural network architectures. - The session concludes with reflections on the enduring relevance and evolution of language models in the field of natural language processing. 21:04 *🧮 Language Model Development Pre-2010* - Explanation of early methods to develop language models. - Discussion of Shannon's paper from 1948 outlining language model development. - Overview of the challenges and limitations of early language models. 22:32 *📊 Quantifying Language Model Performance* - Introduction to evaluating language model performance. - Explanation of the challenges in quantifying language model effectiveness due to the lack of a definitive correct answer. - Illustration of the evaluation process using an example sentence and the concept of word prediction difficulty. 29:17 *📏 Metric Relating to Compression and Shannon's Work* - Introduction to a metric related to language compression inspired by Shannon's work. - Explanation of encoding words with binary strings based on their probabilities. - Discussion on how the metric, perplexity, measures the efficiency of language compression. 38:21 *📈 Evaluation and Practical Applications of Perplexity* - Discussion on the practical evaluation of perplexity using a test set. - Exploration of the relationship between perplexity and model quality. - Overview of historical perplexity scores achieved by various language models and their implications. 44:09 *📊 Correlation between Perplexity and Task Accuracy* - Perplexity correlates almost perfectly with translation accuracy. - General language perplexity correlates extremely well with task accuracy on different tasks. 45:59 *📈 Impact of Perplexity Reduction on Model Performance* - Lower perplexity corresponds to better model performance. - Graphs illustrate the relationship between perplexity reduction and resource investment. 53:17 *🧠 Introduction to Memory and Attention Models* - Explains the concept of memory and attention in neural network models. - Introduces the limitations of Markov models and the need for attention mechanisms. 01:05:03 *🧠 Understanding Query, Key, and Value Operations in LLMs* - Query, key, and value operations in LLMs enable prediction of the next word based on contextual information. - The process involves querying a lookup table using the query, retrieving the key, and using it to predict the next word. 01:07:03 *🔄 Transitioning from Argmax to Softmax* - Argmax selection poses challenges in neural networks due to its discontinuous nature. - Softmax provides a smooth alternative for selecting the best choice, enabling meaningful training. 01:15:08 *📊 Evolution of Attention Mechanisms in Language Modeling* - The attention mechanism, popularized by the "Attention is All You Need" paper, revolutionized language modeling. - Transformers utilize attention mechanisms to process input sequences efficiently and capture long-term dependencies. 01:18:05 *🖥️ Efficient Computation with Attention Mechanisms* - Attention mechanisms efficiently compute long-term context by utilizing matrix operations. - Matrix multiplication and softmax operations allow for effective computation of attention scores. 01:27:10 *🖥️ Transition from CPUs to GPUs for efficient computation* - The shift from CPUs to GPUs revolutionized the efficiency of running applications like language models. - GPUs allowed for faster computation of complex operations like softmax, significantly altering the research landscape. 01:29:19 *🧠 Understanding GPU architecture and parallel computing* - GPUs operate as parallel computers with multiple threads running simultaneously. - Threads within blocks share block memory, enabling fast data exchange and computation. 01:34:24 *🧮 Efficient matrix multiplication on GPUs* - Matrix multiplication on GPUs involves loading data into block memory, performing computations within blocks, and minimizing reads from global memory. - Leveraging shared memory and parallel processing allows for efficient computation of matrix multiplication. 01:47:04 *💡 Maximizing GPU performance for ML applications* - ML applications benefit from efficient GPU performance, measured in ML operations per second (MLops). - Optimizing data formats, such as using smaller floating-point values, enhances GPU efficiency. 01:53:20 *🚀 GPU Optimization: Why Speed Matters* - GPU programming optimization is crucial for efficient computation. - Efficient GPU programming enables faster computation, which is essential for large-scale models like LLMs. 01:53:49 *🔍 Scaling in LLMs: Model Size vs. Training Data* - The performance of LLMs depends on the interplay between model size and training data. - Increasing model size and training data improves the model's ability to generalize and understand complex patterns. 01:54:42 *📊 Compute Optimization Formula: The Chinchilla Formula* - The Chinchilla Formula extrapolates the expected perplexity of LLMs based on model size and training data. - It suggests a proportional relationship between model size and training data for optimal performance. 02:11:35 *📊 Cost Comparison and Scaling Laws* - Understanding the costs associated with large language models (LLMs). - Comparing and incorporating costs related to data acquisition and model scaling. 02:13:09 *💰 Cost Considerations in Model Development* - Addressing cost considerations in LLM development, especially focusing on compute and data. - Exploring scenarios where reducing specific costs benefits certain stakeholders. 02:14:46 *🔄 Data Reusability and Synthesis* - Discussing the reusability of training data and its diminishing returns. - Exploring the potential of synthetic text generation and its current applications. 02:17:29 *📰 Importance of Training Data Quality* - Highlighting the significance of high-quality training data, such as from sources like the New York Times. - Discussing challenges in quantifying and assessing the quality of training data. 02:18:19 *🛑 Addressing Token Exhaustion Concerns* - Discussing concerns related to token exhaustion and its potential impact on model training. - Exploring strategies to address token scarcity, including alternative data sources. 02:32:52 *🧠 Exploring Formal Logic in Language Models* - Introducing a formal logic approach to understanding language model behavior. - Discussing the use of logical operations to manipulate and analyze model behavior. 02:34:57 *🔄 RASP Language and Multi-Layered Models* - Explaining the RASP language and its deterministic, logical approach to modeling. - Highlighting the benefits of using multiple layers in language model architectures. 02:36:30 *🔍 Potential and Limitations of Formal Logic in LLMs* - Discussing the implications of formal logic approaches for understanding model capabilities. - Exploring the feasibility of implementing complex tasks using logical formulations in language models. Made with HARPA AI

Lmao this talk is trash

Wanted to hear more!

This is awesome. I loves these tools and Taxi data

If Harvard wasn’t so woke. Only woke DEI look down upon Extension students.

Thanks.

Thanks.

Thanks for the informative talk. Can you please share the slides?

Why 360? How about a normal Video?

you can just imagine formulas

Can you upload this with the slide not in VR

Going to need to reupload this, the slides are distorted.

make normal videos please

Thanks very much for this presentation. It helps a lot. Could you kindly tell me the tool you use for demo?

why does every harvard presentation have such a long preamble. just skip the first 10 minutes if you don't want to sit through it.

I'd love to have access to the data to play around with in ArcGIS Pro.

Harvard should not exist.

'Promosm' 🤪

could you share slides?

This was such an awesome video! Thank you HDSI for posting this.

I like your style!!!! #1 YT views provider -> Promo>SM!

Thanks for sharing the tutorial video. Could you share the R code and slides if possible, thanks!

p̷r̷o̷m̷o̷s̷m̷ 😑

Would it be possible to have a recorded version for those who registered?

Thanks so much indeed for this video. Helps me a lot

Are their any open datasets for fair AI ?

Great work and thanks for sharing! Is it just me or the video is a lil bit asynchronous of the voice and the pictures?

Thank you. Great panel discussion!