People often ask me, “what resources do you use to keep on top of machine learning”? I spent some time curating some of my favorite resources and I figured this would be a fine place to share them.
But first, there is some bad news.
The best resource I know of for staying abreast of developments in the field is Twitter.
I know, this is not what you (or I) wanted to hear, but time and time again Twitter has shown me new papers or new findings that have had significant impact on what my team does. Before we dive into my recommendations on how to use Twitter as a professional whilst maintaining your sanity, I’ll talk about some of my other favorite resources.
2: Podcasts
I have listened to data science/machine learning podcasts regularly for the last 7 years and they have continuously shaped my understanding and improved my depth in machine learning. In fact, I recommend listening to podcasts to anyone, from people looking to get into the field to people who are trying to understand how to apply the latest advancements to their work.
On reflection, one of my favorite things about listening to a technical Podcast is it’s limitation: ideas cannot be expressed using images or text. This limitations forces people to explain complex concepts using only intuition. It’s pretty difficult to explain how an LSTM works using equations if you can’t write them down, but explaining them without the equations gives you a much deeper grasp on what is actually going on.
The other interesting thing podcasts can do is introduce you to the vernacular of the field. This is an important step for being able to communicate with other data scientists but it also forms a vocabulary for being able to read and quickly understand papers (and blog posts) in the field.
I’ve found that once you have a basic understanding of the big picture items in machine learning, you can pick up almost any paper and understand most of it’s contents. Below are my recommendations for podcasts.
- DataFramed In this podcast Hugo Browne Anderson talks to real practitioners in the field from numerous companies spanning many industries. It is well produced and nearly every guest is on point, providing practical advice and prospective on how data science works in the real world.
- This Week In Machine Learning and AI This podcast alternates from technical discussions with experts around the world (both practitioners and academics) as well as guests from companies which offer AI/ML solutions. I really like how Sam Charrington alternates between technical, practical, and tool considerations of the machine learning space. Guests are almost always excellent and Sam does a really great job surveying the landscape of how machine learning is applied across industries.
- Talking Machines The focus of the first few seasons is conveying various aspects of machine learning to the listener, and includes interviews with the fathers of deep learning. These seasons are probably my favorite machine learning podcasts ever with a balance of technical depth and great interviews.
Artificial Intelligence Podcast Lex Fridman is a leader in the field of AI and machine learning and has the ability to get some of the most interesting/most important guests from the field. He’s interviewed the VP of the YouTube Search algorithm, the most important figures in deep learning (Ian Goodfellow, Francois Chollet, Andrew Ng, Michael Jordan, John Hopfield), and regularly has guests which stretch the imagination around artificial intelligence (Elon Musk, Whitney Cummings). My favorite episode ever is his interview with Yann LeCun, anyone interested in deep learning/artificial intelligence would gain from listening.
TBD. I’ve been so inspired by podcasts that I’m co-hosting my own with my good friend Michael Aita! Stay tuned to the blog for details.
3: Newsletters
I’ve found subscription newsletters are a great way to stay on top of various parts of the industry. While often many of the papers/talks highlighted in the newsletter I have already heard about, I learn something new from almost every letter and gaining the prospective of someone who pays close attention to the field is really useful for getting the big picture.
I also find that newsletters are typically more focused than podcasts. You can get curated overviews of the papers/events within a very specific topic of interest, providing in depth insight into various topic area’s.
Lastly, newsletters are hard to find! There isn’t a newsletter search engine so the only way I find it is when people share or talk about it on social media (remember stupid Twitter? This is one of it’s uses). Here are a collection of my favorites:
- Sebastian Ruder NLP News This is a monthly newsletter that covers what is happening in the Natural Language Processing space. Sebastian has a clear mastery of this field (and made some of significant contributions to state of the art NLP). There hasn’t been a newsletter since late last year, but I hope he picks it back up.
- ChinAI Jeffrey Ding provides translations of key Chinese documents related to AI and machine learning policy as well as provides an overview of what is moving in China. Fascinating insight into China’s aggressive and rapidly changing application of AI.
- Papers With Code This isn’t precisely a newsletter but it does provide a nice overview of impactful papers in the field, and it is an excellently curated and clean view of who is doing what across the field of machine learning and artificial intelligence. I read at least one paper from each notification and sometimes several are interesting enough for me to read.
4: Books
Books are an obvious way to learn more about the field and due to the medium, they typically allow you to dive deeper into the problems at hand. That said, their are some books that I can easily pick up a lot from and others that are written so poorly or opaquely that they don’t clarify anything.
I tend to gravitate towards books that are focused on expressing the information as clearly as possible and incorporate code to go along with the examples. Below is a selection of my favorite from across genre’s.
- Introduction to Statistical Learning This is my favorite book ever on the process of building and validating machine learning models. It is exceptionally clearly written, with just the right amount of depth to give a good foundational background for concepts without getting to far into theory. It has code examples in R but don’t let that scare you away.
- Forecasting: Principals and Practice Rob Hyndman is a legend in the field of forecasting and this book is a good illustration why. This book offers everything you need to get from no forecasting capabilities at all to being able to understand how to read an ACF and how to properly validate a forecasting model. It is also demonstrated in R, but check out this translation to python.
- Safari Books Online Safari Books Online (O’Reilly Publishing) is an excellent resource for anyone doing technical things. They tend to publish books that are aimed at practitioners which offer the right balance of theory and practice. Some of my favorite books across subjects: Practical Statistics for Data Scientists, Deep Learning Illustrated, Deep Reinforcement Learning Hands On, Graph Algorithms. These books are all available outside of the subscription as well.
5: Other Things
Obviously one of the best way to keep apprised of what is going on the field is to read papers. If you’ve already subscribed to newsletters you’ll typically get info on some of the more interesting papers in a given field, but not always.
One reason I put papers last is that you need some baseline understanding of the foundations as well as the terminology before you can really absorb the more technical content of papers. I found podcasts to be especially helpful for this. Additionally, there are some research groups that put out exceptional content that is at the state of the art.
- Standford NLP Stanford is doing world class research in NLP and offers the lectures and content of it’s CS224 class for free (among other classes). Just the suggested readings alone are an invaluable collection of impactful and important papers, when combined with the lectures and code with corrected assignments it’s difficult to imagine a better place to come up to speed on what is happening in NLP.
- Fast AI This course which is refreshed yearly is one of the cleanest, best put together, and fastest-to-master courses on machine learning and deep learning I’ve seen anywhere. Jeremy Howard teaches with a top down style, starting with practical examples that you can start working with after the first lecture, and diving down into the deepest details of how each algorithm works. You’ll need some introduction to code, but there isn’t any better place to start.
- Random papers I find on twitter (see below)
1: Twitter
With all the information and exciting resources above, how could their be any more? You know when you see news headlines of the “Danger of releasing GPT-2 into the wild” or some new bert related breakthrough? Those are on twitter days or weeks in advance. It’s not only the big important stories, people from all around share interesting analysis, groundbreaking results, and interesting conference summaries.
The problem is, too much time on Twitter is directly and causally linked to degradation in human sanity. Twitter is a place where insanity is amplified, argued out of context, and supported by people you thought you respected. That said, I have found that with careful strategy you can avoid most of the mine fields and get quite a bit of value out of it.
The key to having a productive relationship with twitter is to carefully choose the people you follow and liberally unfollow people who post content that draws you into your unhappy place. Unfortunately you’ll find folks who post both valuable content and content that is hyper political/annoying/hypertension-inducing. For those folks, I suggest you air on the side of unfollowing.
Those of you who aren’t Twitter users might scoff at this and try to talk to me about how I should have an open mind to a variety of ideas. This simply doesn’t reflect the depth that some Twitter users have fallen. If you want to use Twitter to primarily stay on top of the field, be choosy. If you want something else, feel free to dive into the insanity. Here is a list of people that I have found tend to post interesting stuff: @debbiebere, @hugginface, @chrmanning, @ClementDelangue, @lexfridman, @_inesmontani, @jjding99, @malco_barrett, @honnibal, @juliasilge, @wesmckinn, @drob, @ylecun, @karpathy, @jeremyphoward, @seb_ruder, @fchollet.
Notable Ommissions
I did not include resources that are obvious (Google, YouTube) even though I use them all the time. Additionally, I didn’t list any of the major MOOC platforms (coursera, udemy, etc…). This isn’t because they aren’t useful, it’s just I haven’t taken any of the courses from these platforms since 2014. In general, I’ve found these platforms to offer very informative/useful content, but i’ve never found these to be the fastest way to learn for me. One notable exception is Coursera’s Introduction To Deep Learning taught by Andrew Ng.
In this course you will build a neural network and implement gradient descent from scratch while getting instruction from one of the worlds best. They’ve recently updated the course (when I took it we used octave as the programming langauge) and while I haven’t taken it I’ve heard it’s good.
Do you have other resources worth adding to the list? Twitter accounts worth following? Let me know in the comments below!
Also if you want to you could follow me on Twitter… @data_mike_j