Why do certain approaches work better than others when designing and training deep neural networks?
With this question in mind, a PhD Computer Science student from Stellenbosch leveraged his background in mathematical statistics to provide a deeper theoretical understanding of what's happening behind the scenes when these networks train successfully - and why they often don't!
This hard-earned expertise has already landed Dr Arnu Pretorius the position of research scientist at the newly-established InstaDeep office in Cape Town. Founded in Tunisia and based in London, the company specialises in the development of highly innovative machine learning algorithms and their use in cutting-edge applications. One of InstaDeep's latest innovations, for example, is the result of a joint research project with DeepMind, one of the top Artificial Intelligence (AI) labs in the world.
Pretorius was also the only student from an African institution to present a paper at the International Conference for Machine Learning in Sweden in 2018, and in 2019 he presented at the Neural Information Processing Systems (NeurIPS) conference in Canada. These two conferences are among the world's largest gatherings for experts in Artificial Intelligence (AI) and Machine Learning (ML).
Pretorius, who will be receiving his PhD in Computer Science at SU's December 2019 graduation ceremony, says he became interested in machine learning while studying in the former MIH Media Lab with a NASPERS bursary: “Machine learning and neural networks, a subfield of ML, involve a wonderful coming together of different ideas with their origin in statistics, applied mathematics, engineering, physics and computer science. And you can apply these to solve relevant and practical problems," he explains.
But he soon came to realise that there was limited theoretical understanding of exactly why and how certain algorithms worked better than others.
After completing his MComm in mathematical statistics (cum laude), he approached Dr Steve Kroon in the Computer Science Division with his idea of making a theoretical contribution to the field. With a PhD in mathematical statistics, Dr Kroon's research in machine learning and artificial intelligence focuses on neural networks, Bayesian methods, and adversarial search.
Dr Kroon is also part of the Maties Machine Learning group, and participated in the first two Deep Learning Indabas that took place in Africa from 2017-2018. The aim of the Deep Learning Indaba is to provide a platform for students and researchers interested in machine learning and Artificial Intelligence to learn from the best in the world, and to share ideas and best practice from diverse fields.
Pretorius attended all three Indabas so far, in time becoming a mentor and running practical sessions: “I have learned so much at these gatherings, met wonderful new people and made new friends. Mentorship from the Indaba community has had a huge impact on my career thus far. If it weren't for the Indabas, I would not have been where I am now."
It was at the 2018 Deep Learning Indaba in Stellenbosch that he met the co-founder and CEO of InstaDeep, Karim Beguir. “He is passionate about machine learning's potential in Africa – both in terms of its application and the people. I was personally very inspired by his vision for the future of the continent."
As a research scientist at InstaDeep, Pretorius plans to maintain contact with machine learning groups at SU and other institutions: “It is so exciting to know that South Africans will have the opportunity to work on high profile projects in their own country."
His advice to students in this field? “Be prepared to work incredibly hard. This is such an interesting field, but it takes an enormous effort to stay up to date. AI technology is changing incredibly fast, and you need to stay on top of the latest developments. That said, the field is also highly stimulating and rewarding, so it's definitely worth your while!"
Pretorius will receive his PhD degree at Stellenbosch University's December graduation ceremony this week, together with another 28 PhD, 26 MSc and 145 BScHons-graduates from SU's Faculty of Science. Pretorius' research was supported by a bursary from the Centre for Artificial Intelligence Research (CAIR) and a Harry Crossley postgraduate scholarship, both from SU.
On the photo above, Dr Steve Kroon and Dr Arnu Pretorius after the December 2019 graduation ceremony at SU. Photo: Stefan Els
Dr Steve Kroon
Computer Science Division, Stellenbosch University
Tel: 021 808 9375/4232
Dr Arnu Pretorius
Research scientist, InstaDeep