 |
| Dr John Kelleher from DIT |
Enssat school was very honored to receive Dr John Kelleher from the Dublin Institute of technology for a three-day visit. Our third year computing students benefited from a talk on "deep neural network". He also joined the mobility conference as it was the international week at Enssat and he gave a very useful presentation of DIT and the city of Dublin. We currently have a partnership with 5 students studying at DIT this year. Dr Kelleher finished his French visit by giving a seminar to the university researchers of IRISA on "Tackling Long-Distance Dependencies in Recurrent Neural Network Language Models using Attention"(1).
 |
| John Kelleher presenting DIT at the international mobility conference. |
(1) Abstract: In this talk I will present ongoing research from my group at DIT that looks to tackle the problem of long-distance dependencies in language models by extending Recurrent Neural Network Language Models (RNN-LMs) with an attention mechanism. As part of the talk I will present results that show that an "attentive'' RNN-LM (with 11M parameters) achieves a better perplexity than larger RNN-LMs (with 66M parameters) and achieves performance comparable to an ensemble of 10 similar sized RNN-LMs. I will also present results that show that an "attentive'' RNN-LM needs less contextual information to achieve similar results to the state-of-the-art on the wikitext2 dataset. The talk will conclude with a review of the current architecture and a discussion on potential future directions for extending the research.