multi-task deep learning in the software development domain · multi-task deep learning (multi-task...
TRANSCRIPT
Chair of Software Engineering for Business Information Systems (sebis) Faculty of InformaticsTechnische Universität Münchenwwwmatthes.in.tum.de
Multi-task Deep Learning in the Software Development domainSilvia Severini, Garching, 04.11.19Advisor: Ahmed Elnaggar and Davide Bacciu
▪ Introduction▪ Background▪ Research questions▪ Approach▪ Datasets ▪ Results and discussion▪ Conclusion▪ Future research
Outline
© sebis 1Final Presentation Master Thesis – Silvia Severini
The Software Development Domain
© sebis 2
High-quality software:
● nowadays almost every aspect of life relies on it (health, transportation, entertainment)
● Costly
● Hard-work for developers
=> Application of Machine Learning techniques
“Deep learning has achieved competitive performance against previous
algorithms on about 40 SE tasks” [cit]
Final Presentation Master Thesis – Silvia Severini
Problem
When applying Deep learning to software development domain tasks, we face three main problems:
● Data scarcity
● Energy consumption
● Manipulation of source code
Source code summarization for Python
© sebis 3Final Presentation Master Thesis – Silvia Severini
▪ Introduction▪ Background▪ Research questions▪ Approach▪ Datasets ▪ Results and discussion▪ Conclusion▪ Future research
Outline
© sebis 4Final Presentation Master Thesis – Silvia Severini
Natural Language Processing on Source Code
● Context awareness
● Unlimited vocabulary
● Data preprocessing and tokenization
public class HelloWorld { public static void main ( String [ ] args ) { System . out . println ( " Hello, World " ) ; } }
© sebis 5Final Presentation Master Thesis – Silvia Severini
Multi-task Deep Learning
(Multi-task Deep Learning) given m learning tasks {Ti }i=1m where all the tasks or a subset of them
are related, multi-task learning aims to help improve the learning of a model for Ti by using the knowledge contained in all or some of the m tasks.
Why?
● Implicit data augmentation● Attention focusing● Representation bias● Regularization
=> Augment of the generalization capabilities [1]
© sebis 6Final Presentation Master Thesis – Silvia Severini
▪ Introduction▪ Background▪ Research questions▪ Approach▪ Datasets ▪ Results and discussion▪ Conclusion▪ Future research
Outline
© sebis 7Final Presentation Master Thesis – Silvia Severini
Research questions
Can multi-task deep learning be beneficial for tasks in the software development domain?1
How far is multi-task deep learning from state-of-the-art solutions in the software development domain?
2
Could the model be trained with the English language and programming languages together?
3
How does training on multiple tasks of the software development domain simultaneously compare to training on each task separately act?
4
© sebis 8Final Presentation Master Thesis – Silvia Severini
▪ Introduction▪ Background▪ Research questions▪ Approach▪ Datasets ▪ Results and discussion▪ Conclusion▪ Future research
Outline
© sebis 9Final Presentation Master Thesis – Silvia Severini
Transformer model
State-of-the-art for sequence-to-sequence problem manipulation:
● Parallelizable computation
● Faster training
● Manipulation of long-range dependencies
● Encoder - Decoder
Transformer model by Vaswani et al. (2017)
© sebis 10Final Presentation Master Thesis – Silvia Severini
Model architecture
● Multi-task
○ Decoder-only with attention
○ Vocabulary from the language model
○ Multiple loss functions optimized concurrently
● Single-task
○ Each task is trained independently
○ No language model objective
○ Encoder-decoder with attention
© sebis 11Final Presentation Master Thesis – Silvia Severini
Experimental setup
Leibniz-Rechenzentrum (LRZ) available machines
● TensorFlow based library maintained by the Google Brain team
● Three phases:○ Data generation○ Training○ Decoding and evaluation
Hardware:
© sebis 12Final Presentation Master Thesis – Silvia Severini
▪ Introduction▪ Background▪ Research questions▪ Approach▪ Datasets ▪ Results and discussion▪ Conclusion▪ Future research
Outline
© sebis 13Final Presentation Master Thesis – Silvia Severini
Dataset
© sebis 14Final Presentation Master Thesis – Silvia Severini
Dataset and preprocessing
1. 7 supervised tasks from “Deep learning in
software engineering” [Li et al.]
2. Unsupervised Language Model with 5
languages
● Substitution of char, string and numbers with
specific tokens
● Linearization of the code snippets
● Specific Tokenizer for each language
© sebis 15Final Presentation Master Thesis – Silvia Severini
▪ Introduction▪ Background▪ Research questions▪ Approach▪ Datasets ▪ Results and discussion▪ Conclusion▪ Future research
Outline
© sebis 16Final Presentation Master Thesis – Silvia Severini
Single-task learning
© sebis 17Final Presentation Master Thesis – Silvia Severini
Multi-task learning
0.78
0.74
0.70
0.66
0.62
0.58
50k 100k 150k 200k 250k 300k 350k 400k 450k 500k 550k 600k
© sebis 18Final Presentation Master Thesis – Silvia Severini
Single-task vs Multi-task
● MTL performed better on
summarization tasks
● Overfitting avoided for MTL
© sebis 19Final Presentation Master Thesis – Silvia Severini
Comparison with state-of-the-art
● First and last do not have BLEU
counterpart
● Improvement over 4 tasks
© sebis 20Final Presentation Master Thesis – Silvia Severini
Program Synthesis
Partial results
Small MTL Big MTL
50k 100k 150k 200k 250k 300k 350k 400k 450k 500k 550k 600k 650k
65%
74%
0.78
0.74
0.70
0.66
0.62
0.58
© sebis 21Final Presentation Master Thesis – Silvia Severini
▪ Introduction▪ Background▪ Research questions▪ Approach▪ Datasets ▪ Results and discussion▪ Conclusion▪ Future research
Outline
© sebis 22Final Presentation Master Thesis – Silvia Severini
Conclusions
● We investigate the application of Multi-task learning in the software development domain
● We trained single-task and multi-tasks models based on the Transformer architecture
● Consider the structural information of the source code to deal with English and programming
languages
● We achieved better performances than the state-of-the-art papers on 4 tasks
● Multi-task learning is promising for this domain, given enough resources available
© sebis 23Final Presentation Master Thesis – Silvia Severini
▪ Introduction▪ Background▪ Research questions▪ Approach▪ Datasets ▪ Results and discussion▪ Conclusion▪ Future research
Outline
© sebis 24Final Presentation Master Thesis – Silvia Severini
Future research
● Deeper hyperparameters exploration
● Combination of different tasks to understand which give benefits
● Better evaluate the effectiveness of the language model
● Starting point for future research in the software development domain
© sebis 25Final Presentation Master Thesis – Silvia Severini
Technische Universität MünchenFaculty of InformaticsChair of Software Engineering for Business Information Systems
Boltzmannstraße 385748 Garching bei München
Tel +49.89.289.Fax +49.89.289.17136
wwwmatthes.in.tum.de
Silvia Severini
17132