Implementation of a Recommendation System for Software Engineering using n-gram language models as described by Santos et al. (2016). The recommender has been implemented using the KaVE Project.
This project is part of the course Advanced Software Engineering - FS 18 and follows a given project description.
For the installation of the recommender you can either use the maven repository dependency, or compile this project. The steps required are outlined in the corresponding subchapters.
To use it in your Maven build add:
<repositories>
<repository>
<id>sebastian-ngram-recommender</id>
<url>https://packagecloud.io/sebastian/ngram-recommender/maven2</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
</repositories>and the dependency:
<dependency>
<groupId>ch.uzh.ase.ngram</groupId>
<artifactId>ngram-recommender</artifactId>
<version>LATEST</version>
</dependency>Run the following commands to clone the project to your local file system:
git clone https://github.com/kuserich/ngram-recommendation-system.git
cd ngram-recommendation-system
mvn clean installAfter you have run the commands the .jar file should be available and can be added to your project.
The n-gram sentences recommender has been trained with the static repository data set provided by the KaVE Project. All repositories are processed using the SentenceExtractor.
Note that the amount of data that is extracted might easily exceed i) space on commodity hard disk and ii) memory requirements on commodity hardware.
Using the static repository data and the current configuration of the SentenceExtractor the output of the extraction process will be around 332GB, which will give a total of 1730 language models.
A recommender, which is represented in NgramRecommenderClient, can either be trained using new data or a language model from the models directory can be loaded.
Own Training Set
The user has the possibility to add its own API sentences, this is particularly of interest if the user has his own projects which are not open for the public.
public void train(String trainFile) throws IOException {
...
}Trained Set
The user has the possibility to use a model file, which is mostly used by widely used libraries.
public void setModelNameFromFileName(String modelFile) {
...
}After the NgramRecommenderClient has been trained, the recommender can accept inputs and predict the next word. Notice that usually a prediction includes an entire API sentence. That is, rather than the single subsequent token that follows given the input, the recommender would predict the most likely API sentence which can be of any length between 1 and n-1 such that n is the n in "n-grams" which the model created during training. Nevertheless, as we're using the predictions with the Event data from the KaVE Project and comparing to the selection that was made, we reduced the number of API tokens that are predicted to 1.
public Set<Tuple<IMethodNAme, Double>> query(StringList stringToCompare) {
Set<Tuple<IMethodName, Double>> recommendation = new HashSet<>();
...
return recommendation;
}Example file
NgramRecommenderClient nrc = new NgramRecommenderClient("models/NUnit.Framework.xml");
System.out.println(nrc.query(new StringList("NUnit.Framework.Assert,AreEqual")));All the models can be found in the "models" directory.
We chose to use Travis CI to have automated tests and automated deployment of the artifacts. Sharing is caring. ;)
Thanks to the commitment to open-source the Travis-CI takes care of the building, testing, deployment, and test coverage sharing of the project. The current build processes can be found here.
language: java
jdk: oraclejdk8
(1)install:
- "mvn -N io.takari:maven:wrapper -Dmaven=3.3.9"
- "./mvnw --show-version --errors --batch-mode test-compile dependency:go-offline"
(2)script: "./mvnw --show-version --errors --batch-mode -Prun-its clean verify"
(3)test:
- mvn test -B -q
(4)cache:
directories:
- $HOME/.m2
(5)deploy:
provider: script
script: "cp .travis.settings.xml $HOME/.m2/settings.xml && mvn deploy"
skip_cleanup: true
on:
all_branches: true
(6)after_success:
- mvn clean -DTRAVIS_JOB_ID=$TRAVIS_JOB_ID cobertura:cobertura coveralls:report(1) - This step installs all the dependencies, if they are not locally avaialable.
(2) - This step ensures a clean setup.
(3) - This step runs the test and prepares a test report.
(4) - This is not really a build step, it just saves dependencies and other required fragments for the maven build in cache.
(5) - The deployment step pushes the artifact to the Maven Repository at packetcloud.io.
(6) - The last step of the build script pushes the test report to Coveralls. Important to note is that the Travis CI job id is added to the coverall report. That is, each artifact is assigned an ID which is used accross in both, Coveralls and packagecloud.
The artifacts are deployed to packagecloud.io and versioned as follows:
0.1.${env.TRAVIS_BUILD_NUMBER}-${env.TRAVIS_BRANCH}
env.TRAVIS_BUILD_NUMBER: Every build in Travis CI gets a number, starting with 1.
env.TRAVIS_BRANCH: As we build every commit, the branch name is added to the artifacts name.
As outlined in the sixth build step in Section Building, the test report is pushed to Coveralls. The build number as well as the branch of the build are the same as of the Travis CI task.
We extracted and trained language models on the entire Context data set for a total of 1730 models. Using the Event data set with a total of 11'546'219 events registered, we evaluated the accuracy of our n-gram language model for a set of 2-gram, 3-gram, 4-gram, and 5-gram language models by comparing the predicted token from the language model with the actual selection. We were able to find models for a total of 1792 out of 11'546'219 events. Out of these, only 78 were predicted accurately (1:1). The overall accuracy of our recommender thus is ~4.35%.
We were unable to reproduce the results outlined in Santos et al. (2016). We believe that this may be due to the following reasons:
i) We ommitted evaluating the accuracy for non-selected but proposed tokens (i.e. rank position)
ii) A large number of language models is relatively small (small number of sentences with small number of tokens)
iii) The overlap between Event data and Context data is unclear
iv) A large portion of accuracy might be lost due to namespace specific bucketizing
v) Due to branching statements, some API sentences get astronomically large, which skews the distribution (relative probability)
For contribution please get in contact with the representatives of the Software Evolution and Architecture Lab (s.e.a.l) of the University of Zurich.
Markus Göckeritz
Julian Iff
Sebastian Pinegger
This project is licensed under the Apache License 2.0.