08-runtime-and-performance-analytics.Rmd

# Runtime and Performance Analytics

In this chapter, we discuss the field of performance and runtime analytics. Using Kitchenham’s method
[@kitchenham2004procedures], we have narrowed down the scope of this survey.
For inspiration, we first explored five recent papers on runtime and performance analytics
published at top conferences and journals. These five papers were selected because the papers discuss the "software"
side of performance and runtime analytics, which is consistent with the scope of this book.
We have chosen to focus on the field of performance Vs. energy consumption.
This choice was made since studies on energy consumption
refer to a contemporary and thriving research area
of the runtime and performance analytics field.

## Introduction
Energy consumption is an important factor in the day-to-day usage of software. Especially in the
field of software development for mobile devices, considering energy consumption determines the 
battery-life and limits the usage time of the device. Additionally, battery life is seen as a very
important attribute for most smartphone owners. 92% of potential smartphone
buyers consider battery life as a significant factor in their selection criteria [@OOC2017]. 
From the same research, it is also found that 66% of smartphone owners would pay more for a device 
with longer battery life, given that, on average, 63% of the users are unsatisfied with their current devices' battery life. Although the importance of energy efficient software is clear, programmers lack the knowledge on
the best practices in order to reduce software energy consumption. Also, the current educational material is not focused on this field either [@PHA2016]. In an analysis of questions posted on StackOverflow.com, on the topic of energy efficiency, performed by Pinto et. al. [@PCL2014], it was found that although programmers 
had questions related to energy efficiency, they rarely received appropriate advice. 
With over 2 billion daily smartphone and tablet users worldwide [@NPPPZL2017] it is evident that 
this lack of knowledge needs to be addressed. To help programmers close this knowledge gap, 
hardware-based tools have been introduced. These tools can determine the energy profile of an 
application with high precision. However, for these measurements, costly hardware components need to be 
acquired [@NPPPZL2017]. Software-based solutions solve the problem of requiring expensive 
hardware and are easier in use, but also less precise.

In this chapter, we represent the current state of energy efficiency in software development for 
apps. For this, we set up the following three research questions:

* **RQ1:** What is the state of the art of energy efficiency in software development for mobile 
apps?
* **RQ2:** What is the state of practice of energy efficiency in software development for mobile 
apps?
* **RQ3:** What future work needs to be done in the field of energy efficiency in software 
development for mobile apps?

In the following paragraph, the methodology of the study we have followed will be presented. 
After that, each research question will be answered. In the last paragraph of this chapter 
the conclusion on energy efficiency in software development for apps will be made.

## Methodology
In order to answer our research questions, we have retrieved over 30 papers from the selected 
field. These papers have been found by searching on Google Scholar with a set of filters. 
First off, only papers from the following journals and conferences have been selected:

* ACM Transactions on Software Engineering Methodology (TOSEM),
* Empirical Software Engineering (EMSE),
* IEEE Transactions on Software Engineering (TSE),
* Information and Software Technology (IST),
* Journal of Systems and Software (JSS),
* ACM Computing Surveys (CSUR),
* International Conference on Software Engineering (ICSE),
* Foundations of Software Engineering (SIGSOFT FSE),
* International Conference on Automated Software Engineering (ASE),
* Working Conference on Mining Software Repositories (MSR)
* Symposium on Operating Systems Design and Implementation (OSDI)

Furthermore, during the selection of the papers,
we gave priority to papers that are published not too long ago; preferably after 2012.
But if a paper has been cited many times and its findings are interesting,
it is also considered in our survey.

Given these two criteria the following search queries have been made:

* "Android AND Energy Efficiency AND Software"
* "Energy AND Android AND Runtime AND Analysis AND performance AND Software"

From the selected papers some backward and forward references have also been selected for us to get the 
complete picture. All papers are also checked for relevance to the topic.

To answer **RQ1**, a more in-depth investigation into available tools and guidelines for the 
energy efficiency is performed. For **RQ2**, the focus lies on what tools, guidelines and best 
practices are currently being used in mobile software development. Finally, **RQ3** is answered by
finding papers used for both **RQ1** and **RQ2**.

## RQ1: State of the Art

Many researchers found that there is an increasing demand for energy efficient software. In 
their research, they often propose state of the art approaches which assist developers. These 
approaches can be separated into two main categories: tools and guidelines. This section will 
contain an overview of all state of the art tools and guidelines that satisfy our search 
criteria. This overview is used to answer **RQ1**.

### Tools
The table below shows a quick overview of the tools used to answer **RQ1**.

  |Reference   |  Tool                |  Description                            |
  |------------|-----------------------|------------------------------------------|
  |@CA2018      | Leafactor             | Analyze code smells                      |
  |@LWXM2017    | NavyDroid             | Locate energy inefficiencies             |
  |@KKK2016     | Static analysis tool  | Identify graphical energy bugs           |
  |@SKHA2018    | APOA                  | Compare energy consumption of apps       |
  |@BCBR2017    | Energy Patch          | Detect, validate and repair energy leaks |
  |@NPPPZL2017B | PETrA                 | Measure energy consumption of apps       |
  |@PSCS2018    | jStanley              | Detect and improve energy bugs in Java   |

_Leafactor_

Using static code analysis and automatic refactoring, Leafactor is able to apply
Android-specific optimizations of energy efficiency. The possible optimizations are
indicated by a leaf icon, and the fixed priority is provided by the official Android lint
documentation. This priority reflects the severity of the energy performance, from 1 to 10
with 10 being the most severe energy consumption. Leafactor is able to detect five known
energy-related optimizations.

_NavyDroid_

NavyDroid is a tool created on top of the Java Pathfinder (JPF). Being constructed as a
strengthened DFA (deterministic finite automaton), it can accurately simulate the paused
state, the killed state, and related state transitions of an activity. NavyDroid can
detect complex patterns of wake lock misuses (for example multiple lock acquisitions). 

_Static analysis tool_

The proposed static analysis is a novel static optimization technique for eliminating
drawing commands to produce energy-efficient apps. The technique is exploiting the
insight that static analysis is able to predict future behavior of the app. With the
examples of _loop invariant texture analysis_, _packing_, and _identical frames detection_
it indicates total energy savings up to 44% of the total energy consumption of the device.

_APOA_

A recommendation system which can be used in any marketplace for helping users and
developers to compare apps in terms of performance. As an input, APOA uses a set of metrics
and rating of apps in Comma Separated Value (CSV) format, as well as optimization metrics 
(considering the context of usage). The result is a Pareto optimal front, from which the user selects the most
preferred solution.

_Energy Patch_

A framework that can systematically detect and fix energy bugs in mobile apps, in a scalable
fashion. It uses a combination of static and dynamic analysis techniques to detect, validate,
and repair energy bugs in Android apps. This enables EnergyPatch to quickly narrow down to
the potential program paths along which energy bugs may occur.

_PETrA_

A novel software-based tool for measuring the energy consumption of Android apps. PETrA
(Power Estimation Tool for Android) measures the energy consumption of Android apps by
relying on the tools and APIs provided with the publicly available Project Volta. This
means that all smartphones equipped with Android 5.0 or higher are compatible with that. The tool
provides similar performance to hardware-based solutions.

_jStanley_

jStanley is an Eclipse plugin that helps developers to detect energy bugs.
jStanley identifies in programs the use of Java collections,
check if they are energy efficient and have good runtime performance
and make recommendations for the usage of functions,
which are more efficient as alternatives.
Not only energy leaks are highlighted by the tool, but also better alternatives are provided.
The tool is driven by a set of CSV files that contains energy consumption and runtime values of a wide
variety of methods. These values are harvested on a specific device of the researchers.
With the help of these values, the possible performance boost is calculated.

Most of the state of the art tools can be classified as performing either measurement of energy 
consumption or code analysis. Although plenty of such tools have been proposed, with low citation 
numbers, none of them seem to have had a big influence on research. There is, however,
the possibility, given the growing demand for energy efficiency, that the amount and quality of tools
will increase in the near future.

### Guidelines
Another aspect of the state of the art is the currently proposed guidelines for energy awareness 
and improvements. To answer **RQ1**, seven papers have been selected that present guidelines. The 
research by Cruz et al. [@CA2017] shows that, interestingly enough, the best practices provided by 
Google falls short in addressing energy consumption. However, they come up with guidelines that do 
give improvements. Their guideline shows that correct usage of Android methods such as 
_iewHolder_, _DrawAllocation_, _WakeLock_, _ObsoleteLayoutParam_, and _Recycle_ improved energy 
efficiency. From the research by McIntosh et al.[@MSA2018] it is noted that machine learning is
now also implemented by excited developers into their apps, having an impact on the mobile device's
battery life. They have combined empirical measurements of different machine learning algorithm 
implementations with complexity theory to provide the guideline that some implementations of 
algorithms, such as J48, MLP, and SMO, generally perform better than others. However, they also 
recommend that for optimal results developers must consider their own specific application since 
many factors, e.g. dataset size, can influence the performance.
Finally, Li et al. [@LH2014] investigated commonly recommended energy-saving practices, to see if they
are valid. The topics mentioned in this paper include the following aspects:
optimization of the sending of HTTP packets, 
efficient memory usage, sufficient array lengths,
valid static invocations, and correct field access.
This work gives guidance to mobile app developers
on bundling network packets up to a certain size and using certain 
coding practices for reading array length information,
adequately accessing class fields, and performing 
invocations all led to reduced energy consumption.

## RQ2: State of Practice
Now that we have an overview of the state of the art regarding energy consumption on Android-driven 
mobile devices, we investigate what is the current state of practice. In this section, we try to 
find an answer to **RQ2** stated in the introduction, by going through the papers we have retrieved and 
using the methodology described in Section 2. First, we study how different software engineering 
approaches affect energy consumption and if developers have any knowledge or training regarding 
energy saving programming techniques. Furthermore, we try to see if there are any tools being used, 
which help programmers to decrease energy consumption. To make this more specific, we divide **RQ2** into 
sub-research questions:

* **SQ1:** What are the current development approaches used in practice and how does that affect 
energy consumption?
* **SQ2:** What do programmers currently know and do about energy saving app development?

### Answering SQ1
The programming language used by developers when implementing new mobile apps
is one of the things that is taken into consideration for better energy usage and performance.
There have been a number of studies into the effects of using a certain 
programming language for building software applications. Java is still the most used language for Android 
applications. According to Oliveira et al. [@OOC2017], Java is actually not a good choice when 
limiting energy consumption is one of your priorities. One of the contributions of Oliveira's et al. paper is a 
study where hybrid implementations of applications are compared with the original, pure Java 
implementations, in terms of energy consumption and performance. The paper suggests that hybrid 
implementations might be a good solution, but the problem is that the study was done on just four 
Android applications. The paper did use the Rosetta Code Repository, but this does not represent 
commercial software development.

Another comparable recent research paper investigated the energy-delay product (EDP), which is 
defined as "a weighted function of the energy consumption and run-time performance 
product" [@GKLS2018]:

$E * T^w$

where $E$ is the total energy used to complete a task, $T$ the time and $w$ is a value that represents 
how important energy savings are with regard to time. The advantage of using this function is that 
performance is not neglected. The results in the paper show that the programming language should be 
chosen depending on the programming task. From these results, it is clear that Java is not performing well when 
comparing it to other languages, as it can be seen in the table below. However, it should be noted that
the paper does not test this on actual Android applications, but rather on the Rosetta Code Repository.

![Programming Languages Average Weighted EDP Ranking, retrieved from [@GKLS2018]](figures/EDPranking.jpeg)

The above two papers are examples of research studies that suggest that the current state of practice, 
i.e. developing applications in Java only, is not in line with the current state of the art on energy 
efficiency. In spite of these research papers, developers are still working mostly with Java. One 
possible explanation is that Java is simply compatible with almost every system, because of the Java 
Virtual Machine (JVM). We suggest that further research should investigate why developers are
sticking to Java. It seems that the availability of software libraries, the needed learning time,
the available bug-fixing tools, and the current expertise of developers
might have a significant impact on their choices.

### Answering SQ2
In the previous sections, we have seen examples of the state of the art, as well as software development 
methods that could lead to improvements in energy consumption. However, this knowledge only holds 
value when the community can get developers willing to use these approaches. There have been a number of 
studies that look into what is actually happening.

One example is the paper by Moura et al. [@MPEC2015]. In this paper, a study is conducted by 
looking at a large number of energy-aware commits in GitHub. This 
analysis has yielded a list of approaches that are being used by developers in practice. These 
include frequency and voltage scaling, usage of power efficient libraries, and more. The study notes 
that the vast majority of the commits focus on the lower levels of the software stack. Furthermore, 
only 16.2% of the commits were related to the use of more efficient libraries or data structures. There 
is also a number of software qualities that have been shown to be popular when considering energy consumption.
These include correctness, responsiveness, and performance. Another point that 
researches have addressed is whether software developers were certain of their energy-saving commits. The 
paper suggests that there are definitely cases where developers are not certain about their energy 
saving changes. This might be caused by the fact that there are a few user-friendly tools available 
for aiding developers in making energy-aware decisions. One example of such a tool that attempts to 
assist the developer in making energy friendly decisions is jStanley [@PSCS2018]. However, both 
this paper itself and our own replication study, indicate that current attempts on efficient energy consumption are not anywhere close to ideal. Most of the studies are limited to
the static analysis of the source code and with the usability of the presented tools to be poor.
This prevents such techniques of being used in practice. Additionally, there is the problem of how to 
actually measure the energy being consumed. A number of papers explain how hard this task actually
is. Moura et al. [@MPEC2015] also alluded to the fact that developers that do use third-party 
energy-management tools often lack trust in the accuracy of the tools that they are using. The
measurement of energy-consumption is not actually as straightforward as one may think. In a number
of papers, it is explained that hardware-based solutions are accurate enough, but require expensive
components. This is a serious issue, especially in the field of mobile software development. 
On the other hand, software-based solutions are cheaper but less accurate. Now there is work being 
done on creating more accurate software solutions [@@NPPPZL2017]. Early studies show promising 
results in combining the best of both worlds in order to obtain improved results.

## RQ3: Future research
In this section, we will take a look at the suggested future research directions that have been 
mentioned in the papers used in this survey. We attempt to extract some general trends from the 
future research suggestions that are included. A similar opinion among almost all of the papers 
discussed in this section is that there is an agreement on the fact
that the research field of energy consumption is still not very mature.
The conclusions of the majority of papers mention that more research is required.
However, in this survey, we found that there is a consensus regarding the belief that
it is hard for researchers to make strong claims, mainly due to the doubts 
about both the measurement methods and
the limitations of the available research methodologies on the field. 

One of the fields of research we have looked at, in the previous sections, was the one referred to the relationship between programming languages and energy consumption. Multiple papers suggest that the programming 
languages used for implementation have an influence on the energy consumption and performance 
[@GKLS2018; @OOC2017]. These papers, however, do not look into the influence of specific 
features of the programming languages on energy consumption and performance.

We came across a number of tools which can be used to aid 
developers to manage the energy consumption of applications. However, to our 
knowledge, there are no studies reporting on the actual use of such tools. For future research, 
we suggest the investigation of the number of developers that actually use such tools.
Furthermore, it is important for researchers to investigate which features should be included in tools for optimal use.
For example, we replicated the paper that introduces jStanley, a tool which can be used for energy and performance 
optimization [@PSCS2018]. We faced the issue that this tool cannot implement the suggested 
improvements in an easy and efficient way. In particular, we analyzed the AssertJ open source project 
from GitHub, and we got more than 300 energy saving suggestions.
A user has to click on every suggestion to actually implement the recommended improvement. This is tedious work.
Similarly, a lot of the papers that mention tools that could be used in practice are currently 
being tested with benchmarks. However, it is always the question whether these benchmarks are 
representative enough of the real world to hold actual merit.
Additionally, the outcomes of the papers introducing these tools are not comparable. Testing all 
these tools on the same projects would give better insights into what tools are actually useful for 
the developers to use them in terms of achieving as much energy efficiency as possible. 

Even though to our knowledge, there have not been any studies yet referring to whether the tools described in 
the current research studies are actually been used in practice, there have been some studies into energy-saving 
practices on GitHub. For instance, studies taking into account commit messages [@BLXWT2016; @MPEC2015] have 
shed some light into the state of practice. These studies looked into the commit messages 
containing keywords related to energy consumption and classified the corresponding code. The 
future research directions that are indicated by such papers is that it will be important to 
verify that the energy-saving commits actually have an impact on the overall performance of the 
software. That is both in terms of energy consumption, as well as metrics for runtime performance
and usability. Similarly, future work could be done on verifying that the energy-saving techniques 
are consistent across different software platforms.

Finally, it is to be noted that nearly all the investigated research studies are based on open-source projects from GitHub 
repositories. Research on proprietary and closed-source software could possibly lead to different 
results and would, therefore, be interesting to be considered.

## Conclusion
Having answered the research questions of this chapter in the sections above, we can conclude that 
energy awareness with regard to developing applications for mobile devices, and more specific 
Android, needs more attention. A lot of research is done in the field, resulting in guidelines and 
tools that help developers. But these guidelines are very generic and also the given energy 
awareness information is project-specific. The tools presented by recent research papers hold claims to improvements, but for 
example, the validated tool jStanley is not as easy to use as it is claimed in the original paper. More research 
into the claims made in the papers describing these tools is needed.
In practice, it is observed that the main language for creating Android apps (Java) is not the most 
energy efficient option. Furthermore, developers often do not give enough priority to energy-saving 
options and when they do they are often unsure about the effects of their changes.