Test suite

[mateor]

I need to add a test suite. When run as a test, it will ignore clean-up and then diff patched and unpatched at minimum.

A test suite would be cool, although i do not know how needed it is here, since we haven't pushed anything truly "broken" in a long time. Bugs in non-linux machines, yes, but I won't be able to craft a test suite for that...

This is just something to think about. Maybe a prep for the play store.

[xee5ch]

I would welcome the idea because I am getting back into using it, and now want to help you hack it. Do not know much about testing shell scripts, so naturally I googled it.

SO: Unit tests for shells scripts

shunit2: One of the answes from above, a unit test framework for shell scripts

[mateor]

I am out of town this week in an area with weak WiFi...I may not be able to really test much until next Sunday, although I will be working on OpenPdroid update each night. But the pull requests have looked good in passing...I WILL review them when I return if not sooner.

In order to really institute a testing framework, we would need a way to easily pass values and call some stub methods. Currently, I have several hundred ROMs of various vintage/origin that I run big changes through- not ideal.

Our logging is pretty robust, just about everything that can go wrong or that we need to know is reported there. The auto-patcher command-line arguments allow us to pass values for romtype, patchdate and mods...perhaps just adding a test flag which then skipped the compiling/decompiling steps is the next step in creating a testing framework.

Of course that would not protect us from any issues with compiling/recompiling...but some things simply require a full run to test. I just fixed a nasty bug related to recompiling for 2.9.7.

I will also be thinking on this more. I also have much of my Auto-patcher 3.0 worked out in my head...if you are interestead we can talk about that as well.

Thanks

[xee5ch]

I am out of town this week in an area with weak WiFi...I may not be able to really test much until next Sunday, although I will be working on OpenPdroid update each night. But the pull requests have looked good in passing...I WILL review them when I return if not sooner.

No worries. I just started to get into it and waited impatiently
yesterday after your lightning fast responses. I am just itchy to get
back into even some light coding, and I use PDroid all the time, and I
did not recall it was written in bash. I just checked out the tickets
and thought I could help with low-hanging fruit.

In order to really institute a testing framework, we would need a way to easily pass values and call some stub methods. Currently, I have several hundred ROMs of various vintage/origin that I run big changes through-not ideal.

Agreed. I saw the other ticket (meateor/autopatcher#116) and I started
my own functionalize branch and will begin moving everything below
BASH_SOURCE (kind of like my if __name__ == "__main__" construct in
Python; no idea bash had it) into functions and make them it to
components, then see how "testable" we can make it. If my last patch
goes through, and you have no objections, I will start work on that
next.

Our logging is pretty robust, just about everything that can go wrong or that we need to know is reported there. The auto-patcher command-line arguments allow us to pass values for romtype, patchdate and mods...perhaps just adding a test flag which then skipped the compiling/decompiling steps is the next step in creating a testing framework.

Agreed, you logging is great. I really appreciate how easy your stuff
was/is to use. As for the testing flag as the first step, agreed. I
just thought you would want to build something robust on account of how
many ROMs you deal with already. A bash unit-testing framework sounds
extreme, but I am not sure how to thoroughly right tests just using bash
without worrying about another batch of code and my own quality concerns
(my sucky coding, not yours). But you are the boss, and I am willing to
start small this way.

Of course that would not protect us from any issues with compiling/recompiling...but some things simply require a full run to test. I just fixed a nasty bug related to recompiling for 2.9.7.

Well I would love to help with that too.

I will also be thinking on this more. I also have much of my Auto-patcher 3.0 worked out in my head...if you are interestead we can talk about that as well.

I am all ears. Let me know.

Thanks

[mateor]

The testing unit I had in mind is not really for anyone's benefit but mine and yours and any other contributors. When I was working out the new apk_was_copied routine, I had to run through entire patching processes for each edit, it took forever. I would like to be able to run ./auto_patcher -t ROM.zip openpdroid --Mms.apk and then just test the Mms.apk process. Even if that doesn't quite happen, just the ability to pass parameters instead of extracting from the build.prop makes a difference.

I sometimes come across situations where I want to test what happens with the script when something like apk_was_copied is combined with provisional use. Then I have to scourr user-submitted test logs for a rom with those matching symptoms. So a framework to test those sort of situations would be a huge improvement and time saver. Then we could simply think up test cases as they cross our minds and add them to a test-list. So when we run ./auto_patcher --test-suite or whatever, it can just print out a brief report after running through all the tests...18 of 18 tests passed or whatever. My inspiration for this was some experience with rails, and part of my motivation is to make test-driven-development a more natural part of my toolbox.

Some deeper investigation of man getopt will be required.

I will briefly write up my next stage plan for the autopatcher in the appropriate issue and will link to it below.

[xee5ch]

Cool, let me know what your thoughts are and I would be very down to help. As you might have guessed, #123 ideas about partitioning files came up because I was thinking about the testing suite stuff at the same time. But I am not sure that is the best way forward for you and others, so we will talk later and see what you think.

UPDATE: And the getopt stuff sounds interesting, forgot to mention. And more importantly excuse the terrible formatting in my previous comment. I responded by email and I failed several attempts at getting quotes to work properly.

C'est la vie.

[mateor]

I am working on the "3.0.0" update and part of that is going to involve filtering out garbage files from the diff. As we make patches now, we simply decompile the files and diff them. So any meaningless change in unrelated files, even just registers, is also part of our patching process and acts as an additional point of failure.

I am working on this now. and my current method for my test scripts is to hardcode in "--ignore=$FILE" when calling diff. But that only handles the files that we know about, and I think that basically every class in these jar files is a candidate to end up in our diffs. This is especially important to watch as the update involves splitting the diffs from .jar patches to .smali patches. No need to carry useless patches there at all.

I bring this up here, because I would like to add a test that explicitly checks for just the .smali patches we need, and deletes the rest. This could be distributed to patch builders and then we could simply set it and forget it.

[xee5ch]

Sounds cool. How can I help you with this moving forward then?

[mateor]

I hope to have the test suite begin pretty soon. I don't thin it needs to be very complicated. The tests I would like to run don't have to be too complicated. Things like

• size of patches being applied fits a predetermined size range
• md5 of all patched binaries change (ensure patches are actually applied)

etc. Ideally, everytime I find/fix a bug, I would pop over a write a quick test so it gets caught in the future. And then periodically we can just run the entire test suite and just let it run, with an output something like "28 out of 28 tests passed".

Ideally, the same test suite could be used to test just one apk/jar from the files.txt...so if OpenPDroid modifies 5 jars/apks, we could call the test to just operate on the Mms for example. Currently, I modify the OpenPDroid files.txt to just read Mms.apk and then return it to original state when done. I would like a less error-prone approach. I am still open to the structure that you sent before, I just didn't know how you intended to implement it. If you have an idea, feel free to pitch it.

I realize that this seems a lot for a bach script. But the update I am working on now should hopefully move us pretty close to universal patching and hopefully stock-rom support. If that gets humming as well as our current support for custom roms, then I will move forward and create an apk installer for the Play Store. The installer that I intend to use will leverage the bash scripts as part of a package management system, so this bash script will actually power the OpenPdroid installer in the Play Store. When we are considering using these scripts as a way to modify running devices...I just want to be real careful.

I know that these things are possible, I have done them on my personal devices as of last year. But I have been trying to think of ways to distribute that aren't dangerous, you know? Having an ever-growing test suite is part of that.

[mateor]

I wouldn't say that this is a test structure...but I am using something similar to what I brought up above while working on an update. This script splits the jar patches into smali patches (split_patch_opd.py) and this erases all errant patches that we don't need for OpenPDroid support (split_opd.sh

[xee5ch]

Sorry for the delayed response. I had no real grand plan with how to implement the testing suite, I just searched around for bash/other shell scripting language testing frameworks out of curiosity, as to not reinvent the wheel if we want to get more and more complicated with it. I am not particularly wed to any idea, just want to help you out any way I can.

In any event, I need to get back into this. I would probably start by cleaning up some of the code and make it all functions, current issue #116 , and then your main loop calling in all the proper functions and in the right order. Later you could even serialize them with xargs to get speed, optionally.

But the bottom line for the functions would be, testing each unit of operation independently, and that would allow to test what you want, and then allow to chain components together without just running the whole auto-patcher process or doing manual steps. Does that make sense?