Add experimental compilation options tuners in Python #580

Artix18 · 2018-07-24T14:44:19Z

These commits implement more functions in the python binders,
and also add different options tuning experiments in Python.

This function might be useful in the future for compilation options search. One of the available options allows us to set the maximum shared memory used.

This function gives the time of execution of a given kernel with an input and specified options. Useful for benchmarking purposes.

Allow the user to set manually the new option privateDepth.

Given a CudaMappingOptions instance, return a Python dictionary that represents all the given options and their values.

ftynse

Looks good until 58caf8a (I've reviewed that code before in different PRs).

This review is on 58caf8a only. Consider calling your file utils.py instead of my_utils.py, it's a shared repository ;) Or chose another meaningful name.

Generally, I advise against using a flow based on global variables. From this commit alone it is not possible to see who else uses them and whether this use is consistent, so you probably want to split your PR into commits differently.

There are several places where sequential positions in a feature vector get translated to mapping options and back. It would be nice to have that localized in a single place, like a special bidirectional map stored somewhere. This would avoid problems when you want to change the list and forget one of the "translation" places.

I guess my last question is to @nicolasvasilache , what is the purpose if this code? Is it a one-off throw-away experiment? Do we expect to reuse it? Knowing this would help me adjust the expected code quality.

ftynse · 2018-07-25T15:45:35Z

python/experimental/options_search/my_utils.py

+
+def get_convolution_example(size_type="default", inp_sz_list=[]):
+    global INIT_INPUT_SZ
+    INIT_INPUT_SZ = 7


isn't it assigned 7 in line 10 anyway?

Yes but the idea is to have other types of examples. And so the INIT_INPUT_SZ would change.

ftynse · 2018-07-25T15:47:46Z

python/experimental/options_search/my_utils.py

+    print(options.tolist())
+
+def set_tc(tc_code_arg, tc_name_arg):
+    global tc_code, tc_name


I don't see these variables declared in this file. Also, do you really want to hide global variable assignment behind a function?

I think it's the (almost the) simplest way to do.

One a bit simpler way would be to call these two functions in the example generation function.

Global variables are never a good idea. Especially if you want to run stuff in parallel eventually. I suggest creating a simple class Config, putting these variables inside and passing instances of this class around. The flow of data is clear.

Global variables are gone. Thank you for the advice!

ftynse · 2018-07-25T15:47:51Z

python/experimental/options_search/my_utils.py

+    tc_name = tc_name_arg
+
+def set_inp(inp_arg):
+    global inp


ftynse · 2018-07-25T15:47:59Z

python/experimental/options_search/my_utils.py

+    inp = inp_arg
+
+def set_vars(tc_prog_arg, inp_arg, cat_val_arg, cat_sz_arg):
+    global tc_prog, inp, cat_val, cat_sz


ftynse · 2018-07-25T15:48:20Z

python/experimental/options_search/my_utils.py

+def catVec_to_optVec(catVec):
+    global cat_val
+    opt = [cat_val[i][catVec[i]] for i in range(NB_HYPERPARAMS)]
+    #if(opt[18] * opt[17] > 1024)


we prefer not to commit commented-out code, especially full of magic constants

This code is obsolete, but it was supposed to help handling the constraints on blocks and threads.
I can remove it.

If it's obsolete, definitely not commit it

ftynse · 2018-07-25T15:57:27Z

python/experimental/options_search/my_utils.py

+def getAllDivs(inp, maxp2=8):
+    p2 = []
+    pp=1
+    for i in range(maxp2+1):


use a list comprehension instead of the loop and mutable temporaries?

ftynse · 2018-07-25T15:58:07Z

python/experimental/options_search/my_utils.py

+        for sz in elem.shape:
+            l += computeDivs(sz)
+    my_list = list(set(l+p2))
+    return np.sort(my_list).tolist()


it feels like a standard sorted(my_list) would have worked just fine here

ftynse · 2018-07-25T15:58:23Z

python/experimental/options_search/my_utils.py

+    return np.sort(my_list).tolist()
+
+def computeCat(inp_arg):
+    global cat_sz, cat_val, inp


these global variables are not defined in this file

ftynse · 2018-07-25T15:58:33Z

python/experimental/options_search/my_utils.py

+    cat_val = []
+
+    divs = getAllDivs(inp)
+    #divs2 = getAllDivs([np.array([tc.tclib.shared_memory_size()])])


commented-out code

Uncommenting this line and one below has the effect of enabling maxsharedoption.
I could certainly add a facultative (global) boolean variable in the initialization.

ftynse · 2018-07-25T16:04:07Z

python/experimental/options_search/my_utils.py

+    divs = getAllDivs(inp)
+    #divs2 = getAllDivs([np.array([tc.tclib.shared_memory_size()])])
+
+    cat_val.append([0,1,2])                    #0


I suppose the comments in this function relate the index of the respective value in the vector of tunable parameters. Would be nice to have at least a comment about that.

More generally, I would suggest having a bi-directional mapping between indices and names in one place and just using it everywhere else.

I find this much more readable

allowed_values = [[] for i in range(...)] allowed_values[MapOptIdx.nTiledDims] = [i + 1 for i in range(6)]

Will work on that then.

Done. What do you think?

facebook-github-bot · 2018-07-25T21:09:34Z

Thank you for your pull request. We require contributors to sign our Contributor License Agreement, and yours has expired.

Before we can review or merge your code, we need you to email [email protected] with your details so we can update your status.

facebook-github-bot · 2018-07-26T12:00:55Z

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Facebook open source project. Thanks!

Also introduce a new folder where will be all the files related to experimentingnew ways of tuning options.

generate_genetic_options will save a file containg the best found options by the genetic algorithm. eval_genetic_options would benchmark these options from the file.

These options must be given in their vector format. See my_utils for more info

Generate a graph of the performance of a random search for a given kernel and inputs sizes.

Generates a set of random options and tries to predict the execution time.

This algorithm uses a variant of experience replay, and one policy per option to predict.

Generate options using UCT algorithm. Useful for benchmarking.

nicolasvasilache

trying to unsubscribe, don't see a way other than approving

Artix18 requested a review from nicolasvasilache July 24, 2018 14:45

facebook-github-bot added the CLA Signed label Jul 24, 2018

Artix18 force-pushed the experimental_tuner branch 8 times, most recently from 3cacf60 to caf5130 Compare July 25, 2018 12:15

Jules Pondard added 4 commits July 25, 2018 05:20

Add shared_memory_size function to PyBinds

f8bdb8e

This function might be useful in the future for compilation options search. One of the available options allows us to set the maximum shared memory used.

Add profile_kernel function to PyBinds

ac4ce08

This function gives the time of execution of a given kernel with an input and specified options. Useful for benchmarking purposes.

Add privateDepth function to PyBinds

4cd991c

Allow the user to set manually the new option privateDepth.

Add getDict function to PyBinds

c607c68

Given a CudaMappingOptions instance, return a Python dictionary that represents all the given options and their values.

Artix18 force-pushed the experimental_tuner branch from caf5130 to 978b4de Compare July 25, 2018 12:21

Artix18 requested a review from ftynse July 25, 2018 14:33

ftynse reviewed Jul 25, 2018

View reviewed changes

Artix18 force-pushed the experimental_tuner branch from 978b4de to 5aa8ad7 Compare July 26, 2018 12:00

Artix18 force-pushed the experimental_tuner branch 2 times, most recently from 034a112 to dc73868 Compare July 27, 2018 09:38

Jules Pondard added 7 commits July 30, 2018 11:34

Add utilities needed for all the different tuning experiments

c3bf96c

Also introduce a new folder where will be all the files related to experimentingnew ways of tuning options.

Add two files to generate and test options of the Genetic algorithm

f9f430a

generate_genetic_options will save a file containg the best found options by the genetic algorithm. eval_genetic_options would benchmark these options from the file.

Add a file to quickly compare different set of options

56610d2

These options must be given in their vector format. See my_utils for more info

Add random search baseline

41f57f8

Generate a graph of the performance of a random search for a given kernel and inputs sizes.

Implement a gradient boosting execution time predictor

418345e

Generates a set of random options and tries to predict the execution time.

Add a RL actor-critic like algorithm for benchmarking

72f2170

This algorithm uses a variant of experience replay, and one policy per option to predict.

Add vanilla MCTS/UCT algorithm for options

90baabe

Generate options using UCT algorithm. Useful for benchmarking.

Artix18 force-pushed the experimental_tuner branch from dc73868 to 90baabe Compare July 30, 2018 09:34

nicolasvasilache approved these changes Dec 11, 2020

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add experimental compilation options tuners in Python #580

Add experimental compilation options tuners in Python #580

Artix18 commented Jul 24, 2018 •

edited

Loading

ftynse left a comment

ftynse Jul 25, 2018

Artix18 Jul 26, 2018

ftynse Jul 25, 2018

Artix18 Jul 26, 2018

ftynse Jul 26, 2018

Artix18 Jul 27, 2018

ftynse Jul 25, 2018

ftynse Jul 25, 2018

ftynse Jul 25, 2018

Artix18 Jul 26, 2018

ftynse Jul 26, 2018

ftynse Jul 25, 2018

Artix18 Jul 26, 2018

ftynse Jul 25, 2018

Artix18 Jul 26, 2018

ftynse Jul 25, 2018

ftynse Jul 25, 2018

Artix18 Jul 26, 2018

ftynse Jul 25, 2018

Artix18 Jul 26, 2018

Artix18 Jul 27, 2018

facebook-github-bot commented Jul 25, 2018

facebook-github-bot commented Jul 26, 2018

nicolasvasilache left a comment

Add experimental compilation options tuners in Python #580

Are you sure you want to change the base?

Add experimental compilation options tuners in Python #580

Conversation

Artix18 commented Jul 24, 2018 • edited Loading

ftynse left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

facebook-github-bot commented Jul 25, 2018

facebook-github-bot commented Jul 26, 2018

nicolasvasilache left a comment

Choose a reason for hiding this comment

Artix18 commented Jul 24, 2018 •

edited

Loading