Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prepare internal API for shared use between PB and WM callbacks [1/3] #1396

Merged
merged 15 commits into from
Apr 19, 2016

Conversation

hmmr
Copy link
Contributor

@hmmr hmmr commented Apr 17, 2016

This is Part 1/3 of #1382 broken up and reassembled into new commits.

After the merge of TTB branch, and in anticipation of the merge of HTTP API branch, this PR proposes a code reorg consisting of the following steps:

  1. Move functions shared between PB and WM callbacks from riak_kv_ts_svc to riak_kv_ts_util;
  2. Further move functions get_data, put_data, delete_data, query and compile_to_per_quantum_queries (used to produce coverage results, with riak_kv_ts_util:sql_to_cover) into the newly created module riak_kv_ts_api. These will constitute the TS internal API.
  3. Move code serving the INSERT query from riak_kv_ts_svc into riak_kv_qry. Now the call to riak_kv_qry:submit("INSERT ...") will actually take effect (previously, the 'submit' part was a no-op while all the validations, conversions and the eventual call of riak_kv_ts_util:put_data happened as a side effect).

Andrei Zavada added 10 commits April 17, 2016 05:13
with tweaks to decode_query_permissions return value per dialyzer hints.
* use provided human descriptions for compiler errors instead of forcing
  {MnemonicAtom, Description} through "~p" into #rpberrresp.errmsg;
* handle lexer errors (they are signalled via {'EXIT', {Msg, Stacktrace}},
  so make sure ony Msg is reported to the client.
after the maps and other not-simple field types were removed from riak_ql.
@hmmr hmmr changed the title Prepare internal API for shared use between PB, TTB and WM callbacks Prepare internal API for shared use between PB and WM callbacks [1/3] Apr 17, 2016
@@ -118,8 +257,10 @@ maybe_await_query_results(_) ->
% we can't use a gen_server call here because the reply needs to be
% from an fsm but one is not assigned if the query is queued.
receive
Result ->
Result
{ok, Result} ->
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are simply guarding against bad results? For proper responses, this should be the same.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's dialyzer wisperings. Because this function returns whatever it receives, dialyzer has no way of knowing what the return type would be. By constraining the terms on reception to only be one of {ok, Result} or {error, Reason}, we give dialyzer a useful hint.

Andrei Zavada added 2 commits April 18, 2016 02:52
This ensures that both TTB and PB versions of riakc_ts:query get the same
{ok, {[], []}} on empty result set.
@@ -87,3 +87,9 @@
["The '", ParamName, "' parameter is part the primary key, and must have an ",
"equals clause in the query but the ", atom_to_list(Op), " operator was used."])
).

-define(
E_TOO_MANY_SUBQUERIES(N),
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why have this as a macro instead of a function? Looks like it's only called in one place anyway.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm following the local conventions here (alternatively, following the path of least resistance).

@hmmr
Copy link
Contributor Author

hmmr commented Apr 18, 2016

@javajolt All eventually required, yes. In particular, basho/riak_pb#190 is required for #1397, and so also is basho/riak-erlang-client#277. The r_t one uses emended error messages, and is therefore needed for #1396.

My recommendation is to merge the riak_pb PR first, then erlang-client, then proceed with this one.

@hazen
Copy link

hazen commented Apr 18, 2016

Here are explanation of failures:

  • ts_simple_aggregation_math/ts_simple_div_by_zero - Got back divide_by_zero instead of Divide by Zero
  • ts_cluster_comprehensive/ts_unicode - Result from creating table now needs to be {ok, {_,_}}
  • ts_cluster_riak_shell_regression_log - Got back too_many_subqueries but expected Too many subqueries (7)

@hazen
Copy link

hazen commented Apr 18, 2016

Test Results:
ts_simple_activate_table_pass_2-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                       : pass
ts_simple_aggregation-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                                 : pass
ts_simple_aggregation_fail-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                            : pass
ts_simple_aggregation_math-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                            : pass
ts_simple_api-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                                         : pass
ts_simple_create_table_already_created-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                : pass
ts_simple_create_table_dup_primary_key-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                : pass
ts_simple_create_table_no_primary_key-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                 : pass
ts_simple_create_table_not_null_pk_fields-riak_kv_bitcask_backend,riak_kv_eleveldb_backend             : pass
ts_simple_create_table_pass_1-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                         : pass
ts_simple_create_table_split_key-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                      : pass
ts_simple_describe_table-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                              : fail
ts_simple_div_by_zero-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                                 : pass
ts_simple_get-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                                         : pass
ts_simple_get_not_found-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                               : pass
ts_simple_insert-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                                      : pass
ts_simple_insert_incorrect_columns-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                    : pass
ts_simple_latin1_create_table_not_allowed-riak_kv_bitcask_backend,riak_kv_eleveldb_backend             : pass
ts_simple_put-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                                         : pass
ts_simple_put_all_datatypes-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                           : pass
ts_simple_put_all_null_datatypes-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                      : pass
ts_simple_put_bad_date-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                                : pass
ts_simple_put_invalid_data-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                            : pass
ts_simple_put_non_existent_bucket-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                     : pass
ts_simple_recompile_ddl-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                               : pass
ts_simple_select-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                                      : pass
ts_simple_select_compare_two_fields_not_allowed-riak_kv_bitcask_backend,riak_kv_eleveldb_backend       : pass
ts_simple_select_double_in_key-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                        : pass
ts_simple_select_incompatible_type_float_not_allowed-riak_kv_bitcask_backend,riak_kv_eleveldb_backend  : pass
ts_simple_select_incompatible_type_integer_not_allowed-riak_kv_bitcask_backend,riak_kv_eleveldb_backend: pass
ts_simple_select_missing_field_in_pk_not_allowed-riak_kv_bitcask_backend,riak_kv_eleveldb_backend      : pass
ts_simple_select_not_found-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                            : pass
ts_simple_select_unexpected_token_not_allowed-riak_kv_bitcask_backend,riak_kv_eleveldb_backend         : pass
ts_simple_select_where_has_no_lower_bounds_not_allowed-riak_kv_bitcask_backend,riak_kv_eleveldb_backend: pass
ts_simple_select_where_has_no_upper_bounds_not_allowed-riak_kv_bitcask_backend,riak_kv_eleveldb_backend: pass
ts_simple_single_key_ops-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                              : pass
ts_simple_unicode_create_table_not_allowed-riak_kv_bitcask_backend,riak_kv_eleveldb_backend            : pass
riak_shell_test_connecting-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                            : pass
riak_shell_test_connecting_error-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                      : pass
riak_shell_test_disconnecting-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                         : pass
riak_shell_test_reconnecting-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                          : pass
ts_cluster_activate_table_pass_1-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                      : pass
ts_cluster_aggregation-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                                : pass
ts_cluster_comprehensive-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                              : pass
ts_cluster_coverage-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                                   : fail
ts_cluster_create_table_pass_1-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                        : pass
ts_cluster_create_table_via_sql_SUITE-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                 : fail
ts_cluster_keys_SUITE-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                                 : pass
ts_cluster_put_pass_1-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                                 : pass
ts_cluster_quantum_boundaries_SUITE-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                   : pass
ts_cluster_riak_shell_basic_sql-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                       : fail
ts_cluster_riak_shell_regression_log-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                  : fail
ts_cluster_select_pass_1-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                              : pass
ts_cluster_select_pass_2-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                              : pass
ts_cluster_select_pass_3_sorted_on_key-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                : pass
ts_cluster_unicode-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                                    : pass
ts_degraded_activate_table_fail_1-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                     : pass
ts_degraded_aggregation-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                               : pass
ts_degraded_select_pass_1-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                             : pass
ts_degraded_select_pass_2-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                             : pass
ts_simple_activate_table_pass_1-riak_kv_bitcask_backend,riak_kv_eleveldb_backend                       : pass
---------------------------------------------
5 Tests Failed
56 Tests Passed
That's 91.80327868852459% for those keeping score

@hazen
Copy link

hazen commented Apr 18, 2016

  • ts_simple_describe_table is returning a list of lists instead of a list of tuples for data
  • ts_cluster_coverage function clause problem calling riak_kv_ts_svc:compile in ts_cluster_coverage:test_quanta_range
  • ts_cluster_create_table_via_sql_SUITE fails in describe_test. Probably same as above.
  • ts_cluster_riak_shell_basic_sql same describe failure
  • ts_cluster_riak_shell_regression_log-riak_kv_bitcask_backend got {{badmatch,{error,{already_started,riak_shell}}},[{riak_shell_app,boot_TEST,1,[{file,"src/riak_shell_app.erl"},{line,43}]},{riak_shell_test_util,shell_init,0,[{file,"tests/riak_shell_test_util.erl"},{line,49}]}. Guessing previous test failure left the shell running. Unrelated to this PR.

@hazen
Copy link

hazen commented Apr 19, 2016

👍 169492a

@hazen hazen merged commit a237d8e into riak_ts-develop Apr 19, 2016
@hazen hazen deleted the reorg-az-ts_internal_api branch April 19, 2016 04:30
@hmmr
Copy link
Contributor Author

hmmr commented Apr 19, 2016

What's the "seventy" code for Amen?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants