Che APB installation fails on minishift with default configuration

[ibuziuk]

Steps to reproduce:

• Follow the instructions from README.md
• Select Che (APB) from the catalog

• Use default setup in the wizard

• ERROR: in My Project Provisioned Service for Che would have the following error:

infoThe service is not yet ready. Error provisioning ServiceInstance of ClusterServiceClass (K8S: "1882ffca5d72b1084e9107e3485f5066" ExternalName: "dh-eclipse-che-apb") at ClusterServiceBroker "ansible-service-broker": Status: 403; ErrorMessage: ; Description: User does not have sufficient permissions; ResponseError:

minishift version - v1.20.0+53c500a

[1] https://github.com/ansibleplaybookbundle/eclipse-che-apb#requirements

[l0rd]

@ibuziuk can you please verify if you are able to deploy any APB?

[ibuziuk]

yeah, sure I will

[ibuziuk]

@l0rd Apache HTTP Server (httpd) seems to work fine:

[l0rd]

@ibuziuk are you able to retrieve the provisioning logs as described here?

[ibuziuk]

@l0rd after some attempts I was finally able to start che via apb and start a workspace:

Not sure though what did the trick. I will do installation setup from scratch again and update docs if needed. Funny thing that apb bootstrap results in 403 error for me, but nevertheless installing che via apb worked even without this step:

Running APB image: docker.io/ansibleplaybookbundle/apb-tools:canary
Targetting minishift host: tcp://192.168.42.121:2376
Contacting the ansible-service-broker at: https://asb-openshift-automation-service-broker.192.168.42.121.nip.io/openshift-automation-service-broker/v2/bootstrap
Error: Attempt to bootstrap Broker returned status: 403
Unable to bootstrap Ansible Service Broker.

[ibuziuk]

@l0rd so far I was able to run the che via apb only after granting cluster-admin role to developer:

oc adm policy add-cluster-role-to-user cluster-admin developer

this does not sound like to be smth. expected, right ?

[ibuziuk]

not related to this issue but also send a PR with typo fix in README - #20

[l0rd]

@ibuziuk granting developer the admin role is needed to run apb CLI tool. But that is needed only for development.

If installing Che using the service catalog UI, the role of you user doesn't matter. It would be useful if you could provide the provisioning logs (if any) as requested in the comment above.

[ibuziuk]

@l0rd I was able to retrieve only ASB logs [1]. provisioning namespace / pod were not created

[1] https://pastebin.com/rteAQAD1

[ghost]

@ibuziuk @l0rd but isn't it smth that we ultimately want to achieve? One installation per cluster - an admin does it, or whoever has admin privileges.

I can confirm I have the same with OKD 3.10

[l0rd]

@eivantsov what do you mean? Here the problem is that @ibuziuk is not able to successfully provision Che using Che APB. And he cannot get any log about why the APB is failing to start (except User doesn't have enough permissions and I suspect that the User is the ASB user, not @ibuziuk user).

Regarding the admin privileges:

• we should not require admin privileges because we don't really need them to use Che APB (c.f. Changing ASB configuration should not be a requirement #17 (comment))
• since it is completely possible to have 2 distinct users with admin privileges, requiring such privileges won't help blocking a second Che instances to be started on the cluster

[ghost]

@l0rd I mean don't we want to have a limited number of users (admins only) to deploy Che?

For me it works now only after I granted OpenShift user with cluster-admin privileges. So I had to:

1. Add cluster-admin role to OpenShift user
2. Edit cm to change sandbox role to admin

Not one or the other, but both. So, different issues.

Since the installation will require admin privileges anyway (for stacks and editing configmap - sandbox role to admin), I don't see any issues with requiring a user to be a cluster admin. Maybe not for upstream though.

Currently, I do not see any relevant logs related to User doesn't have permissions.

[l0rd]

@l0rd I mean don't we want to have a limited number of users (admins only) to deploy Che?

I agree. We have been discussing it on the corresponding issue #18

Since the installation will require admin privileges anyway (for stacks and editing configmap - sandbox role to admin)

Upstream doesn't need to build and create imagestreams for stacks right?
And I think we should not change sandbox role to admin anymore. We should rather manually create the RoleBinding. It has the benefit of not requiring admin rights and can be done after provisioning. This was discussed here and here.
We need to avoid requiring admin privileges, it limits Che adoption. I hope we agree on that.

[ghost]

@l0rd yes, keep provisioning on failure and then manually create sa and rolebinding. Looks ok.

But i am still puzzled with the original problem - why only an admin OpenShift user can provision Che?

[l0rd]

But i am still puzzled with the original problem - why only an admin OpenShift user can provision Che?

I don't know. And I cannot test it myself right now. Maybe ASB log have some hint?

[ghost]

@l0rd I can only see what is obvious. 403 is the response to PUT call:

172.17.0.6 - - [14/Aug/2018:15:44:19 +0000] "PUT /osb/v2/service_instances/e83638d1-9fd8-11e8-b49e-0242ac110009?accepts_incomplete=true HTTP/1.1" 403 65
time="2018-08-14T15:44:19Z" level=info msg="Request: \"PUT /osb/v2/service_instances/e83638d1-9fd8-11e8-b49e-0242ac110009?accepts_incomplete=true HTTP/1.1\\r\\nHost: broker.openshift-automation-service-broker.svc:1338\\r\\nAccept-Encoding: gzip\\r\\nContent-Length: 1039\\r\\nContent-Type: application/json\\r\\nUser-Agent: Go-http-client/1.1\\r\\nX-Broker-Api-Originating-Identity: kubernetes eyJ1c2VybmFtZSI6ImRldmVsb3BlciIsInVpZCI6IiIsImdyb3VwcyI6WyJzeXN0ZW06YXV0aGVudGljYXRlZDpvYXV0aCIsInN5c3RlbTphdXRoZW50aWNhdGVkIl0sImV4dHJhIjp7InNjb3Blcy5hdXRob3JpemF0aW9uLm9wZW5zaGlmdC5pbyI6WyJ1c2VyOmZ1bGwiXX19\\r\\nX-Broker-Api-Version: 2.13\\r\\n\\r\\n{\\\"service_id\\\":\\\"1882ffca5d72b1084e9107e3485f5066\\\",\\\"plan_id\\\":\\\"17835fe3e1d51c1136eecc730e0ef738\\\",\\\"organization_guid\\\":\\\"3d0309fa-9fd8-11e8-bbb1-54e1ad81a005\\\",\\\"space_guid\\\":\\\"3d0309fa-9fd8-11e8-bbb1-54e1ad81a005\\\",\\\"parameters\\\":{\\\"che_data_pvc_quantity\\\":\\\"1Gi\\\",\\\"che_debug_server\\\":false,\\\"che_image_tag\\\":\\\"nightly\\\",\\\"che_infra_kubernetes_pvc_precreate__subpaths\\\":true,\\\"che_infra_kubernetes_pvc_quantity\\\":\\\"1Gi\\\",\\\"che_infra_kubernetes_pvc_strategy\\\":\\\"unique\\\",\\\"che_jdbc_db_host\\\":\\\"postgres\\\",\\\"che_jdbc_db_name\\\":\\\"dbche\\\",\\\"che_jdbc_db_password\\\":\\\"pgchepassword\\\",\\\"che_jdbc_db_port\\\":\\\"5432\\\",\\\"che_jdbc_db_username\\\":\\\"pgche\\\",\\\"che_keycloak_admin_password\\\":\\\"admin\\\",\\\"che_keycloak_admin_username\\\":\\\"admin\\\",\\\"che_keycloak_client__id\\\":\\\"che-public\\\",\\\"che_keycloak_realm\\\":\\\"che\\\",\\\"che_log_level\\\":\\\"INFO\\\",\\\"che_predefined_stacks_reload__on__start\\\":true,\\\"che_server_deployment_stragety\\\":\\\"Recreate\\\",\\\"che_server_image_pull_policy\\\":\\\"IfNotPresent\\\",\\\"che_workspace_auto_start\\\":false},\\\"context\\\":{\\\"clusterid\\\":\\\"2fcdfdaf-9fd8-11e8-8551-0242ac110006\\\",\\\"namespace\\\":\\\"myproject\\\",\\\"platform\\\":\\\"kubernetes\\\"}}\""

I have also found this https://trello.com/c/KO5c6Ixp/345-13-37-approach-to-user-impersonation-service-account-privileges-for-apbs-and-the-broker

[ghost]

And it's not just Che APB that behaves like that but all of them?

I think we miss some important info.

[ibuziuk]

Have created an issue in ansible-service-broker - openshift/ansible-service-broker#1056
@tchughesiv @jcpowermac maybe you have some ideas ?

[ghost]

I talked to @ruromero and he seems to have an identical setup but can provision APB as a normal user.

[ibuziuk]

@eivantsov hmmm.. but for you it is still failing without cluster admin right ?

[ghost]

Yes, still fails for normal users