[WIP] Add kubernetes manifest files

[eggshell]

This will add a directory named deployment which contains all of the
necessary kubernetes manifest files to deploy onto a user's
managed kubernetes platform of choice. These have been written to
be platform-agnostic, but have not yet been tested for use with
Minikube.

Documentation will need to be added, but I want to get opinions
and have discussions with the geonode community before moving
further. I have a running deployment using these configs here:

https://eggshell.us-south.containers.appdomain.cloud/

Signed-off-by: Cullen Taylor [email protected]

[eggshell]

I had to make some minor modifications to a few of the docker images in order to get this to work. I have opened pull requests in those repos to be thorough:

GeoNode/geonode-docker#13
GeoNode/geoserver-docker#43

[afabiani]

@eggshell nice, very interesting.

Can you please create an issue with label "feature" associated to this PR describing better what this PR does?

Also, if feasible, would be cool to think to some integration testing. Something similar to what we do on .travis.yml when starting the docker-compose.
(not an expert of kubernetes, so just guessing here)

[codecov]

Codecov Report

Merging #3911 into master will decrease coverage by 7.75%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3911      +/-   ##
==========================================
- Coverage   54.17%   46.42%   -7.76%     
==========================================
  Files         235      266      +31     
  Lines       15224    18362    +3138     
  Branches     2270     2779     +509     
==========================================
+ Hits         8248     8524     +276     
- Misses       6241     9199    +2958     
+ Partials      735      639      -96

Impacted Files	Coverage Δ
geonode/geoserver/ows.py	25.67% <0%> (-58.11%)	⬇️
geonode/security/utils.py	34.2% <0%> (-38.97%)	⬇️
geonode/geoserver/upload.py	8.45% <0%> (-36.45%)	⬇️
geonode/geoserver/helpers.py	22.65% <0%> (-17.84%)	⬇️
geonode/social/signals.py	57.03% <0%> (-17.61%)	⬇️
geonode/messaging/consumer.py	37.31% <0%> (-12.69%)	⬇️
geonode/geoserver/views.py	41.44% <0%> (-10.26%)	⬇️
geonode/proxy/views.py	35.92% <0%> (-10.24%)	⬇️
geonode/base/templatetags/base_tags.py	30% <0%> (-7.65%)	⬇️
geonode/messaging/producer.py	55.55% <0%> (-7.41%)	⬇️
... and 66 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cc94e82...6bcd1d7. Read the comment docs.

[eggshell]

@afabiani thanks for your time! I will work on a writeup and post the issue here once it is submitted.

re: integration testing. I haven't personally done a kube deploy in travis, but it seems like Minikube in Travis is the way forward on that. This is nice because I want people to be able to deploy to their local minikube cluster if they don't have PaaS resources, so I'll investigate that further.

[francbartoli]

@eggshell This is great! Thanks for opening it. I will definitively have a try with my local minikube. Btw I have just tried to load a common shapefile on your public instance and it fails but we can investigate further when I have it deployed locally. Maybe is it a matter of the resources given to the k8s pod?
Out of curiosity is that running on IBM Cloud kubernetes cluster?

Again I would move the folder deployment under a new folder scripts/kubernetes for now.

[francbartoli]

Move the folder under a new one scripts/kubernetes

[eggshell]

@francbartoli Thanks for the feedback and functional testing! I debugged some today and fixed the issue with uploading shapefiles where you'd get a 413 from nginx. The problem there was that there was that the kubernetes ingress controller (which sits in front of geonode's static nginx) had its client_max_body_size set too low, so I went ahead and upped that to 100M to match geonode's config. Still trying to figure out a way to make it persist as I manually fixed it and reloaded the ingress controller's nginx config. Will update my commit as soon as I have that hammered out.

However, shapefile uploading is now hindered by a new error:

Unable to obtain connection: Cannot create PoolableConnectionFactory (Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.)

Which seems to be tomcat/catalina related. It would seem these are not pulling in environment variables to check where the postgres service actually is, but rather assuming it is living locally, which I definitely want to help work around, but am a little lost on how to fix at the moment. If you have any ideas of where this lingering config might be, let me know!

To answer your other question, this is indeed running on an IBM Cloud kubernetes cluster. I do want to make this is as platform-agnostic as possible, so will be testing with other major providers hopefully. And of course, I do want to get this running in minikube so others can play with it without the need to pay money for a hosted cluster.

Will also update my commit to reflect your requested filepath change. Thanks again!

[francbartoli]

Which seems to be tomcat/catalina related. It would seem these are not pulling in environment variables to check where the postgres service actually is, but rather assuming it is living locally, which I definitely want to help work around, but am a little lost on how to fix at the moment. If you have any ideas of where this lingering config might be, let me know!

I have added a couple of comments, I don't know why but it seems to be that the variable GEODATABASE_URL has been removed from .env of production. This would be a bug because in this case the django container is not connecting to the postgres service name. Could you open a ticket in case my comments will solve the issue?

[eggshell]

@francbartoli so I went ahead and added the GEODATABASE_URL environment variable to the celery and django configs, as well as to the geoserver config just to be sure, and it is getting injected into the container successfully. However, something appears to still be overwriting it back to localhost:5432. Fixing this required logging into the geoserver webui and changing the datastore properties.

This seems to have propagated out to /geoserver_data/data/workspaces/geonode/geonode_data/datastore.xml, and uploading is now possible. I am wondering if there is some sort of timing issue where GEODATABASE_URL is not getting set before settings.py runs and parses it?

[francbartoli]

@eggshell I’ll look into this, just to make sure that we are not getting into a false problem: did you test from scratch after the removal of all previous docker volumes?

[francbartoli]

@eggshell I'm currently trying to run the deployment on minikube. I'm assuming the commands to execute are:

kubectl apply -f scripts/kubernetes/*.yaml

for each yaml file. Is that correct? Does the order matter? Anything else? Thanks for providing some guidance

[eggshell]

@francbartoli Looks like it was a false problem. I had the /geoserver_data dir mounted to an nfs, so switching to a statically provisioned volume on the workers fixed it. The problem with the ingress not setting nginx's client_max_body_size has also been corrected. I pushed both of those changes yesterday.

You have the command to deploy down correctly. The ingress definition in geonode-deployment.yaml needs to be tweaked in order to work on minikube. I have been looking into it this morning using this blog post. Working on a writeup for the whole PR and some rudimentary docs today.

[francbartoli]

@eggshell sweet, that definitively would help!

[eggshell]

@francbartoli @afabiani I opened #3924 as requested.

[afabiani]

@eggshell very nice, good job. Waiting for @francbartoli feedbacks.

@olivierdalang might also provide some feedbacks on this.

[francbartoli]

@eggshell at your earliest convenience you can add a k8s docs part and then we can start together to looking at a circleCI setup

[afabiani]

@eggshell @francbartoli is this PR finished? Any other request?

[francbartoli]

@afabiani I would wait for some docs as we don't have any urgency with this. What do you think?

[afabiani]

@francbartoli sure, absolutely agree. Just wanted to be sure this wasn't stalled.

[eggshell]

@afabiani @francbartoli agree on docs, I am on some planes today but will throw them up asap. a few minor changes will be incoming as well but this should be wrapping up soon. Thank you again for your time!

[francbartoli]

@eggshell did you have time to add some docs?

[gannebamm]

@eggshell did you have time to add some docs?

Just out of curiosity: What is the current status of this PR? Are these (https://github.com/eggshell/geonode-on-k8s) the mentioned docs?

[giohappy]

@francbartoli I think these types of contributions can be accepted if active maintenance is guaranteed. @eggshell seems to have abandoned this PR so I would be in favour of closing it.

[francbartoli]

@giohappy we can close it now and as soon as I have time I will be happy to add:
[ ] remove dind
[ ] docs
[ ] deploy on minikube

and then volunteer to maintain it