Skip to content
This repository has been archived by the owner on Oct 30, 2018. It is now read-only.

Clean install issues on Ubuntu 12.04 #107

Closed
zvetaM opened this issue Apr 20, 2016 · 9 comments
Closed

Clean install issues on Ubuntu 12.04 #107

zvetaM opened this issue Apr 20, 2016 · 9 comments

Comments

@zvetaM
Copy link

zvetaM commented Apr 20, 2016

I've been following the instructions on how to install data.gov.uk on a clean Ubuntu 12.04 installation, Option 2: Fresh machine preparation, and I've had some problems:

  1. sudo adduser co -u 510 -G sudo does not work. Instead I use sudo adduser co -uid 510 and manually add the user co to the group sudo in /etc/groups

  2. the command sudo gem install librarian-puppet -v 1.0.3 fails with the error that ruby >= 1.9.3 is not installed, even though I did run the previous command sudo apt-get install ruby1.9.3 rubygems git. I decided to install ruby 2.2 using the commands

sudo apt-get install python-software-properties
sudo apt-add-repository ppa:brightbox/ruby-ng
sudo apt-get update
sudo apt-get install ruby2.2

librarian-puppet -v 1.0.3 now installed successful.

  1. When cloning the repo to the machine I used the commands
    sudo mkdir /vagrant
    sudo chown co /vagrant
    sudo chgrp co /vagrant
    ...
    git checkout master
    ln -s /vagrant/dgu-vagrant-puppet/src /vagrant/src
    ln -s /vagrant/dgu-vagrant-puppet/puppet/ /vagrant/puppet

As the commands in the instructions fail due to user co not being allowed to touch the root directory without sudo

  1. When running sudo /vagrant/puppet/install_puppet_dependancies.sh I get the following error
    Unable to load puppet, the puppet gem is required for :git and :path source.

I installed puppet with sudo gem install puppet. Now I get the following error:

/var/lib/gems/2.2.0/gems/librarian-puppet-1.0.3/lib/librarian/puppet/source/local.rb:83:in `evaluate_modulefile': uninitialized constant Puppet::ModuleTool::ModulefileReader (NameError)

In the comment rodjek/librarian-puppet#296 (comment) I learned that this seems to be a problem that can be dealt with by downgrading to puppet3

How can I downgrade to puppet3?

Thank you for your answers
davidread pushed a commit that referenced this issue Apr 20, 2016
Improvements from #107 - fixes various issues with the "Fresh machine…
3bcc6bf 
… preparation"

Also:
* updated location of CKAN repos that are cloned
* checked this all works, from "Virtual Machine creation" through CKAN install.
@davidread
Copy link
Contributor

@zvetaM Thanks for such useful feedback and suggestions!

To be honest, I wrote the 'fresh machine' instructions without trying them, so I'm very grateful for you sorting them out.

I've committed a change to address all these points: 3bcc6bf

  1. I changed -G to --group. I guess this is variation between versions of adduser.
    2 & 3 Thanks - looks good. I've pasted those in.
  2. I've added a line for installing the correct version of the puppet gem.

@zvetaM
Copy link
Author

zvetaM commented Apr 21, 2016 •
edited
Loading

Thank you. Unfortunately, following the new installation steps, things still don't work on my side.

When I run sudo puppet apply /vagrant/puppet/manifests/site.pp I get the following error:
Could not autoload package: constant Puppet::Type::Package not defined at /etc/puppet/modules/sudo/manifests/package.pp:59 on node ckan.openstacklocal and the command returns the value 1.

I tried combining different ruby and puppet versions. (ruby2.0 and ruby2.2, puppet 2.7.26, 2.7.18, 2.7.20, ...) and all fail, although all with different errors.

For example, on Puppet 2.7.18 it fails when executing sudo puppet apply /vagrant/puppet/manifests/site.pp with
/var/lib/gems/2.2.0/gems/puppet-2.7.18/lib/puppet/defaults.rb:164: warning: duplicated key at line 165 ignored: :queue_type
warning: iconv couldn't be loaded, which is required for UTF-8/UTF-16 conversions
Could not load confine test 'operatingsystem': cannot load such file -- puppet/provider/confine/operatingsystem
Could not load confine test 'operatingsystem': cannot load such file -- puppet/provider/confine/operatingsystem
Could not load confine test 'operatingsystem': cannot load such file -- puppet/provider/confine/operatingsystem
Could not load confine test 'operatingsystem': cannot load such file -- puppet/provider/confine/operatingsystem
Could not autoload package: constant Puppet::Type::Package not defined at /etc/puppet/modules/sudo/manifests/package.pp:59 on node ckan.openstacklocal

While on puppet 2.7.20 it fails at the step sudo ./install_puppet_dependancies.sh with /var/lib/gems/2.2.0/gems/puppet-2.7.20/lib/puppet/util/monkey_patches.rb:128:in `to_proc': can't modify frozen Symbol (RuntimeError)

If I run the next command source ~/ckan/bin/activate && cd /src/ckan , I get bash: /home/co/ckan/bin/activate: No such file or directory ...

@davidread
Copy link
Contributor

Here are the versions used in the vagrant route and which I've confirmed work.

$ ruby --version
ruby 1.8.7 (2012-02-08 patchlevel 358) [x86_64-linux]
$ gem list
librarian-puppet (1.0.3)
puppet (2.7.19)
...
$ puppet --version
2.7.19

If 'puppet apply' hasn't worked then you won't be able to do the following steps.

@zvetaM
Copy link
Author

zvetaM commented Apr 22, 2016 •
edited
Loading

Thank you, David!

I installed the 1.8.7 version of Ruby by first installing rvm (as user co) and then issuing the Ruby 1.8.7 install using rvm:

curl -L get.rvm.io | bash -s stable
source ~/.rvm/scripts/rvm
rvm requirements
rvm install 1.8.7

And then reinstalling puppet and librarian puppet under this version of Ruby.
=>
Librarian-puppet won't install under Ruby 1.8.7, as the newer versions of highline no longer supports it. We have to install highline version 1.6.1 before installing librarian-puppet
--> see rodjek/librarian-puppet#286

gem install highline --version 1.6.1

In order to get 'puppy apply' working, I had to

  1. Add to /vagrant/puppet/manifests/site.pp:
    group { 'puppet': ensure => 'present' }

  2. Create the symbolic link
    ln -s /vagrant/dgu-vagrant-puppet/pypi /vagrant/pypi

  3. Install PasteScript==1.7.5
    /home/co/ckan/bin/pip install --no-index --find-links=file:///vagrant/pypi PasteScript==1.7.5

  4. Install Pylons==0.9.7
    /home/co/ckan/bin/pip install --no-index --find-links=file:///vagrant/pypi Pylons==0.9.7

Whenever we want to use sudo as user co, it is better to execute
sudo -H bash
source ~/.rvm/scripts/rvm

davidread pushed a commit that referenced this issue Apr 25, 2016
Need old ruby 1.8.7 version to match what the Vagrant box installs, a…
b7151dc 
…nd what the puppet scripts are designed for. Tips from: #107 (comment)
@davidread
Copy link
Contributor

Thanks again @zvetaM

I've updated the README and site.pp to nearly all of this: b7151dc

However I'm not sure why you needed to pip install PasteScript and Pylons separately as they are in the puppet stuff already: https://github.com/datagovuk/dgu-vagrant-puppet/blob/master/puppet/modules/dgu_ckan/manifests/init.pp#L70-L72

@zvetaM
Copy link
Author

zvetaM commented Apr 25, 2016

Thank you!

Regarding Pylons and PasteScript, Puppet apply kept failing at

err: /Stage[main]/Dgu_ckan/Dgu_ckan::Pip_package[Pylons==0.9.7]/Exec[pip_install_Pylons==0.9.7]/returns: change from notrun to 0 failed: /home/co/ckan/bin/pip install --no-index --find-links=file:///vagrant/pypi --log-file /home/co/ckan/pip.log Pylons==0.9.7 returned 1 instead of one of [0] at /etc/puppet/modules/dgu_ckan/manifests/pip_package.pp:23

(and the same for PasteScript)

Previous instances of these errors I was able to resolve, but these two simply won't go away, so I decided to install them manually.

davidread pushed a commit that referenced this issue Apr 25, 2016
Tips for installing troublesome python packages, from #107 (comment)
8b86735
@davidread
Copy link
Contributor

Ok, I've added a note about these too: 8b86735

@zvetaM
Copy link
Author

zvetaM commented Apr 26, 2016 •
edited
Loading

Hy!

I also encountered some problems with Drupal;

When I executed the last step in the first block of code in 4. Drupal --> Install the DGU Drupal Distribution, drush --yes --verbose site-install dgu --db-url=mysql://co:pass@localhost/dgu --account-name=admin --account-pass=admin --site-name='Slo portal'
I received the following error:

WD cron: PDOException: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'dgu.ckan_dataset_history' doesn't exist: DELETE FROM {ckan_dataset_history}                                                         [error]
WHERE  (timestamp < :db_condition_placeholder_0) ; Array
(
    [:db_condition_placeholder_0] => 1458989203
)
 in ckan_dataset_cron() (line 348 of /var/www/drupal/dgu/profiles/dgu/modules/contrib/ckan/ckan_dataset/ckan_dataset.module).

which I ignored for the time being.

  1. But then in the step 5. Drupal content, when I wanted to change admin's password using "drush uli" (in /var/www/drupal/dgu), after entering the new password into the web gui and clicking save, I received the error:

The website encountered an unexpected error. Please try again later.

Through enabling the dblog and syslog Drupal modules I managed to get more details about the error:

$drush wd-show:
 6  26/Apr 12:57  php       error     EntityMetadataWrapperException: Unknown data property field_user_account_uid. in EntityStructureWrapper->getPropertyInfo() (line 339 of 
                                      /var/www/drupal/dgu/profiles/dgu/modules/contrib/ent                                                                                    
 5  26/Apr 12:57  user      error     EntityMetadataWrapperException: Unknown data property field_user_account_uid. in EntityStructureWrapper->getPropertyInfo() (line 339 of 
                                      /var/www/drupal/dgu/profiles/dgu/modules/contrib/ent

By the way, when I tried to enable dblog Drupal module, I received the following error. But afterwards it seemed as if dblog was activated fine anyway, since drush wd-show works fine...

$drush --yes en dblog
The following extensions will be enabled: dblog
Do you really want to continue? (y/n): y

exception 'DatabaseSchemaObjectExistsException' with message 'Table field_data_field_dataset_reference already exists.' in /var/www/drupal/dgu/includes/database/schema.inc:660                                      [error]
Stack trace:
#0 /var/www/drupal/dgu/includes/database/database.inc(2720): DatabaseSchema->createTable('field_data_fiel...', Array)
#1 /var/www/drupal/dgu/modules/field/modules/field_sql_storage/field_sql_storage.module(269): db_create_table('field_data_fiel...', Array)
#2 [internal function]: field_sql_storage_field_storage_create_field(Array)
#3 /var/www/drupal/dgu/includes/module.inc(922): call_user_func_array('field_sql_stora...', Array)
#4 /var/www/drupal/dgu/modules/field/field.crud.inc(180): module_invoke('field_sql_stora...', 'field_storage_c...', Array)
#5 /var/www/drupal/dgu/profiles/dgu/modules/contrib/features/includes/features.field.inc(268): field_create_field(Array)
#6 [internal function]: field_base_features_rebuild('dgu_data_set', 'field_base')
#7 /var/www/drupal/dgu/profiles/dgu/modules/contrib/features/features.module(562): call_user_func_array('field_base_feat...', Array)
#8 /var/www/drupal/dgu/profiles/dgu/modules/contrib/features/features.module(945): features_invoke('field_base', 'features_rebuil...', 'dgu_data_set')
#9 /var/www/drupal/dgu/profiles/dgu/modules/contrib/features/features.module(971): _features_restore('rebuild', Array)
#10 /var/www/drupal/dgu/profiles/dgu/modules/contrib/features/features.module(257): features_rebuild()
#11 [internal function]: features_flush_caches()
#12 /var/www/drupal/dgu/includes/module.inc(951): call_user_func_array('features_flush_...', Array)
#13 /var/www/drupal/dgu/includes/common.inc(7625): module_invoke_all('flush_caches')
#14 /home/co/.composer/vendor/drush/drush/commands/core/drupal/environment_7.inc(145): drupal_flush_all_caches()
#15 /home/co/.composer/vendor/drush/drush/commands/pm/pm.drush.inc(1122): drush_module_enable(Array)
#16 [internal function]: drush_pm_enable('dblog')
#17 /home/co/.composer/vendor/drush/drush/includes/command.inc(361): call_user_func_array('drush_pm_enable', Array)
#18 /home/co/.composer/vendor/drush/drush/includes/command.inc(212): _drush_invoke_hooks(Array, Array)
#19 [internal function]: drush_command('dblog')
#20 /home/co/.composer/vendor/drush/drush/includes/command.inc(180): call_user_func_array('drush_command', Array)
#21 /home/co/.composer/vendor/drush/drush/lib/Drush/Boot/BaseBoot.php(64): drush_dispatch(Array)
#22 /home/co/.composer/vendor/drush/drush/drush.php(70): Drush\Boot\BaseBoot->bootstrap_and_dispatch()
#23 /home/co/.composer/vendor/drush/drush/drush.php(11): drush_main()
#24 {main}
  1. In /var/log/ckan/ckan.log I noticed several such messages:
[Tue Apr 26 09:06:18 2016] [notice] caught SIGTERM, shutting down
[Tue Apr 26 09:07:41 2016] [notice] Apache/2.2.22 (Ubuntu) DAV/2 PHP/5.3.10-1ubuntu3.22 with Suhosin-Patch mod_wsgi/3.3 Python/2.7.3 configured -- resuming normal operations_

I also noticed few of these error messages, but couldn't find any cause for them:
_PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 64 bytes) in /var/www/drupal/dgu/profiles/dgu/modules/contrib/features/includes/features.user.inc on line 276, referer: http://172.16.95.122/admin/
  1. If I click on "modules" in the Drupal GUI, I get the following error (using drush wd-show)
52  26/Apr 13:33  php       error     DatabaseSchemaObjectExistsException: Table field_data_field_dataset_reference already exists. in DatabaseSchema->createTable() (line 660 of                                          
                                       /var/www/drupal/dgu/includes/database/schema.inc
  1. When I run Database-update in the Drupal GUI, I get the following error:
An AJAX HTTP error occurred. HTTP Result Code: 200 Debugging information follows. Path: http://172.16.95.122/update.php?op=selection&token=qDjdfHEKm1Fj3YiJhxpqs4AOhROow32nFIklNqPGw84&id=409804&op=do StatusText: OK ResponseText: Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 71 bytes) in /var/www/drupal/dgu/includes/bootstrap.inc on line 969 (napaka 8)

I hope that this isn't too much... I've followed the instructions to the letter, except in certain cases - for example, adding the admin user in CKAN:

sudo -u www-data /home/co/ckan/bin/paster --plugin=ckan sysadmin add admin --config=/var/ckan/ckan.ini

Thank you.

@davidread davidread mentioned this issue Apr 26, 2016
Open
@davidread
Copy link
Contributor

I've put this in a new issue: #108

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@davidread @zvetaM