sudo apt-get update sudo apt-get install mysql-server sudo apt-get install git gcc curl zlib1g-dev libxml2-dev imagemagick libmagickcore-6.q16-dev libmagickwand-6.q16-dev openjdk-11-jdk-headless make libsqlite3-dev g++ nodejs
# For debian: openjdk-17-jre-headless
# This is important! do NOT install the mysql client sudo apt remove libmysqlclient-dev sudo apt install libmariadbd-dev
# How to go around a proxy… # On your local machine, connect to iself… ssh -N -D 54321 localhost
# On the remote machine, forward the remote port to the local one… ssh -R9080:localhost:54321 remote-machine
# Then in the remote machine… export http_proxy=socks5://localhost:9080 export https_proxy=socks5://localhost:9080 # For culr edit ~/.curlrc proxy = socks5://localhost:9080
# From the muscat user gpg –keyserver keyserver.ubuntu.com –recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB curl -sSL get.rvm.io | bash
source ~/.rvm/scripts/rvm
# For rails 5 #rvm pkg install openssl #rvm install 2.7.5 –with-openssl-dir=$HOME/.rvm/usr #rvm use 2.7.5 #rvm gemset create rails #rvm use 2.7.5@rails
rvm install 3.2 rvm gemset create rails7 rvm use 3.2.2@rails7
git clone github.com/rism-digital/muscat cd muscat git checkout develop-rails-7 cd config cp sample_database.yml database.yml cp sample_application.rb application.rb cp muscat-custom-sample.scss ../vendor/assets/stylesheets/muscat-custom.scss
cd .. #back to muscat root bundle install EDITOR=nano rails credentials:edit #copy the empty credential block
# Create db
sudo mysql # Or log in with a user that has user creation privileges
# Create the user, substitute with an appropriate user and password!
# CREATE DATABASE muscat_development CHARACTER SET utf8 COLLATE utf8_general_ci; # CREATE USER ‘rism’@‘localhost’; # ALTER USER ‘rism’@‘localhost’ IDENTIFIED WITH mysql_native_password BY ‘password’; # GRANT ALL ON muscat_development.* TO ‘rism’@‘localhost’;
# edit config/database.yml to match # NOTE socket is /var/run/mysqld/mysqld.sock
#sudo apt remove libmysqlclient-dev #sudo apt remove libmariadbd-dev #sudo apt install libmariadbd-dev #gem install mysql2 -v ‘0.5.3’ #gem uninstall mysql2 #bundle install rake db:migrate
This document describes how to bootstrap into the webapplication. The documentation is updated for Ubuntu 18.04 LTS and Debian 10 (Buster).
The following other libraries and programs are needed
-
Ruby > 2.7
-
Apache 2.4
-
MySQL > 8.0.4 or MariaDB 10.x
-
Git
-
Passenger for Apache 2
-
rails
-
ruby-bundler
-
libmysqlclient-dev
-
imagemagick
-
libmagickwand-dev
-
Java RE 11+
NOTE From Muscat 6.1 MySQL *8.0.4 is required* for autocomplete and comments to properly work, as the REGEX library was changed. MariaDB 10.x, as provided by Debian 10 (Buster) also seems to work properly.
Depending on your system, a package for Mysql 8 may not exist. In this case use the official package provided by Oracle dev.mysql.com/downloads/repo/apt/
NOTE: Download the latest version or the GPG keys may have been expired! Copy the file from the link provided in that page, there is no need for an account, follow the “No thanks, just start my download” link. In this case it points to dev.mysql.com/get/mysql-apt-config_0.8.15-1_all.deb
wget https://dev.mysql.com/get/mysql-apt-config_0.8.15-1_all.deb sudo dpkg -i mysql-apt-config_0.8.15-1_all.deb
(obviously subsitute with the exact file name in the link). dpk will prompt a screen to configure tha package, select option to configure the server version and select 8, then exit.
Then update and install mysql:
sudo apt-get update sudo apt-get install mysql-server
When prompted, select *use legacy passwords* as not all connectors for the moment work with the new system. This will be updated in the future. Also install the newest versions of the connector libraries:
sudo apt-get install libmysqlclient21 libmysqlclient-dev
If you are using Debian 10 (Buster), the default MariaDB server (mariadb-server, that pulls 10.3) and clients (ruby-mysql2) are just fine.
Muscat 6 requires Ruby 2.3 or higher and Bundler 2.1.4 or higher. Rails (5.2) is installed with the correct version with bundle.
Recent Ubuntu releases (from 18.4 LTS) and Debian 10 (Buster) provide come Ruby 2.5 by default so no actions are necessary. On Debian Buster, Bundler 2.1.4 is provided as a backport. Versions for Passenger, Java (as required for Solr) are also fine.
sudo apt-get update sudo apt-get install mysql-server libmysqlclient-dev # If mysql was not installed separately sudo apt-get install apache2 git ruby ruby-dev gcc zlib1g-dev libxml2-dev imagemagick libmagickcore-6.q16-dev libmagickwand-6.q16-dev openjdk-11-jdk-headless passenger make libapache2-mod-passenger libsqlite3-dev g++ default-libmysqld-dev nodejs sudo a2enmod passenger
Passenger also wants to compile a binary module, this is optional but will enable it
sudo mkdir /var/www/.passenger sudo chown -R www-data:www-data /var/www/.passenger
Get the sources if necessary (github.com/rism-ch/muscat and github.com/rism-ch/muscat-guidelines):
git clone https://github.com/rism-ch/muscat.git --recursive sudo gem install bundler # We are compatibile with 2.1 now sudo bundle install #--deployment # deployment is only for the production system
Alternatively, you can just download and use the stable release tarballs found in github.com/rism-ch/muscat/releases.
Install base configuration:
cd muscat/config cp sample_database.yml database.yml cp sample_application.rb application.rb
Install the base css:
cd muscat/config cp muscat-custom-sample.scss ../vendor/assets/stylesheets/muscat-custom.scss
Make sure the directory is owned by the right user (www-data on Ubuntu or Debian)
sudo chown -R www-data:www-data muscat/
Set up databases access:
sudo mysql # Or log in with a user that has user creation privileges
Create the user, substitute with an appropriate user and password!
CREATE DATABASE muscat_development CHARACTER SET utf8 COLLATE utf8_general_ci; CREATE USER 'rism'@'localhost'; ALTER USER 'rism'@'localhost' IDENTIFIED WITH mysql_native_password BY 'password'; GRANT ALL ON muscat_development.* TO 'rism'@'localhost';
Remember username and pass should be the same as in database.yml. Then, migrate the database (that us, update the structure):
bundle exec rake db:migrate # development
NOTE: in a production environment all the tasks must declare RAILS_ENV=production to run. For local development this is not necessary
sudo RAILS_ENV=production bundle exec rake db:migrate ## on a production system
Add basic dataset:
bundle exec rake db:seed
All the secret keys are now stored in the rails 5.2 credentials file, which is not included in the repo. A blank on must be created with:
rails credentials:edit
This will automatically create credentials.yml.enc and the master key, along with a cookie secret key. If you are not familiarised with this Rails 5.2 feature, this guide is very didactic: youtu.be/BHgvPPr2nLE.
The user and password are needed for background search and export, and should be a muscat user with minimum guest access. So, to use the other services on Muscat, such as history, add to this file:
export: user: "xx" password: "xx"
Remember that they key must be readable from the user running apache, so check the permissions and change the user is needed:
chown www-data:www-data config/master.key
Muscat 7.1 now supports Solr 8.8 installed externally. This means for example that it is possible to use a Solr cluster installed on a different server. The internal Solr 5.5 server still works, but will not be supported anymore in the future since the bundled Solr unsupported and has problems with modern Java versions. The Muscat core should work with any installation of Solr, here a sample installation procedure is shown but it should be adapted for each installation.
NOTE this was tested on Linux distributions only. Grab a copy of the official distribution (8.11.1 at the time of writing) and unpack it in a suitable place
wget https://www.apache.org/dyn/closer.lua/lucene/solr/8.11.1/solr-8.11.1.tgz?action=download -O solr-8.11.1.tgz tar -xvzf solr-8.11.1.tgz
Solr needs two directories, one for data and one for the binaries. The latter is generally /opt/solr, and the data installation for this example will be /data/solr. Extract just the installation script:
tar xzf solr-8.11.1.tgz solr-8.11.1/bin/install_solr_service.sh --strip-components=2
And execute it:
sudo ./install_solr_service.sh solr-8.11.1.tgz -d /data/solr -p 8983
This will create the directory layout as above. It will also create a solr user. For the complete reference see solr.apache.org/guide/8_8/taking-solr-to-production.html Now copy over the Muscat core:
cp -R $MUSCAT_HOME/solr-configuration/muscat /data/solr/data cp $MUSCAT_HOME/solr-configuration/jar-8.8-linux/ThemaxQuery-1.0-SNAPSHOT.jar /data/solr/data/lib/
Where /data/solr/ in the directory specified above with -d. Solr includes a startup script so
systemctl daemon-reload service solr start
Should start it. The logfiles in this case are found in /data/solr/logs.
Grab a copy of the official distribution (8.11.1 at the time of writing) and unpack it in a suitable place
wget https://www.apache.org/dyn/closer.lua/lucene/solr/8.11.1/solr-8.11.1.tgz?action=download -O solr-8.11.1.tgz tar -xvzf solr-8.11.1.tgz
When using this distribution, you can copy the muscat core bundled with Muscat directly into the binary distibution
cp -R $MUSCAT_HOME/solr-configuration/muscat solr-8.11.1/server/solr cp $MUSCAT_HOME/solr-configuration/jar-8.8-linux/ThemaxQuery-1.0-SNAPSHOT.jar solr-8.11.1/server/solr/lib/
NOTE The bundled ThemaxQuery-1.0-SNAPSHOT.jar contains the PAE incipit indexer used in Muscat. A version for macOS 11 is provided in jar-8.8-mac, the linux version should work with any recent Ubuntu/Debian system. In muscat/solrconfig.xml the path for this jar is hardcoded <lib dir=“../../jar”/> so it will find it in the stock Solr installation. For a different installation the path should made to point to the correct jar file path. Solr should run at this point:
sudo -u www-data ./bin/solr start -p 8983
Or an appropriate startup script may be used
Install solr with brew, use solr-8.11.1 at least
brew install solr
As above, copy the configuration and jar file:
cp -R $MUSCAT_HOME/solr-configuration/muscat /usr/local/var/lib/solr cd /usr/local/Cellar/solr/8.11.1/contrib mkdir themax cp $MUSCAT_HOME/solr-configuration/jar-8.8-mac/ThemaxQuery-1.0-SNAPSHOT.jar /usr/local/Cellar/solr/8.11.1/contrib/themax
NOTE depending on the Homebrew version used, the path for the configuration might be /opt/homebrew/var/lib/solr
Edit /usr/local/var/lib/solr/muscat-index/solrconfig.xml To point to the correct jar. Remove <lib dir=“../../jar”/> and add:
<lib dir="${solr.install.dir:../../../..}/contrib/themax" regex=".*\.jar"/>
to the list of lib directories at the top. This will tell Solr to load the jar files in the contrib/themax directory. Restart Solr:
brew services restart solr
By default Solr listens on port 8983 with a core named muscat. To change this edit the files
config/sunspot.yml
To change the core name or port
# Make sure solr is not running as root if used in production sudo -u www-data bundle exec rake RAILS_ENV=production sunspot:solr:start # else just do: bundle exec rake sunspot:solr:start
NOTE sunspot.yml and blacklight.yml now point to the muscat core as used for 8.8. To run with 5.5 and the default configuration, edit those files to use
-
production: port
8983coreproduction -
development: port
8982coredevelopment
Make sure Solr and Mysql are running. Default (development) startup:
rails s -e development
Try to open $IP_ADDRESS:3000/
For startup in production mode see “Starting Daemons in Production Mode”
For refreshing an installation in production mode (with sudo, as root or sudo -u www-data depending on the permissions on the installation dir):
bundle exec rake RAILS_ENV=production assets:clean bundle exec rake RAILS_ENV=production assets:precompile
A default administrative user has been created as part of the installation process. To log in, go to http://ip:3000/admin and log in with the following credentials:
username: [email protected] password: Password1234
It is advised that you delete this account after creating a new administrative user in the admin interface.
rake sunspot:reindex
In production mode run:
RAILS_ENV=production rake sunspot:reindex
Specify only a model:
rake sunspot:reindex[,Person]
Do reindex in 1 record batches, useful if reindex crashes to see in which one (very slow to start)
rake sunspot:reindex[1]
It is possible to run muscat with rvm/rbenv, passanger standalone and nginx as a fronted.
First, create a dedicated user for Muscat:
sudo adduser muscat
Make then sure the user can not login via ssh. Add it temporarly to sudoers for the next step.
Install rvm following the instructions here: rvm.io/rvm/install. If you are behind a proxy you need to edit .curlrc for muscat user and add:
proxy = the-proxy.com:1234
Then proceed with the installation and add the needed ruby version (here 2.7.5)
rvm install 3.2.2 rvm gemset create rails7 rvm use 3.2.2 rvm --default use 3.2.2@rails7
You can disable muscat from sudoers after this step. Make sure the gem form passenger is in the Gemfile:
gem 'passenger'
Edit ‘/etc/systemd/system/muscat.service` for the application startup
[Unit] Description=Muscat After=network-online.target [Service] Restart=always RestartSec=5 TimeoutSec=5 User=muscat Group=muscat WorkingDirectory=/INSTALLATION/DIR/muscat ExecStart=/home/muscat/.rvm/bin/rvm 3.2.2@rails do bundle exec passenger start [Install] WantedBy=multi-user.target
Change ‘/INSTALLATION/DIR/` to the actual installation dir. Then Edit `Passengerfile.json` for the basic configuration
{
"environment": "production",
"port": 8081,
// Passenger is managed by systemd
"daemonize": false,
}
here is a sample nginx configuration:
# These are some "magic" Nginx configuration options that aid in making
# WebSockets work properly with Passenger Standalone. Please learn more
# at http://nginx.org/en/docs/http/websocket.html
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
// Same as Passenger.json
upstream muscat {
server localhost:8081;
}
# Virtual host configuration for muscat.
server {
server_name www.muscat.com;
root /var/www/muscat-dir-for-nginx;
listen 443 ssl http2;
listen [::]:443 ssl http2;
keepalive_timeout 70;
ssl_certificate /CERTS/fullchain.pem;
ssl_certificate_key /CERTS/privkey.pem;
include ssl_include;
# ignore all location requests that start with a dot.
# Return 404 to make it look like it's not found (otherwise
# a 403 would say it is there, you just can't access it)
location ~ /\. {
deny all;
access_log off;
return 404;
}
if (-f /etc/nginx/muscat-test.unavailable) {
return 503;
}
error_page 503 @custom503;
location @custom503 {
root /var/www/muscat-dir-for-nginx/www;
rewrite ^(.*)$ /nginx_errors/503.html break;
internal;
}
location / {
proxy_pass http://muscat;
proxy_http_version 1.1;
proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Scheme $scheme;
proxy_buffering off;
}
}
server {
server_name muscat.com;
listen 80;
listen [::]:80;
# Since we're just redirecting from this configuration, we don't need to count it in the
# access logs.
access_log off;
# Ensure any challenges from Let's Encrypt are served.
# directory on the server for this.
location ~ /\.well-known/acme-challenge {
root /var/www/muscat-dir-for-nginx/www;
}
# Redirect all other traffic to the muscat instance.
location / {
return 301 https://muscat.com/$request_uri;
}
}
Change ‘muscat.com` to the actual address, and create muscat-dir-for-nginx and muscat-dir-for-nginx/www Create the file `/etc/nginx/ssl_include` for the SSL configuration:
add_header Strict-Transport-Security "max-age=63072000"; ssl_stapling on; ssl_stapling_verify on; ssl_session_tickets on; ssl_session_timeout 24h; ssl_session_cache shared:SSL:100m; ssl_session_ticket_key /etc/nginx/ssl/ticket.key; ssl_dhparam /etc/nginx/ssl/dhparam.pem; ssl_prefer_server_ciphers on; ssl_protocols TLSv1.2 TLSv1.3; ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384; resolver 8.8.8.8 [2001:4860:4860::8888] 1.1.1.1 [2606:4700:4700::1111] 8.8.4.4 1.0.0.1;
DelayedJob now is a bit more tricky, as different queues get different worker pools:
sudo -u www-data RAILS_ENV=production bin/delayed_job start --pool=reindex,triggers,folders:10 --pool=sub_reindex:10 --pool=resave --pool=export
Other ways to start delayed_job:
sudo RAILS_ENV=production bin/delayed_job start rake jobs:work # Run in foreground rake jobs:workoff # Run all jobs in foreground and exit rake jobs:clear # Clear failed jobs ## in production it is then... RAILS_ENV=production bundle exec rake jobs:<xxx>
Depending on the env, remove sudo if necessary. RAILS_ENV is needed only on production, and has to come after sudo.
It is also good to block access to Solr:
sudo iptables -A INPUT -p tcp -s localhost --dport 8983 -j ACCEPT sudo iptables -A INPUT -p tcp --dport 8983 -j DROP
In version 7.0 the crono scheduler was removed and a normal crontab plus a script is provided. The default crontab file should be linked in /etc/conn.d. Edit the muscat_crontab to set PATH_TO to the correct muscat installation, you will need to do it by hand.
On debian/ubuntu this should work:
ln -s config/muscat_crontab /etc/cron.d chmod +x /etc/cron/muscat_crontab sudo service cron restart
If your crontab still has this line:
0 2 * * * cd $PATH_TO && $PATH_TO/bin/rake blacklight:delete_old_searches[7] RAILS_ENV=production
It is save to delete it as it is now taken care but the global muscat crontab. NOTE If you ever change the cron file name, it CANNOT contains dots! Or it will silently fail.
Add to /etc/mysql.cnf
[mysqld] innodb_buffer_pool_size = 8G innodb_log_buffer_size = 512M innodb_log_file_size = 1G innodb_write_io_threads = 16 innodb_flush_log_at_trx_commit = 0
To speed up imports. (See here for more)
On a full muscat DB there can be many many _old versions_ in the versions table, since version snapshots are kept for each saved item. When doing development this can be quite annoying since it can take up to 30 minutes to restore a db.
sed '/INSERT INTO `versions`/d' muscat_dump.sql > muscat_no_versions.sql
Removes all the old versions. From 30 minutes to 9.
This concerns really only the internal use of the source validator job, which can validate al the sources periodically. Often it is useful to add blanket exclusions if some data will not be corrected. They must be placed in the folder ‘validation_exclusions` in config/, with subfolders containing the exclusions for each model, for example:
config/validation_exclusions/source/exclusions.yml
is the configuration file for sources. This is a sample configuration
exclude:
"650":
tags:
"0":
from_file_list: 650_0_exclude
"588":
tags:
"a":
and_rules:
id_prefix: "234"
creation_date: "2023-06-06"
Every tag specifies some rules for the subtags to be excluded:
-
‘from_file_list`: read ids to be excluded from a file, one by line
-
‘id_prefix`: ignore records with the id starting with these digits
-
‘creation_date`: ignore records created exactly on this date
Other rules will be implemented in the future.
No rules are provided by default in muscat and none should be pushed to the repo since the use depends on the user data.
On older Ubuntu systems add the following repositories
deb http://ppa.launchpad.net/brightbox/ruby-ng/ubuntu trusty main deb https://oss-binaries.phusionpassenger.com/apt/passenger trusty main
(On some systems it is necessary to uncomment therubyracer from the gemfile) On Ubuntu 18.04 only passenger is needed:
deb https://oss-binaries.phusionpassenger.com/apt/passenger bionic main
(Add /etc/apt/sources.list or create a relevant file in /etc/apt/sources.list.d) Also make sure that universe repositories are enabled!
On 18.40, also enable GPG verification of Passenger:
sudo apt-get install -y dirmngr gnupg sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 561F9B9CAC40B2F7 sudo apt-get install -y apt-transport-https ca-certificates
Then install all the dependencies and setup passenger:
In Muscat’s default configuration, users are created and authenticated in Muscat’s own local database. This authentication method is called ‘:database_authenticatable`, and if it is fine to you, there is no need to modify nor configure anything further.
There is an alternative Single Sign On (SSO) authentication for corporate environments, and it is implemented using SAML integration. If you are interested, please follow this steps as described:
-
First, run the generator with ‘bin/rails g muscat:install_saml` to get the required configuration files.
-
Add :saml_authenticatable as one of the AUTHENTICATION_METHODS in ‘config/application.rb` (order of methods is not important)
-
SAML users will have the role defined by ‘RISM::SAML_AUTHENTICATION_CREATE_USER_ROLE` which is “guest” by default. Edit `config/application.rb` and uncomment `RISM::SAML_AUTHENTICATION_CREATE_USER_ROLE` and set the desired role if you prefer another one.
-
Customize the new devise login form that the generator created at ‘“app/views/active_admin/devise/sessions/new.html.erb”`. At least, edit the file and remove the unused link to the IdP or define yours if it’s none of the two.
-
Configure the SAML settings at ‘config/initializers/devise_saml_authenticatable.rb`
- To automatically parse IdP's metadata, parse the remote url just before ruby_saml's settings ``` ... idp_metadata_parser = OneLogin::RubySaml::IdpMetadataParser.new config.saml_config= idp_metadata_parser.parse_remote "https://remote-sso-server.example.org/saml/idp/metadata" config.saml_configure do |settings| ... ```
-
Configure the mapping between SAML user attributes (keys) and application user attributes (values).
-
Muscat must be restarted if any configuration is changed.
The SSO authentication method is added as convenience to Muscat to serve some external users, but it is not supported by upstream developers. If you need support, please contact its authors: coditramuntana.com/en/contact .
Example Apache configuration:
# create /usr/local/etc/apache/Includes/default.conf
# and add default site:
<VirtualHost *:80>
# ServerName www..org
DocumentRoot /var/rails/rism-ch/public
<Directory "/var/rails/rism-ch/public">
Options All -Indexes +ExecCGI -Includes +MultiViewsny
Require all granted
</Directory>
RailsEnv production
</VirtualHost>
Double check permissions in the muscat installation. Also make sure DocumentRoot and Directory point to public/ in the muscat installation location.
Start Apache and the related services in production mode (see below).
Alternatively, you can take advantatge of the Apache itk module, that simplifies not only having more than one Muscat in a single server using virtual hosts, but also the whole file permissions and ownership altogether. This Apache module (mpm-itk.sesse.net/) makes the Apache web server to run as a plain user, so it is the owner of the files, and it is no longer necessary file ownership via chown or users via sudo. Apache-itk is an official Debian and Ubuntu package, so:
apt install libapache2-mpm-itk
And add the following stanza in the apache configuration file, for example just before the DocumentRoot declaration:
<IfModule mpm_itk_module> AssignUserId muscat_linux_user_name muscat_linux_group </IfModule>
where muscat_linux_user_name and muscat_linux_group are the first two fields of the output of the id shell command.