.. _ref_filehooks:

FileHooks
=========

The fileHooks project (https://sharing-codeability.uibk.ac.at/development/sharing/file-hooks) is a simple infrastructure for forwarding 
events from GitLab to the GitSearch REST service at http://sharing_search:8080/api/gitlab/eventListener.

The services GitLab and Elasticsearch are considered backend services.
It is responsible for the data collection and preparation.
This section describes the fileHooks used in GitLab and the infrastructure setup.
Finally, some tips to handle errors are provided.




Infrastructure Setup
--------------------

It is currently assumed that all services run on the same host as separate docker containers.
The setup of the containers and the host server is discussed in the following.
Lastly, the manual installation procedure for file hooks is given as a reference.


Container Setup
~~~~~~~~~~~~~~~

Subsequently, the setup for GitLab, PlantUml Elasticsearch is shown.

The setup of the Services GitLab search and MySQL are discussed in the section :ref:`ref_git_search`.

To create all containers for the backend in production, a docker-compose script is provided in ``src/main/docker/gitlab-setup/``.

It can be executed as follows:
  - ``cd src/main/docker/gitlab-setup/``
  - ``. .env`` the .env file can be found in the KeePass file. It contains the secrets for the containers. (Remark: the entry for the FILEHOOKS_CONFIG_FILE is missing.)
  - ``docker-compose create``
  - ``docker-compose start``    

The following environment variables are set within the config files.
No modification should be required for those if the correct config file is used.

- ``GITLAB_HOME``: Directory where data generated by GitLab is persisted
- ``EXTERNAL_URL``: External Url of the GitLab instance
- ``GITLAB_HOSTNAME``: Hostname of GitLab
- ``ES_HOME``: Directory where data generated by Elasticsearch is persisted
- ``FILEHOOKS_CONFIG_FILE``: Name of the file in ``filehooks/conf/`` used to configure the filehooks code

+----------------------+----------------------------------------+------------------------------------------------+
| Environment variable | Production                             | Development                                    |
+======================+========================================+================================================+
| GITLAB_HOME          | /mnt/qt-sharing-codeability/srv/gitlab | /mnt/qt-codeability-austria/sharing/srv/gitlab |
+----------------------+----------------------------------------+------------------------------------------------+
| ES_HOME              | /mnt/qt-sharing-codeability/es         | /mnt/qt-codeability-austria/sharing/es         |
+----------------------+----------------------------------------+------------------------------------------------+
| EXTERNAL_URL         | https://sharing-codeability.uibk.ac.at | https://sharing.codeability-austria.uibk.ac.at |
+----------------------+----------------------------------------+------------------------------------------------+
| GITLAB_HOSTNAME      | sharing-codeability.uibk.ac.at         | sharing.codeability-austria.uibk.ac.at         |
+----------------------+----------------------------------------+------------------------------------------------+
| GITLAB_HOSTNAME      | conf/production.ini                    | conf/staging.ini                               |
+----------------------+----------------------------------------+------------------------------------------------+
| FILEHOOKS_CONFIG_FILE| TODO                                   | TODO                               |
+----------------------+----------------------------------------+------------------------------------------------+

.. note::

    If the container is set up from scratch (there are no persisted data available), a password for the user ``root`` has to be specified using the web interface.
    For the development and production server, this password should be added to KeePass.
    Alternatively, the password can also be set directly in the GitLab container:

    ``docker exec -it sharing_gitlab gitlab-rake 'gitlab:password:reset[root]'``

Installing the Filehooks package
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In the previous section, the container infrastructure is set up.
When this is successfully done, the filehooks code needs to be installed in the GitLab container.
There is a script in the ``setup`` directory of the file-hooks project for this job:

.. code-block::

   ./install_filehooks_locally.sh

This script copies files from the repository into the GitLab container
and sets up the code such that it is run whenever GitLab emits an event.


The GitLab container can be accessed interactively by running

.. code-block::

    docker exec -it sharing_gitlab /bin/bash

Server Setup
~~~~~~~~~~~~

To make GitLab reachable from outside, conduct the following steps after connecting to the server via ``ssh``:

1. Add the following snippet to ``/etc/apache2/sites-enabled/default-ssl.conf``:

   .. code-block::

       # Sharing
       <VirtualHost *:443>
           SSLProxyEngine On
           AllowEncodedSlashes NoDecode
           SSLProxyVerify none
           SSLProxyCheckPeerCN off
           SSLProxyCheckPeerName off
           SSLProxyCheckPeerExpire off

           ##### Portainer #####

           RewriteRule ^/portainer$ /portainer/ [R,L]
           ProxyPass /portainer/ http://sharing-codeability.uibk.ac.at:9000/
           ProxyPassReverse /portainer/ http://sharing-codeability.uibk.ac.at:9000/
           ProxyPass /portainer/api/websocket/ http://sharing-codeability.uibk.ac.at:9000/api/websocket/
           ProxyPassReverse /portainer/api/websocket/ http://sharing-codeability.uibk.ac.at:9000/api/websocket/

           #####################
           ProxyPass / https://sharing-codeability.uibk.ac.at:10083/ nocanon
           ProxyPassReverse / https://sharing-codeability.uibk.ac.at:10083/


           ErrorLog ${APACHE_LOG_DIR}/error.sharing-codeability.uibk.ac.at.log
           CustomLog ${APACHE_LOG_DIR}/access.sharing-codeability.uibk.ac.at.log combined

           # Michael further tools settings
           Include /etc/apache2/codeAbility/sharing/*.conf
       </VirtualHost>

.. note::

    Please review the configuration above carefully. Gitlab is very sensitive, when run behind a reverse proxy!

2. Add the following snippet to ``/etc/apache2/sites-enabled/000-default.conf``:

   .. code-block::

       <VirtualHost *:80>
           # ...
           Redirect / https://sharing-codeability.uibk.ac.at
       </VirtualHost>

3. ``sudo systemctl restart apache2``


Afterward, the containers can be started:

1. Navigate to ``/mnt/qt-sharing-codeability/file-hooks`` (prod).
2. Pull this repository for the latest updates

   - It could be necessary to ``git reset --hard`` the repository before, because some scripts replace confidential variables during installation.
3. Navigate to ``setup`` and set up the containers (see above)


Manual File Hook Setup
~~~~~~~~~~~~~~~~~~~~~~

.. note::

    Just for reference.

As a reference on how to add other file hooks to GitLab, the steps to install the file hook ``trigger_project_update.py`` are given below:

1. Install python requirements:

   .. code-block:: bash

       pip3 install --upgrade setuptools
       pip3 install -r requirements.txt

2. Create API-Token with the user ``root`` and the scopes ``api``, ``read_api``, ``read_repository``
3. Add the API-Token in ``conf.production.ini`` (section ``gitlab``, key ``token``)
4. Install the ``filehooks`` package

   .. code-block:: bash

       pip3 install .

5. Install java

   .. code-block:: bash

       apt-get install openjdk-8-jdk

6. Initialize the indices in elasticsearch using the script ``create_index.py``. This script can be deleted once the indexes were created successfully.

   .. code-block:: bash

       python3 create_index.py

7. Add the file ``trigger_project_update.py`` to ``/opt/gitlab/embedded/service/gitlab-rails/file_hooks`` in the GitLab container
8. Ensure that the script ``trigger_project_update.py`` has the permissions ``755``
9. Validate the file hooks:

   .. code-block:: bash

       gitlab-rake file_hooks:validate


Infrastructure Update
---------------------

Subsequently, a guide for updating GitLab and the filehooks is provided.

Update Guide GitLab
~~~~~~~~~~~~~~~~~~~

1. Navigate to the root directory of filehooks repository.
2. Create a backup of GitLab with the script ``setup/backup_sharing_gitlab.sh``
3. Navigate to the parent directory of ``GITLAB_HOME`` and copy the mounted volume, e.g.,

   .. code-block::

       cp -a srv srv_2021_01_31

4. Change the GitLab version in the file ``setup/sendmail/Dockerfile``.
5. Run the script ``setup/setup-infrastructure.sh``
6. Wait & check if GitLab starts successfully. Usually it takes about 10 minutes until GitLab reaches the status ``healthy``. It may very well be that the status is ``unhealthy`` for a period of time along the way.
7. Install the filehooks in GitLab using the script ``setup/install_filehooks_locally.sh``.
8. Check if the filehooks work properly.

.. note::

    When upgrading the GitLab version, follow the `upgrade recommendations <https://docs.gitlab.com/ee/policy/maintenance.html#upgrade-recommendations>`_ from GitLab.



Update Guide Filehooks
~~~~~~~~~~~~~~~~~~~~~~

1. Check out the version of the code which should be deployed somewhere in the file system.

2. Run ``setup/install_filehooks_locally.sh``.

Errors
------

In case a container crashes, it should start automatically.
Consequently, it should not be necessary to start any container manually after the setup was executed successfully.

.. warning::

    If the GitLab container crashes, the python-package ``filehooks`` is not re-installed automatically. (TODO check whether this is still true)
    Hence, new or changed files will not be added to elastic search.
    You have to install the filehooks (see update guide filehooks)! And do a complete reindexing, to ensure a consistent index.

Subsequently, the logging systems for GitLab and FileHooks are discussed.


GitLab
~~~~~~

GitLab has an advanced logging system distributed over many log files. Details can be found in the `GitLab documentation <https://docs.gitlab.com/ee/administration/logs.html>`_.
For example, the command ``docker logs -f -n 10 sharing_gitlab`` can be used to inspect the logs.

FileHooks
~~~~~~~~~

- ``/var/log/gitlab/gitlab-rails/file_hook.log``: Fatal errors (e.g., unexpected exceptions) are logged in this file.
- ``/var/log/gitlab/gitlab-rails/trigger_project_update.log``: General logging information for the fileHook ``trigger_project_update.py`` are logged in this file.

GitSearch Indexer
=================

.. _ref_gitsearch_indexer:

The GitSearch Indexer listens to requests via the REST service at http://sharing_search:8080/api/gitlab/eventListener.
It is responsible for validating and updating the Elasticsearch index.


This GitSearch Indexer does two tasks:

1. Health check and validation: It informs the user who modified the project via email if the metadata information is incomplete or invalid after a modification in a repository was conducted.
Validation happens on the ``master``-branch of all projects in the group ``sharing``. It also checks projects in all other groups, however if they do not contain meta data, the check is skipped.  

The indexer will mainly be triggered by push events, but also by moving or renaming a project and or groups/namespaces.
The check proceeds as follows:
First, the root directory of the repository is checked for files named ``metadata.json``, ``metadata.yaml``, or ``metadata.yml``.
There must be exactly one such file, otherwise the check fails.
Subsequently, the correctness of all metadata files is validated (also dependent metadata files, if it is a collection).
If an error occurred, an email is sent to the user who pushed the changes.

Meta data checks comprise:
 - the syntactical correctness of the metadata file as yaml or json file (results in an error)
 - the presence of the required fields (results in an error)
 - the presence of the required fields in the dependent metadata files (results in an error)
 - checks against the vocabulary service at https://oeresource.logic.at/en/meta/api/v1?format=json  (results in a warning)

The check fails if there is an error, but is accepted if there are only warnings. In both cases the author is informed by e-mail.

2. It keeps the Elasticsearch index up-to-date by adding/updating/deleting files according to the triggered GitLab event.
Only the ``main``-branch (or ``master`` if ``main`` does not exist) and the group ``sharing`` (including subgroups and all subprojects) are indexed in Elasticsearch.
Metadata files (``metadata.json``, ``metadata.yaml``, or ``metadata.yml``) at the project root are indexed in the alias ``metadata``.

Finally, the GitSearch Indexer provides functionality, to recreate the index and to recheck all projects. During this task all event-processing is paused.