Repo organization

This topic explains what you’ll find in the GitHub repository used by ConfigSync. See also Set up a GitHub repo for ConfigSync.

Set up a GitHub repo for ConfigSync

ConfigSync uses a GitHub repository to store the configuration files. The repository should be created independently and shared with the ConfigSync service via the Helm chart values. This document describes the process of setting up a GitHub repository for Fusion config files.

Create a new repository

Create a new repository in the github organization where the ConfigSync should store the config files.

Create a deploy key

Create a deploy key for the new repository. The deploy key should have write access to the repository. See the GitHub documentation for instructions.

Add the deploy key and repository information to the ConfigSync values

Option 1: Add the deploy key in the Helm chart

fusion-config-sync:
  # other values
  sshKey: <base64-encoded-private-key>
  cfg:
    github:
      repo: <repository-url> # use the ssh url e.g. [e-mail]:OWNER/REPO.git
      branch: <customer-branch> # you can leave it as is
      mainBranch: <main-branch> # most probably it's `main`

If outbound port 22 is blocked, you can use SSH over HTTPS with a URL like ssh://[e-mail]:443/OWNER/REPO.git. See the GitHub documentation for details. You still need a deploy key for this option.

If port 443 is also blocked, you can use HTTPS with a Personal Access Token (PAT).

To encode the private key to base64 you can use the following command:

cat ~/.ssh/id_rsa | base64

Option 2: Add the deploy key as an external secret

Alternatively, you can store the deploy key as an external secret and reference it in the ConfigSync values.

Create a secret with the deploy key:

apiVersion: v1
kind: Secret
metadata:
  name: config-sync-ssh
type: Opaque
data:
  config: SG9zdCAqCiAgU3RyaWN0SG9zdEtleUNoZWNraW5nIG5v
  id_rsa: <base64-encoded-private-key>

To encode the private key to base64 you can use the following command:

cat ~/.ssh/id_rsa | base64

Update the ConfigSync values to reference the secret with the deploy key:

fusion-config-sync:
  # other values
  sshKey: null
  additionalVolumeMounts:
    - name: config-sync-ssh-key
      mountPath: /root/.ssh
      readOnly: true
  additionalVolumes:
    - name: config-sync-ssh-key
      secret:
        secretName: config-sync-ssh
        defaultMode: 256

Additional configuration

github.* settings control the remote GitHub repo, branch, and path within the repo.

github connection properties, common for pub/sub modes

cfg:
  github:
    repo: https://github.com/lucidworks/cloud-config-sync.git # git repo, mandatory
    branch: dev # git branch, mandatory
    mainBranch: main # main branch, mandatory
    dir: "/tmp/config-sync-repo" # repo dir, default "/tmp/config-sync-repo"
    path: / # configs path in the repo dir, default "/"
    username: "fusion-config-sync" # git user name, default "fusion-config-sync"
    email: "[e-mail]" # git user email, mandatory, default "[e-mail]"

Update the ConfigSync deployment

Update the ConfigSync deployment with the new values.

Branch and Path

If you need to share the same GitHub repo for different clusters or environments, you can use the cfg.github.branch setting to specify the branch in the GitHub repo where the config files are stored.

Additionally (or alternatively) ConfigSync supports a cfg.github.path (referred to as root in the future) setting to specify the path in the GitHub repo where the config files are stored. By default, ConfigSync stores the config files in the root of the GitHub repo.

Apps and Object Types

Under the root directory, there is a directory for each app, such as app1, app2. Under each app directory, there are directories for each object type, such as query-pipelines, blobs, etc.

System Objects

ConfigSync uses _lw_system as the app name for system objects. This directory is always present in the repo and contains system objects like users, roles, solr configs, etc.

Reserved Files

Additionally, ConfigSync has set of reserved files in the root directory:

• .lock - A lock file to prevent concurrent updates
• .gitattributes - Used to list LFS files
• .gitignore - Used to ignore files that should not be stored in the repo

Important

These files should not be modified or deleted.

Example repo structure

|-- root
|   |-- _lw_system
|   |   |-- users
|   |   |   |-- user1.json
|   |   |   |-- user2.json
|   |   |-- roles
|   |   |   |-- role1.json
|   |   |   |-- role2.json
|   |   |-- solr-configs
|   |   |   |-- config1
|   |   |   |   |-- solrconfig.xml
|   |   |   |   |-- managed-schema.xml
|   |-- app1
|   |   |-- query-pipelines
|   |   |   |-- pipeline1.json
|   |   |   |-- pipeline2.json
|   |   |-- blobs
|   |   |   |-- blob1.json
|   |   |   |-- blob2.json
|   |-- app2
|   |   |-- query-pipelines
|   |   |   |-- pipeline3.json
|   |   |   |-- pipeline4.json
|   |   |-- blobs
|   |   |   |-- blob3.json
|   |   |   |-- blob4.json
|-- .lock
|-- .gitattributes
|-- .gitignore

ConfigSync uses the Fusion links API to determine app association for Fusion objects. However, it does not track links in GitHub as it uses the directory structure to maintain app association for an object.