System Administration

by

Ansible – Protecting Sensitive Data

Ansible Automation routinely requires the use of sensitive values. You can use Ansible Vault to provide a way to encrypt and decrypt, and manage sensitive data such as passwords, certificates, keys, and API tokens

Encrypt the File

Step 1: Create the YAML file named prod.yaml

    ansible-vault create prod.yaml

    Step 2: After running the command, the terminal will prompt you to enter the password that will be used to encrypt the file

    New Vault Password: 
    Confirm New Vault Password:

    Step 3: Ansible will open up the system editor to allow you to define the value that should be encrypted. 

    Username: Secret-Information-1
    Password: Secret-Information-2

    Step 4: Upon saving the file, Ansible will encrypt its content and place the output on disk

    $ANSIBLE_VAULT;1.1;AES256
    xxxxxxxxxxxxxxxxxxxxxx
    yyyyyyyyyyyyyyyyyyyyyy

    The file is now encrypted!

    Decrypt the File

    Run the following command in the terminal to validate that the variable definition file is decrypted and values are injected into the playbook. 

    ansible-playbook -i inventory --extra-vars "@prod.yml" --ask-vau;t-pass playbook-enc.yaml

    The output of the playbook will show the content decrypted in the managed host

    References:

    • Red Hat Certified Engineer (RHCE) Ansible Automation Study Guide (Alexc Soto Bueno)
    by

    Ansible Execution Strategies

    Ansible execution is versatile enough that we can modify how and when tasks are executed. The settings can be made globally or at the play level. The number of parallel threads is determined by fork. The default is 5

    Execution StrategiesExplanationExample
    linearThis is the default. The task is executed simultaneously against all the hosts using the forks, and then the next series of hosts until the batch is done before going to the next task At ansible.cfg
    ….
    [defaults]
    strategy = linear
    fork=10

    Or at Play Level
    – name: web servers
    hosts: webservers
    strategy: linear
    debugTask execution is like the linear strategy, but controlled by an interactive debug sessionAt ansible.cfg
    ….
    [defaults]
    strategy = debug
    fork=10

    Or at Play Level
    – name: web servers
    hosts: webservers
    strategy: debug
    freeAnsible will not wait for other hosts to finish before queueing for more tasks on other hosts. It prevents blocking new tasks for hosts that have already completed.At ansible.cfg
    ….
    [defaults]
    strategy = free
    fork=10

    Or at Play Level
    – name: web servers
    hosts: webservers
    strategy: free

    Rolling Updates Strategies.

    Fork is based on the hardware limitation. The more powerful your servers are in terms of processing resources, the higher the fork parameters can be set. But there are occasions where the number of parallel executions is determined by software/application restrictions. For example, if you have a rolling updates for your webserver, you can use the “serial” parameters to instruct how many hosts should be updated at a time. In such a way, you can avoid simultaneously avoid simultaneous to avoid downtime.

    Execution StrategiesExplanationExample
    serialExecute all the hosts in the same batch before moving to the next batch

    You can use an absolute number or a percentage    		– name: Serial Example
    hosts: webservers
    serial: “50%”

    tasks:
    – name: First tasks

    References:

    • Red Hat Certified Engineer (RHCE) Ansible Automation Study Guide (Alexc Soto Bueno)
    by

    Ansible-Playbook Commonly Used Optional Arguments

    These are commonly used Ansible-Playbook Optional Arguments

    FlagDescription
    -i, –inventorySpecify inventory host path or comma-seperated host list
    -b, –becomeRun operations with become (not imply password prompting). Uses existing privilege escalation like sudo, dzdo, runas etc
    -K –ask-become-passAsk for the privilege escalation password
    –become-method <become-method>Privilege escalation method to use. Default is sudo.
    –become-password-file <BECOME_PASSWORD_FILE>Privilege escalation password file
    –become-user <BECOME_USER>Run the operation as this user. Default is root
    -f <FORKS>, –fSpecify the number of parallel processes to use. Default is 5
    -e, –extra-varsAdditional variables as key=value. When specifying a filecontaining a set of variables, prepend the file with @
    -t <Tags> –tags <TAGS>Only run plays and tasks tagged with these values
    –ask-vault-password, –ask-vault-passAsk foir vault password

    References:

    • Red Hat Certified Engineer (RHCE) Ansible Automation Study Guide (Alexc Soto Bueno)
    by

    Disabling ipv6 on Rocky Linux 8 with Ansible

    If you wish to disable ipv6 on Rocky Linux 8, there is a wonderful writeup on the script found at https://github.com/juju4/ansible-ipv6/blob/main/tasks/ipv6-disable.yml which you may find useful. If you just need to disable it temporarily without disruption (assuming you have not been using ipv6 at all)

    - name: Disable IPv6 with sysctl
  ansible.posix.sysctl:
    name: "{{ item }}"
    value: "1"
    state: "present"
    reload: "yes"
  with_items:
    - net.ipv6.conf.all.disable_ipv6
    - net.ipv6.conf.default.disable_ipv6
    - net.ipv6.conf.lo.disable_ipv6

    If you can tolerate a bit of disruption, you may want to take a look at putting it at the network configuration and restarting it

    - name: RedHat | disable ipv6 in sysconfig/network
  ansible.builtin.lineinfile:
    dest: /etc/sysconfig/network
    regexp: "^{{ item.regexp }}"
    line: "{{ item.line }}"
    mode: '0644'
    backup: true
    create: true
  with_items:
    - { regexp: 'NETWORKING_IPV6=.*', line: 'NETWORKING_IPV6=NO' }
    - { regexp: 'IPV6INIT=.*', line: 'IPV6INIT=no' }
  notify:
    - Restart network
    - Restart NetworkManager
  when: ansible_os_family == 'RedHat'
    by

    Using Ansible to get Flexlm License Information and copy to Shared File Environment

    You can use Ansible to extract Flexlm information from a remote license server, which is stored in a central place where you can display the information.

    I use crontab to extract the information every 15 min and place it in a central place so that users can check the license availability. 

    - name: Extract Information from ANSYS Lic Server and extract to file
  block:
    - name: Get FlexLM License Info
      ansible.builtin.shell: "/usr/local/ansys_inc/shared_files/licensing/linx64/lmutil lmstat -c ../license_files/ansyslmd.lic -a"
      register: lmstat_output

    - name: Save FlexLM License Output to File on ANSYS Lic Server
      copy:
        content: "{{ lmstat_output.stdout }}"
        dest: "/var/log/ansible_logs/ansys_lmstat.log"

    - name: Get FlexLM Output from Remote Server
      fetch:
        src: "/var/log/ansible_logs/ansys_lmstat.log"
        dest: "/usr/local/lic_lmstat_log/ansys_lmstat.log"
        flat: yes

    The fetch command is useful for fetching files from remote machines and storing them locally in a file tree. For more information, do take a look at Fetch files from remote nodes

    At crontab, I fetch the file every 15min

    */15 * * * * /root/ansible_cluster/run_lmstat_licsvr.sh

    The run_lmstat_licsvr.sh is simply to call the ansible playbook to run the ansible script above.

    by

    Optimizing Ansible Performance: Serial Execution

    By default, Ansible parallelises tasks on multiple hosts simultaneously and speeds up automation in large inventories. But sometimes, this is not ideal in a load-balanced environment, where upgrading the servers simultaneously may cause the loss of services. How do we use Ansible to run the updates at different times? I use the keyword “serial” before executing the roles universal package. 

    - hosts: standalone_nodes
  become: yes
  serial: 1 
  roles:
        - linux_workstation

    Alternatively, you can use percentages to indicate how many will upgrade at one time. 

    - hosts: standalone_nodes
  become: yes
  serial: 25%
  roles:
        - linux_workstation

    References:

    1. How to implement parallelism and rolling updates in Ansible
    by

    Optimizing Ansible Performance: Implementing Parallelism with Forks

    Ansible’s parallel processes are known as forks, and the default number of forks is five. In other words, Ansible attempts to run automation jobs on 5 hosts simultaneously. The more forks you set, the more resources are used on the Ansible control node.

    How do you implement? Just edit the ansible.cfg file. Look for the “forks” parameters. You can use the command “ansible-config view” to view ansible.cfg output. 

    [defaults]
    inventory = inventory
    private_key_file = ~/.ssh/xxxxxx
    become = true
    become_user = root
    timeout = 30
    forks = 10
    log_path = /var/log/ansible.log
    display_skipped_hosts=yes
    display_ok_hosts=yes
    display_failed_stderr=yes
    show_custom_stats=yes
    verbosity = 0

    References and Other Useful Information:

    1. How to implement parallelism and rolling updates in Ansible
    2. Ansible Update Management: Serial Execution and Percentage Indicators
    by

    Ansible Delayed Error Handling with Rescue Blocks: Chrony Setup Example

    A recap there are 2 main use of Blocks in Ansible. The first write-up can be found at Grouping Tasks with Block in Ansible

    1. Apply conditional logic to all the tasks within the block. In such a way, the logic only need to be declared once
    2. Apply Error handling especially when recovering from an error condition.

    Today, we will deal with Point 2 in this blog entry

    According to Ansible Documentation found at https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_blocks.html

    Rescue blocks specify tasks to run when an earlier task in a block fails. This approach is similar to exception handling in many programming languages. Ansible only runs rescue blocks after a task returns a ‘failed’ state. Bad task definitions and unreachable hosts will not trigger the rescue block.

    Here is my simple example for implementation

    - name: Check current Timezone
  command: timedatectl show --property=Timezone --value
  register: timezone_output
  changed_when: false

- name: Configure Timezone to Asia/Singapore
  command: timedatectl set-timezone Asia/Singapore
  when: timezone_output.stdout != "Asia/Singapore"

- name: Install and Configure Chrony Service Block
  block:
    - name: Install Chrony package
      dnf:
        name: chrony
        state: present

    - name: Configure Chrony servers
      lineinfile:
        path: /etc/chrony.conf
        line: "server sg.pool.ntp.org iburst"
        insertafter: '^#.*server 3.centos.pool.ntp.org iburst'
        state: present

    - name: Enable Chrony service
      service:
        name: chronyd
        state: started
        enabled: yes
  rescue:
    - name: Print when Errors
      debug:
        msg: 'Something failed at Chrony Setup'
  when:
    - ansible_os_family == "RedHat"
    - ansible_distribution_major_version == "8"
    by

    Efficient Task Grouping with Ansible: Timezone Configuration Example

    Ansible allows us to logically group a set of tasks together together, and…..

    1. Apply conditional logic to all the tasks within the block. In such a way, the logic only need to be declared once
    2. Apply Error handling especially when recovering from an error condition.

    We will deal with Point 1 in this blog entry.

    Point 1: Conditional Logic

    - name: Check current Timezone
  command: timedatectl show --property=Timezone --value
  register: timezone_output
  changed_when: false

- name: Configure Timezone to Asia/Singapore
  command: timedatectl set-timezone Asia/Singapore
  when: timezone_output.stdout != "Asia/Singapore"

- name: Install and Configure Chrony Service Block
  block:
    - name: Install Chrony package
      dnf:
        name: chrony
        state: present

    - name: Configure Chrony servers
      lineinfile:
        path: /etc/chrony.conf
        line: "server sg.pool.ntp.org iburst"
        insertafter: '^#.*server 3.centos.pool.ntp.org iburst'
        state: present

    - name: Enable Chrony service
      service:
        name: chronyd
        state: started
        enabled: yes
  when:
    - ansible_os_family == "RedHat"
    - ansible_distribution_major_version == "8"

    Reference:

    Blocks

    by

    Automating Security Patch Logs and MS-Team Notifications with Ansible on Rocky Linux 8

    If you have read the blog entry Using Ansible to automate Security Patch on Rocky Linux 8, you may want to consider capturing the logs and send notification to MS-Team if you are using that as a Communication Channel. This is a follow-up to that blog.

    Please look at Part 1: Using Ansible to automate Security Patch on Rocky Linux 8

    Writing logs (Option 1: Ansible Command used if just checking)

    Recall that in Option 1: Ansible Command used if just checking, Part 1a & Part 1b, you can consider writing to logs in /var/log/ansible_logs

    - name: Create a directory if it does not exist
      file:
        path: /var/log/ansible_logs
        state: directory
        mode: '0755'
        owner: root
      when:
        - ansible_os_family == "RedHat"
        - ansible_distribution_major_version == "8"



    - name: Copy Results to file
      ansible.builtin.copy:
        content: "{{ register_output_security.results | map(attribute='name') | list }}"
        dest: /var/log/ansible_logs/patch-list_{{ansible_date_time.date}}.log
      changed_when: false
      when:
        - ansible_os_family == "RedHat"
        - ansible_distribution_major_version == "8"

    Notification (Option 1: Ansible Command used if just checking)

    You can write to MS Team to provide a short notification to let the Engineers knows that the logs has been written to /var/log/ansible_logs

    - name: Send a notification to MS-Teams that Test Run (No Patching) is completed
      run_once: true
      uri:
        url: "https://xxxxxxx.webhook.office.com/webhookb2/xxxxxxxxxxxxxxxxxxxxxxxxx"
        method: POST
        body_format: json
        body:
          title: "Test Patch Run on {{ansible_date_time.date}}"
          text: "Test Run only. System has not been Patched Yet. Logs saved at: /var/log/ansible_logs/patch-list_{{ansible_date_time.date}}.log"
      when:
        - register_update_success is defined
        - ext_permit_flag == "no"

    Writing to MS-Team to capture the success Or failure of the Update (Option 2: Ansible Command used when ready for Patching)

    - name: Send a notification to MS-Teams Channel if Upgrade failed
      run_once: true
      uri:
        url: "https://xxxxx.webhook.office.com/webhookb2/xxxxxx"
        method: POST
        body_format: json
        body:
          title: "Patch Run on {{ansible_date_time.date}}"
          text: "Patch Update has Failed"
      when:
        - register_update_success is not defined
        - ext_permit_flag == "yes"



    - name: Send a notification to MS-Teams Channel if Upgrade failed
      run_once: true
      uri:
        url: "https://entuedu.webhook.office.com/webhookb2/xxxxxx"
        method: POST
        body_format: json
        body:
          title: "Patch Run on {{ansible_date_time.date}}"
          text: "Patch Update is Successful. Logs saved at: /var/log/ansible_logs/patch-list_{{ansible_date_time.date}}.log"
      when:
        - register_update_success is defined
        - ext_permit_flag == "yes"