Building the Future Today with HPC

At SC20, Intel’s Trish Damkroger, vice president and general manager of HPC at Intel, shows how Intel and its partners are building the future of HPC today through hardware and software technologies that accelerate the broad deployment of advanced HPC systems. (Credit: Intel Corporation)

Using Intel Cluster Checker (Part 3)

Framework Definition (FWD) Selection and Definition

you wish to select a framework definitions. You can do it by using the command

% clck-analyze -X list
Framework definition file path: /usr/local/intel/cc2019/clck/2019.10/etc/fwd/

Framework definition list:
avx512_performance_ratios_priv
avx512_performance_ratios_user
basic_internode_connectivity
basic_shells
benchmarks
bios_checker
clock
cluster
cpu_admin
cpu_base
cpu_intel64
cpu_user
dapl_fabric_providers_present
dgemm_cpu_performance
environment_variables_uniformity
ethernet
exclude_hpl
file_system_uniformity
hardware
health
health_admin
health_base
health_extended_user
hpcg_cluster
hpcg_single
hpl_cluster_performance
hyper_threading
imb_allgather
imb_allgatherv
imb_allreduce
imb_alltoall
imb_barrier
imb_bcast
imb_benchmarks_blocking_collectives
imb_benchmarks_non_blocking_collectives
imb_gather
imb_gatherv
imb_iallgather
imb_iallgatherv
imb_iallreduce
imb_ialltoall
imb_ialltoallv
imb_ibarrier
imb_ibcast
imb_igather
imb_igatherv
imb_ireduce
imb_ireduce_scatter
imb_iscatter
imb_iscatterv
imb_pingping
imb_pingpong_fabric_performance
imb_reduce
imb_reduce_scatter
imb_reduce_scatter_block
imb_scatter
imb_scatterv
infiniband_admin
infiniband_base
infiniband_user
intel_dc_persistent_memory_capabilities_priv
intel_dc_persistent_memory_dimm_placement_priv
intel_dc_persistent_memory_events_priv
intel_dc_persistent_memory_firmware_priv
intel_dc_persistent_memory_kernel_support
intel_dc_persistent_memory_mode_uniformity_priv
intel_dc_persistent_memory_namespaces_priv
intel_dc_persistent_memory_priv
intel_dc_persistent_memory_tools_priv
intel_hpc_platform_base_compat-hpc-2018.0
intel_hpc_platform_base_core-intel-runtime-2018.0
intel_hpc_platform_base_high-performance-fabric-2018.0
intel_hpc_platform_base_hpc-cluster-2018.0
intel_hpc_platform_base_sdvis-cluster-2018.0
intel_hpc_platform_base_sdvis-core-2018.0
intel_hpc_platform_base_sdvis-single-node-2018.0
intel_hpc_platform_compat-hpc-2018.0
intel_hpc_platform_compliance_tcl_version
intel_hpc_platform_core-2018.0
intel_hpc_platform_core-intel-runtime-2018.0
intel_hpc_platform_cpu_sdvis-single-node-2018.0
intel_hpc_platform_firmware_high-performance-fabric-2018.0
intel_hpc_platform_high-performance-fabric-2018.0
intel_hpc_platform_hpc-cluster-2018.0
intel_hpc_platform_kernel_version_core-2018.0
intel_hpc_platform_libfabric_high-performance-fabric-2018.0
intel_hpc_platform_libraries_core-intel-runtime-2018.0
intel_hpc_platform_libraries_sdvis-cluster-2018.0
intel_hpc_platform_libraries_sdvis-core-2018.0
intel_hpc_platform_libraries_second-gen-xeon-sp-2019.0
intel_hpc_platform_linux_based_tools_present_core-intel-runtime-2018.0
intel_hpc_platform_memory_sdvis-cluster-2018.0
intel_hpc_platform_memory_sdvis-single-node-2018.0
intel_hpc_platform_minimum_memory_requirements_compat-hpc-2018.0
intel_hpc_platform_minimum_storage
intel_hpc_platform_minimum_storage_sdvis-cluster-2018.0
intel_hpc_platform_minimum_storage_sdvis-single-node-2018.0
intel_hpc_platform_mount
intel_hpc_platform_perl_core-intel-runtime-2018.0
intel_hpc_platform_rdma_high-performance-fabric-2018.0
intel_hpc_platform_sdvis-cluster-2018.0
intel_hpc_platform_sdvis-core-2018.0
intel_hpc_platform_sdvis-single-node-2018.0
intel_hpc_platform_second-gen-xeon-sp-2019.0
intel_hpc_platform_subnet_management_high-performance-fabric-2018.0
intel_hpc_platform_version_compat-hpc-2018.0
intel_hpc_platform_version_core-2018.0
intel_hpc_platform_version_core-intel-runtime-2018.0
intel_hpc_platform_version_high-performance-fabric-2018.0
intel_hpc_platform_version_hpc-cluster-2018.0
intel_hpc_platform_version_sdvis-cluster-2018.0
intel_hpc_platform_version_sdvis-core-2018.0
intel_hpc_platform_version_sdvis-single-node-2018.0
intel_hpc_platform_version_second-gen-xeon-sp-2019.0
iozone_disk_bandwidth_performance
kernel_parameter_preferred
kernel_parameter_uniformity
kernel_version_uniformity
local_disk_storage
lsb_libraries
lshw_disks
lshw_hardware_uniformity
memory_uniformity
mpi
mpi_bios
mpi_environment
mpi_ethernet
mpi_libfabric
mpi_local_functionality
mpi_multinode_functionality
mpi_prereq_admin
mpi_prereq_user
network_time_uniformity
node_process_status
opa_admin
opa_base
opa_user
osu_allgather
osu_allgatherv
osu_allreduce
osu_alltoall
osu_alltoallv
osu_barrier
osu_bcast
osu_benchmarks_blocking_collectives
osu_benchmarks_non_blocking_collectives
osu_benchmarks_point_to_point
osu_bibw
osu_bw
osu_gather
osu_gatherv
osu_iallgather
osu_iallgatherv
osu_iallreduce
osu_ialltoall
osu_ialltoallv
osu_ialltoallw
osu_ibarrier
osu_ibcast
osu_igather
osu_igatherv
osu_ireduce
osu_iscatter
osu_iscatterv
osu_latency
osu_mbw_mr
osu_reduce
osu_reduce_scatter
osu_scatter
osu_scatterv
perl_functionality
precision_time_protocol
privileged_user
python_functionality
rpm_snapshot
rpm_uniformity
second-gen-xeon-sp
second-gen-xeon-sp_parallel_studio_xe_runtimes_2019.0
second-gen-xeon-sp_priv
second-gen-xeon-sp_user
select_solutions_redhat_openshift_base
select_solutions_redhat_openshift_plus
select_solutions_sim_mod_benchmarks_base_2018.0
select_solutions_sim_mod_benchmarks_plus_2018.0
select_solutions_sim_mod_benchmarks_plus_second_gen_xeon_sp
select_solutions_sim_mod_priv_base_2018.0
select_solutions_sim_mod_priv_plus_2018.0
select_solutions_sim_mod_priv_plus_second_gen_xeon_sp
select_solutions_sim_mod_user_base_2018.0
select_solutions_sim_mod_user_plus_2018.0
select_solutions_sim_mod_user_plus_second_gen_xeon_sp
services_status
sgemm_cpu_performance
shell_functionality
single
std_libraries
stream_memory_bandwidth_performance
syscfg_settings_uniformity
tcl_functionality
tools

Node Roles:

The role annotation keyword is used to assign a node to one or more roles. A role describes the intended functionality of a node.

For example, the following nodefile defines 4 nodes: node1 is a head and compute node; node2, node3, and node4 are compute nodes; and node5 is disabled.

node1 # role: head
node2 # role: compute
node3 # role: login

Valid node role values are described below.

  • boot – Provides software imaging / provisioning capabilities.
  • compute – Is a compute resource (mutually exclusive with enhanced).
  • enhanced – Provides enhanced compute resources, for example, contains additional memory (mutually exclusive with compute).
  • external – Provides an external network interface.
  • head – Alias for the union of boot, external, job_schedule, login, network_address, and storage.
  • job_schedule – Provides resource manager / job scheduling capabilities.
  • login – Is an interactive login system.
  • network_address – Provides network address to the cluster, for example, DHCP.
  • storage – Provides network storage to the cluster, like NFS.

More Information:

User Guide:

Using Intel Cluster Checker (Part 2)

Continuation of Using Intel Cluster Checker (Part 1)

Setup Environment Variables

In you .bashrc, add the following

export CLCK_ROOT=/usr/local/intel/cc2019/clck/2019.10
export CLCK_SHARED_TEMP_DIR=/scratch/tmp

Command Line Options

-c / –config=FILE: Specifies a configuration file. The default configuration file is CLCK_ROOT/etc/clck.xml.

-C / –re-collect-data: Attempts to re-collect any missing or old data for use in analysis. This option only applies to data collection.

-D / –db=FILE: Specifies the location of the database file. This option works in clck-analyze and in clck, but not currently in clck-collect.

-f / –nodefile: Specifies a nodefile containing the list of nodes, one per line. If a nodefile is not specified for clck or clck-collect, a Slurm query will be used to determine the available nodes. If no nodefile is specified for clck-analyze, the nodes already present in the database will be used.

-F / –fwd=FILE: Specifies a framework definition. If a framework definition is not specified, the health framework definition is used. This option can be used multiple times to specify multiple framework definitions. To see a list of available framework definitions, use the command line option -X list.

-h / –help: Displays the help message.

-l / –log-level: Specifies the output level. Recognized values are (in increasing order of verbosity)**: alert, critical, error, warning, notice, info, and debug. The default log level is error.

-M / –mark-snapshot: Takes a snapshot of the data used in an analysis. The string used to mark the data cannot contain the comma character “,” or spaces. This option only applies to analysis.

-n / –node-include: Displays only the specified nodes in the analyzer output.

-o / –logfile: Specifies a file where the results from the run are written. By default, results are written to clck_results.log.

-r / –permutations: Number of permutations of nodes to use when running cluster data providers. By default, one permutation will run. This option only applies to data collection.

-S / –ignore-subclusters: Ignores the subcluster annotations in the nodefile. This option only applies to data collection.

-t / –threshold-too-old: Sets the minimum number of days since collection that will trigger a data too old error. This option only applies to data analysis.

-v / –version: Prints the version and exits.

-X / –FWD_description: Prints a description of the framework definition if available. If the value passed is “list”, then it prints a list of found framework definitions.

-z / –fail-level: Specifies the lowest severity level at which found issues fail. Recognizes values are (in increasing order of severity)**: informational, warning, and critical. The default level at which issues cause a failure is warning.

–sort-asc: Organizes the output in ascending order of the specified field. Recognized values are “id”, “node”, and “severity”.

–sort-desc: Organizes the output in descending order of the specified field. Recognized values are “id”, “node”, and “severity.”

For more  information about the available command line options and their uses, run Intel® Cluster Checker with the -h option, or see the man pages.

More Information:

  1. Using Intel Cluster Checker (Part 1)
  2. Using Intel Cluster Checker (Part 2)
  3. Using Intel Cluster Checker (Part 3)

User Guide:

Using Intel Cluster Checker (Part 1)

What is Intel Cluster Checker?

Intel® Cluster Checker provides tools to collect data from the cluster, analysis of the collected data, and provides a clear report of the analysis. Using Intel® Cluster Checker helps to quickly identify issues and improve utilization of resources.

Intel® Cluster Checker verifies the configuration and performance of Linux®-based clusters through analysis of cluster uniformity, performance characteristics, functionality and compliance with Intel® High Performance Computing (HPC) specifications. Data collection tools and analysis provide actionable remedies to identified issues. Intel® Cluster Checker tools and analysis are ideal for use by developers, administrators, architects, and users to easily identify issues within a cluster.

Installing Intel Cluster Checker Using Yum Repository

If you are using Yum Installation, do take a look at Intel Cluster Checker 2019 Installation

If not, you can untar the package if you have the tar.gz

Environment Setup

# source /usr/local/intel/2018u3/bin/compilervars.sh intel64
# source /usr/local/intel/2018u3/mkl/bin/mklvars.sh intel64
# source /usr/local/intel/2018u3/impi/2018.3.222/bin64/mpivars.sh intel64
# source /usr/local/intel/2018u3/parallel_studio_xe_2018/bin/psxevars.sh intel64
# export MPI_ROOT=/usr/local/intel/2018u3/impi/2018.3.222/intel64
# source /usr/local/intel/cc2019/clck/2019.10/bin/clckvars.sh

Create a nodefile and put the hosts in

% vim nodefile
node1
node2
node3

Running Intel Cluster Checker

*Make sure you have SSH  login to the nodes without password. See SSH Login without Password

% clck -f nodefile

Examples of run…..

Running Collect

................................................................................................................................................................................................................
Running Analyze

SUMMARY
Command-line: clck -f nodefile
Tests Run: health_base
**WARNING**: 3 tests failed to run. Information may be incomplete. See clck_execution_warnings.log for more information.
Overall Result: 8 issues found - HARDWARE UNIFORMITY (2), PERFORMANCE (2), SOFTWARE UNIFORMITY (4)
-----------------------------------------------------------------------------------------------------------------------------------------
8 nodes tested: node010, node[003-009]
0 nodes with no issues:
8 nodes with issues: node010, node[003-009]
-----------------------------------------------------------------------------------------------------------------------------------------
FUNCTIONALITY
No issues detected.

HARDWARE UNIFORMITY
The following hardware uniformity issues were detected:
1. The InfiniBand PCI physical slot for device 'MT27800 Family [ConnectX-5]' 

PERFORMANCE
The following performance issues were detected:
1.Zombie processes detected.
1 node: node010
2. Processes using high CPU.
7 nodes: node010, node[003,005-009]

SOFTWARE UNIFORMITY
The following software uniformity issues were detected:
1. The OFED version, 'MLNX_OFED_LINUX-4.5-1.0.1.0 (OFED-4.5-1.0.1)', is not uniform.....
5 nodes: node[003-004,006-007,009]
2. The OFED version, 'MLNX_OFED_LINUX-4.3-1.0.1.0 (OFED-4.3-1.0.1)', is not uniform.....
3 nodes: node010, node[005,008]
3. Environment variables are not uniform across the nodes.
.....
4. Inconsistent Ethernet driver version.
.....

See the following files for more information: clck_results.log, clck_execution_warnings.log

Intel MPI Library Troubleshooting

If you are an admin and if you make sure their cluster is set up to work with the Intel® MPI Library, do the following

% clck -f nodefile -F mpi_prereq_admin

If you are non-privileged user and if you make sure their cluster is set up to work with the Intel® MPI Library, do the following

% clck -f nodefile -F mpi_prereq_user

More Information:

  1. Using Intel Cluster Checker (Part 1)
  2. Using Intel Cluster Checker (Part 2)
  3. Using Intel Cluster Checker (Part 3)

User Guide:

Intel® Enpirion® Multi-Rail Power Sequencing and Monitoring

The Multi-Rail Power Sequencer and Monitor is a programmable module within the Intel MAX 10 FPGA and MAX V CPLD. The sequencer can monitor up to 144 power rails to meet the full range of power requirements for FPGAs, ASICs, CPUs, and other processors. It can be easily configured and scaled with our user-friendly Platform Designer GUI.

Intel FPGA Technology Day (IFTD 2020)

Event Overview

Intel® FPGA Technology Day (IFTD) is a one-day virtual event that showcases the latest Intel FPGA products and solutions through a series of webinars and demonstrations from Intel, partners, and customers. During this day-long event, you will learn how Intel® FPGAs, Intel® eASIC structured ASICs, Intel® SmartNICs, and Intel® Enpirion® power solutions can shorten your design cycle, help to overcome your many design challenges, and accelerate business innovation. In the IFTD 2020 virtual exhibition space, you will see Intel® FPGA technology in action in the edge, in data centers, and in the cloud. IFTD 2020 takes place on November 18, 2020.

To Register see https://plan.seek.intel.com/IFTD-2020_RegPage?trackSrc=Invite_registernow_CTA

Keynote

  • Accelerate Your Future with the broad Intel Portfolio including Intel FPGAs
  • Accelerate your Cloud and Enterprise Applications with Intel Data Center Solutions including Intel FPGAs
  • Conquer Your Most Difficult Embedded Challenges
  • How to use Intel FPGAs and eASIC Structured ASICs to develop solutions for a wide array of applications

Find CPU and GPU Performance Headroom using Roofline Analysis

Join Technical Consulting Engineer and HPC programming expert Cedric Andreolli for a session covering:

  • How to perform GPU headroom and GPU caches locality analysis using Advisor Roofline extensions for oneAPI and OpenMP
  • An introduction to a new memory-level Roofline feature that helps pinpoint which specific memory level (L1, L2, L3, or DRAM) is causing the bottleneck
  • A walkthrough of Intel Advisor’s improved user interface

To see video, see https://techdecoded.intel.io/essentials/find-cpu-gpu-performance-headroom-using-roofline-analysis/#gs.fpbz93

Intel Launches 11th Gen Intel Core and Intel Evo (code-named “Tiger Lake”)

Intel released 11th Gen Intel® Core™ mobile processors with Iris® Xe graphics (code-named “Tiger Lake”). The new processors break the boundaries of performance with unmatched capabilities in productivity, collaboration, creation, gaming and entertainment on ultra-thin-and-light laptops. They also power the first class of Intel Evo platforms, made possible by the Project Athena innovation program. (Credit: Intel Corporation)

  • Intel launches 11th Gen Intel® Core™ processors with Intel® Iris® Xe graphics, the world’s best processors for thin-and-light laptops1, delivering up to 2.7x faster content creation2, more than 20% faster office productivity3 and more than 2x faster gaming plus streaming4 in real-world workflows over competitive products.
  • Intel® Evo™ platform brand introduced for designs based on 11th Gen Intel Core processors with Intel Iris Xe graphics and verified through the Project Athena innovation program’s second-edition specification and key experience indicators (KEIs).
  • More than 150 designs based on 11th Gen Intel Core processors are expected from Acer, Asus, Dell, Dynabook, HP, Lenovo, LG, MSI, Razer, Samsung and others.

Accelerate Insights with AI and HPC Combined by Intel

In this presentation the presenter will address those questions and give an overview of respective technology for Artificial Intelligence, including hardware platforms & software stacks with a special focus on how to enable successful development of AI solutions. The presenter will look into how to do this on the datacenter technology you know and use today, as well as specific technology for AI workloads. This will also be illustrated with practical customer examples