What's Catalog?

The multiplicity of data that describes the characteristic of a node and its belongings, these data are gathered through various tools and protocols.

Catalog Classification

  • By catalog source
    • dmi
    • ohai
    • driveid
    • bmc
    • megaraid
    • ...
  • By trigger mode

    • Auto Catalog: The catalog that auto-gathered immediately when a node is newly discovered (Computer Node)
    • Passive Catalog: The catalog that gathered after the node has been discovered by triggering a catalog workflow. (Mgmt Server/Switch/Pdu…)
  • By execution environment:

    • Microkernel based catalog (ohai/dmi/bmc/smart/driveId…)

    • Local catalog (some ipmi, snmp related)

  • By sku scope:

    • common catalog
    • sku specified (racadm, redfish, ...)

Catalog Document

  • Raw Command Output
FRU Device Description : Builtin FRU Device (ID 0)
 Chassis Type          : Rack Mount Chassis
 Chassis Serial        : QTFCEV4120280
 Board Mfg Date        : Tue Apr  8 01:48:00 2014
 Board Mfg             : Quanta
 Board Product         : S210-X12RS V2
 Board Serial          : QTF3EV41400432
 Board Part Number     : 31S2RMB00H0
 Product Manufacturer  : Quanta
 Product Name          : S210-X12RS V2
 Product Serial        : QTFCEV4120280

FRU Device Description : AST2300
 Chassis Type          : Rack Mount Chassis
 Chassis Serial        : QTFCEV4120280
 Board Mfg Date        : Tue Apr  8 01:48:00 2014
 Board Mfg             : Quanta
 Board Product         : S210-X12RS V2
 Board Serial          : QTF3EV41400432
 Board Part Number     : 31S2RMB00H0
 Product Manufacturer  : Quanta
 Product Name          : S210-X12RS V2
 Product Serial        : QTFCEV4120280
  • Parsed Catalog:
{
    "createdAt": "2016-04-25T07:47:15.317Z",
    "data": {
        "Builtin FRU Device (ID 0)": {
            "Board Mfg": "Quanta",
            "Board Mfg Date": "Tue Apr  8 01:48:00 2014",
            "Board Part Number": "31S2RMB00H0",
            "Board Product": "S210-X12RS V2",
            "Board Serial": "QTF3EV41400432",
            "Chassis Type": "Rack Mount Chassis",
            "Product Manufacturer": "Quanta",
            "Product Name": "S210-X12RS V2",
            "Product Serial": "QTFCEV4120280"
        },
        "AST2300": {
            "Board Mfg": "Quanta",
            "Board Mfg Date": "Tue Apr  8 01:48:00 2014",
            "Board Part Number": "31S2RMB00H0",
            "Board Product": "S210-X12RS V2",
            "Board Serial": "QTF3EV41400432",
            "Chassis Type": "Rack Mount Chassis",
            "Product Manufacturer": "Quanta",
            "Product Name": "S210-X12RS V2",
            "Product Serial": "QTFCEV4120280"
        }
    },
    "id": "571dcb832e99e68d3cfe6824",
    "node": "571dcb631facd49c3c5659ff",
    "source": "ipmi-fru",
    "updatedAt": "2016-04-25T07:47:15.317Z"
}

Catalog Graphs

module.exports = {
    friendlyName: 'Discovery',
    injectableName: 'Graph.Discovery',
    options: {...},
    tasks: [
        {
            label: 'bootstrap-ubuntu',
            taskName: 'Task.Linux.Bootstrap.Ubuntu'
        },
        {
            label: 'catalog-dmi',
            taskName: 'Task.Catalog.dmi'
        },
        {
            label: 'catalog-ohai',
            taskName: 'Task.Catalog.ohai',
            waitOn: {
                'catalog-dmi': 'finished'
            }
        },
        {
            label: 'catalog-bmc',
            taskName: 'Task.Catalog.bmc',
            waitOn: {
                'catalog-ohai': 'finished'
            },
            ignoreFailure: true
        },
        {
            label: 'catalog-lsall',
            taskName: 'Task.Catalog.lsall',
            waitOn: {
                'catalog-bmc': 'finished'
            },
            ignoreFailure: true
        },
        ...
    ]
};

Catalog Tasks

  • Catalog via command:
module.exports = {
    friendlyName: 'Catalog lsall',
    injectableName: 'Task.Catalog.lsall',
    implementsTask: 'Task.Base.Linux.Catalog',
    options: {
        commands: [
            'sudo lspci -nn -vmm',
            'sudo lshw -json',
            'sudo lsblk -o KNAME,TYPE,ROTA; echo BREAK; sudo lsscsi --size'
        ]
    },
    properties: {
        catalog: {
            type: 'lsall'
        }
    }
};
  • Catalog via script
module.exports = {
    friendlyName: 'Catalog Drive IDs',
    injectableName: 'Task.Catalog.Drive.Id',
    implementsTask: 'Task.Base.Linux.Catalog',
    options: {
        commands: [
            {
                command: 'sudo node get_driveid.js',
                downloadUrl: '/get_driveid.js?nodeId='
            }
        ]
    },
    properties: {
        catalog: {
            type: 'driveId'
        }
    }
};

Catalog Job

Catalog Parser

  • How to link a task with a parser?
  • Fixed Parser:
var dmi = "sudo dmidecode";


/**
 * Input data format:
 *     data.stdout: The command stdout
 *     data.error: Indicate whether there is any error while executing the command
 */
CommandParser.prototoype[dmi] = function(data) {
    try {
        ... //do something parse
        return Promise.resolve({ source: 'dmi', data: parsedResult, store: true});
    } catch(err) {
        return Promise.reject( {source: 'dmi', error: err });
    }
  • Match Parser:
matchParsers.ipmiUserList = {
    regex: /^sudo ipmitool -c user list \d+$/,
    parsefunction: function(data) {
        try {
            ... //do something parse
            return Promise.resolve({ source: 'ipmi-user-list', data: parsedResult, store: true});
        } catch(err) {
            return Promise.reject( {source: 'ipmi-user-list', error: err });
        }
    }
};

Remote Catalog Procedure

  • Job

subscribeRequestCommands

sbuscribeRespondCommands

  • Southbound APi Service

RequestCommands

RespondCommands

  • Nodes

GET /tasks

POST /tasks

Touch Microkernel

What's Overlayfs?

Misc

Why a lot catalog task is set to ignoreFailure?

Exercise

(1) Design a Linux catalog task which parse the ls -l for the given folder or file and store the result into source ls.

For example, if the given folder is /var/opt, the ls -l /var/opt output is:

$ ls -l /var/opt
total 645504
drwxr-xr-x    24 yuanf yuanf      4096 Apr 19 16:21 ./
drwxr-xr-x     4 root  root       4096 Mar  8 17:39 ../
-rw-rw-r--     1 yuanf root          0 Feb 20 21:31 a.js
-rw-rw-r--     1 yuanf yuanf     18242 Feb 24 15:29 .babel.json
drwxrwxr-x     5 yuanf yuanf      4096 Feb 13 10:22 backup/
lrwxrwxrwx     1 yuanf yuanf        46 Aug  2  2016 alias -> /home/rackhd/personal/dotfiles/files/bash-alias

Then you need to parse the output into JSON data (the folder . and .. should be discarded):

[
    {
        "file_type": "file",
        "permission": {
            "user": "rw-"
            "group": "rw-"
            "others": "r--",
        },
        "link_count": 1,
        "owner": "yuanf",
        "group": "root",
        "size": 0,
        "modified_time": "Feb 20 21:31",
        "name": "a.js"
    },
    {
        "file_type": "file",
        "permission": {
            "user": "rw-"
            "group": "rw-"
            "others": "r--",
        },
        "link_count": 1,
        "owner": "yuanf",
        "group": "yuanf",
        "size": 18242,
        "modified_time": "Feb 24 15:29",
        "name": ".babel.json"
    },   
    {
        "file_type": "dir",
        "permission": {
            "user": "rwx"
            "group": "rwx"
            "others": "r-x",
        },
        "link_count": 5,
        "owner": "yuanf",
        "group": "yuanf",
        "size": 4096,
        "modified_time": "Feb 13 10:22",
        "name": "backup"
    }, 
    {
        "file_type": "link",
        "permission": {
            "user": "rwx"
            "group": "rwx"
            "others": "rwx",
        },
        "link_count": 1,
        "owner": "yuanf",
        "group": "yuanf",
        "size": 46,
        "modified_time": "Aug  2  2016",
        "name": "alias",
        "link_origin": "/home/rackhd/personal/dotfiles/files/bash-alias"
    }
]

Firstly, you need create a parser to get above JSON data, and pass the provided unit-test.

Then you need to inject the task into discovery workflow, so that when a node is newly discovered, the ls catalog can be auto created.

results matching ""

    No results matching ""