TextFSM - Changing a state

Last updated on Feb 24, 2020

What?

TextFSM is a python module for parsing (badly) formatted text. It was originally designed to parse the output of cli from different devices - in particular, the network.

In TextFSM there are three important entities: input, list of rules and output.

The CLI output I’m going to call input (hehehe) because that’s the information that we need to parse. That’s the conclusion we’re feeding to the FSM. TextFSM, which processes (parses) it according to the list of rules. The resulting text goes into output.

How?

TextFSM is a final state machine. In the simplest case, in TextFSM there are two states: Start (must be present in the template file) and EOF - an implicit state to which the transition happens when FSM reaches the end of the file.

The FSM alrorithm is something like this:

FSM reads a line from input;
FSM checks if this line matches the rules from the list of rules. List of the rules for each line of input is checked from top to bottom, starting from the first line of the list of rules;
If a line matches one of the rules in the list of rules, the FSM executes the action and goes to the next line of input, checking it against the list of rules again, starting on the first line of the list, again. Also, in this step, FSM can fill in the variable values.

More information on the algorithm, actions and rules can be found in the documentation. (see References)

Why?

If we have a simple output with repeating values, we can use only one state, filling the string with the values of matched variables. If the output is less structured, you will most likely have to use additional states.

FSM with one state (except EOF)

One stage is enough when input clearly defines the separator between the lines of output, as for example here:

[ Input ]

Server group radius
    Sharecount = 1  sg_unconfigured = FALSE
    Type = standard  Memlocks = 1
    Server(10.14.121.20:1814,1815) Transactions:
    Authen: 6	Author: 0	Acct: 0
    Server_auto_test_enabled: FALSE
     Keywrap enabled: FALSE
    Server(10.3.121.20:1814,1815) Transactions:
    Authen: 3	Author: 33	Acct: 333
    Server_auto_test_enabled: FALSE
     Keywrap enabled: FALSE
    Server(10.4.12.26:1814,1815) Transactions:
    Authen: 4	Author: 0	Acct: 0
    Server_auto_test_enabled: FALSE
     Keywrap enabled: FALSE
    Server(10.129.111.20:1814,1815) Transactions:
    Authen: 5	Author: 0	Acct: 555
    Server_auto_test_enabled: FALSE
     Keywrap enabled: FALSE
    Server(172.30.2.123:1812,1813) Transactions:
    Authen: 0	Author: 0	Acct: 0
    Server_auto_test_enabled: TRUE
     Keywrap enabled: FALSE
Server group SERVER_GROUP_2
    Sharecount = 1  sg_unconfigured = FALSE
    Type = standard  Memlocks = 1
    Server(172.30.10.123:1812,1813) Transactions:
    Authen: 2249	Author: 0	Acct: 22424
    Server_auto_test_enabled: TRUE
     Keywrap enabled: FALSE

This structure can be parsed this way:

[ List of rules ]

Value Filldown NAME (\S+)
Value Filldown SHARECOUNT (\d+)
Value Filldown SG_UNCONFIGURED (TRUE|FALSE)
Value Filldown TYPE (\w+)
Value Filldown MEMLOCKS (\d+)
Value Required SERVER_IP (\d+\.\d+\.\d+\.\d+)
Value AUTH_PORT (\d+)
Value ACC_PORT (\d+)
Value AUTO_TEST_ENABLED (TRUE|FALSE)
Value KEYWRAP_ENABLED (TRUE|FALSE)
Value AUTHEN_COUNT (\d+)
Value AUTHOR_COUNT (\d+)
Value ACCT_COUNT (\d+)

Start
  ^Server group ${NAME}
  ^.*Sharecount = ${SHARECOUNT}.*sg_unconfigured = ${SG_UNCONFIGURED}
  ^.*Type = ${TYPE}.*Memlocks = ${MEMLOCKS}
  ^.*Server\(${SERVER_IP}\:${AUTH_PORT},${ACC_PORT}\)
  ^.*Authen: ${AUTHEN_COUNT}.*Author: ${AUTHOR_COUNT}.*Acct: ${ACCT_COUNT}
  ^.*Server_auto_test_enabled: ${AUTO_TEST_ENABLED}
  ^.*Keywrap enabled: ${KEYWRAP_ENABLED} -> Record

graph TD;
  Start-->EOF;

[ Output ]

['NAME', 'SHARECOUNT', 'SG_UNCONFIGURED', 'TYPE', 'MEMLOCKS', 'SERVER_IP', 'AUTH_PORT', 'ACC_PORT', 'AUTO_TEST_ENABLED', 'KEYWRAP_ENABLED', 'AUTHEN_COUNT', 'AUTHOR_COUNT', 'ACCT_COUNT']
['radius', '1', 'FALSE', 'standard', '1', '10.14.121.20', '1814', '1815', 'FALSE', 'FALSE', '6', '0', '0']
['radius', '1', 'FALSE', 'standard', '1', '10.3.121.20', '1814', '1815', 'FALSE', 'FALSE', '3', '33', '333']
['radius', '1', 'FALSE', 'standard', '1', '10.4.12.26', '1814', '1815', 'FALSE', 'FALSE', '4', '0', '0']
['radius', '1', 'FALSE', 'standard', '1', '10.129.111.20', '1814', '1815', 'FALSE', 'FALSE', '5', '0', '555']
['radius', '1', 'FALSE', 'standard', '1', '172.30.2.123', '1812', '1813', 'TRUE', 'FALSE', '0', '0', '0']
['SERVER_GROUP_2', '1', 'FALSE', 'standard', '1', '172.30.10.123', '1812', '1813', 'TRUE', 'FALSE', '2249', '0', '22424']

Note the line Keywrap enabled:. It goes in the end of each section and triggers the the Record action - which writes the accumulated values to output.

FSM with two states (except EOF)

[ Input ]

total number of groups:3

following RADIUS server groups are configured:
        group radius:
                server: all configured radius servers
                deadtime is 0
        group RADIUS_GROUP_A:
                server: 10.65.101.22 on auth-port 1814, acct-port 1815
                server: 10.110.101.22 on auth-port 1814, acct-port 1815
                server: 10.194.101.22 on auth-port 1814, acct-port 1815
                server: 10.10.10.27 on auth-port 1814, acct-port 1815
                deadtime is 0
                vrf is management
                Source interface mgmt0
        group RADIUS_GROUP_B:
                server: 172.40.2.14 on auth-port 1812, acct-port 1813
                deadtime is 10
                Source interface loopback0

[ List of rules ]

Value Required NAME (\S+)
Value List SERVER (\d+\.\d+\.\d+\.\d+|all configured radius servers)
Value DEADTIME (\d+)
Value List AUTH_PORT (\d+)
Value List ACCT_PORT (\d+)
Value VRF (\S+)
Value SOURCE_INTERFACE (\S+)

Start
  ^.*group ${NAME}: -> Group

Group
  ^.*server:\s*${SERVER}\s*on auth-port ${AUTH_PORT}, acct-port ${ACCT_PORT}
  ^.*server:\s*${SERVER} -> AllServers
  ^.*vrf is ${VRF}
  ^.*Source interface ${SOURCE_INTERFACE} -> Record Start
  ^.*deadtime is ${DEADTIME}

AllServers
  ^.*deadtime is ${DEADTIME} -> Record Start

graph TD;
  Start-->AllServers;
  AllServers-->Start;
  Start-->EOF;

[ Output ]

['NAME', 'SERVER', 'DEADTIME', 'AUTH_PORT', 'ACCT_PORT', 'VRF', 'SOURCE_INTERFACE']
['radius', ['all configured radius servers'], '0', [], [], '', '']
['RADIUS_GROUP_A', ['10.65.101.22', '10.110.101.22', '10.194.101.22', '10.10.10.27'], '0', ['1814', '1814', '1814', '1814'], ['1815', '1815', '1815', '1815'], 'management', 'mgmt0']
['RADIUS_GROUP_B', ['172.40.2.14'], '10', ['1812'], ['1813'], '', 'loopback0']

Ok, but why do we need 2 states? Because in the group radius the last line before the next group is deadtime is 0, and in other groups the last line is Source interface. To process these different groups, you need different rule lists, which go to different states (Start and AllServers).

The condition that triggers the transition to AllServers state - the string server: all configured radius servers, that is getting matched by this rule: ^.*server:\s*${SERVER} -> AllServers.

In the state AllServers FSM reaches the line deadtime is, writes the state and returns to the state Start.

If FSM stays in the state Start, the state of the collected variables is getting written after the Source interface line is getting matched.