# Collectl2plotly

## How to
To use you need to install it via pip:
```
    pip install ./Collectl2plotly
```

After that you can use it like that:
```
    Collectl2plotly <options>
```
For informations of the options use "--help" or go to the [options](#options) description.

## Collectl commad to make Datafiles
Important is "-sZ" with out Collectl2plotly can not get the data from the file for the plots
```
    collectl -f <filename that will be created> -i <interval>:<interval> -sZ
```

## Options

| Option          | Takes | Description  | Default |
| --------------- | ----- | ------------ | ------- |
| -s, --source | a path to a directory with collectl datafiles(.raw.gz) or collectl datafile file directly. | It is multiple useable! It will be used as collectl sources to get the Plotdata. | If no sources is given as parameter it search in the current dir |
| -c, --collectl  | how to call collectl | is the collectl that will be called to get the data from the sources | collectl without a path |
| -p, --plotlypath | a path to a plotly javascript libery | is needed for the plot in the html files | plotly.js next to skript |
| -d, --destination | a path to a directory | is where a directory with the html files will be created | current directory |
| --configpath | a path to config file | is a python file with the plot and merge settings see [here](#config)| a default config |
| --shorten /<br> --notshorten | - | enable or disable shorten of commands with parameters/options only to file/command names. <br> examples:<br> python ~/scripts/script.py 1 --> script.py <br>ls -lisa --> ls | enabled |
| --coarsest | - | enable shorten of commands only to command names.<br> If enabled --shorten is ignored!<br> examples: <br> python ~/scripts/script.py 1 --> python <br> ls -lisa --> ls | disabled |
| --filtercmds / --notfiltercmds | - | enable or disable filtering | enabled |
| --filtertype | filtertype <br> see --help for detail | to select which filter to be used | see --help |
| --filtervalue | any value (string, int, ...) | is passed to the filter function as string | - |
| --force | - | override existing plot directory if exist | - |


## Config
The config is python file with following varibles to create spezial plots or/and with different values instead of the default.

#### NEEDED_VALUES
Specifies which values are merged under which names in a dict.
As key is the name of the merged values and the dict value is config with:

**'keys'**:<br>
This are a list of collectl table headers of the values which are given into the merger function.

**'merger'**:<br>
Is a tuple with the merger function name and a dict with the parameter of the merger. See [merger](#merger) to know what merger and parameter exist.

**'base_value'**:<br>
Is the startvalue of merger.

If 'merger' or 'base_value' is not given it use the default from [NEEDED_VALUES_DEFAULTS](#needed_values_defaults).

Example:
```python
NEEDED_VALUES = {
    'CPU': {
        'keys':['PCT'],
        'merger': ('x2oneaddition_float', {'operator': '/', 'operator_value': 100}),
        'base_value': 0.0,
    },
    'syscalls': {
        'keys':['RSYS', 'WSYS'],
        'merger': ('x2oneaddition_int', {}),
    },
}
```


#### NEEDED_VALUES_DEFAULTS
Defines the defaults of NEEDED_VALUES.

Example:
```python
NEEDED_VALUES_DEFAULTS = {
    'default_base_value': 0,
    'default_merger': ('x2oneaddition_int', {}),
}
```

#### NAME_SPEZIAL_PARAMETER_CONFIG
Specifies for which commands a script/program name is searched (dict key)and
which parameters the command has, that takes another value that is not the name (dict value). This is required to identify the name.

Important:<br>
Versions and path are ignored on the command example: python3 and /bin/python2 are both only python.
Bash -c and sh -c are an exception and are treated separately in the code.

Example:
```python
NAME_SPEZIAL_PARAMETER_CONFIG = {
    'java': ['-cp', '-classpath'],
    'bash': [],
    'sh': [],
    'perl': [],
    'python': ['-m', '-W', '-X', '--check-hash-based-pycs', '-c'],
}
```

#### COMAND_BLACKLIST
Is a list of commands which are ignored. It use the shorted command to check it is in blacklist or not.

Example:
```python
COMAND_BLACKLIST = [
    'collectl',
]
```

#### PLOTS_CONFIG
Is a list of plots settings of the plots to be displayed on the pages.
```
PLOTS_CONFIG = [
    {Plot1},
    {Plot2},
    ...
]
```

A plot settings is dict with following keys:

**'name'**:<br>
Is the name of the Plot this will be displayed in the index of the websites.

**'needed_key'**:<br>
Is a name from NEEDED_VALUES this values are used as y axis data in the plot.

**'relative-absolute_xaxis'**:<br>
Is a bool if True add buttons to the plot that switch the xaxis from absolute Date to Runtime and back.
If not given it is False.

**'plotly_settings'**:<br>
Is a settings dict in Plotly style, here it is possible to set everything about the appearance of the plots.<br>
Hints for plotly_settings:
+ Use [Plotly wiki](https://plot.ly/javascript/) javascript  to search settings because the javascript objekt looks same like the python dict.
+ If you use same settings in multiple plots you can put in a dict variable first and included on the right possition like "**variable".
+ {host} will replaced with the hostname in title of plot and in y/xaxis.

Example:
```python
PLOTS_CONFIG = [
    {
        'name': 'CPU load',
        'needed_key': 'CPU',
        'relative-absolute_xaxis': True,
        'plotly_settings': {
            'data': {
                'type': 'scattergl',
                'mode': 'markers',
            },
            'layout': {
                'height': 500,
                'title': {'text': 'CPU load {host}', 'y': 1},
                'xaxis': {'title': 'Date'},
                'yaxis': {'title': 'CPU load'},
                'showlegend': True,
                **barchart_buttons
            },
        },
    },
]
```

#### PLOTLY_COLORS
Is a list of colors to be used for the plots.
Repeats itself when there are not enough colors for the plot.

Example:
```python
PLOTLY_COLORS=[
    'rgb(31, 119, 180)', 'rgb(255, 127, 14)',
    'rgb(44, 160, 44)', 'rgb(214, 39, 40)',
    'rgb(148, 103, 189)', 'rgb(140, 86, 75)',
    'rgb(227, 119, 194)', 'rgb(127, 127, 127)',
    'rgb(188, 189, 34)', 'rgb(23, 190, 207)',
]
```

#### PLOTLY_STATIC_COLORS
Is to give one comand a specific Color.

Key is the displayed name in plot.<br>
Value is color as string.

Example:
```python
PLOTLY_STATIC_COLORS = {
    'example.py': 'rgb(254, 1, 154)',
}

# if not needed do:
PLOTLY_STATIC_COLORS = {}
```

## Filter

### Existing filter
| Name | Name of function | Description |  filtervalue default |
| ---- | ---------------- | ----------- |  ------------------- |
| hardvalue | filter_hardvalue | Filter out all that are above 'filtervalue' when adding from largest to smallest and not zero. | 95(%) |
| average | filter_average | Filters out those commands that are on average below the 'filtervalue' relative to the overall average. <br>Default: filtervalue=5(percent) | 5(%) |

#### Understanding example for the filter
this how the filtering is to understand this is not so in the Code! <br>
hardvalue:
```
    filtervalue = 95
    cmd1 = [0, 0]
    cmd2 = [1, 0]
    ...
    values are all values from diffrent commands sum together
    values = [0, 1, 4, 4, 5, 6]

    values in percentage calculate
    example on 5:
    25 =  5 * 100 / 20(sum of all values)

    new_values are [0, 5, 20, 20, 25, 30]
    sort from greates to smallest
    new_values = [30, 25, 20, 20, 5, 0]

    go over new_values and sum until it is greater than or equal to filtervalue, the follow values are filterd out
    30 + 25 + 20 + 20 == 95
    => [6, 5, 4, 4] == [cmd6, cmd5, cmd4, cmd3]
```
average:
```
    filtervalue = 5
    cmd1 = [0, 0]
    cmd2 = [0, 2]
    ...
    values are the averages from diffrent commands
    values = [0, 1, 4, 4, 5, 6]

    calculate values in percentage
    example on 5:
    150 =  5 * 100 / 3,33(avarage of all values)

    if percentage is less than filtervalue the cmd is filterd out

    end result [1, 4, 4, 5, 6] == [cmd2, cmd3, cmd4, cmd5, cmd6]
```

### Add new filter
In default it gives only two filter but in filter_func.py it is possible to add more.
If you add one Filter extend 'FILTER_FUNCTIONS' in Collectl2plotly.py with your Filter.

#### Arguments of a filter function
1. Is a dict that looks like:
```python
{
    'number_of_values': X all,
    'commands':{
        cmd:{'number_of_values': X command, metric:SUM of command, ...}
    },
    metrics:SUM of all commands, ...
}
```
2. Is a string from the option filtervalue or None if no filtervalue given as option on execution.

#### Return
A filter function must return a dict that structure is like follow:
```
    {
        metric: [command that is not filter out, ...],
        ... for all metrics
    }
```

## Merger
A merger is a function that merge a values from collectl to existing base value. It is also used to formed values from collectl.


### Existing merger

| Name | Description | return type |
| ---- | ----------- | ----------- |
| x2oneaddition_int | convert collectl value to int and sum all values | int |
| x2oneaddition_float | convert collectl value to float and sum all values | float |


 | possible argument | take | describtion | supported merger |
 | ----------------- | ---- | ----------- | ---------------- |
 | round | int | Rounding merged collectl values on given point (Python round function). | x2oneaddition_* |
 | operator | basic python operater as str (/ , * , -, +, %) | Specifies which operator should be used to calculate a value from parameter operator_value and the collectl values who already merged. This will be merged to the base value. | x2oneaddition_* |
 | operator_value | int/float | value that must be given to use operator for use see operator. | x2oneaddition_* |

### Add new Merger
In default it gives only two filter but in value_merger.py it is possible to add more.
If one merger is added in value_merger.py he must be added in the global variable MERGER in Collectl2plotly.py.

#### Arguments of a merger function
1. base value is the initial value for merging on it. Normaly has the same type like the return value
because the return value can possible the next base value.
2. (args) All following arguments are the values(str) from the collectl that were assigned in [NEEDED_VALUES](#needed_values).
3. (kwargs) Are the Parameter that were assigned in [NEEDED_VALUES](#needed_values) for the merger.

#### Return
A merger function must return a value that can be the base value of his merger.
Because merger can be multiple called for the same time if multiple samples exist.