readme.md 10.5 KB
Newer Older
Julius Metz's avatar
Julius Metz committed
1 2 3 4

# Collectl2plotly

## How to
5
To use you need to install it via pip:
Julius Metz's avatar
Julius Metz committed
6
```
7
    pip install ./Collectl2plotly
Julius Metz's avatar
Julius Metz committed
8 9
```

10
After that you can use it like that:
Julius Metz's avatar
Julius Metz committed
11
```
12
    Collectl2plotly <options>
Julius Metz's avatar
Julius Metz committed
13
```
14 15
For informations of the options use "--help" or go to the [options](#options) description.

Julius Metz's avatar
Julius Metz committed
16 17 18 19 20
## Collectl commad to make Datafiles
Important is "-sZ" with out Collectl2plotly can not get the data from the file for the plots
```
    collectl -f <filename that will be created> -i <interval>:<interval> -sZ
```
Julius Metz's avatar
Julius Metz committed
21 22 23

## Options

Julius Metz's avatar
Julius Metz committed
24
| Option          | Takes | Description  | Default |
Julius Metz's avatar
Julius Metz committed
25
| --------------- | ----- | ------------ | ------- |
26
| -s, --source | a path to a directory with collectl datafiles(.raw.gz) or collectl datafile file directly. | It is multiple useable! It will be used as collectl sources to get the Plotdata. | If no sources is given as parameter it search in the current dir |
Julius Metz's avatar
Julius Metz committed
27
| -c, --collectl  | how to call collectl | is the collectl that will be called to get the data from the sources | collectl without a path |
Julius Metz's avatar
Julius Metz committed
28
| -p, --plotlypath | a path to a plotly javascript libery | is needed for the plot in the html files | plotly.js next to skript |
Julius Metz's avatar
Julius Metz committed
29
| -d, --destination | a path to a directory | is where a directory with the html files will be created | current directory |
Julius Metz's avatar
Julius Metz committed
30
| --configpath | a path to config file | is a python file with the plot and merge settings see [here](#config)| a default config |
Julius Metz's avatar
WC  
Julius Metz committed
31
| --shorten /<br> --notshorten | - | enable or disable shorten of commands with parameters/options only to file/command names. <br> examples:<br> python ~/scripts/script.py 1 --> script.py <br>ls -lisa --> ls | enabled |
32
| --coarsest | - | enable shorten of commands only to command names.<br> If enabled --shorten is ignored!<br> examples: <br> python ~/scripts/script.py 1 --> python <br> ls -lisa --> ls | disabled |
Julius Metz's avatar
Julius Metz committed
33 34
| --filtercmds / --notfiltercmds | - | enable or disable filtering | enabled |
| --filtertype | filtertype <br> see --help for detail | to select which filter to be used | see --help |
Julius Metz's avatar
Julius Metz committed
35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130
| --filtervalue | any value (string, int, ...) | is passed to the filter function as string | - |
| --force | - | override existing plot directory if exist | - |


## Config
The config is python file with following varibles to create spezial plots or/and with different values instead of the default.

#### NEEDED_VALUES
Specifies which values are merged under which names in a dict.
As key is the name of the merged values and the dict value is config with:

**'keys'**:<br>
This are a list of collectl table headers of the values which are given into the merger function.

**'merger'**:<br>
Is a tuple with the merger function name and a dict with the parameter of the merger. See [merger](#merger) to know what merger and parameter exist.

**'base_value'**:<br>
Is the startvalue of merger.

If 'merger' or 'base_value' is not given it use the default from [NEEDED_VALUES_DEFAULTS](#needed_values_defaults).

Example:
```python
NEEDED_VALUES = {
    'CPU': {
        'keys':['PCT'],
        'merger': ('x2oneaddition_float', {'operator': '/', 'operator_value': 100}),
        'base_value': 0.0,
    },
    'syscalls': {
        'keys':['RSYS', 'WSYS'],
        'merger': ('x2oneaddition_int', {}),
    },
}
```


#### NEEDED_VALUES_DEFAULTS
Defines the defaults of NEEDED_VALUES.

Example:
```python
NEEDED_VALUES_DEFAULTS = {
    'default_base_value': 0,
    'default_merger': ('x2oneaddition_int', {}),
}
```

#### NAME_SPEZIAL_PARAMETER_CONFIG
Specifies for which commands a script/program name is searched (dict key)and
which parameters the command has, that takes another value that is not the name (dict value). This is required to identify the name.

Important:<br>
Versions and path are ignored on the command example: python3 and /bin/python2 are both only python.
Bash -c and sh -c are an exception and are treated separately in the code.

Example:
```python
NAME_SPEZIAL_PARAMETER_CONFIG = {
    'java': ['-cp', '-classpath'],
    'bash': [],
    'sh': [],
    'perl': [],
    'python': ['-m', '-W', '-X', '--check-hash-based-pycs', '-c'],
}
```

#### COMAND_BLACKLIST
Is a list of commands which are ignored. It use the shorted command to check it is in blacklist or not.

Example:
```python
COMAND_BLACKLIST = [
    'collectl',
]
```

#### PLOTS_CONFIG
Is a list of plots settings of the plots to be displayed on the pages.
```
PLOTS_CONFIG = [
    {Plot1},
    {Plot2},
    ...
]
```

A plot settings is dict with following keys:

**'name'**:<br>
Is the name of the Plot this will be displayed in the index of the websites.

**'needed_key'**:<br>
Is a name from NEEDED_VALUES this values are used as y axis data in the plot.

131 132 133 134
**'relative-absolute_xaxis'**:<br>
Is a bool if True add buttons to the plot that switch the xaxis from absolute Date to Runtime and back.
If not given it is False.

Julius Metz's avatar
Julius Metz committed
135 136 137 138 139 140 141 142 143 144 145 146 147
**'plotly_settings'**:<br>
Is a settings dict in Plotly style, here it is possible to set everything about the appearance of the plots.<br>
Hints for plotly_settings:
+ Use [Plotly wiki](https://plot.ly/javascript/) javascript  to search settings because the javascript objekt looks same like the python dict.
+ If you use same settings in multiple plots you can put in a dict variable first and included on the right possition like "**variable".
+ {host} will replaced with the hostname in title of plot and in y/xaxis.

Example:
```python
PLOTS_CONFIG = [
    {
        'name': 'CPU load',
        'needed_key': 'CPU',
148
        'relative-absolute_xaxis': True,
Julius Metz's avatar
Julius Metz committed
149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287
        'plotly_settings': {
            'data': {
                'type': 'scattergl',
                'mode': 'markers',
            },
            'layout': {
                'height': 500,
                'title': {'text': 'CPU load {host}', 'y': 1},
                'xaxis': {'title': 'Date'},
                'yaxis': {'title': 'CPU load'},
                'showlegend': True,
                **barchart_buttons
            },
        },
    },
]
```

#### PLOTLY_COLORS
Is a list of colors to be used for the plots.
Repeats itself when there are not enough colors for the plot.

Example:
```python
PLOTLY_COLORS=[
    'rgb(31, 119, 180)', 'rgb(255, 127, 14)',
    'rgb(44, 160, 44)', 'rgb(214, 39, 40)',
    'rgb(148, 103, 189)', 'rgb(140, 86, 75)',
    'rgb(227, 119, 194)', 'rgb(127, 127, 127)',
    'rgb(188, 189, 34)', 'rgb(23, 190, 207)',
]
```

#### PLOTLY_STATIC_COLORS
Is to give one comand a specific Color.

Key is the displayed name in plot.<br>
Value is color as string.

Example:
```python
PLOTLY_STATIC_COLORS = {
    'example.py': 'rgb(254, 1, 154)',
}

# if not needed do:
PLOTLY_STATIC_COLORS = {}
```

## Filter

### Existing filter
| Name | Name of function | Description |  filtervalue default |
| ---- | ---------------- | ----------- |  ------------------- |
| hardvalue | filter_hardvalue | Filter out all that are above 'filtervalue' when adding from largest to smallest and not zero. | 95(%) |
| average | filter_average | Filters out those commands that are on average below the 'filtervalue' relative to the overall average. <br>Default: filtervalue=5(percent) | 5(%) |

#### Understanding example for the filter
this how the filtering is to understand this is not so in the Code! <br>
hardvalue:
```
    filtervalue = 95
    cmd1 = [0, 0]
    cmd2 = [1, 0]
    ...
    values are all values from diffrent commands sum together
    values = [0, 1, 4, 4, 5, 6]

    values in percentage calculate
    example on 5:
    25 =  5 * 100 / 20(sum of all values)

    new_values are [0, 5, 20, 20, 25, 30]
    sort from greates to smallest
    new_values = [30, 25, 20, 20, 5, 0]

    go over new_values and sum until it is greater than or equal to filtervalue, the follow values are filterd out
    30 + 25 + 20 + 20 == 95
    => [6, 5, 4, 4] == [cmd6, cmd5, cmd4, cmd3]
```
average:
```
    filtervalue = 5
    cmd1 = [0, 0]
    cmd2 = [0, 2]
    ...
    values are the averages from diffrent commands
    values = [0, 1, 4, 4, 5, 6]

    calculate values in percentage
    example on 5:
    150 =  5 * 100 / 3,33(avarage of all values)

    if percentage is less than filtervalue the cmd is filterd out

    end result [1, 4, 4, 5, 6] == [cmd2, cmd3, cmd4, cmd5, cmd6]
```

### Add new filter
In default it gives only two filter but in filter_func.py it is possible to add more.
If you add one Filter extend 'FILTER_FUNCTIONS' in Collectl2plotly.py with your Filter.

#### Arguments of a filter function
1. Is a dict that looks like:
```python
{
    'number_of_values': X all,
    'commands':{
        cmd:{'number_of_values': X command, metric:SUM of command, ...}
    },
    metrics:SUM of all commands, ...
}
```
2. Is a string from the option filtervalue or None if no filtervalue given as option on execution.

#### Return
A filter function must return a dict that structure is like follow:
```
    {
        metric: [command that is not filter out, ...],
        ... for all metrics
    }
```

## Merger
A merger is a function that merge a values from collectl to existing base value. It is also used to formed values from collectl.


### Existing merger

| Name | Description | return type |
| ---- | ----------- | ----------- |
| x2oneaddition_int | convert collectl value to int and sum all values | int |
| x2oneaddition_float | convert collectl value to float and sum all values | float |


 | possible argument | take | describtion | supported merger |
 | ----------------- | ---- | ----------- | ---------------- |
 | round | int | Rounding merged collectl values on given point (Python round function). | x2oneaddition_* |
288
 | operator | basic python operater as str (/ , * , -, +, %) | Specifies which operator should be used to calculate a value from parameter operator_value and the collectl values who already merged. This will be merged to the base value. | x2oneaddition_* |
Julius Metz's avatar
Julius Metz committed
289 290 291 292
 | operator_value | int/float | value that must be given to use operator for use see operator. | x2oneaddition_* |

### Add new Merger
In default it gives only two filter but in value_merger.py it is possible to add more.
293
If one merger is added in value_merger.py he must be added in the global variable MERGER in Collectl2plotly.py.
Julius Metz's avatar
Julius Metz committed
294 295 296 297

#### Arguments of a merger function
1. base value is the initial value for merging on it. Normaly has the same type like the return value
because the return value can possible the next base value.
Julius Metz's avatar
Julius Metz committed
298 299
2. (args) All following arguments are the values(str) from the collectl that were assigned in [NEEDED_VALUES](#needed_values).
3. (kwargs) Are the Parameter that were assigned in [NEEDED_VALUES](#needed_values) for the merger.
Julius Metz's avatar
Julius Metz committed
300 301 302 303 304

#### Return
A merger function must return a value that can be the base value of his merger.
Because merger can be multiple called for the same time if multiple samples exist.