Commit c8e45f02 authored by Matthias Lieber's avatar Matthias Lieber

reworked readme

parent dd32b479
# Collectl2plotly
TODO: what is collectl2plotly? add link to http://collectl.sourceforge.net
## How to
To use you need to install it via pip:
```
......@@ -11,21 +13,23 @@ After that you can use it like that:
```
Collectl2plotly <options>
```
For informations of the options use "--help" or go to the [options](#options) description.
For description of the options use "--help" or go to the [options](#options) description.
## Collectl commad to make Datafiles
Important is "-sZ" with out Collectl2plotly can not get the data from the file for the plots
## Collectl command to create input files for Collectl2plotly
Collectl2plotly plots process data from collectl.
To record process data, use the collectl option "-sZ".
```
collectl -f <filename that will be created> -i <interval>:<interval> -sZ
collectl -f <filename that will be created> -i :<sample interval in seconds> -sZ
```
You can also use collectl data from multiple cluster nodes that was recorded at the same time.
## Options
| Option | Takes | Description | Default |
| --------------- | ----- | ------------ | ------- |
| -s, --source | a path to a directory with collectl datafiles(.raw.gz) or collectl datafile file directly. | It is multiple useable! It will be used as collectl sources to get the Plotdata. | If no sources is given as parameter it search in the current dir |
| -c, --collectl | how to call collectl | is the collectl that will be called to get the data from the sources | collectl without a path |
| -p, --plotlypath | a path to a plotly javascript libery | is needed for the plot in the html files | plotly.js next to skript |
| -s, --source | a path to a directory with collectl datafiles(.raw.gz) or collectl datafile directly. | It is multiple useable! It will be used as collectl sources to get the Plotdata. | If no source is given it searches in the current dir |
| -c, --collectl | how to call collectl | is the collectl that will be called to get (replay) the data from the sources | collectl without a path |
| -p, --plotlypath | a path to a plotly javascript libary | is needed for the plot in the html files | plotly.js next to skript |
| -d, --destination | a path to a directory | is where a directory with the html files will be created | current directory |
| --configpath | a path to config file | is a python file with the plot and merge settings see [here](#config)| a default config |
| --shorten /<br> --notshorten | - | enable or disable shorten of commands with parameters/options only to file/command names. <br> examples:<br> python ~/scripts/script.py 1 --> script.py <br>ls -lisa --> ls | enabled |
......@@ -37,7 +41,12 @@ Important is "-sZ" with out Collectl2plotly can not get the data from the file f
## Config
The config is python file with following varibles to create spezial plots or/and with different values instead of the default.
The config is a python file with the following varibles to create special plots or/and with different values instead of the default.
Usually, the default should be ok and there is no need to write a custom config.
TODO: describe what merging is
TODO: describe what filtering is
#### NEEDED_VALUES
Specifies which values are merged under which names in a dict.
......@@ -203,41 +212,35 @@ PLOTLY_STATIC_COLORS = {}
| hardvalue | filter_hardvalue | Filter out all that are above 'filtervalue' when adding from largest to smallest and not zero. | 95(%) |
| average | filter_average | Filters out those commands that are on average below the 'filtervalue' relative to the overall average. <br>Default: filtervalue=5(percent) | 5(%) |
#### Understanding example for the filter
this how the filtering is to understand this is not so in the Code! <br>
#### Filter example
hardvalue:
```
filtervalue = 95
cmd1 = [0, 0]
cmd2 = [1, 0]
...
values are all values from diffrent commands sum together
values: for each command the sum of all recorded values: cmd1, cmd2, ... cmd6
values = [0, 1, 4, 4, 5, 6]
values in percentage calculate
calculate percentages from values relative to sum of all values
example on 5:
25 = 5 * 100 / 20(sum of all values)
25 = 5 * 100 / 20 (sum of all values)
new_values are [0, 5, 20, 20, 25, 30]
sort from greates to smallest
new_values = [30, 25, 20, 20, 5, 0]
go over new_values and sum until it is greater than or equal to filtervalue, the follow values are filterd out
go over new_values and sum up until sum is greater than or equal to filtervalue, the remaining values are filtered out
30 + 25 + 20 + 20 == 95
=> [6, 5, 4, 4] == [cmd6, cmd5, cmd4, cmd3]
```
average:
```
filtervalue = 5
cmd1 = [0, 0]
cmd2 = [0, 2]
...
values are the averages from diffrent commands
values: for each command the average of all recorded values: cmd1, cmd2, ... cmd6
values = [0, 1, 4, 4, 5, 6]
calculate values in percentage
calculate percentages from values relative to average of all values
example on 5:
150 = 5 * 100 / 3,33(avarage of all values)
150 = 5 * 100 / 3.33 (average of all values)
if percentage is less than filtervalue the cmd is filterd out
......@@ -245,8 +248,8 @@ average:
```
### Add new filter
In default it gives only two filter but in filter_func.py it is possible to add more.
If you add one Filter extend 'FILTER_FUNCTIONS' in Collectl2plotly.py with your Filter.
Two filter functions are pre-defined. Additional filters can be added in filter_func.py.
If you add one filter extend 'FILTER_FUNCTIONS' in Collectl2plotly.py with your filter.
#### Arguments of a filter function
1. Is a dict that looks like:
......@@ -262,7 +265,7 @@ If you add one Filter extend 'FILTER_FUNCTIONS' in Collectl2plotly.py with your
2. Is a string from the option filtervalue or None if no filtervalue given as option on execution.
#### Return
A filter function must return a dict that structure is like follow:
A filter function must return a dict, structured like this:
```
{
metric: [command that is not filter out, ...],
......@@ -271,8 +274,9 @@ A filter function must return a dict that structure is like follow:
```
## Merger
A merger is a function that merge a values from collectl to existing base value. It is also used to formed values from collectl.
A merger is a function that merge values from collectl to existing base value. It is also used to format values from collectl.
TODO: not clear what a base value is.
### Existing merger
......@@ -289,16 +293,16 @@ A merger is a function that merge a values from collectl to existing base value.
| operator_value | int/float | value that must be given to use operator for use see operator. | x2oneaddition_* |
### Add new Merger
In default it gives only two filter but in value_merger.py it is possible to add more.
If one merger is added in value_merger.py he must be added in the global variable MERGER in Collectl2plotly.py.
Two mergers are pre-defined. Additional ones can be added in value_merger.py.
If one merger is added in value_merger.py it must be added in the global variable MERGER in Collectl2plotly.py.
#### Arguments of a merger function
1. base value is the initial value for merging on it. Normaly has the same type like the return value
1. base value is the initial value for merging on it. Normally it has the same type like the return value
because the return value can possible the next base value.
2. (args) All following arguments are the values(str) from the collectl that were assigned in [NEEDED_VALUES](#needed_values).
3. (kwargs) Are the Parameter that were assigned in [NEEDED_VALUES](#needed_values) for the merger.
#### Return
A merger function must return a value that can be the base value of his merger.
Because merger can be multiple called for the same time if multiple samples exist.
A merger function must return a value that can be the base value of this merger
(mergers can be called multiple times for the same time stamp if multiple samples exist).
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment