Commit c8e45f02 authored by Matthias Lieber's avatar Matthias Lieber

reworked readme

parent dd32b479
# Collectl2plotly # Collectl2plotly
TODO: what is collectl2plotly? add link to http://collectl.sourceforge.net
## How to ## How to
To use you need to install it via pip: To use you need to install it via pip:
``` ```
...@@ -11,21 +13,23 @@ After that you can use it like that: ...@@ -11,21 +13,23 @@ After that you can use it like that:
``` ```
Collectl2plotly <options> Collectl2plotly <options>
``` ```
For informations of the options use "--help" or go to the [options](#options) description. For description of the options use "--help" or go to the [options](#options) description.
## Collectl commad to make Datafiles ## Collectl command to create input files for Collectl2plotly
Important is "-sZ" with out Collectl2plotly can not get the data from the file for the plots Collectl2plotly plots process data from collectl.
To record process data, use the collectl option "-sZ".
``` ```
collectl -f <filename that will be created> -i <interval>:<interval> -sZ collectl -f <filename that will be created> -i :<sample interval in seconds> -sZ
``` ```
You can also use collectl data from multiple cluster nodes that was recorded at the same time.
## Options ## Options
| Option | Takes | Description | Default | | Option | Takes | Description | Default |
| --------------- | ----- | ------------ | ------- | | --------------- | ----- | ------------ | ------- |
| -s, --source | a path to a directory with collectl datafiles(.raw.gz) or collectl datafile file directly. | It is multiple useable! It will be used as collectl sources to get the Plotdata. | If no sources is given as parameter it search in the current dir | | -s, --source | a path to a directory with collectl datafiles(.raw.gz) or collectl datafile directly. | It is multiple useable! It will be used as collectl sources to get the Plotdata. | If no source is given it searches in the current dir |
| -c, --collectl | how to call collectl | is the collectl that will be called to get the data from the sources | collectl without a path | | -c, --collectl | how to call collectl | is the collectl that will be called to get (replay) the data from the sources | collectl without a path |
| -p, --plotlypath | a path to a plotly javascript libery | is needed for the plot in the html files | plotly.js next to skript | | -p, --plotlypath | a path to a plotly javascript libary | is needed for the plot in the html files | plotly.js next to skript |
| -d, --destination | a path to a directory | is where a directory with the html files will be created | current directory | | -d, --destination | a path to a directory | is where a directory with the html files will be created | current directory |
| --configpath | a path to config file | is a python file with the plot and merge settings see [here](#config)| a default config | | --configpath | a path to config file | is a python file with the plot and merge settings see [here](#config)| a default config |
| --shorten /<br> --notshorten | - | enable or disable shorten of commands with parameters/options only to file/command names. <br> examples:<br> python ~/scripts/script.py 1 --> script.py <br>ls -lisa --> ls | enabled | | --shorten /<br> --notshorten | - | enable or disable shorten of commands with parameters/options only to file/command names. <br> examples:<br> python ~/scripts/script.py 1 --> script.py <br>ls -lisa --> ls | enabled |
...@@ -37,7 +41,12 @@ Important is "-sZ" with out Collectl2plotly can not get the data from the file f ...@@ -37,7 +41,12 @@ Important is "-sZ" with out Collectl2plotly can not get the data from the file f
## Config ## Config
The config is python file with following varibles to create spezial plots or/and with different values instead of the default. The config is a python file with the following varibles to create special plots or/and with different values instead of the default.
Usually, the default should be ok and there is no need to write a custom config.
TODO: describe what merging is
TODO: describe what filtering is
#### NEEDED_VALUES #### NEEDED_VALUES
Specifies which values are merged under which names in a dict. Specifies which values are merged under which names in a dict.
...@@ -203,41 +212,35 @@ PLOTLY_STATIC_COLORS = {} ...@@ -203,41 +212,35 @@ PLOTLY_STATIC_COLORS = {}
| hardvalue | filter_hardvalue | Filter out all that are above 'filtervalue' when adding from largest to smallest and not zero. | 95(%) | | hardvalue | filter_hardvalue | Filter out all that are above 'filtervalue' when adding from largest to smallest and not zero. | 95(%) |
| average | filter_average | Filters out those commands that are on average below the 'filtervalue' relative to the overall average. <br>Default: filtervalue=5(percent) | 5(%) | | average | filter_average | Filters out those commands that are on average below the 'filtervalue' relative to the overall average. <br>Default: filtervalue=5(percent) | 5(%) |
#### Understanding example for the filter #### Filter example
this how the filtering is to understand this is not so in the Code! <br>
hardvalue: hardvalue:
``` ```
filtervalue = 95 filtervalue = 95
cmd1 = [0, 0]
cmd2 = [1, 0] values: for each command the sum of all recorded values: cmd1, cmd2, ... cmd6
...
values are all values from diffrent commands sum together
values = [0, 1, 4, 4, 5, 6] values = [0, 1, 4, 4, 5, 6]
values in percentage calculate calculate percentages from values relative to sum of all values
example on 5: example on 5:
25 = 5 * 100 / 20(sum of all values) 25 = 5 * 100 / 20 (sum of all values)
new_values are [0, 5, 20, 20, 25, 30] new_values are [0, 5, 20, 20, 25, 30]
sort from greates to smallest sort from greates to smallest
new_values = [30, 25, 20, 20, 5, 0] new_values = [30, 25, 20, 20, 5, 0]
go over new_values and sum until it is greater than or equal to filtervalue, the follow values are filterd out go over new_values and sum up until sum is greater than or equal to filtervalue, the remaining values are filtered out
30 + 25 + 20 + 20 == 95 30 + 25 + 20 + 20 == 95
=> [6, 5, 4, 4] == [cmd6, cmd5, cmd4, cmd3] => [6, 5, 4, 4] == [cmd6, cmd5, cmd4, cmd3]
``` ```
average: average:
``` ```
filtervalue = 5 filtervalue = 5
cmd1 = [0, 0] values: for each command the average of all recorded values: cmd1, cmd2, ... cmd6
cmd2 = [0, 2]
...
values are the averages from diffrent commands
values = [0, 1, 4, 4, 5, 6] values = [0, 1, 4, 4, 5, 6]
calculate values in percentage calculate percentages from values relative to average of all values
example on 5: example on 5:
150 = 5 * 100 / 3,33(avarage of all values) 150 = 5 * 100 / 3.33 (average of all values)
if percentage is less than filtervalue the cmd is filterd out if percentage is less than filtervalue the cmd is filterd out
...@@ -245,8 +248,8 @@ average: ...@@ -245,8 +248,8 @@ average:
``` ```
### Add new filter ### Add new filter
In default it gives only two filter but in filter_func.py it is possible to add more. Two filter functions are pre-defined. Additional filters can be added in filter_func.py.
If you add one Filter extend 'FILTER_FUNCTIONS' in Collectl2plotly.py with your Filter. If you add one filter extend 'FILTER_FUNCTIONS' in Collectl2plotly.py with your filter.
#### Arguments of a filter function #### Arguments of a filter function
1. Is a dict that looks like: 1. Is a dict that looks like:
...@@ -262,7 +265,7 @@ If you add one Filter extend 'FILTER_FUNCTIONS' in Collectl2plotly.py with your ...@@ -262,7 +265,7 @@ If you add one Filter extend 'FILTER_FUNCTIONS' in Collectl2plotly.py with your
2. Is a string from the option filtervalue or None if no filtervalue given as option on execution. 2. Is a string from the option filtervalue or None if no filtervalue given as option on execution.
#### Return #### Return
A filter function must return a dict that structure is like follow: A filter function must return a dict, structured like this:
``` ```
{ {
metric: [command that is not filter out, ...], metric: [command that is not filter out, ...],
...@@ -271,8 +274,9 @@ A filter function must return a dict that structure is like follow: ...@@ -271,8 +274,9 @@ A filter function must return a dict that structure is like follow:
``` ```
## Merger ## Merger
A merger is a function that merge a values from collectl to existing base value. It is also used to formed values from collectl. A merger is a function that merge values from collectl to existing base value. It is also used to format values from collectl.
TODO: not clear what a base value is.
### Existing merger ### Existing merger
...@@ -289,16 +293,16 @@ A merger is a function that merge a values from collectl to existing base value. ...@@ -289,16 +293,16 @@ A merger is a function that merge a values from collectl to existing base value.
| operator_value | int/float | value that must be given to use operator for use see operator. | x2oneaddition_* | | operator_value | int/float | value that must be given to use operator for use see operator. | x2oneaddition_* |
### Add new Merger ### Add new Merger
In default it gives only two filter but in value_merger.py it is possible to add more. Two mergers are pre-defined. Additional ones can be added in value_merger.py.
If one merger is added in value_merger.py he must be added in the global variable MERGER in Collectl2plotly.py. If one merger is added in value_merger.py it must be added in the global variable MERGER in Collectl2plotly.py.
#### Arguments of a merger function #### Arguments of a merger function
1. base value is the initial value for merging on it. Normaly has the same type like the return value 1. base value is the initial value for merging on it. Normally it has the same type like the return value
because the return value can possible the next base value. because the return value can possible the next base value.
2. (args) All following arguments are the values(str) from the collectl that were assigned in [NEEDED_VALUES](#needed_values). 2. (args) All following arguments are the values(str) from the collectl that were assigned in [NEEDED_VALUES](#needed_values).
3. (kwargs) Are the Parameter that were assigned in [NEEDED_VALUES](#needed_values) for the merger. 3. (kwargs) Are the Parameter that were assigned in [NEEDED_VALUES](#needed_values) for the merger.
#### Return #### Return
A merger function must return a value that can be the base value of his merger. A merger function must return a value that can be the base value of this merger
Because merger can be multiple called for the same time if multiple samples exist. (mergers can be called multiple times for the same time stamp if multiple samples exist).
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment