Log configuration
PyPPL
has fancy logs. You can define how they look like (theme) and what messages to show (levels).
Built-in log themes
We have some built-in themes:
green_on_black (default):
green_on_white:
blue_on_black:
blue_on_white:
magenta_on_black:
magenta_on_white:
If you don't like them, you can also disable them:
To use them, just specify the name in your pipeline configuration file:
dict(default=dict(
logger = dict(
theme = 'magenta_on_white'
)
))
PyPPL(logger_theme = 'green_on_black').start(...).run()
"theme"
to False
(false
for json
)
If you set theme
to True
, then default theme green_on_black
is used.
Level groups
Unlike python's default logging system, we have group of levels. The group is like the python logging level. However, here we have more, and with different values:
GROUP_VALUES = dict(
TITLE = 80,
SUBTITLE = 70,
STATUS = 60,
CRITICAL = 50,
ERROR = 40,
WARNING = 30,
INFO = 20,
DEBUG = 10,
NOTSET = 0
)
And each group has different levels
LEVEL_GROUPS = dict(
TITLE = ['PROCESS'],
SUBTITLE = ['DEPENDS', 'DONE'],
STATUS = ['WORKDIR', 'CACHED', 'P_DONE'],
CRITICAL = ['INFO', 'BLDING', 'SBMTING', 'RUNNING', 'JOBDONE', 'KILLING'],
ERROR = ['ERROR'],
WARNING = ['WARNING', 'RTRYING'],
INFO = ['PYPPL', 'PLUGIN', 'TIPS', 'CONFIG'],
DEBUG = ['DEBUG'],
)
So if you set PyPPL(logger_level = 'INFO')
; that means the levels in groups with value >= 20 will be shown.
Logger level | Meaning |
---|---|
PROCESS |
Mark when a process initiates |
DEPENDS |
Show dependencies of a process |
ERROR |
Show errors |
INFO |
Some information |
DONE |
Mark when a process is done |
BLDING |
Mark when a job starts to build |
SBMTING |
Mark when a job starts to submit |
RUNNING |
Mark when a job starts to run |
RTRYING |
Mark when a job starts to retry |
CACHED |
Mark when a job is cached |
PYPPL |
When pipeline starts |
TIPS |
Show tips |
CONFIG |
Show configuration is read from file |
WORKDIR |
Show the workdir for a process |
JOBDONE |
Mark when a job is done |
You can modify based on the group:
# also show debug message, but don't show the tips
PyPPL(logger_level = 'info', logger_leveldiffs = ["+DEBUG", "-TIPS"])
Define your own theme
Let's see how the built-in theme looks like first:
in pyppl/logger.py
:
THEMES = dict(
green_on_black = dict(
TITLE = '{s.BRIGHT}{f.CYAN}',
SUBTITLE = '{f.MAGENTA}',
STATUS = '{f.YELLOW}',
CRITICAL = '{f.GREEN}',
ERROR = '{f.RED}',
WARNING = '{s.BRIGHT}{f.YELLOW}',
DEBUG = '{s.DIM}{f.WHITE}',
NOTSET = ''
),
# other themes
}
\033[30m
for black (check here).
You can save it in configurations:
# pyppl.toml
[logger.theme]
TITLE = '{s.BRIGHT}{f.CYAN}'
SUBTITLE = '{f.MAGENTA}'
STATUS = '{f.YELLOW}'
CRITICAL = '{f.GREEN}'
ERROR = '{f.RED}'
WARNING = '{s.BRIGHT}{f.YELLOW}'
DEBUG = '{s.DIM}{f.WHITE}'
NOTSET = ''
Or pass it at runtime:
PyPPL(logger_theme = dict(TITLE = '{s.BRIGHT}{f.CYAN}', ...))
Log to file
By default, PyPPL
will not log to a file until you set a file path to {"logger": {"file": "/path/to/logfile"}}
in the configuration. Or you can specfiy False
to it to disable logging to file. If you set it to True
, a default log file will be used, which is: "./<pipeline name>.pyppl.log"
Note
Themes are not applied to handler to log to file.
Progress bar
Job status and progress are indicated in the log with progress bar:
PBAR_MARKS = {
STATES.INIT : ' ',
STATES.BUILDING : '~',
STATES.BUILT : '-',
STATES.BUILTFAILED : '!',
STATES.SUBMITTING : '+',
STATES.SUBMITTED : '>',
STATES.SUBMITFAILED : '$',
STATES.RUNNING : '>',
STATES.RETRYING : '-',
STATES.DONE : '=',
STATES.DONECACHED : 'z',
STATES.DONEFAILED : 'x',
STATES.ENDFAILED : 'X',
STATES.KILLING : '<',
STATES.KILLED : '*',
STATES.KILLFAILED : '*',
}
Note that if a cell represents multiple jobs, it has a priority as listed below:
STATES = Diot(
INIT = '00_init',
BUILDING = '99_building',
BUILT = '97_built',
BUILTFAILED = '98_builtfailed',
SUBMITTING = '89_submitting',
SUBMITTED = '88_submitted',
SUBMITFAILED = '87_submitfailed',
RUNNING = '78_running',
RETRYING = '79_retrying',
DONE = '67_done',
DONECACHED = '66_donecached',
DONEFAILED = '68_donefailed',
ENDFAILED = '69_endfailed',
KILLING = '59_killing',
KILLED = '57_killed',
KILLFAILED = '58_killfailed',
)
But if the progress bar belongs to a job (shown when a job is submitted or done), the status of the job has the highest priority. So in the above example, if the progress bar belongs to job #1:
[JOBDONE] [1/9] [>----] Done: x.x% | Running: x
^ Indicating current job (">" for running)
=
.