Skip to content

Change log

3.1.1 (2020.5.10)

  • Fix preset default still overwrite runtime config

3.1.0 (2020.4.22)

  • Fix error when keys in args are not strings
  • Fix utils.filesig when dead link found in a directory
  • Try to keep the basename for input files if they are symbolic links
  • Fix ppldir being overwritten by runtime_config if it is set explictly
  • Use textwrap.dedent for scripts
  • Add repr for ProcSet
  • Adopt latest python-simpleconf, python-varname and liquidpy

3.0.3 (2020.1.22)

  • Make better alignment in runner script file
  • Refine "pyppl profile" command
  • Add force cache mode

3.0.2 (2020.1.15)

  • Add status for plugin API proc_postrun
  • Copy input keys while copying a process
  • Fix extra empty lines in log
  • Add check for threading resource availablility
  • Add full traceback for proc definition
  • Fix default runner config not loaded if running with different runner profile

3.0.1 (2019.12.31)

  • Fix job.data.i.x not pointing to files in <indir>
  • Remove unnecessary job attribute from building __eq__
  • Fix channel buliding for procs
  • Fix _anything2procs not using procset argument when a list of ProcSets passed
  • Change runner DEFAULT_POLL_INTERVAL to 1 sec
  • Fix return value of kill of local runner (must be a bool value).
  • Fix attr.ib properties not copied during proc coping
  • Add proc props for plugins
  • Move timer to pyppl_rich plugin
  • Compute input, output and script for process before jobs created, as they are not thread-safe.

3.0.0 (2019.12.26)

  • Fundamental:
    • Rewrite Proc and Job using attrs and attr_property
  • Plugins:
    • Refine plugin system
    • Move some functions to plugins
    • Re-organize log levels and enable plugins to add levels
  • Runners:
    • Make runners also plugins
  • Other:
    • Optimize and fix other bugs

2.3.2 (2019.12.5)

  • Infer job cache status when not job.cache file generated (main thread quits before job done)
  • Remove force cache mode
  • Use diot instead of python-box

2.3.1 (2019.11.29)

  • Fix global config.args been overwritten when multiple pipelines in one session
  • Wrap long process descriptions in log
  • Fix kwargs for Proc constructor not triggering __setattr__

2.3.0 (2019.11.18)

  • Add check for output names
  • Fix deepcopy problem in proc.copy
  • Remove template functions, as they are not necessary for the framework itself
  • Add keys in kwargs of Proc construct in sets
  • Fix other bugs

2.2.0 (2019.10.23)

  • Add forced cache #78
  • Add CLI hooks for plugins
  • Fix job not submitting while retrying

2.1.4 (2019.10.11)

  • Optimize CLI "pyppl list/clean" to show results immediately instead of loading all processes.
  • Fix CLI option "-before" validation.
  • Fix error when some objects cannot be deeply copied while coping a process.
  • Fix while @ in job names for sge runner.
  • Use TEMPDIR instead for _fsutil to avoid creating a subdirectory under it threading-unsafely.
  • Enhance if some SGE fails to generate stdout/err file.

2.1.3 (2019.8.21)

  • Rename plugins from pyppl- to pyppl_ due to inability of dashes in entry_point

2.1.2 (2019.8.19)

  • Save 'var' type of input as string to avoid error when reading it
  • Report plugin version in logger

2.1.1 (2019.8.8)

  • Drop proc.tplenvs, use proc.envs instead

2.1.0 (2019.8.5)

  • Add plugin system
  • Move flowchart and report as separate plugins
  • Add template function glob1

2.0.0 (2019.6.28)

  • Change versioning back to semantic
  • Drop support for python <= 3.5
  • Adopt more of 3rd-party packages
  • Use poetry to manage dependencies and publish package
  • Rewrite runner system and make it easier to write a custom runner
  • Rewrite logger system
  • Migrate bin/pyppl to pyppl/console.py
  • Use a state machine to manage job states
  • Rewrite job queue strategy
  • Adopt pytest for testing
  • Change Aggr to ProcSet and rewrite some methods
  • Show a more intuitive example in README
  • Move docs to readthedocs
  • Optimize Job class to save memories
  • Fix other bugs

2019.2.20

  • Use date as version naming.
  • Adopt colorama for teminal colors.
  • Allow configs to update recursively.
  • Remove python 3.3 support.
  • Fix #48: unicode loaded when config file is in json

1.4.3 (2019.1.28)

  • Change path shortening cutoff to total length.
  • Fix a bug for template function fn (filename/stem).
  • Fix logger formatter conflicts.

1.4.2 (2019.1.23)

  • Add runner subcommand for bin/pyppl to show available runners.
  • Allow length of the progress bar configurable.
  • Allow the paths to be shortened in logs.
  • Fix other bugs.

1.4.1 (2019.1.9)

  • Add render function for template to allow variables to be rendered in template.
  • Allow reset for list parameter in parameters
  • Add callback for Parameter

1.4.0 (2018.12.20)

  • Implement jobmgr with PriorityQueue, and put job building, submission and polling in the same queue
  • Remove file locks while job initiation to speed up
  • Other bug fixes.

1.3.1 (2018.11.8)

  • Fix pipeline not quitting if retry times reached.
  • Optimize CPU usage.

1.3.0 (2018.11.2)

  • Use threading instead of multiprocess to save resources and rewrite queue system
  • Change cclean to acache (after cache)
  • Change nthread to nsub (# threads used to submit jobs)
  • Remove six dependency
  • Cleanup pipeline when halt (Ctrl-c hit or error encountered while errhow = halt)

1.2.0 (2018.9.28)

  • Use liquidpy as default template engine (see how to migrate in FAQ).
  • Fix other bugs.

1.1.2 (2018.9.13)

  • Optimize check server alive for ssh runner.
  • Rwrite command line tool: bin/pyppl
  • Add commands to allow subcommand in command line argument parser.
  • Allow params to parse arguments arbitrarily.
  • Allow help information to be interpolated from outsite for params.
  • Add alias support for params and commands.
  • Rewrite API generator and regenerate API docs.
  • Fix other bugs.

1.1.1 (2018.8.30)

  • Allow progress bar to be collapsed in stream log.
  • Remove loky dependency, so PyPPL could run on cygwin/msys2.
  • Add ~/pyppl.yml, ~/pyppl.yaml, ~/pyppl and ~/pyppl.json to default configuration files.
  • Update docs.
  • Fix other bugs.

1.1.0 (2018.8.20)

  • Let pipeline halt if any job fails (#33).
  • Add KeyboardInterupt handling.
  • Add Warning message when a process is locked (another instance is running)
  • Clean up utils and runners source codes
  • Support and add tests to travis for python2.7 and python3.3+ for both OSX and Linux.
  • Allow global functions/values to be used in builtin templates.
  • Add shortcut for lambda function in builtin templates.
  • Add nones and transpose method for Channel.

1.0.1 (2018.7.31)

  • Change the default path of flowchart and log from script directory to current directory.
  • Rewrite the way of attribute setting for aggregations.
  • Introduce modules for aggregations.
  • Allow setting attributes from Proc constructor.
  • Implement #33, Ctrl-c now also halts the pipeline, and hides the exception details.

1.0.0 ! (2018.7.10)

  • Fix runner name issue #31.
  • Use mkdocs to generate documentations and host them on GitHub pages.
  • Keep stdout and stderr when a job is cached: #30.
  • Allow command line arguments to overwrite Parameter's type.
  • Host the testing procedures with Travis.
  • Fix other bugs.

0.9.6 (2018.6.8)

  • Auto-delegate common proc config names to aggr
  • Add proc.origin to save the original proc id for copied procs
  • Remove brings, add proc.infile to swith '{{in.(infile)}}' to job.indir path, original path or realpath of the input file
  • Add process lock to avoid the same processes run simultaneously
  • Use built-in Box instead of box.Box
  • Merge template function Rvec and Rlist into R, add repr
  • Fix #29 and #31, and fix other bugs

0.9.5 (2018.3.6)

  • Add proc.dirsig to disable/enable calculating signatures from deep directories
  • Add Jobmgr class to handle job distribution
  • Allow channel.rowAt and colAt to return multiple rows and columns, respectively
  • Allow empty channel as input (process will be skipped)
  • Refine all tests
  • Rewrite thread-safe file system helper functions
  • Add specific exception classes
  • Report line # when error happens in template
  • Add progress bar for jobs
  • Allow stdout and stderr file as output

0.9.4 (2017.12.27)

  • Add yaml support for config file (#26).
  • Allow empty list for input files.
  • Merge continuous job ids in log (Make the list shorter).
  • More details when internal template failed to render (#25)
  • Ignore .yaml config files if yaml module is not installed.
  • sleep before testing isRunning to avoid all jobs running it at the same time.
  • Use repr to output p.args and p.props.
  • Merge Proc attributes profile and runner. Profile is now an alias of runner, and will be removed finally.

0.9.3 (2017.11.20)

  • Beautify parameters help page.
  • Enable multithreading for job construction and cache checking (set by proc.nthread).
  • Uniform multiprocessing/threading.
  • Fix Aggr delegate problems.
  • Add ProcTree to manage process relations.
  • Report processes will not run due to prior processes not ran.
  • Add cclean option for enable/disable cleanup (output check/export) if a job is cached.
  • Add tooltip for flowchart svg.
  • Fix job id not saved for runner_sge.
  • Fix resume assignment issue.
  • Rewrite proc.log function so that logs from jobs do not mess up when they are multithreaded.
  • Fix params loaded from file get overwriten.
  • Add coverage report.

0.9.2 (2017.10.23)

  • Add profile for Proc so different procs can run with different profiles.
  • Add delegate for Aggr.
  • Add get, repCol, repRow, rowAt method for Channel.
  • Dont't sleep for batch interval if jobs are cached or running.
  • Add header argument for Channel.fromFile.
  • Fix a bunch of bugs.

0.9.1 (2017.10.6)

  • Fix issues reported by codacy
  • Fix an issue checking whether output files generated
  • Deepcopy args and tplenvs when copy a process
  • Refer relative path in p.script (with "file:" prefix) where p.script is defined
  • Fix a utils.dictUpdate bug
  • Template function 'Rlist' now can deal with python list
  • Add log for export using move method (previously missed)
  • Allow Aggr instance to set input directly
  • Switch default showing of parameters loaded from object or file to False
  • Optimize utils.varname
  • Add warning if more input data columns than number of input keys
  • Fix output channel key sequence does not keep
  • Use object id instead of name as key for PyPPL nexts/paths in case tag is set in pipeline configrations

0.9.0 (2017.9.22)

  • Change class name with first letter capitalized
  • Add resuming from processes (#20)
  • Fix #19
  • Group log configuration
  • Make flowchart themeable and configurable
  • Make attributes of Proc clearer
  • Add tutorials
  • Make tests more robust
  • Enhancer templating, support Jinja2
  • Set attributes of processes in aggregation with set

0.8.1 (2017.8.4)

  • Add partial echo
  • Add interactive log from the script
  • Add partial export
  • Add log themes and filters
  • Add compare to command line tool
  • Fix bugs

0.8.0 (2017.8.1)

  • Add slurm and dry runner
  • Fix bugs when more than 2 input files have same basename
  • Add indent mark for script, specially useful for python
  • Make stdout/stderr flushes out for instant runners when p.echo = True
  • Add sortby, reverse for channel.fromPath
  • Add command line argument parse
  • Fix a bug that threads do not exit after process is done

0.7.4 (2017.7.18)

  • Docs updated (thanks @marchon for some grammer corrections)
  • Some shortcut functions for placeholders added
  • Check running during polling removed
  • Logfile added
  • p.args['key'] can be set also by p.args.key now
  • Bug fixes

0.7.3 (2017.7.3)

  • Config file defaults to ~/.pyppl.json (~/.pyppl also works)
  • Callfront added
  • Empty input allowed
  • Same basename name allowed for input files of a job
  • Description of a proc added
  • Aggr Optimized
  • Bug #9 Fixed
  • Private key supported for ssh runner
  • Feature #7 Implemented

0.7.2 (2017.6.20)

  • Optimize isRunning function (using specific job id)
  • Support python3 now
  • Test on OSX
  • More debug information for caching
  • Bug fixes

0.7.1 (2017.6.15)

  • Move pyppl-cli to bin/pyppl
  • channel.collapse now return the most common directory of paths
  • Report oringinal file of input and bring files
  • Show number of omitted logs
  • Bug fixes

0.7.0 (2017.6.13)

  • Add colored log
  • Put jobs in different directories (files with same basename can be used as input files, otherwise it will be overwritten).
  • Add configuration checkrun for pyppl allow runner.isRunning to be disabled (save resources on local machine).
  • Add built-in functions for placeholders; lambda functions do not need to call (just define)
  • File placeholders (.fn, .bn, .prefix, etc) removed, please use built-in functions instead.
  • Add an alias p.ppldir for p.tmpdir to avoid confusion.
  • Update command line tool accordingly
  • Split base runner class into two.

0.6.2 (2017.5.30)

  • Update docs and fix compilation errors from gitbook
  • Change pyppl.dot to pyppl.pyppl.dot;
  • Add channel.fromFile method;
  • Add aggr.addProc method;
  • Fix proc/aggr copy bugs;
  • Fix utils.varname bugs;
  • Fix bugs: channel._tuplize does not change list to tuple any more.
  • Add fold/unfold to channel;
  • cache job immediately after it's done;
  • remove proc in nexts of its depends when its depends are reset;
  • add dir for input files, prefix for output files;
  • Fix utilfs.dirmtime if file not exists;
  • add pyppl-cli;
  • Change rc code, make it consistent with real rc code.

0.6.1 (2017.4.27)

  • Overwrite input file if it exists and not the same file;
  • fix varname bug when there are dots in the function name;
  • Add brings feature;
  • Add features to README, and brings to docs

0.6.0 (2017.4.26)

  • Set job signature to False if any of the item is False (that means expected files not exists); - Do cache by job itself;
  • Make it possible to cache and export successful jobs even when some jobs failed
  • Host docs in gitbook
  • Init job with log func from proc;
  • Add docstring for API generation;
  • Redefine return code for outfile not generated;
  • Error ignore works now;
  • Rewrite runner_local so it fits other runners to extend;
  • Fix proc depends on mixed list of procs and aggrs

0.5.0 (2017.4.18)

  • Fix local runner not waiting (continuiously submitting jobs);
  • Add property alias for aggr;
  • Output cleared if job not cached
  • Fix bugs when export if outfiles are links;
  • change default export method to move;
  • add id and tag to calculate suffix for proc;
  • add timer;
  • add isRunning for job so that even if the main thread quit, we can still retrieve the job status;

0.4.0 (2017.4.13)

  • Add files (array) support for input;
  • Recursive update for configuration;
  • Add aggregations;
  • Move functions to utils;
  • Separate run for runners to submit and wait;
  • Add use job class for jobs in a proc;
  • Use "1,2 3,4" for channel.fromArgs for multi-width channels;
  • Add rbind, cbind, slice for channel;
  • Add alias for some proc properties;
  • Remove callfront for proc;
  • Add export cache mode;
  • Add gzip export support (#1);
  • Unify loggers;
  • Use job cache instead of proc cache so that a proc can be partly cached;
  • Rewrite buildInput and buildOutput;
  • Use job to construct runners;

0.2.0 (2017.3.14)

  • Basic functions

Initiate (2017.1.27)