Fixed issue where pipelines with a dependency would run on a time schedule even if the upstream pipeline didn’t run (and vice versa).
Fixed output of next scheduled pipelines to better reflect DAG structures.
Directed acyclic graph (DAG) pipelines - where the output of one
pipeline can feed into another - are now available using the
maestroOutputs and maestroInputs tags.
Pipelines that input into a downstream pipeline should use the
maestroOutputs tag. Pipelines that receive input from an
upstream pipeline should use the maestroInputs tag
(#98).
New function show_network for visualizing the
connections between pipelines that are connected in a DAG.
MaestroSchedule gains new methods
get_network() (returns a data.frame) and
show_network() (returns a visualization using
{DiagrammeR}).
Added catch-all maestro tag to identify a function
as a pipeline without specifying other configurations.
create_pipeline to allow for
interactive creation of pipelines that default to skip.Fixed issue with suggest_orch_frequency when using
different styles of frequency (e.g., 1 day vs. daily) in a single
schedule.
Fixed issue where pipeline sourcing failures were appearing as successful runs in status outputs.
This version refactors much of the code base to rely on R6 classes for pipelines and schedules. Pay careful attention to the breaking changes to see how existing code may be impacted.
Schedules are now represented as an R6 object of class
<MaestroSchedule>. build_schedule()
returns a MaestroSchedule object that can be passed to
run_schedule() as normal. To access the schedule table run
get_schedule().
run_schedule() no longer returns a list of
$status and $artifacts but now
returns/modifies the MaestroSchedule object. Status can be accessed
using get_status(schedule) and artifacts via
get_artifacts(schedule)
suggest_orch_frequency() now takes a
<MaestroSchedule> object.
Data example_schedule removed from the
package.
Skipped pipelines are no longer shown in the CLI output of
run_schedule().
It is now required that all pipeline names are unique. The names
of each maestro pipeline function must be unique across the project to
support the implementation of DAGs. build_schedule() will
abort if any non-unique names are detected.
Added functions get_schedule(),
get_status(), and get_artifacts() for
interacting with <MaestroSchedule> objects.
Added function invoke() to instantly run a pipeline
in a schedule.
New tags maestroHours, maestroDays, and
maestroMonths allows running of pipelines on specific hours
of day, days of week, days of month, or months of year (#100).
maestroFrequency tag now accepts the values hourly,
daily, weekly, biweekly, monthly, quarterly, and yearly. Argument
orch_frequency to run_schedule() also accepts
these values.
Changed from example_schedule data the pipeline with
a schedule of 1 minute to 30 minutes in keeping with best practices for
minimum pipeline frequency.
suggest_orch_frequency now uses the smallest
interval between any two pipelines (#99).
Error messages on unintentional overwrites from
create_*() functions correctly reference name of path or
directory that was to be overwritten.
Fixed cli output of run_schedule() to not show
skipped pipelines in the next run portion.
Fixed cli output to correctly handle counting of successful runs when pipelines are skipped.
Performance improvements to build_schedule()
(#101).
Creater functions create_pipeline() and
create_maestro no longer have default arguments for the
path to where the scripts are created. Users must explicitly define
these paths.
Argument log_file in run_schedule() no
longer defaults to ./maestro.log but instead defaults to
NULL.
create_* now take a boolean
overwrite argument to make the overwriting of existing
pipelines, projects, and orchestrators more explicit.run_schedule() now returns a list with status and
artifacts instead of just a data.frame of the status. Artifacts are any
values returned from pipelines. Pipelines that return nothing will have
no artifacts.suggest_orch_frequency() to provide
a suggestion of what frequency to use for the orchestrator.run_schedule() now correctly outputs
the total number of pipelines (#81) and correctly outputs number of
errors.maestroFrequency tag now adheres to a more human-readable format like “1 day”, “2 hours”, “4 weeks”, etc.
orch_frequency argument in
run_schedule() also takes more human-readable format
identical to maestroFrequency tag.
maestroInterval tag removed
orch_interval argument to
run_schedule() removed.
create_maestro() and
create_orchestrator() now use the argument
type instead of extension for defining what
script type to use for the orchestrator.
Changed last_parsing_errors() to
last_build_errors(); changed functions of the form
last_runtime_*() to last_run_*().
Additional columns added to the output of
run_schedule(): pipeline_started and
pipeline_ended to indicate the start and end times of a
pipeline execution; next_run to indicate when the next run
should be based on the frequency of the pipeline and
orchestrator.
Pipelines now show as skipped if they are not scheduled.
Added hex logo
Backend improvements to schedule checking
Timestamps are formatted to specified time zone.
run_schedule() cli output suggests to use
last_run_errors() or last_run_warnings() if
any errors or warnings were found.