A fork of logstash-config with a new transpile command that converts Logstash Pipelines into Elasticsearch Ingest Pipeline syntax.
baffo provides a Go parser for Logstash configuration files, based on the original Logstash Treetop grammar.
It uses pigeon to generate the parser from a PEG (Parsing Expression Grammar).
This fork adds the transpile command, allowing you to convert existing Logstash pipelines into Elasticsearch ingest pipelines — no temporary files needed.
⚠️ This package is under active development. API stability is not guaranteed.
go install github.com/herrBez/baffo/cmd/baffo@latestThe word baffo means moustache in Italian, chosen to clearly indicate that this project is a fork of the original Mustache. The original name is inspired by the original Logstash Logo (wooden character with an eye-catching mustache).
baffo is a CLI tool to check, lint, format, and transpile Logstash configuration files.
The transpile command transpiles a Logstash Pipelines to one or more Elasticsearch Ingest Pipelines:
baffo transpile file.confFor the transpilation we have different flags at disposal:
add_default_global_on_failure: whether to add a default global on failure processordeal_with_error_locally: whether to deal with the errors locally à là Logstash (e.g., by adding the tag on error by default)fidelity: whether we want to keep the correct the if-else semantic, i.e., calculating the condition only oncepipeline_threshold: determine how many processors will cause the creation of a new pipeline when converting if-else statementsadd_cleanup_processor: whether we add a final remove processor to remove temporary fields created by the transpiler (and the@metadatafield)inline: whether the positional arguments are the script or a path to the script (default)
By default, we try to keep the semantics as close as possible with the original Logstash Pipeline. To obtain idiomatic pipelines, consider using the following settings:
baffo transpile file.conf \
--deal_with_error_locally=false \
--pipeline_threshold=10 \
--add_default_global_on_failure=true \
--fidelity=false
⚠️ Disclaimer: Semantic equivalence between the input Logstash pipelines and the generated Elasticsearch ingest pipelines is not formally guaranteed. The output should not be used in production without careful review and testing.
If you need a quick conversion without bothering creating a file you can:
baffo transpile /dev/stdin --deal_with_error_locally=false \
--pipeline_threshold 10 \
--add_default_global_on_failure=true \
--fidelity=false \
--inline=true \
'filter{ mutate { add_field => {"[test]" => "foo"}}}'
The program outputs a JSON Dictionary with one entry per generated pipeline.
The main-pipeline is called main-pipeline-<filename>.
You can use tools like yq and jq to manipulate them.
More Details To get more details about the plugins that are currently supported for conversions we refer to the dedicated documentation.
Testsuite To verify that the Elasticsearch ingest pipelines generated from Logstash pipelines behave as expected, we use the Baffo Testsuite. This testsuite provides a collection of sample (Logstash) pipelines and automated checks to assess semantic equivalence between the original Logstash configuration and the transpiled Elasticsearch pipelines.
The check command verifies the syntax of Logstash configuration files:
baffo check file.confThe lint command checks for problems in Logstash configuration files.
The following checks are performed:
- Valid Logstash configuration file syntax
- No comments in exceptional places (these are comments, that are valid by the Logstash configuration file syntax, but but are located in exceptional or uncommon locations)
- Precence of an
idattribute for each plugin in the Logstash configuration
If the --auto-fix-id flag is passed, each plugin gets automatically an ID. Be aware, that this potentially reformats
the Logstash configuration files.
baffo lint --auto-fix-id file.confWith the f
6436
ormat command, baffo returns the provided configuration files in a standardized format (indentation,
location of comments). By default, the reformatted file is print to standard out. If the flag --write-to-source
is provided, the Logstash config files are reformatted in place.
baffo format --write-to-source file.confUse the --help flag to get more information about the usage of the tool.
- Get and install pigeon.
- Run
go generatein the root directory of this repository.
The project is a fork of Logstash Config by Lucas Bremgartner (breml)
This fork adds transpile support for Elasticsearch ingest pipelines.