Team is a required top-level organizational and authorization construct.
Folder is an optional organizational structure to hold
Schema is required by Horreum to define the meta-data associated with a
It allows Horreum to process the JSON content to provide validation, charting, and change detection
Schema defines the following;
- An optional expected structure of a Dataset via JSON validation schemas
Labelsthat define how to use the data in the JSON document
Transformers, to transform uploaded JSON documents into one or more datasets.
A Schema can apply to an entire Run JSON document, or parts of a Run JSON document
Label is required to define how metrics are extracted from the JSON document and processed by Horreum.
There are 2 types of Labels:
Metrics Label: Describes a metric to be used for analysis, e.g. “Max Throughput”, “process RSS”, “startup time” etc
Filtering Label: Describes a value that can be used for filtering
Datasetsand ensuring that datasets are comparable, e.g. “Cluster Node Count”, “Product version” etc
Label can be defined as either a
Metrics label, a
Filtering label, or both.
Filtering Labels are combined into
Fingerprints that uniquely identify comparable Datasets within uploaded Runs.
Extractor is a required JSONPath expression that refers to a section of an uploaded JSON document. An Extractor can return on of;
- A scalar value
- An array of values
- A subsection of the uploaded JSON document
NoteIn the majority of cases, an Extractor will simply point to a single, scalar value
NoteIn the majority of cases, the Combination Function is simply an Identity function with a single input and does not need to be defined
You can think of a `test`` as a repo for the results of a particular benchmark, i.e. a benchmark performs a certain set of actions against a system under test
Runs can have different configurations, making them not always directly comparable, but the Run results stored under one Test can be filtered by their
Fingerprint to ensure that all
Datasets used for analysis are comparable
Run is a particular single upload instance of a
Run is associated with one or more
Schemas in order to define what data to expect, and how to process the JSON document
Transformers are typically used to;
- Restructure the JSON document. This is useful where users are processing JSON documents that they do not have control of the structure, and the structure is not well defined
- Split a
RunJSON output into multiple, non-comparable
Datasets. This is useful where a benchmark iterates over a configuration and produces a JSON output that contains multiple results for different configurations
Schema can have 0, 1 or multiple
NoteIn the majority of cases, the
Rundata does not need to be transformed and there is a one-to-one direct mapping between
Dataset. In this instance, an
Identity Transformeris used and does not need to be defined by the user
NoteIn most cases, there is a 1:1 relationship between a
Dataset, when the
Datasetis expected to have one unified set of results to be analyzed together
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.