pax_global_header00006660000000000000000000000064137014572570014525gustar00rootroot0000000000000052 comment=d71af1b7caf5ef9c65fc84dbf3cf55319afc35b5 trapperkeeper-metrics-1.3.1/000077500000000000000000000000001370145725700160445ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/.gitignore000066400000000000000000000003271370145725700200360ustar00rootroot00000000000000pom.xml pom.xml.asc *jar /lib/ /classes/ /target/ /checkouts/ .lein-deps-sum .lein-repl-history .lein-plugins/ .lein-failures .nrepl-port /resources/locales.clj /dev-resources/i18n/bin /resources/**/Messages*.class trapperkeeper-metrics-1.3.1/.travis.yml000066400000000000000000000002021370145725700201470ustar00rootroot00000000000000language: clojure lein: 2.7.1 matrix: include: - jdk: openjdk8 script: ./ext/travisci/test.sh notifications: email: false trapperkeeper-metrics-1.3.1/CHANGELOG.md000066400000000000000000000122401370145725700176540ustar00rootroot00000000000000## 1.3.0 * [TK-489](https://tickets.puppetlabs.com/browse/TK-489) - Add trapperkeeper-authorization support. See that project for configuration. The jolokia access configuration still defaults to disallowing remote access, so this needs to be overridden by the user if using tk-authorization. ## 1.2.3 * [TK-486](https://tickets.puppetlabs.com/browse/TK-486) - Deprecate the v1 endpoint, so it can be removed in a future release. Going forward all metrics gathering should go through the v2 endpoint, access to which can be flexibly controlled via jolokia access rules. ## 1.2.2 * Publicly published version of 1.2.1. ## 1.2.1 This is an internal-only release. * [PE-28468](https://tickets.puppetlabs.com/browse/PE-28468) - Disable the v1 metrics endpoint and restrict v2 to localhost-only access, to fix a CVE where sensitive Puppet data could appear in metrics names. ## 1.2.0 * Update Jolokia to 1.6.0. This was a security release of Jolokia, though we do not use the affected features of Jolokia it is still recommended to upgrade. ## 1.1.0 * [TK-442](https://tickets.puppetlabs.com/browse/TK-442) - Updates dropwizard from 3.1.2 to 3.2.2. NOTE: This is done by bumping the dependency on clj-parent which also brings in a new version of tk-jetty9. Java 7 is no longer supported. ## 1.0.0 This is a major backwards incompatible feature release. The major new feature is support for exporting to graphite. See the [documentation](documentation/configuration.md) for details about how to enable graphite exporting. The breaking changes are: * The `metrics` section of the configuration has changed to allow configuration of jmx and graphite reporters per domain instead of globally. See the [documentation](documentation/configuration.md) for details about the new format. * Domains must be specified as keywords instead of strings in the clojure API. * `initialize-registry-settings` has been renamed to `update-registry-settings` Additional changes: * Allow multiple services to register `default-metrics-allowed` lists through the `update-registry-settings` service function to limit the number of metrics sent to graphite Jira Links: * [TK-361](https://tickets.puppetlabs.com/browse/TK-361) - Move pe-trapperkeeper-metrics code, including graphite exporting, into OSS trapperkeeper-metrics. * [TK-393](https://tickets.puppetlabs.com/browse/TK-393) - Make jmx enabled per-registry. Deprecate enabling jmx globally * [TK-394](https://tickets.puppetlabs.com/browse/TK-394) - Only accept metrics domain as keyword instead of string * [TK-436](https://tickets.puppetlabs.com/browse/TK-436) - Add support for appending registry settings from multiple services ## 0.6.0 This is a feature release. * [TK-433](https://tickets.puppetlabs.com/browse/TK-433) - Add a new `get-server-id` services function. * [TK-430](https://tickets.puppetlabs.com/browse/TK-430) - Bump clj-parent to 0.3.3 and i18n to 0.6.0 ## 0.5.0 This is a feature release that also contains bug fixes. * [TK-404](https://tickets.puppetlabs.com/browse/TK-404) - Added a new "v2" API that is based on the [Jolokia](https://jolokia.org) library. Jolokia provides a superset of functionality found in the current "v1" metrics API. Refer to 'document/configuration.md' for more information on setting up Jolokia with Trapperkeeper. The [Jolokia Documentation](https://jolokia.org/reference/html/protocol.html) has the specifics on the new metrics API. * [TK-427](https://tickets.puppetlabs.com/browse/TK-427) - Ensure the JSON request is fully parsed before closing the input stream of the request * [TK-394](https://tickets.puppetlabs.com/browse/TK-394) - Tolerate either keywords or strings for metric domains * Remove explicit ring-core, servlet-api, and slf4j-api deps * Switch to using lein parent (and puppetlabs/clj-parent) for most dependency versions, which also now requires Leiningen 2.7.1 or greater ## 0.4.2 This is a bug fix release. * Don't require JMX to be enabled for the metrics endpoint to work. Now, if the `metrics-webservice` is added to the config, the `/metrics` endpoint will always be registered. ## 0.4.1 This is a maintenance release. * Bump puppetlabs/ring-middleware dependency from 0.3.1 to 1.0.0. ## 0.4.0 This is a feature release. * Add an `initialize-registry-settings` function to the MetricsService protocol. The implementation of this function in the trapperkeeper-metrics service in this repo is not yet implemented and currently just throws an error. ## 0.3.0 This is a minor feature, maintenance, and bugfix release. * Introduce i18n library and lay groundwork for future i18n work * Update project.clj to prefer explicit dependencies instead of implicit transitive dependencies to resolve conflicts * Extract common ring utils into puppetlabs/ring-middleware ## 0.2.0 This is a feature release. * Add the ability to configure multiple metrics registries * Add a new metrics-server service (ported from the PuppetDB `/metrics` API) for querying JMX metrics ## 0.1.2 This is a feature release. * [TK-252](https://tickets.puppetlabs.com/browse/TK-252) Always build a metrics registry, deprecate metrics.enabled setting * Add `mean-millis` and related utility fns trapperkeeper-metrics-1.3.1/CODEOWNERS000066400000000000000000000002621370145725700174370ustar00rootroot00000000000000# This will cause the puppetserver-maintainers group to be assigned # review of any opened PRs against the branches containing this file. * @puppetlabs/puppetserver-maintainers trapperkeeper-metrics-1.3.1/CONTRIBUTING.md000066400000000000000000000010241370145725700202720ustar00rootroot00000000000000# How to contribute Third-party patches are essential for keeping Puppet Labs open-source projects great. We want to keep it as easy as possible to contribute changes that allow you to get the most out of our projects. There are a few guidelines that we need contributors to follow so that we can have a chance of keeping on top of things. For more info, see our canonical guide to contributing: [https://github.com/puppetlabs/puppet/blob/master/CONTRIBUTING.md](https://github.com/puppetlabs/puppet/blob/master/CONTRIBUTING.md) trapperkeeper-metrics-1.3.1/LICENSE000066400000000000000000000014011370145725700170450ustar00rootroot00000000000000 Trapperkeeper Metrics - A library to help make it easier to track metrics in other Trapperkeeper applications Copyright (C) 2005-2015 Puppet Labs Inc Puppet Labs can be contacted at: info@puppetlabs.com Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. trapperkeeper-metrics-1.3.1/Makefile000066400000000000000000000000441370145725700175020ustar00rootroot00000000000000include dev-resources/Makefile.i18n trapperkeeper-metrics-1.3.1/README.md000066400000000000000000000043571370145725700173340ustar00rootroot00000000000000# `trapperkeeper-metrics` [![Build Status](https://travis-ci.org/puppetlabs/trapperkeeper-metrics.svg?branch=master)](https://travis-ci.org/puppetlabs/trapperkeeper-metrics) [![Clojars Project](http://clojars.org/puppetlabs/trapperkeeper-metrics/latest-version.svg)](http://clojars.org/puppetlabs/trapperkeeper-metrics) `trapperkeeper-metrics` is a library intended to help make it easier to track metrics in other Trapperkeeper applications. It includes: * a TK service that manages the life cycle of your metrics registry * config-driven control of metrics and metrics reporting * other miscellaneous utility functions for working with metrics * support for configuring metrics to be reported to Graphite * support for filtering metrics before reporting them to Graphite For more detailed information (what this library does and doesn't do, more detailed tips on how to write code against it, future plans, etc.), check out the [documentation](./documentation/index.md). See the [configuration docs](./documentation/configuration.md) for info on how to configure the service, including Graphite reporting. See the [api docs](./documentation/api.md) for service API documentation. See the [metrics reporting and filtering](./documentation/metrics_reporting_and_filtering.md) for information on how metrics filtering works. ## Example code One example, a Trapperkeeper service which uses the metrics service in conjunction with a Ring handler, is included with this project ([source code](./examples/ring_app/README.md)). ## HTTP Metrics with `comidi` To get the most value out of this library, use it in concert with [comidi](https://github.com/puppetlabs/comidi) and [trapperkeeper-comidi-metrics](https://github.com/puppetlabs/trapperkeeper-comidi-metrics) (to take advantage of the built-in HTTP metrics; see the trapperkeeper-comidi-metrics docs) and the [Trapperkeeper Status Service](https://github.com/puppetlabs/trapperkeeper-status) (to expose the most useful metrics data from your app via HTTP). The `trapperkeeper-comidi-metrics` repo contains a working example app that illustrates how to tie everything together. #Support Please log tickets and issues at our [Jira Tracker](https://tickets.puppetlabs.com/issues/?jql=project%20%3D%20Trapperkeeper). trapperkeeper-metrics-1.3.1/dev-resources/000077500000000000000000000000001370145725700206325ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/dev-resources/Makefile.i18n000066400000000000000000000130201370145725700230440ustar00rootroot00000000000000# -*- Makefile -*- # This file was generated by the i18n leiningen plugin # Do not edit this file; it will be overwritten the next time you run # lein i18n init # # The locale in which our messages are written, and for which we therefore # have messages without any further effort MESSAGE_LOCALE=en # The name of the package into which the translations bundle will be placed BUNDLE=puppetlabs.trapperkeeper_metrics # The list of names of packages covered by the translation bundle; # by default it contains a single package - the same where the translations # bundle itself is placed - but this can be overridden - preferably in # the top level Makefile PACKAGES?=$(BUNDLE) LOCALES=$(basename $(notdir $(wildcard locales/*.po))) BUNDLE_DIR=$(subst .,/,$(BUNDLE)) BUNDLE_FILES=$(patsubst %,resources/$(BUNDLE_DIR)/Messages_%.class,$(MESSAGE_LOCALE) $(LOCALES)) FIND_SOURCES=find src -name \*.clj # xgettext before 0.19 does not understand --add-location=file. Even CentOS # 7 ships with an older gettext. We will therefore generate full location # info on those systems, and only file names where xgettext supports it LOC_OPT=$(shell xgettext --add-location=file -f - /dev/null 2>&1 && echo --add-location=file || echo --add-location) LOCALES_CLJ=resources/locales.clj define LOCALES_CLJ_CONTENTS { :locales #{$(patsubst %,"%",$(MESSAGE_LOCALE) $(LOCALES))} :packages [$(patsubst %,"%",$(PACKAGES))] :bundle $(patsubst %,"%",$(BUNDLE).Messages) } endef export LOCALES_CLJ_CONTENTS i18n: msgfmt # Update locales/messages.pot update-pot: locales/messages.pot locales/messages.pot: $(shell $(FIND_SOURCES)) | locales @tmp=$$(mktemp $@.tmp.XXXX); \ $(FIND_SOURCES) \ | xgettext --from-code=UTF-8 --language=lisp \ --copyright-holder='Puppet ' \ --package-name="$(BUNDLE)" \ --package-version="$(BUNDLE_VERSION)" \ --msgid-bugs-address="docs@puppet.com" \ -k \ -kmark:1 -ki18n/mark:1 \ -ktrs:1 -ki18n/trs:1 \ -ktru:1 -ki18n/tru:1 \ -ktrun:1,2 -ki18n/trun:1,2 \ -ktrsn:1,2 -ki18n/trsn:1,2 \ $(LOC_OPT) \ --add-comments --sort-by-file \ -o $$tmp -f -; \ sed -i.bak -e 's/charset=CHARSET/charset=UTF-8/' $$tmp; \ sed -i.bak -e 's/POT-Creation-Date: [^\\]*/POT-Creation-Date: /' $$tmp; \ rm -f $$tmp.bak; \ if ! diff -q -I POT-Creation-Date $$tmp $@ >/dev/null 2>&1; then \ mv $$tmp $@; \ else \ rm $$tmp; touch $@; \ fi # Run msgfmt over all .po files to generate Java resource bundles # and create the locales.clj file msgfmt: $(BUNDLE_FILES) $(LOCALES_CLJ) # force rebuild of locales.clj if its contents is not the # the desired one ifneq ($(shell cat $(LOCALES_CLJ) 2> /dev/null),$(shell echo '$(subst ','\'',$(LOCALES_CLJ_CONTENTS))')) .PHONY: $(LOCALES_CLJ) endif $(LOCALES_CLJ): | resources @echo "Writing $@" @echo "$$LOCALES_CLJ_CONTENTS" > $@ resources/$(BUNDLE_DIR)/Messages_%.class: locales/%.po | resources msgfmt --java2 -d resources -r $(BUNDLE).Messages -l $(*F) $< resources/$(BUNDLE_DIR)/Messages_$(MESSAGE_LOCALE).class: locales/messages.pot | resources msgfmt --java2 -d resources -r $(BUNDLE).Messages -l $(MESSAGE_LOCALE) $< # Use this to initialize translations. Updating the PO files is done # automatically through a CI job that utilizes the scripts in the project's # `bin` file, which themselves come from the `clj-i18n` project. locales/%.po: | locales @if [ ! -f $@ ]; then \ touch $@ && msginit --no-translator -l $(*F) -o $@ -i $<; \ fi resources locales: @mkdir $@ help: $(info $(HELP)) @echo .PHONY: help define HELP This Makefile assists in handling i18n related tasks during development. Files that need to be checked into source control are put into the locales/ directory. They are locales/messages.pot - the POT file generated by 'make update-pot' locales/$$LANG.po - the translations for $$LANG Only the $$LANG.po files should be edited manually; this is usually done by translators. You can use the following targets: i18n: refresh all the files in locales/ and recompile resources update-pot: extract strings and update locales/messages.pot locales/LANG.po: create translations for LANG msgfmt: compile the translations into Java classes; this step is needed to make translations available to the Clojure code and produces Java class files in resources/ endef # @todo lutter 2015-04-20: for projects that use libraries with their own # translation, we need to combine all their translations into one big po # file and then run msgfmt over that so that we only have to deal with one # resource bundle trapperkeeper-metrics-1.3.1/dev-resources/logback-test.xml000066400000000000000000000005401370145725700237320ustar00rootroot00000000000000 %d %-5p [%c{2}] %m%n trapperkeeper-metrics-1.3.1/dev-resources/puppetlabs/000077500000000000000000000000001370145725700230115ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/dev-resources/puppetlabs/trapperkeeper/000077500000000000000000000000001370145725700256625ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/dev-resources/puppetlabs/trapperkeeper/services/000077500000000000000000000000001370145725700275055ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/dev-resources/puppetlabs/trapperkeeper/services/metrics/000077500000000000000000000000001370145725700311535ustar00rootroot00000000000000metrics_service_test/000077500000000000000000000000001370145725700353215ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/dev-resources/puppetlabs/trapperkeeper/services/metricsjolokia-access-permissive.xml000066400000000000000000000003221370145725700431130ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/dev-resources/puppetlabs/trapperkeeper/services/metrics/metrics_service_test trapperkeeper-metrics-1.3.1/dev-resources/ssl/000077500000000000000000000000001370145725700214335ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/dev-resources/ssl/ca.pem000066400000000000000000000041531370145725700225240ustar00rootroot00000000000000-----BEGIN CERTIFICATE----- MIIGCzCCA/OgAwIBAgIBATANBgkqhkiG9w0BAQsFADBVMVMwUQYDVQQDDEpQdXBw ZXQgRW50ZXJwcmlzZSBDQSBnZW5lcmF0ZWQgb24gY2VudG9zNy52bSBhdCArMjAx OC0wMS0wMiAxOTo0NzozMyArMDAwMDAeFw0xODAxMDExOTQ4MThaFw0yMzAxMDEx OTQ4MThaMFUxUzBRBgNVBAMMSlB1cHBldCBFbnRlcnByaXNlIENBIGdlbmVyYXRl ZCBvbiBjZW50b3M3LnZtIGF0ICsyMDE4LTAxLTAyIDE5OjQ3OjMzICswMDAwMIIC IjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA1mZIjkBpXzWY0Etw5jWOSWmv YuoJWxpAcpl9exw/hhMf/4gaqOv1Xa6s200+xAZsohuUk6qyZfg62PGuH9lTcd6o RE8+FB7GYi+Io8xKrkifNAJWHZimA8rBn4lyBtwIfxC3zjEJZo7u7zNwahZc1+kH Byu8jUASnNLcUE3+WSs9tOXSCMY8ewu5FinFi5oNiAULkrxUxgJz/Sahr/AIDiJO Mwfl+1Bj2DS4TZCKZK1zHLw5kxKJjkvdM6YentetsbppwL+YPmLrG/MuB3ftlNqZ G3tuRBrFWgHVKoXkaySyWuOWyvEMiYmsj/3R53eo6fsEQs/co0J3zZo84ydGG+yD /g1BbXk+Gk7nes6Twl6/QVUKq/Q5xgIe+qEL1ufFdV9WWHGNA89Rwm08W+WUu9A6 0bUJRQvzPat3SNq79AYJCEj+bwTU65neXreKswqRnsysKzrPjlOz37QF6+qUHORN M0QiL+H3442+84O/aSk/0YI2fE9X/0ekw97XaRavBp6EX/ltaMdyPVW0iyf26Bpk fans7qajVhyaAmtKUMDdzWxZMG3kVH22+aYbSImwBJaaLqowM2RlCkjs1ui76bui F7B8EmJZLHTC1qt1Sz6fSVVmpLu19vcadqpSHvRE0vYFQPPGqseJwh3A6y4LA3Yo PxlDXBXSiFS6j/JbGUUCAwEAAaOB5TCB4jA3BglghkgBhvhCAQ0EKgwoUHVwcGV0 IFJ1YnkvT3BlblNTTCBJbnRlcm5hbCBDZXJ0aWZpY2F0ZTAOBgNVHQ8BAf8EBAMC AQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUbc/TLNk6zyecUfKnzNX+alop NpcwZwYDVR0jBGAwXqFZpFcwVTFTMFEGA1UEAwxKUHVwcGV0IEVudGVycHJpc2Ug Q0EgZ2VuZXJhdGVkIG9uIGNlbnRvczcudm0gYXQgKzIwMTgtMDEtMDIgMTk6NDc6 MzMgKzAwMDCCAQEwDQYJKoZIhvcNAQELBQADggIBACkWc3RKTYxBP/uP0m88F8OP GdwKJKQFek2MS9nbFKp4c5aErhEPGw5wDCoQvhUUcdD6e8THGiquauZ/xdCNukX/ Q61g05h0rzyAdJLVkREUosQzCPBNnPg7Mnun2w7ImKh79R9RLw5yRCxgLYjs2fua x9g5QawS9RrYAdW9EUgmLtz7cS9+gg2FhdE/xYuQ+gQOq44i3+4U4ZWJO7qmC/SG Aqos6+N8cbufuAnRf8t/84rFMcBSseNsykP/TU0I72A4dytJpBd6I8J/Gizo+b03 xr0QhCtVI8AVL+yhU5LSbmbFgtOYX/mUpDOGxKJgs3LDE9dhVce5RRgj53itSPjU me5pgnBH7fkHkWFj15R2gJgOOYqMW454vKhUEJSb59y3/1eK1nS/JwoXjifSacLJ s0eewrOaX53U0Agdgdyu2hAQgRHwQJgGLv+H3rRyyyx4TcoQhBpnVzE1LwtA0Unn h2zsszRCpcA1vN7yTmyAUyHyHMhXlJLZ6ryH6GSycXJtc4SI15YTGlLmIr12ziSl CmN6+R7mOL/3PNrhwuxDUTMKAL+/Ir6HHSnj0vXnnOohZpHXHPaxsc9Qctso978R dlbk8jv3D28gOrXccwKOnYAwHM8m6VG5ZxGHO1FfwB8YDsS337ajfhX5mePL2UWS sg+BZ6FhveVHc1uEgbTt -----END CERTIFICATE----- trapperkeeper-metrics-1.3.1/dev-resources/ssl/cert.pem000066400000000000000000000037351370145725700231030ustar00rootroot00000000000000-----BEGIN CERTIFICATE----- MIIFoTCCA4mgAwIBAgIBBTANBgkqhkiG9w0BAQsFADBVMVMwUQYDVQQDDEpQdXBw ZXQgRW50ZXJwcmlzZSBDQSBnZW5lcmF0ZWQgb24gY2VudG9zNy52bSBhdCArMjAx OC0wMS0wMiAxOTo0NzozMyArMDAwMDAeFw0xODAxMDEyMTExNDVaFw0yMzAxMDEy MTExNDVaMBQxEjAQBgNVBAMMCWxvY2FsaG9zdDCCAiIwDQYJKoZIhvcNAQEBBQAD ggIPADCCAgoCggIBAK2DUJH5QElXSkEMcuSfhYS4Pf6XKfrGu0z1OKbPlY+3zSbH qIjuArR1fNw7EOk5PJRku0JnQTuzcu7i39FxzDHKnca+m0UjemDi9odzFGRhdEpg NfDLGnLoAkWHs6tRsL24rnc9SySjZc6u+Nf7Jdv+rh6i5qkAby3Bp+lR9zhLBp7m R/LkRQlK15iTXHlfdOuAifDzgtPxTHnwrWzuXvfel49x6yyahWdWnqNKl7e+naKK j32laXxEHIq3FOTDCXZqYOHVWyGVd0cBKnJr1jaycBwZHEeHczHUsJUFyzcnhVGe gOqs5CFc18j+joiQ+zkLXUOBc9RpOkqwZB6OjqdfgIJQdV/ER4LHkT2OZGAEt/e8 tq+x1Pfwey0qiFuet7bgSDwbQ/5ZgBBpCvJPrYI9AiPiFLQcXB+79T2iaa6w7m3E 4VvpbUqFp9p38Fr/EiZ6X0CiPBq4j51FNNH8F0GyxBTO/yzFLEqyhm9+CxEnhlll GmfCd4MNwdI5/3HsAlR+ksJwNxWQqY5PcafrzfT1dySHmxSjZ4aaNb24mpGSln7r qhYRC6SQ+aJBwxi68dgz0UFME1fHv19nZMdiPhvdmLM2BW7a+gp6OFMoM+MmGMSF tgWHNk1aNTTUfv9+OQ11JCX5oikXQYwPLWjDboUixn3VhRUTAjWxzvSvcI8TAgMB AAGjgbwwgbkwNwYJYIZIAYb4QgENBCoMKFB1cHBldCBSdWJ5L09wZW5TU0wgSW50 ZXJuYWwgQ2VydGlmaWNhdGUwDgYDVR0PAQH/BAQDAgWgMCAGA1UdJQEB/wQWMBQG CCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBQBQiU9 MHqKu5IXojRDhPk2oIVgVTAfBgNVHSMEGDAWgBRtz9Ms2TrPJ5xR8qfM1f5qWik2 lzANBgkqhkiG9w0BAQsFAAOCAgEAnEGeANnyRGR/SRAmrCmwlEDH6NdEDecl+FKn JJXeb6z1csK3jp5AHvA9Et3PsFxjj4/CuLKjmdizo2VJJbqtUcSEujn02Z3FYW+q UEqXqFV2zeBcoSrenUizTZHFQV5uV4RbA9nWNC0AixHHVQaaU0Cy7W9s+T+sooFX jphgN0fow1a4jHm0gfmhdpJR01Zl/gCpZh3tIoG8Xz5cVisKPLWCamN4643B3EnE SjxeRPP/WYlwV4LsOVWbM5bNoGceca4LpvL20F2umLOAqfXi8mVlgyoHqVD0muXu 3A9NDRmjx7YlG7e+vxtpgEUJ7tAwTA31j3IwTlFV8L9J/nrVJtrD90y7BQ9CgdKJ UcBCZS/j6Ujc7RMrk9OA59KlkBI9KkwwFzyz6PBMBcieWUZ8bSLHQc799lx8Hqcd hbGXIbES3JWw3QzCiNW/DiejF6+XSfUDji5N1k9n/1dqvP8ir4BwT/K9KKHwwCH6 E+k8RLvfJW/IDkj/d8o1+n5Yp6vBSHUCULlWmS9/Fb6vJZBnIJ9xHVjUiODO4H+Q EiAESrETqdwkCEbjOu3jKquLZzwViqllQWRJm2IHRCaVQXkfzBORdJcofZ0yASYK djSFDzl5rDDtqTk5T6tvieeIZfK09s+0T4akPvqS88zFkcgjUqTRZaRbwM62Zw3P IDIf6HA= -----END CERTIFICATE----- trapperkeeper-metrics-1.3.1/dev-resources/ssl/key.pem000066400000000000000000000062531370145725700227340ustar00rootroot00000000000000-----BEGIN RSA PRIVATE KEY----- MIIJKAIBAAKCAgEArYNQkflASVdKQQxy5J+FhLg9/pcp+sa7TPU4ps+Vj7fNJseo iO4CtHV83DsQ6Tk8lGS7QmdBO7Ny7uLf0XHMMcqdxr6bRSN6YOL2h3MUZGF0SmA1 8MsacugCRYezq1Gwvbiudz1LJKNlzq741/sl2/6uHqLmqQBvLcGn6VH3OEsGnuZH 8uRFCUrXmJNceV9064CJ8POC0/FMefCtbO5e996Xj3HrLJqFZ1aeo0qXt76dooqP faVpfEQcircU5MMJdmpg4dVbIZV3RwEqcmvWNrJwHBkcR4dzMdSwlQXLNyeFUZ6A 6qzkIVzXyP6OiJD7OQtdQ4Fz1Gk6SrBkHo6Op1+AglB1X8RHgseRPY5kYAS397y2 r7HU9/B7LSqIW563tuBIPBtD/lmAEGkK8k+tgj0CI+IUtBxcH7v1PaJprrDubcTh W+ltSoWn2nfwWv8SJnpfQKI8GriPnUU00fwXQbLEFM7/LMUsSrKGb34LESeGWWUa Z8J3gw3B0jn/cewCVH6SwnA3FZCpjk9xp+vN9PV3JIebFKNnhpo1vbiakZKWfuuq FhELpJD5okHDGLrx2DPRQUwTV8e/X2dkx2I+G92YszYFbtr6Cno4Uygz4yYYxIW2 BYc2TVo1NNR+/345DXUkJfmiKRdBjA8taMNuhSLGfdWFFRMCNbHO9K9wjxMCAwEA AQKCAgAVe3aUPBnberU0zIfFCoMLZYw3YxvNcqFp68pxzK2lt4nszTdujJlFIm6t DZsQZQe9D6OnmwUWkScfsHMpiT9Qwtc57gtnSfcy26bOKk9SBkbRlovXHi5OH694 UE7MXTbWpnDjTnFpPdmy9zLT+sI7NJL2NBD+x5D8R0e0uXM0QwW00qoBxoVNmziK KWR/mnPtXGK3tE50UyG2zzeJjxGu+p4lImONiCwQGUdWzDtwgmcd4q3VTo961Sv7 eKNbl1IqmQ0ZqMK5q7myztAkjnldGEaNFAbOTFTUzJJ4QNliIMB+1HL9mx47izUk K3AWNG0GKQQJAZQHrmeLK7+ZkOjUV+aKy8Ow//C0QkSf/EpvqnZVuaZNS17OZIC7 hAc4kg0ZaDlLw3MFw8Y/oz+KxRBH02lYIA19tTgX6urI5ItPMluR4R/VLZvNFdlD 62SbjQxFGneUr4su+O9a0uX8etaCtMYCrQGF8rSN8r7etC884jqHUpFqXJHWY9GS u6+sGhNfXOk9aZp78vUPz3l4FHhJMXTVzZLRBH9NFtkviDPAXaNAhPvAijq6U0Z8 FBKcJ6HULNExbx+fg6WVuoO/XnWX07XAIDGejpd0ZID7ewurHBeUqlANL1vGHzoP LeU1UA9SOSOOl0T5KLDCcr6EBfTIQbmxRuIRq94IQLZnm3RiAQKCAQEA2SOHUx1X WyUTjS6FV7tuSkL78P322K49MgNYFIbmjJmdM3F9Gdzn1OwPgVYFXN7MDTgQvqIU ut1UFH+WID8MWqGZkW1Hh251p2rqYcYJgBJ6bIJuXaFdbhC5L84DVwkjaD6PKKae CYcqbNUxUA9VUp+tkq4aKKW6MRstgNs2yJo6yxRYMDvSdcY0phIioIC7ndm+O625 n4v0yb29C4T1NFcFHVx3WK5K22SWlNhBxJpIbE3GXQFpoE+QePBDzmfrm9/tjWvn Bpg+qLMqdsSHjblMG4NLPDTOjwQ1WKsFucrVOegf59N9twqSaVyeevnA+NudYqMF RDHn0t+CQqtNkwKCAQEAzJEEQXo1mjx9JIZdESotE+8T5poGePuDTNnj9CkQ05hX +h17hvZaArdbyUFC32cCS8TqRKagDXOHGEL9iTo6a6rHTng8F0DQJYfortpG18mG 3VEERX3y/G38z6ZOZdwoKExev2QDBUSPqtEwdBcV9Nk+r6GYD27gGAes0s+OcHgl TYD98Twk1ZJN+JZejVtOXOoXJ8A4RKb04L128kP41ILMploQ3sDHoX2XrbdHdqBi 1MjiF/02jcBoY7aFZqBKl301yl3gap/rXsJuKo1jja/mZyWONEJhGtGjzLYHGsup vDgu3qVzeqgUOpOvMqkyQjaAxQ+aYRc5yAJN1LKogQKCAQEAn0voSDpbTtt5Z3a8 jitL8yEHEbpO8JGLBXjcklqRIo7De+0DOQkN7u+rRFgmIQa2LfB2oFPp31jmCQ+8 xDVcy9SIA99mvV105PiTsF0HtTRZKefaLchm8YtFQGPjoWNr3VsfzVPG1viZUDRr M++7yZypStV9DVrfqIzmqgSoQSyce0WWmPQaJxrcnwTo8FTgzyHh5IhDiF+1mpce 7CeDd/1rHCosyBs9gYxeyB2Kyj/hZ8r6/lo98kTMmVyyceddOwJhlQGXAAiK11px iaYOSBOrWPfOJwIak2Mj+8OjtLSwDwZS+dD3BmS0GqW/AjrdrfP3l1g/UdNG3jXo qx5fDwKCAQAZH8JnDExjCgTQpJPvlL9+kEFgdBBRcYp5uNUPWq/PXQmJFW/jboQ1 izGBsTs2Vt9oUOH41oIXxNWVvDyJSxZ7vLjHwWqX7ITj5yJDv1kuDXk720RsCID8 UcHX+vPr/LMN5/tHBFB+qCIA97o8t80hikDxryYO1y48pUyrg4RZ+deTAkGuDBX2 CQb8ffNU5x7juHhpkGkYvWyb4YknTgPUjc/2ql8iDIk4I0WqtzTWs/IPcjCag6Xc mQH5JSSTKyKeImml1gvG7KcqtzRyk3TjpPPmPS2O64wjkBXPAZgzyUmqbTgMbkOV j+kUHKlKzo72pN0B1t2+CO02zFmkwuCBAoIBAGwmpD6BOR4Ju6vk/oU+RsUmMWbQ 6JqYlpTnkTnBJsf5PeVg4GT06FhfIK8MXrAY7PRRIzI8U5lE8YJU7MnBYG17eyDU yLjiu7uAwMSR+Lu55OkOiFrqziG98MZ+q+P2xsK2yIKHVV7Iw15j/+eSZjRyWzkl x0HdrYPkVu7xj01NzjPKcYnjBTt/zkU6MKhJoWr1vkQ61VpkGFPrjCpizNA3Aj1G vCJeo9Z+zyxlhNnowtKGMRof9ygFycn9iy4uCZrY0YVm1V2MfYQzXbbsGdBF1opO UT6BjLnPl33sEe21Cki3ToM/QPoeJND++4JVkaQIRmJPC3FQjYfnSH7J7xs= -----END RSA PRIVATE KEY----- trapperkeeper-metrics-1.3.1/documentation/000077500000000000000000000000001370145725700207155ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/documentation/api.md000066400000000000000000000042061370145725700220120ustar00rootroot00000000000000# `MetricsService` API The `MetricsService` protocol provides two functions: `get-metrics-registry` and `update-registry-settings`. ### `get-metrics-registry` Takes an optional registry domain as a keyword. Returns the Dropwizard `MetricRegistry` for that domain, which can then be used to register new metrics. If no registry for this domain previously existed, creates the metric registry. If no domain is provided, returns the default registry. Can be called at any time, regardless of if or when `update-registry-settings` has been called. ### `update-registry-settings` Takes a domain and a map of settings, and registers these settings for the domain. Currently, the only available setting is `default-metrics-allowed`, which takes a vector of strings. This function can safely be called multiple times from different Trapperkeeper services. Since the only setting is currently `default-metrics-allowed`, each call to `update-registry-settings` will append the given `default-metrics-allowed` list to the existing list of allowed metrics for that domain. This means that different services can add metrics to the allowed list that are appropriate for that service, so that no one service needs to know the complete list of allowed metrics for the entire TK application Any calls to this function must be called during the `init` phase of the service lifecycle, since these settings are used to initialize registries and reporters during the metrics service's `start` phase. Example usage: ```clojure (trapperkeeper/defservice my-service [[:MetricsService get-metrics-registry update-registry-settings]] (init [this context] (update-registry-settings :foo {:default-metrics-allowed ["foo" "foo.bar"]) ...) (start [this context] (let [registry (get-metrics-registry :foo)] (core/do-something-with-metrics registry)) ...)) ``` ## Registry settings ### `default-metrics-allowed` This setting allows for specifying a list of metric names (as strings) that will be exported to configured reporters (e.g. Graphite - note that this setting does not apply to JMX). See [Metrics Filtering](./metrics_reporting_and_filtering.md) for more info. trapperkeeper-metrics-1.3.1/documentation/configuration.md000066400000000000000000000065531370145725700241170ustar00rootroot00000000000000# Configuring the `MetricsService` All metrics get registered into a Dropwizard `MetricsRegistry`. Typically, each TK application will have its own metric registry, and multiple TK services will share that registry. The Metrics service has support for having multiple metrics registries. Each registry has its own `domain`, so that if multiple applications with different registries are running together in the same process, they will not have conflicts. There is also a `default` metrics registry that is automatically created. Much of the Metrics service configuration is per-registry. However, the `server-id` settings and the `reporters` settings (for `host`, `port`, and `update-interval-seconds`) are global (across all registries). To configure the Metrics service, edit the `metrics.conf` file in your `conf.d` directory. #### `metrics.conf` Here is a sample config file that illustrates the available settings for metrics: ``` metrics: { # a server id that will be used as part of the namespace for metrics produced # by this server server-id: localhost registries: { my-application: { # specify metrics to include in addition to the default list provided. # The combination of these two lists will be what gets exported to graphite and # other reporters (excluding JMX). metrics-allowed: [ "foo.bar" ] # the default prefix prepended to metric names on the # `metrics-allowed` list before filtering them is `puppetlabs.`. If # this does not match the prefix of an application's metrics, specify a different # prefix here so that metrics will be correctly filtered. metrics-prefix: "my-metrics" # enable and disable specific reporters, and set configuration # options specific for the registry. reporters: { # enable or disable JMX metrics reporter jmx: { enabled: true } # enable or disable graphite metrics reporter for this registry # and optionally override host/port/update-interval. graphite: { enabled: true update-interval-seconds: 5 } } } } # this section is used to configure host, port, and update-interval-seconds # for reporters. Note that reporters are not enabled in this section - # they must be enabled per-registry. reporters: { graphite: { # graphite host host: "127.0.0.1" # graphite metrics port port: 2003 # how often to send metrics to graphite update-interval-seconds: 60 } } metrics-webservice: { jolokia: { # Enable or disable the Jolokia-based metrics/v2 endpoint. # Default is true. enabled: false # Configure any of the settings listed at: # https://jolokia.org/reference/html/agents.html#war-agent-installation servlet-init-params: { # Specify a custom security policy: # https://jolokia.org/reference/html/security.html policyLocation: "file:///etc/puppetlabs//jolokia-access.xml" } } } } ``` trapperkeeper-metrics-1.3.1/documentation/index.md000066400000000000000000000240421370145725700223500ustar00rootroot00000000000000# `trapperkeeper-metrics` Documentation ## What This Library Does The main purpose of this library is to provide some wrapper code around the [Dropwizard Metrics Java Library](https://dropwizard.github.io/metrics/3.1.0/), but there is some other functionality provided as well. Here are the major features available in `trapperkeeper-metrics`: * A `Trapperkeeper` service that handles life cycle management for objects from the Dropwizard Metrics library * Support for reading basic metrics configuration info from a `Trapperkeeper` config file, so that the metrics configuration syntax is consistent across all of your `Trapperkeeper` apps * Other utility functions for creating and interacting with metrics in your application For more detail on these features, read on. ### `MetricRegistry` Life Cycle The main entry point into the Dropwizard Metrics API is a class called [`MetricRegistry`](https://dropwizard.github.io/metrics/3.1.0/apidocs/com/codahale/metrics/MetricRegistry.html). This class requires some basic initialization, so `trapperkeeper-metrics` provides a Trapperkeeper service (`MetricsService`) that manages the life cycle of a `MetricRegistry`. The service includes a function, `get-metrics-registry`, so that all of your other Trapperkeeper services can access the registry and register new metrics with it. For example: ```clj (defservice my-service [[:MetricsService get-metrics-registry]] (init [this context] (let [metrics-registry (get-metrics-registry) my-metric-name (metrics/host-metric-name "localhost" "my-metric") my-timer (.timer metrics-registry my-metric-name)] (metrics/time! my-timer (do-some-work))) context)) ``` See the [source code for the sample app](https://github.com/puppetlabs/trapperkeeper-comidi-metrics/blob/master/dev/example/comidi_metrics_web_app.clj) for a working example. See the utility functions in the `trapperkeeper-metrics` [`puppetlabs.metrics`](../src/puppetlabs/metrics.clj) namespace for some helpers for constructing other kinds of metrics besides just `Timer`. See the [Dropwizard Metrics docs](https://dropwizard.github.io/metrics/3.1.0/) for more info about all of the available metrics types and their features. The `get-metrics-registry` also allows you to specify two additional fields, `registry-key` and `domain` which allow you to create other registries (besides the default given by `(get-metrics-registry)` and allow you to configure the namespace of the reporter for that registry. ```clj (defservice my-service [[:MetricsService get-metrics-registry]] (init [this context] (let [default-metrics-registry (get-metrics-registry) my-metrics-registry (get-metrics-registry "my.metrics.domain") ;; This will create the metric ;; `my.metrics.domain:name=puppetlabs.localhost.my-metric` my-metric-name (metrics/host-metric-name "localhost" "my-metric") my-timer (.timer metrics-registry my-metric-name)] (assert (not= default-metrics-registry my-metrics-registry)) (metrics/time! my-timer (do-some-work))) context) (start [this context] ;; We can retrieve the same metrics-registry later. (let [my-metrics-registry (get-metrics-registry "my.metrics.domain")] (do-some-other-work my-metrics-registry)) context)) ``` For more specific details on the service API, see the [`Metrics Service` API docs](./api.md). ### Configuration & Reporters The `MetricsService` also provides a configuration syntax that users can use to configure the metrics for a running TK app. This means that all TK services can provide a consistent interface for interacting with metrics. For more specific details, see the [`MetricsService` configuration docs](../documentation/configuration.md). ### Utility Functions The [`puppetlabs.metrics`](../src/puppetlabs/metrics.clj) namespace contains some utility functions for working with metrics. See the source code and docstrings for more detail. Here are a few bits of basic info: `time!` is a macro that can be used to time some Clojure forms against an existing `Timer`. e.g.: ```clj (let [my-timer (.timer (get-metrics-registry) "my.metric.name")] (time! my-timer (do-some-work!) (do-some-more-work!))) ``` `host-metric-name` can be used to provide a qualified, namespaced metric name. For best results, it is advisable to use this function to create a name for each of your application's metrics; this will ensure that the metrics are namespaced consistently across services. It will also ensure that metrics are namespaced by hostname, which is critical when consolidating metrics data from multiple hosts/services. You can use the `server-id` value from the `MetricsService` configuration to get the appropriate hostname for your metric. (TODO: this part of the API needs to be fleshed out a bit further. We might want to have a more dynamic way to get the hostname/server-id rather than having to put it into the config file. We may want to tweak some other things as well. *In the interim*, it's probably a good idea to *make it clear in your application/service documentation* that the specific metric namespaces should not be considered part of a formal API and may change in subsequent releases.) `register` can be used to add a metric to an existing `MetricRegistry`. `ratio`, `metered-ratio`, and `gauge` can be used to construct other types of Metrics. ### Low-level HTTP API To enable the HTTP API for accessing individual metrics add the service to your `bootstrap.cfg` and configure the `web-router-service` accordingly: ``` web-router-service { "puppetlabs.trapperkeeper.services.metrics.metrics-service/metrics-webservice" : "/metrics" } ``` #### Listing available metrics ##### Request format To get a list of all available metric names: * Request `/metrics/v1/mbeans`. * Use a `GET` request. ##### Response format Responses return a JSON object mapping a string to a string: * The key is the name of a valid MBean. * The value is a URI to use for requesting that MBean's attributes. #### Retrieving multiple metrics ##### Request format To get a the attributes for multiple metrics at the same time: * Request `/metrics/v1/mbeans`. * Use a `POST` request. * Use a request body which is either a JSON object whose values are metric names, JSON array of metric names or a JSON string of a metric name. ##### Response format The response format, though always JSON, depends on the request format: * Requests with a JSON object will return the a JSON object where the values of the original object have been transformed into the Mbeans' attributes for the metric names. * Requests with a JSON array will return the a JSON array where the items of the original array have been transformed into the Mbeans' attributes for the metric names. * Requests with a JSON string will return the a JSON object of the Mbean's attributes for the given metric name. #### Retrieving an specific metric ##### Request format To get the attributes of a particular metric: * Request `/metrics/v1/mbeans/`, where `` is something that was returned in the list of available metrics specified above. * Use a `GET` request. ##### Response format Responses return a JSON object mapping strings to (strings/numbers/Booleans). For example, using `curl` from localhost: curl 'http://localhost:8080/metrics/v1/mbeans/java.lang:type=Memory' { "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 807403520, "init" : 268435456, "max" : 3817865216, "used" : 129257096 }, "NonHeapMemoryUsage" : { "committed" : 85590016, "init" : 24576000, "max" : 184549376, "used" : 85364904 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } #### Alternatives Since we support sending the metrics data to JMX, there are several existing tools and approaches that can be used to read the data for individual metrics via JMX. JVisualVM is one example; see [pe-puppetserver-jruby-jmx-client](https://github.com/puppetlabs/pe-puppetserver-jruby-jmx-client) for an example of how to do this from a JRuby script. ## What This Library Does *Not* Do ## Notes for Developers For best results, use this library in combination with the [`comidi`](https://github.com/puppetlabs/comidi) library, and then take advantage of the `wrap-with-request-metrics` Ring middleware to track metrics about all HTTP requests made to your application. See the `comidi` docs for more info. ## In The Future There Will Be Robots Some ideas for things we might want to add/change in the future: ### metrics-clojure There is an existing clojure library that wraps Dropwizard Metrics: [`metrics-clojure`](https://github.com/sjl/metrics-clojure). At the time when we originally wrote our metrics code, this library was very out-of-date and didn't support some of the features we needed. It also seemed like a pretty thin facade around the Java library, and we decided that it wasn't worth adding an extra dependency. Since then, it's been updated and should be much more compatible with the more recent versions of Dropwizard Metrics. At a glance, it seems like it would be possible for other TK services to use `metrics-clojure` in combination with `trapperkeeper-metrics` as-is; it mostly just provides utility functions that should work fine with the `MetricRegistry` object surfaced by `trapperkeeper-metrics`. So, if you feel like the abstractions it provides over the Java library are worthwhile, try it out and let us know if something doesn't work properly. At some point in the future we may go ahead and add it as a direct dependency and refactor things in `trapperkeeper-metrics` to use it, but so far there hasn't been a hugely compelling reason to do so. ### Additional Facilities for Exposing Metrics There are a few other ideas floating around for how to make it easier for apps/services that are using `trapperkeeper-metrics` to expose the metrics info to end users. More utility functions for easier integration with status service, other tools for facilitating consumption / visualization / etc. Stay tuned. trapperkeeper-metrics-1.3.1/documentation/metrics_reporting_and_filtering.md000066400000000000000000000056121370145725700276670ustar00rootroot00000000000000# Metrics Reporting and Filtering ## Metrics Reporting Currently, trapperkeeper-metrics has the ability to export metrics to two metrics reporters: JMX and Graphite. In order to set up metrics reporting, all that must be done is for the metrics registry domain to be listed under the `registries` key, and for the desired reporters to be enabled under that. In addition, the Graphite `host`, `port`, and `update-interval-seconds` must be set if reporting to Graphite. ## Metrics Filtering Some applications may have many more metrics than Graphite or other reporters can handle, or than users might find useful to see in Graphite. For this purpose, trapperkeeper-metrics provides the ability to filter the metrics an application has registered before sending them to configured reporters. If no filter is provided, all metrics are allowed through. If a filter is provided, then only the metrics on the `allow` list are allowed through. The `allow` list is made up of 1) the metrics on the `default-metrics-allowed` list, and 2) the metric names in the `metrics-allowed` section of the config. TK application developers who want to have a list of metric names that users cannot change (and don't have to worry about and will always be present after upgrade) should set these with the `default-metrics-allowed` setting. This is done using the [`update-registry-settings` service function](./api.md#update-registry-settings). The `metrics-allowed` setting in the config allows users to add additional metrics (on top of the `default-metrics-allowed`) that they would like to have exported. this is especially useful for metric names that include information from their environment - e.g. a node name. The metric names from the two lists (`default-metrics-allowed` and `metrics-allowed`) are combined and then prefixed with either `puppetlabs.` (where `server-id` is taken from that setting in the config) or the value of the `metrics-prefix` from the config. So, for example, a metric that is named `foo.bar` and prefixed with `puppetlabs.` to be `puppetlabs..foo.bar` could be "allowed" by adding it to the `default-metrics-allowed` list or the `metrics-allowed` config setting as `foo.bar`. If it was instead `some.other.prefix.foo.bar` then in addition to having `foo.bar` added to one of the allowed metrics lists, the `metrics-prefix` would need to be set to `some.other.metrics.prefix` in the config. Note that this filtering does not apply to JMX - if JMX reporting is enabled, all metrics will be sent to it, regardless of whether they are in the `metrics-allowed` list or not. In addition to metric name filtering, not all the metric "fields" are reported to Graphite for histogram/timer metrics. For these metrics, the `min`, `max`, `mean`, `count`, `stddev`, `p50`, `p75`, and `p95` are reported (`p98`, `p99`, `p9999`, `m5rate`, `m10rate`, and `m15rate` are not reported). This is not configurable. trapperkeeper-metrics-1.3.1/examples/000077500000000000000000000000001370145725700176625ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/examples/ring_app/000077500000000000000000000000001370145725700214615ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/examples/ring_app/README.md000066400000000000000000000140041370145725700227370ustar00rootroot00000000000000# Simple Ring App with Metrics Example This example demonstrates how to incorporate the trapperkeeper-metrics service into a simple Ring app. This is based loosely upon the [ring_app example] (https://github.com/puppetlabs/trapperkeeper-webserver-jetty9/tree/master/examples/ring_app) in the trapperkeeper-webserver-jetty9 project. See that example for more information on the use of the `jetty9-service` and the Jetty web server integration with Ring. All code needed to execute this example is located in `./src/examples/ring_app`. The Clojure code is contained in the `ring_app.clj` file. This example configures a metrics registry for the count service and adds two counters to that registry, `count-service-report-me` and `count-service-dont-report-me`. During initialization, the service only includes the `count-service-report-me` counter but not the `count-service-dont-report-me` counter in its list of `:default-metrics-allowed` for the count service registry: ~~~~clj (update-registry-settings :count-service {:default-metrics-allowed ["count-service-report-me"]}) ~~~~ Metric info for the `count-service-report-me` counter would be periodically reported to a Graphite server when the Graphite reporter is enabled. Info for the `count-service-dont-report-me` counter, however, would not be reported to the Graphite server -- whether or not the Graphite reporter is enabled. The JMX reporter is enabled by default in the `ring-example.conf` file. Both the `count-service-report-me` and `count-service-dont-report-me` counters are available via JMX, through the `/metrics` webservice endpoint. ## Launching trapperkeeper and running the app To start up trapperkeeper and launch the sample application, use the following `lein` command while in the root of the trapperkeeper-metrics repo: ~~~~sh lein trampoline run --config ./examples/ring_app/ring-example.conf \ --bootstrap-config ./examples/ring_app/bootstrap.cfg ~~~~ For convenience, the application could also be run instead via the `ring-example` alias: ~~~~sh lein ring-example ~~~~ ### Running the app from the repl To startup the sample application from the repl, use the following `lein` command while in the root of the trapperkeeper-metrics repo: ~~~~sh lein repl ~~~~ The `repl` prompt should display the namespace of the ring-app example. Type `(go)` and press enter in order to launch the app: ~~~~sh examples.ring-app.repl=> (go) ~~~~ ### The `bootstrap.cfg` file The bootstrap config file contains a list of services that trapperkeeper will load up and make available. They are listed as fully-qualified Clojure namespaces and service names. For this example, the bootstrap.cfg looks like this: ~~~~ puppetlabs.trapperkeeper.services.metrics.metrics-service/metrics-webservice puppetlabs.trapperkeeper.services.metrics.metrics-service/metrics-service puppetlabs.trapperkeeper.services.webserver.jetty9-service/jetty9-service puppetlabs.trapperkeeper.services.webrouting.webrouting-service/webrouting-service examples.ring-app.ring-app/count-service ~~~~ This configuration indicates that the metrics services, `WebserverService` and `WebroutingService` from the `trapperkeeper-webserver-jetty9` project, and the count service, defined in the `ring_app.clj` file, are to be loaded. ### The `ring-example.conf` configuration file For the application configuration, a file called `ring-example.conf` provides a fairly minimal configuration for the count service: ~~~~hocon global: { logging-config: ./examples/ring_app/logback.xml } webserver: { port: 8080 } web-router-service: { "puppetlabs.trapperkeeper.services.metrics.metrics-service/metrics-webservice": "/metrics" } metrics: { server-id: "localhost" registries: { count-service: { reporters: { jmx: { enabled: true } graphite: { enabled: false } } } } metrics-webservice: { jolokia: { enabled: true } } reporters: { graphite: { host: "127.0.0.1" port: 2003 update-interval-seconds: 5 } } } ~~~~ The Graphite reporter is disabled by default in the `ring-example.conf` configuration file. Assuming a Graphite server were running on host `foo`, you could make the following changes to the configuration to enable reporting to the `foo` server: ~~~ ... metrics: { ... registries: { count-service: { reporters: { ... graphite: { enabled: true } } } } ... reporters: { graphite: { host: "foo" ... } } ~~~ ### Testing the counter service metrics The counter service increments the `count-service-report-me` and `count-service-dont-report-me` counters each time a request is made to its `/count` endpoint. To view the latest value stored in JMX for each counter, query the `/metrics` webservice. For example, the following request would obtain info for the `count-service-report-me` counter: ~~~~sh curl http://127.0.0.1:8080/metrics/v2/read/count-service:name=puppetlabs.localhost.count-service-report-me ~~~~ Just after startup, this would show that the current counter value is 0: ~~~~json { "request": { "mbean": "count-service:name=puppetlabs.localhost.count-service-report-me", "type": "read" }, "status": 200, "timestamp": 1486831476, "value": { "Count": 0 } } ~~~~ Make a request to the count service's endpoint to increment the counters: ~~~~ curl http://127.0.0.1:8080/count ~~~~ When the `.../metrics/v2` request from above is repeated, the value returned from JMX should have increased by 1: ~~~~json { "request": { "mbean": "count-service:name=puppetlabs.localhost.count-service-report-me", "type": "read" }, "status": 200, "timestamp": 1486831664, "value": { "Count": 1 } } ~~~~ trapperkeeper-metrics-1.3.1/examples/ring_app/bootstrap.cfg000066400000000000000000000005351370145725700241620ustar00rootroot00000000000000puppetlabs.trapperkeeper.services.metrics.metrics-service/metrics-webservice puppetlabs.trapperkeeper.services.metrics.metrics-service/metrics-service puppetlabs.trapperkeeper.services.webserver.jetty9-service/jetty9-service puppetlabs.trapperkeeper.services.webrouting.webrouting-service/webrouting-service examples.ring-app.ring-app/count-service trapperkeeper-metrics-1.3.1/examples/ring_app/logback.xml000066400000000000000000000004521370145725700236060ustar00rootroot00000000000000 %d %-5p [%c{2}] %m%n trapperkeeper-metrics-1.3.1/examples/ring_app/ring-example.conf000066400000000000000000000013511370145725700247200ustar00rootroot00000000000000global: { logging-config: ./examples/ring_app/logback.xml } webserver: { port: 8080 } web-router-service: { "puppetlabs.trapperkeeper.services.metrics.metrics-service/metrics-webservice": "/metrics" } metrics: { server-id: "localhost" registries: { count-service: { reporters: { jmx: { enabled: true } graphite: { enabled: false } } } } metrics-webservice: { jolokia: { enabled: true } } reporters: { graphite: { host: "127.0.0.1" port: 2003 update-interval-seconds: 5 } } } trapperkeeper-metrics-1.3.1/examples/ring_app/src/000077500000000000000000000000001370145725700222505ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/examples/ring_app/src/examples/000077500000000000000000000000001370145725700240665ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/examples/ring_app/src/examples/ring_app/000077500000000000000000000000001370145725700256655ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/examples/ring_app/src/examples/ring_app/repl.clj000066400000000000000000000035471370145725700273320ustar00rootroot00000000000000(ns examples.ring-app.repl (:require [puppetlabs.trapperkeeper.core :as tk] [puppetlabs.trapperkeeper.app :as tka] [puppetlabs.trapperkeeper.bootstrap :as bootstrap] [puppetlabs.trapperkeeper.config :as config] [clojure.tools.namespace.repl :refer (refresh)])) ;; This namespace shows an example of the "reloaded" clojure workflow ;; ( http://thinkrelevance.com/blog/2013/06/04/clojure-workflow-reloaded ) ;; ;; It's based on the pattern from Stuart Sierra's `Component` library: ;; ( https://github.com/stuartsierra/component#reloading ) ;; ;; You can load this namespace up into a REPL and then run `(go)` to boot ;; and run the sample application. Then, you can run `(reset)` at any time ;; to stop the running app, reload all of the necessary namespaces, and start ;; a new instance of the app. This means that you can do iterative development ;; without having to restart the whole JVM. ;; ;; You can also view the context of the application (and all of the ;; trapperkeeper services) via `(context)` (or pretty-printed with ;; `print-context`). (def system nil) (defn init [] (alter-var-root #'system (fn [_] (tk/build-app (bootstrap/parse-bootstrap-config! "./examples/ring_app/bootstrap.cfg") (config/load-config "./examples/ring_app/ring-example.conf")))) (alter-var-root #'system tka/init) (tka/check-for-errors! system)) (defn start [] (alter-var-root #'system (fn [s] (if s (tka/start s)))) (tka/check-for-errors! system)) (defn stop [] (alter-var-root #'system (fn [s] (if s (tka/stop s))))) (defn go [] (init) (start)) (defn context [] @(tka/app-context system)) (defn print-context [] (clojure.pprint/pprint (context))) (defn reset [] (stop) (refresh :after 'examples.ring-app.repl/go)) trapperkeeper-metrics-1.3.1/examples/ring_app/src/examples/ring_app/ring_app.clj000066400000000000000000000027251370145725700301640ustar00rootroot00000000000000(ns examples.ring-app.ring-app (:require [clojure.tools.logging :as log] [puppetlabs.metrics :as metrics] [puppetlabs.trapperkeeper.core :refer [defservice]] [puppetlabs.trapperkeeper.services :refer [service-context]])) (defservice count-service [[:MetricsService update-registry-settings get-metrics-registry get-server-id] [:WebserverService add-ring-handler]] (init [this context] (log/info "Count service starting up") (let [registry (get-metrics-registry :count-service) server-id (get-server-id) counter-to-report (.counter registry (metrics/host-metric-name server-id "count-service-report-me")) counter-to-not-report (.counter registry (metrics/host-metric-name server-id "count-service-dont-report-me"))] (add-ring-handler (fn [_] (.inc counter-to-report) (.inc counter-to-not-report) {:status 200 :headers {"Content-Type" "text/plain"} :body (str "Requests made since startup: " (.getCount counter-to-report))}) "/count")) (update-registry-settings :count-service {:default-metrics-allowed ["count-service-report-me"]}) context)) trapperkeeper-metrics-1.3.1/ext/000077500000000000000000000000001370145725700166445ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/ext/travisci/000077500000000000000000000000001370145725700204705ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/ext/travisci/test.sh000077500000000000000000000000271370145725700220050ustar00rootroot00000000000000#!/bin/bash lein test trapperkeeper-metrics-1.3.1/locales/000077500000000000000000000000001370145725700174665ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/locales/messages.pot000066400000000000000000000015441370145725700220250ustar00rootroot00000000000000# SOME DESCRIPTIVE TITLE. # Copyright (C) YEAR Puppet # This file is distributed under the same license as the puppetlabs.trapperkeeper_metrics package. # FIRST AUTHOR , YEAR. # #, fuzzy msgid "" msgstr "" "Project-Id-Version: puppetlabs.trapperkeeper_metrics \n" "Report-Msgid-Bugs-To: docs@puppet.com\n" "POT-Creation-Date: \n" "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME \n" "Language-Team: LANGUAGE \n" "Language: \n" "MIME-Version: 1.0\n" "Content-Type: text/plain; charset=UTF-8\n" "Content-Transfer-Encoding: 8bit\n" #: src/clj/puppetlabs/trapperkeeper/services/metrics/metrics_core.clj msgid "metrics request must be a JSON array, string, or object" msgstr "" #: src/clj/puppetlabs/trapperkeeper/services/metrics/metrics_core.clj msgid "No mbean ''{0}'' found" msgstr "" trapperkeeper-metrics-1.3.1/project.clj000066400000000000000000000043141370145725700202060ustar00rootroot00000000000000(defproject puppetlabs/trapperkeeper-metrics "1.3.1" :description "Trapperkeeper Metrics Service" :url "http://github.com/puppetlabs/trapperkeeper-metrics" :min-lein-version "2.7.1" :pedantic? :abort :parent-project {:coords [puppetlabs/clj-parent "1.7.26"] :inherit [:managed-dependencies]} :dependencies [[org.clojure/clojure] [prismatic/schema] [puppetlabs/kitchensink] [puppetlabs/trapperkeeper] [puppetlabs/trapperkeeper-authorization] [puppetlabs/ring-middleware] [cheshire] [org.clojure/java.jmx] [ring/ring-defaults] [org.clojure/tools.logging] [io.dropwizard.metrics/metrics-core] [io.dropwizard.metrics/metrics-graphite] [org.jolokia/jolokia-core "1.6.2"] [puppetlabs/comidi] [puppetlabs/i18n]] :plugins [[puppetlabs/i18n "0.6.0"] [lein-parent "0.3.1"]] :source-paths ["src/clj"] :java-source-paths ["src/java"] :deploy-repositories [["releases" {:url "https://clojars.org/repo" :username :env/clojars_jenkins_username :password :env/clojars_jenkins_password :sign-releases false}]] :classifiers [["test" :testutils]] :profiles {:dev {:aliases {"ring-example" ["trampoline" "run" "-b" "./examples/ring_app/bootstrap.cfg" "-c" "./examples/ring_app/ring-example.conf"]} :source-paths ["examples/ring_app/src"] :dependencies [[puppetlabs/http-client] [puppetlabs/trapperkeeper :classifier "test"] [puppetlabs/trapperkeeper-webserver-jetty9] [puppetlabs/kitchensink :classifier "test"]]} :testutils {:source-paths ^:replace ["test"] :java-source-paths ^:replace []}} :repl-options {:init-ns examples.ring-app.repl} :main puppetlabs.trapperkeeper.main) trapperkeeper-metrics-1.3.1/resources/000077500000000000000000000000001370145725700200565ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/resources/jolokia-access.xml000066400000000000000000000010211370145725700234610ustar00rootroot00000000000000 localhost 127.0.0.1 0:0:0:0:0:0:0:1 read list version search trapperkeeper-metrics-1.3.1/src/000077500000000000000000000000001370145725700166335ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/src/clj/000077500000000000000000000000001370145725700174035ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/src/clj/puppetlabs/000077500000000000000000000000001370145725700215625ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/src/clj/puppetlabs/metrics.clj000066400000000000000000000056131370145725700237270ustar00rootroot00000000000000;; Utility functions for working with the Metrics library (ns puppetlabs.metrics (:import (com.codahale.metrics MetricRegistry RatioGauge RatioGauge$Ratio Gauge Metric Metered Sampling Timer) (java.util.concurrent TimeUnit)) (:require [schema.core :as schema])) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;;; Public (schema/defn ^:always-validate host-metric-name :- schema/Str "Given a hostname and a metric name, build a qualified metric name for use with Metrics." [hostname :- schema/Str metric-name :- schema/Str] (MetricRegistry/name "puppetlabs" (into-array String [hostname metric-name]))) (schema/defn ^:always-validate http-metric-name :- schema/Str "Given a hostname and a metric name, build a qualified http metric name for use with Metrics." [hostname :- schema/Str metric-name :- schema/Str] (MetricRegistry/name "puppetlabs" (into-array String [hostname "http" metric-name]))) (schema/defn ^:always-validate register :- Metric "Register a metric with a metrics registry, using the given metric name." [registry :- MetricRegistry metric-name :- schema/Str metric :- Metric] (.register registry metric-name metric)) (schema/defn mean :- Double "Given a Timer or Histogram object, get the current mean value." [sampling :- Sampling] (.. sampling getSnapshot getMean)) (schema/defn mean-millis :- Long "Given a Timer or Histogram object, get the mean sample time in milliseconds." [sampling :- Sampling] (.toMillis TimeUnit/NANOSECONDS (mean sampling))) (schema/defn mean-in-unit :- Long "Given a Timer or Histogram object, get the mean sample time in the specified time unit." [sampling :- Sampling time-unit :- TimeUnit] (.convert time-unit (mean sampling) TimeUnit/NANOSECONDS)) (schema/defn ^:always-validate ratio :- RatioGauge "Given two functions, return a Ratio metric whose value will be computed by calling the first function to retrieve the numerator and the second function to retrieve the denominator" [numerator-fn :- (schema/pred ifn?) denominator-fn :- (schema/pred ifn?)] (proxy [RatioGauge] [] (getRatio [] (RatioGauge$Ratio/of (numerator-fn) (denominator-fn))))) (schema/defn ^:always-validate metered-ratio :- RatioGauge "Given two Metered metrics, construct a Ratio metric whose numerator and denominator are computed by calling the `getCount` method of the Metered metrics." [numerator :- Metered denominator :- Metered] (ratio #(.getCount numerator) #(.getCount denominator))) (schema/defn ^:always-validate gauge :- Gauge "Returns a Gauge metric with an initial value" [value] (proxy [Gauge] [] (getValue [] value))) (defmacro time! "Times the body forms against the given Timer metric" [^Timer t & body] `(.time ~(vary-meta t assoc :tag `Timer) (proxy [Callable] [] (call [] (do ~@body))))) trapperkeeper-metrics-1.3.1/src/clj/puppetlabs/trapperkeeper/000077500000000000000000000000001370145725700244335ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/src/clj/puppetlabs/trapperkeeper/services/000077500000000000000000000000001370145725700262565ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/src/clj/puppetlabs/trapperkeeper/services/metrics/000077500000000000000000000000001370145725700277245ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/src/clj/puppetlabs/trapperkeeper/services/metrics/jolokia.clj000066400000000000000000000102211370145725700320420ustar00rootroot00000000000000(ns puppetlabs.trapperkeeper.services.metrics.jolokia "Clojure helpers for constructing and configuring Jolokia servlets." (:require [clojure.tools.logging :as log] [clojure.walk :as walk] [ring.util.servlet :as ring-servlet] [schema.core :as schema]) (:import [javax.servlet.http HttpServletRequest] [org.jolokia.config ConfigKey] [org.jolokia.util LogHandler] [org.jolokia.http AgentServlet])) (def config-mapping "Inspects the Jolokia ConfigKey Enum and generates a mapping that associates a Clojure keyword with each configuration parameter. The keyword used is the camel-cased identifier that would be used to configure a servlet via a web.xml file. For example, `ConfigKey/AGENT_ID` is associated with the keyword `:agentId`. For a complete list of configuration options, see: https://jolokia.org/reference/html/agents.html#agent-war-init-params" (->> (ConfigKey/values) (map (juxt #(-> % .getKeyValue keyword) identity)) (into {}))) (schema/defschema JolokiaConfig "Schema for validating Clojure maps containing Jolokia configuration. Creates a map of optional keys which have string values using the config-mapping extracted from the ConfigKey enum." (->> (keys config-mapping) (map #(vector (schema/optional-key %) schema/Str)) (into {}))) (def config-defaults "Default configuration values for Jolokia." {;; When set to "false", no debug-level messages are produced by the Jolokia ;; namespace. This is unfortunate for debuggability, but having debug set to ;; "true" causes a lot of useless error messages and backtraces to flood the ;; logs when 404 results are generated by requests for JMX items that ;; do not exist. :debug "false" ;; Don't include backtraces in error results returned by the API. :allowErrorDetails "false" ;; Used by jolokia-access to match IPv4 and IPv6 to "localhost" :allowDnsReverseLookup "true" ;; Load access policy from: resources/jolokia-access.xml :policyLocation "classpath:/jolokia-access.xml" :mimeType "application/json"}) (defn create-servlet-config "Generate Jolokia AgentServlet configuration from a Clojure map" ([] (create-servlet-config {})) ([config] (->> config (merge config-defaults) ;; Validate here to ensure defaults are also valid. (schema/validate JolokiaConfig) walk/stringify-keys))) (defn create-logger "Return an object that implements the Jolokia logging interface using the logger from clojure.tools.logging" [] (reify LogHandler (debug [this message] (log/debug message)) (info [this message] (log/info message)) (error [this message throwable] (log/error throwable message)))) (defn create-servlet "Builds a Jolokia Servlet that uses Clojure logging." [auth-check-fn] (let [check-auth (if auth-check-fn (fn [^HttpServletRequest request] (auth-check-fn (ring-servlet/build-request-map request))) (fn [^HttpServletRequest request] {:authorized true :message ""}))] (if auth-check-fn (log/info "Metrics access control using trapperkeeper-authorization is enabled.") (log/warn "Metrics access control using trapperkeeper-authorization is disabled. Add the authorization service to the trapperkeeper bootstrap configuration file to enable it.")) (proxy [AgentServlet] [] ;; NOTE: An alternative to this method override would be to use defrecord ;; to create a class that can be set as `:logHandlerClass` in the servlet ;; configuration. This requires AOT compilation for the namespace defining ;; the record so that Jolokia can find the resulting class. (createLogHandler [_ _] (create-logger)) (service [request response] (let [{:keys [authorized message]} (check-auth request)] (if-not authorized (ring-servlet/update-servlet-response response {:status 403 :headers {} :body message}) (proxy-super service request response))))))) trapperkeeper-metrics-1.3.1/src/clj/puppetlabs/trapperkeeper/services/metrics/metrics_core.clj000066400000000000000000000375021370145725700331030ustar00rootroot00000000000000(ns puppetlabs.trapperkeeper.services.metrics.metrics-core (:import (com.codahale.metrics JmxReporter MetricRegistry) (com.fasterxml.jackson.core JsonParseException) (com.puppetlabs.trapperkeeper.metrics GraphiteReporter AllowedNamesMetricFilter) (java.util.concurrent TimeUnit) (java.net InetSocketAddress) (com.codahale.metrics.graphite Graphite GraphiteSender)) (:require [clojure.tools.logging :as log] [clojure.java.io :as io] [cheshire.core :as json] [schema.core :as schema] [ring.middleware.defaults :as ring-defaults] [puppetlabs.comidi :as comidi] [puppetlabs.ring-middleware.utils :as ringutils] [puppetlabs.trapperkeeper.services.metrics.metrics-utils :as metrics-utils] [puppetlabs.trapperkeeper.services.metrics.jolokia :as jolokia] [puppetlabs.kitchensink.core :as ks] [puppetlabs.i18n.core :as i18n :refer [trs tru]])) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;;; Schemas (def JmxReporterConfig {:enabled schema/Bool}) (def JolokiaApiConfig {(schema/optional-key :enabled) schema/Bool (schema/optional-key :servlet-init-params) jolokia/JolokiaConfig}) (def MbeansApiConfig {(schema/optional-key :enabled) schema/Bool}) (def WebserviceConfig {(schema/optional-key :mbeans) MbeansApiConfig (schema/optional-key :jolokia) JolokiaApiConfig}) (def BaseGraphiteReporterConfig {:host schema/Str :port schema/Int :update-interval-seconds schema/Int}) (def GraphiteReporterConfig (assoc BaseGraphiteReporterConfig :enabled schema/Bool)) ;; schema for what is read from config file for a registry (def GraphiteRegistryReporterConfig (assoc (ks/mapkeys schema/optional-key BaseGraphiteReporterConfig) :enabled schema/Bool)) (def RegistryReportersConfig {(schema/optional-key :jmx) JmxReporterConfig (schema/optional-key :graphite) GraphiteRegistryReporterConfig}) (def RegistryConfig {(schema/optional-key :metrics-allowed) [schema/Str] (schema/optional-key :metric-prefix) schema/Str (schema/optional-key :reporters) RegistryReportersConfig}) (def RegistriesConfig {schema/Any RegistryConfig}) (def ReportersConfig {(schema/optional-key :graphite) BaseGraphiteReporterConfig}) (def MetricsConfig {:server-id schema/Str (schema/optional-key :registries) RegistriesConfig (schema/optional-key :reporters) ReportersConfig (schema/optional-key :metrics-webservice) WebserviceConfig}) (def RegistryContext {:registry (schema/maybe MetricRegistry) :jmx-reporter (schema/maybe JmxReporter) (schema/optional-key :graphite-reporter) GraphiteReporter}) (def DefaultRegistrySettings {:default-metrics-allowed [schema/Str]}) (def MetricsServiceContext {:registries (schema/atom {schema/Keyword RegistryContext}) :can-update-registry-settings? schema/Bool :registry-settings (schema/atom {schema/Keyword DefaultRegistrySettings}) :metrics-config MetricsConfig}) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;;; Private (schema/defn jmx-reporter :- JmxReporter [registry :- MetricRegistry domain :- (schema/maybe schema/Keyword)] (let [b (JmxReporter/forRegistry registry)] (when domain (.inDomain b (name domain))) (.build b))) (schema/defn initialize-registry-context :- RegistryContext "Create initial registry context. This will include a MetricsRegistry and a JMX reporter, but not a Graphite reporter." [config :- (schema/maybe RegistryConfig) domain :- schema/Keyword] (let [jmx-config (get-in config [:reporters :jmx]) registry (MetricRegistry.)] {:registry registry :jmx-reporter (when (:enabled jmx-config) (doto ^JmxReporter (jmx-reporter registry domain) (.start)))})) (schema/defn construct-metric-names :- #{schema/Str} "Prefixes the metric prefix to each metric name. Returns a set of metric names (duplicates are removed)." [prefix :- schema/Str metric-names :- [schema/Str]] (set (map #(format "%s.%s" prefix %) metric-names))) (schema/defn build-metric-filter :- AllowedNamesMetricFilter [metrics-allowed :- #{schema/Str}] (AllowedNamesMetricFilter. metrics-allowed)) (schema/defn get-metric-prefix :- schema/Str "Determines what the metric prefix should be. If a metric-prefix is set in the config, we use that. Else default to the server-id" [metrics-config :- MetricsConfig domain :- schema/Keyword] (if-let [metric-prefix (get-in metrics-config [:registries domain :metric-prefix])] metric-prefix (format "puppetlabs.%s" (:server-id metrics-config)))) (schema/defn build-graphite-reporter :- GraphiteReporter "Constructs a GraphiteReporter instance for the given registry, with the given allowed metrics, and using the given graphite-sender" [registry :- MetricRegistry metrics-allowed :- #{schema/Str} graphite-sender :- GraphiteSender] (-> (GraphiteReporter/forRegistry registry) (.convertRatesTo (TimeUnit/MILLISECONDS)) (.convertDurationsTo (TimeUnit/MILLISECONDS)) (.filter (build-metric-filter metrics-allowed)) (.build graphite-sender))) (schema/defn build-graphite-sender :- GraphiteSender [graphite-config :- GraphiteReporterConfig ;; The domain is only needed as an argument for testing, which is unfortunate. In the future, it ;; would be nice to add the ability to register a function that could receive a callback when a ;; reporter is added, which could solve the problem of needing this extra argument solely for ;; testing (see PE-17010). domain :- schema/Keyword] (Graphite. (InetSocketAddress. (:host graphite-config) (:port graphite-config)))) (schema/defn add-graphite-reporter :- RegistryContext "Adds a graphite reporter to the given registry context if graphite is enabled in the configuration. Starts up a thread which reports the metrics to graphite on the interval specified in :update-interval-seconds" [registry-context :- RegistryContext graphite-config :- (schema/maybe GraphiteReporterConfig) metrics-allowed :- #{schema/Str} domain :- schema/Keyword] (if (:enabled graphite-config) (let [graphite-sender (build-graphite-sender graphite-config domain) graphite-reporter (build-graphite-reporter (:registry registry-context) metrics-allowed graphite-sender)] (.start graphite-reporter (:update-interval-seconds graphite-config) (TimeUnit/SECONDS)) (assoc registry-context :graphite-reporter graphite-reporter)) registry-context)) (schema/defn get-graphite-config :- (schema/maybe GraphiteReporterConfig) "Merge together the graphite config for the registry with the global graphite config." [config :- MetricsConfig domain :- schema/Keyword] (let [reporter-config (get-in config [:reporters :graphite]) registry-config (get-in config [:registries domain :reporters :graphite]) merged-config (merge reporter-config registry-config)] ;; the default value for enabled is false (if (nil? merged-config) merged-config (update-in merged-config [:enabled] #(if (nil? %) false %))))) (schema/defn get-metrics-allowed :- #{schema/Str} "Get the metrics allowed for the registry. Looks at the metrics-allowed registered for the registry in the registry settings atom using the `update-registry-settings` function as well as the metrics-allowed listed in the config file under the `:metrics-allowed` key. Merges these lists together and then adds the metrics prefix to them, returning a set of prefixed allowed metrics." [metrics-config :- MetricsConfig registry-settings :- {schema/Any DefaultRegistrySettings} domain :- schema/Keyword] (let [metric-prefix (get-metric-prefix metrics-config domain) default-metrics-allowed (get-in registry-settings [domain :default-metrics-allowed]) configured-metrics-allowed (get-in metrics-config [:registries domain :metrics-allowed]) metrics-allowed (concat default-metrics-allowed configured-metrics-allowed)] (construct-metric-names metric-prefix metrics-allowed))) (schema/defn maybe-add-default-to-config :- MetricsConfig "Add a `:default` key with an empty map as the value to the registries config if it is not present." [metrics-config :- MetricsConfig] (update-in metrics-config [:registries :default] #(if (nil? %) {} %))) (schema/defn initialize-registries-from-config :- {schema/Any RegistryContext} "Read through the config and create a MetricsRegistry (+ JMX reporter if configured) for every registry mentioned in it. Also create the default registry if not mentioned in the config. Should be called from `init` of the metrics-service." [metrics-config :- MetricsConfig] (let [registries-config (:registries metrics-config)] (into {} (map (fn [x] {x (initialize-registry-context (get registries-config x) x)}) (keys registries-config))))) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;;; Public (schema/defn ^:always-validate add-graphite-reporters :- MetricsServiceContext "Add Graphite reporters to all registries with Graphite enabled in the config, using the configured settings for each registry. Returns an updated service context. Should be called from `start` of the metrics-service." [service-context :- MetricsServiceContext] (let [config (:metrics-config service-context) registry-settings @(:registry-settings service-context)] (doseq [registry @(:registries service-context)] (let [domain (key registry) graphite-config (get-graphite-config config domain) metrics-allowed (get-metrics-allowed config registry-settings domain) registry-with-graphite-reporter (add-graphite-reporter (val registry) graphite-config metrics-allowed domain)] (swap! (:registries service-context) assoc domain registry-with-graphite-reporter)))) service-context) ;; Note here that the return schema includes registries that could have Graphite reporters. If the ;; registry was in the config, then a Graphite reporter could have been configured for it. Any ;; registries not in the config will not have Graphite reporters. (schema/defn ^:always-validate get-or-initialize-registry-context :- RegistryContext "If a registry exists within the service context for a given domain already, return it. Otherwise initialize a new registry for that domain and return it. Modifies the registries atom in the service context to add the new registry" [{:keys [registries metrics-config]} :- MetricsServiceContext domain :- schema/Keyword] (if-let [metric-registry-context (get @registries domain)] metric-registry-context (let [registry-config (get-in metrics-config [:registries domain]) new-registry-context (initialize-registry-context registry-config domain)] (swap! registries assoc domain new-registry-context) new-registry-context))) (schema/defn ^:always-validate create-initial-service-context :- MetricsServiceContext "Create the initial service context for the metrics-service. Initialize all registries in the config, add them to the `registries` atom, and include that in the service context map, along with an empty atom for `registry-settings` and the metrics config." [metrics-config :- MetricsConfig] (let [config-with-default (maybe-add-default-to-config metrics-config) registries (initialize-registries-from-config config-with-default)] {:registries (atom registries) :can-update-registry-settings? true :registry-settings (atom {}) :metrics-config config-with-default})) (schema/defn lock-registry-settings :- MetricsServiceContext "Switch the `can-update-registry-settings?` boolean to false to show that it is after the `init` phase and registry settings can no longer be set." [context :- MetricsServiceContext] (assoc context :can-update-registry-settings? false)) (schema/defn ^:always-validate update-registry-settings :- {schema/Any DefaultRegistrySettings} "Update the `registry-settings` atom for the given domain. If called again for the same domain, the new settings will be merged in, and lists such as :default-metrics-allowed, will be concat'd together." [context :- MetricsServiceContext domain :- schema/Keyword settings :- DefaultRegistrySettings] (when (= false (:can-update-registry-settings? context)) (throw (RuntimeException. "Registry settings must be initialized in the `init` phase of the lifecycle."))) (let [registry-settings (:registry-settings context) deep-merge-fn (fn [first second] ; first will be nil if no settings exist for this domain, ; and deep-merge-with doesn't like that (ks/deep-merge-with concat (or first {}) second))] ; Swap out the atom by updating the value under the specified domain. ; Update using deep-merge-fn to do a deep merge between the existing settings ; and the new settings, concating values together if two keys match (swap! registry-settings update domain deep-merge-fn settings))) (schema/defn ^:always-validate stop [context :- RegistryContext] (if-let [jmx-reporter (:jmx-reporter context)] (.close jmx-reporter)) (if-let [graphite-reporter (:graphite-reporter context)] (.close graphite-reporter))) (schema/defn ^:always-validate stop-all [service-context :- MetricsServiceContext] (let [registries (:registries service-context)] (doseq [[_ metrics-registry] @registries] (stop metrics-registry)) service-context)) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;;; Comidi (defn build-handler [path] (comidi/routes->handler (comidi/wrap-routes (comidi/context path (comidi/context "/v1" (comidi/context "/mbeans" (comidi/GET "" [] (fn [req] (ringutils/json-response 200 (metrics-utils/mbean-names)))) (comidi/POST "" [] (fn [req] (try (let [metrics (with-open [reader (-> req :body io/reader)] (doall (json/parse-stream reader true)))] (cond (seq? metrics) (ringutils/json-response 200 (map metrics-utils/get-mbean metrics)) (string? metrics) (ringutils/json-response 200 (metrics-utils/get-mbean metrics)) (map? metrics) (ringutils/json-response 200 (ks/mapvals metrics-utils/get-mbean metrics)) :else (ringutils/json-response 400 (tru "metrics request must be a JSON array, string, or object")))) (catch JsonParseException e (ringutils/json-response 400 {:error (str e)}))))) (comidi/GET ["/" [#".*" :names]] [] (fn [{:keys [route-params] :as req}] (let [name (java.net.URLDecoder/decode (:names route-params))] (if-let [mbean (metrics-utils/get-mbean name)] (ringutils/json-response 200 mbean) (ringutils/json-response 404 (tru "No mbean ''{0}'' found" name))))))))) (comp i18n/locale-negotiator #(ring-defaults/wrap-defaults % ring-defaults/api-defaults))))) trapperkeeper-metrics-1.3.1/src/clj/puppetlabs/trapperkeeper/services/metrics/metrics_service.clj000066400000000000000000000077371370145725700336220ustar00rootroot00000000000000(ns puppetlabs.trapperkeeper.services.metrics.metrics-service (:require [puppetlabs.trapperkeeper.core :as trapperkeeper] [puppetlabs.trapperkeeper.services.authorization.authorization-service :as tk-auth] [puppetlabs.trapperkeeper.services.protocols.metrics :as metrics] [puppetlabs.trapperkeeper.services.metrics.metrics-core :as core] [puppetlabs.trapperkeeper.services.metrics.jolokia :as jolokia] [puppetlabs.trapperkeeper.services :as tk-services] [clojure.tools.logging :as log] [schema.core :as schema] [puppetlabs.kitchensink.core :as ks] [puppetlabs.trapperkeeper.core :as tk])) (trapperkeeper/defservice metrics-service metrics/MetricsService [[:ConfigService get-in-config]] (init [this context] (let [config (get-in-config [:metrics]) metrics-config (if (nil? (get-in config [:reporters :jmx :enabled])) config (do (log/warn "Enabling JMX globally is deprecated;" "JMX can only be enabled per-registry.") (ks/dissoc-in config [:reporters :jmx :enabled])))] (schema/validate core/MetricsConfig metrics-config) (log/debug "Creating metrics registries") (core/create-initial-service-context metrics-config))) (start [this context] ;; registry settings can only be updated in the `init` phase of the lifecycle (let [updated-context (core/lock-registry-settings context)] (log/debug "Creating graphite reporters") (core/add-graphite-reporters updated-context))) (stop [this context] (core/stop-all (tk-services/service-context this))) (update-registry-settings [this domain settings] (let [context (tk-services/service-context this)] (core/update-registry-settings context domain settings))) (get-metrics-registry [this] (get-in @(:registries (tk-services/service-context this)) [:default :registry])) (get-metrics-registry [this domain] (let [context (tk-services/service-context this)] (:registry (core/get-or-initialize-registry-context context domain)))) (get-server-id [this] (get-in-config [:metrics :server-id]))) (trapperkeeper/defservice metrics-webservice {:required [[:ConfigService get-in-config] [:WebroutingService add-ring-handler get-route get-server] [:WebserverService add-servlet-handler]] :optional [AuthorizationService]} (init [this context] (when (get-in-config [:metrics :metrics-webservice :mbeans :enabled] false) (log/warn "The v1 metrics endpoint is deprecated and will be removed in a future release." "Use the v2 endpoint instead.") (add-ring-handler this (core/build-handler (get-route this)))) (when (get-in-config [:metrics :metrics-webservice :jolokia :enabled] true) (let [config (->> (get-in-config [:metrics :metrics-webservice :jolokia :servlet-init-params] {}) jolokia/create-servlet-config) ;; NOTE: Normally, these route and server lookups would be done by ;; WebroutingService/add-servlet-handler, but that doesn't properly ;; mount sub-paths at the moment (TK-420). So we explicitly compute ;; these items and use WebserverService/add-servlet-handler instead. route (str (get-route this) "/v2") server (get-server this) options (if (nil? server) {:servlet-init-params config} {:servlet-init-params config :server-id (keyword server)}) auth-service (tk-services/maybe-get-service this :AuthorizationService) auth-check-fn (if auth-service (partial tk-auth/authorization-check auth-service))] (add-servlet-handler (jolokia/create-servlet auth-check-fn) route options))) context) (stop [this context] context)) trapperkeeper-metrics-1.3.1/src/clj/puppetlabs/trapperkeeper/services/metrics/metrics_utils.clj000066400000000000000000000026411370145725700333070ustar00rootroot00000000000000(ns puppetlabs.trapperkeeper.services.metrics.metrics-utils (:require [cheshire.custom :refer [JSONable]] [clojure.java.jmx :as jmx] [puppetlabs.kitchensink.core :as ks])) (defn filter-mbean "Converts an mbean to a map. For attributes that can't be converted to JSON, return a string representation of the value." [mbean] {:post [(map? %)]} (->> mbean (ks/mapvals (fn [v] (cond ;; Nested structures should themselves be filtered (map? v) (filter-mbean v) (instance? java.util.HashMap v) (->> v (into {}) filter-mbean) (satisfies? JSONable v) v :else (str v)))))) (defn all-mbean-names "Return a seq of all mbeans names" [] {:post [(coll? %)]} (map str (jmx/mbean-names "*:*"))) (defn mbean-names "Return a map of mbean name to a link that will retrieve the attributes" [] (->> (all-mbean-names) (map (fn [mbean-name] [mbean-name (format "/mbeans/%s" (java.net.URLEncoder/encode mbean-name "UTF-8"))])) (into (sorted-map)))) (defn get-mbean "Returns the attributes of a given MBean" [name] (when (some #(= name %) (all-mbean-names)) (filter-mbean (jmx/mbean name)))) trapperkeeper-metrics-1.3.1/src/clj/puppetlabs/trapperkeeper/services/protocols/000077500000000000000000000000001370145725700303025ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/src/clj/puppetlabs/trapperkeeper/services/protocols/metrics.clj000066400000000000000000000016211370145725700324420ustar00rootroot00000000000000(ns puppetlabs.trapperkeeper.services.protocols.metrics) (defprotocol MetricsService "A service that tracks runtime metrics for the process." (get-metrics-registry [this] [this domain] "Provides access to a MetricsRegistry where `domain` is the string used to look up the registry. Specifing no `domain` will return the default MetricsRegistry. The `domain` is the name that will appear at the front of the JMX metric. For example in `foo:name=my-metric`, `foo` is the `domain`.") (get-server-id [this] "Get the server-id from the `metrics` -> `server-id` part of the config.") (update-registry-settings [this domain settings] "Allows for specifying settings for a metric registry reporter that don't go into a config file. Must be called during the 'init' phase of a service's lifecycle. Will error if called more than once per metric registry.")) trapperkeeper-metrics-1.3.1/src/java/000077500000000000000000000000001370145725700175545ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/src/java/com/000077500000000000000000000000001370145725700203325ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/src/java/com/puppetlabs/000077500000000000000000000000001370145725700225115ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/src/java/com/puppetlabs/trapperkeeper/000077500000000000000000000000001370145725700253625ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/src/java/com/puppetlabs/trapperkeeper/metrics/000077500000000000000000000000001370145725700270305ustar00rootroot00000000000000AllowedNamesMetricFilter.java000066400000000000000000000012121370145725700344750ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/src/java/com/puppetlabs/trapperkeeper/metricspackage com.puppetlabs.trapperkeeper.metrics; import com.codahale.metrics.Metric; import com.codahale.metrics.MetricFilter; import java.util.Set; // Takes in a whitelist of strings to match against public class AllowedNamesMetricFilter implements MetricFilter{ private final Set allowedMetricNames; public AllowedNamesMetricFilter(Set allowedMetricNames){ this.allowedMetricNames = allowedMetricNames; } @Override public boolean matches(String name, Metric metric) { if (allowedMetricNames.isEmpty()) { return true; } else return allowedMetricNames.contains(name); } } trapperkeeper-metrics-1.3.1/src/java/com/puppetlabs/trapperkeeper/metrics/GraphiteReporter.java000066400000000000000000000255621370145725700331730ustar00rootroot00000000000000package com.puppetlabs.trapperkeeper.metrics; import com.codahale.metrics.Clock; import com.codahale.metrics.Counter; import com.codahale.metrics.Gauge; import com.codahale.metrics.Histogram; import com.codahale.metrics.Meter; import com.codahale.metrics.Metered; import com.codahale.metrics.MetricFilter; import com.codahale.metrics.MetricRegistry; import com.codahale.metrics.ScheduledReporter; import com.codahale.metrics.Snapshot; import com.codahale.metrics.Timer; import com.codahale.metrics.graphite.Graphite; import com.codahale.metrics.graphite.GraphiteSender; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import java.io.IOException; import java.util.Locale; import java.util.Map; import java.util.SortedMap; import java.util.concurrent.TimeUnit; /** * A reporter which publishes metric values to a Graphite server. * A modified version of {@link com.codahale.metrics.graphite.GraphiteReporter}, which sends a reduced set of statistics * for {@link Timer} and {@link Histogram} metrics * * @see Graphite - Scalable Realtime Graphing */ public class GraphiteReporter extends ScheduledReporter { /** * Returns a new {@link Builder} for {@link GraphiteReporter}. * * @param registry the registry to report * @return a {@link Builder} instance for a {@link GraphiteReporter} */ public static Builder forRegistry(MetricRegistry registry) { return new Builder(registry); } /** * A builder for {@link GraphiteReporter} instances. Defaults to not using a prefix, using the * default clock, converting rates to events/second, converting durations to milliseconds, and * not filtering metrics. */ public static class Builder { private final MetricRegistry registry; private Clock clock; private String prefix; private TimeUnit rateUnit; private TimeUnit durationUnit; private MetricFilter filter; private Builder(MetricRegistry registry) { this.registry = registry; this.clock = Clock.defaultClock(); this.prefix = null; this.rateUnit = TimeUnit.SECONDS; this.durationUnit = TimeUnit.MILLISECONDS; this.filter = MetricFilter.ALL; } /** * Use the given {@link Clock} instance for the time. * * @param clock a {@link Clock} instance * @return {@code this} */ public Builder withClock(Clock clock) { this.clock = clock; return this; } /** * Prefix all metric names with the given string. * * @param prefix the prefix for all metric names * @return {@code this} */ public Builder prefixedWith(String prefix) { this.prefix = prefix; return this; } /** * Convert rates to the given time unit. * * @param rateUnit a unit of time * @return {@code this} */ public Builder convertRatesTo(TimeUnit rateUnit) { this.rateUnit = rateUnit; return this; } /** * Convert durations to the given time unit. * * @param durationUnit a unit of time * @return {@code this} */ public Builder convertDurationsTo(TimeUnit durationUnit) { this.durationUnit = durationUnit; return this; } /** * Only report metrics which match the given filter. * * @param filter a {@link MetricFilter} * @return {@code this} */ public Builder filter(MetricFilter filter) { this.filter = filter; return this; } /** * Builds a {@link GraphiteReporter} with the given properties, sending metrics using the * given {@link GraphiteSender}. * * Present for binary compatibility * * @param graphite a {@link Graphite} * @return a {@link GraphiteReporter} */ public GraphiteReporter build(Graphite graphite) { return build((GraphiteSender) graphite); } /** * Builds a {@link GraphiteReporter} with the given properties, sending metrics using the * given {@link GraphiteSender}. * * @param graphite a {@link GraphiteSender} * @return a {@link GraphiteReporter} */ public GraphiteReporter build(GraphiteSender graphite) { return new GraphiteReporter(registry, graphite, clock, prefix, rateUnit, durationUnit, filter); } } private static final Logger LOGGER = LoggerFactory.getLogger(GraphiteReporter.class); private final GraphiteSender graphite; private final Clock clock; private final String prefix; private GraphiteReporter(MetricRegistry registry, GraphiteSender graphite, Clock clock, String prefix, TimeUnit rateUnit, TimeUnit durationUnit, MetricFilter filter) { super(registry, "graphite-reporter", filter, rateUnit, durationUnit); this.graphite = graphite; this.clock = clock; this.prefix = prefix; } @Override public void report(SortedMap gauges, SortedMap counters, SortedMap histograms, SortedMap meters, SortedMap timers) { final long timestamp = clock.getTime() / 1000; // oh it'd be lovely to use Java 7 here try { if (!graphite.isConnected()) { graphite.connect(); } for (Map.Entry entry : gauges.entrySet()) { reportGauge(entry.getKey(), entry.getValue(), timestamp); } for (Map.Entry entry : counters.entrySet()) { reportCounter(entry.getKey(), entry.getValue(), timestamp); } for (Map.Entry entry : histograms.entrySet()) { reportHistogram(entry.getKey(), entry.getValue(), timestamp); } for (Map.Entry entry : meters.entrySet()) { reportMetered(entry.getKey(), entry.getValue(), timestamp); } for (Map.Entry entry : timers.entrySet()) { reportTimer(entry.getKey(), entry.getValue(), timestamp); } graphite.flush(); } catch (IOException e) { LOGGER.warn("Unable to report to Graphite", graphite, e); try { graphite.close(); } catch (IOException e1) { LOGGER.warn("Error closing Graphite", graphite, e1); } } } @Override public void stop() { try { super.stop(); } finally { try { graphite.close(); } catch (IOException e) { LOGGER.debug("Error disconnecting from Graphite", graphite, e); } } } private void reportTimer(String name, Timer timer, long timestamp) throws IOException { final Snapshot snapshot = timer.getSnapshot(); graphite.send(prefix(name, "max"), format(convertDuration(snapshot.getMax())), timestamp); graphite.send(prefix(name, "mean"), format(convertDuration(snapshot.getMean())), timestamp); graphite.send(prefix(name, "min"), format(convertDuration(snapshot.getMin())), timestamp); graphite.send(prefix(name, "stddev"), format(convertDuration(snapshot.getStdDev())), timestamp); graphite.send(prefix(name, "p50"), format(convertDuration(snapshot.getMedian())), timestamp); graphite.send(prefix(name, "p75"), format(convertDuration(snapshot.get75thPercentile())), timestamp); graphite.send(prefix(name, "p95"), format(convertDuration(snapshot.get95thPercentile())), timestamp); reportMetered(name, timer, timestamp); } private void reportMetered(String name, Metered meter, long timestamp) throws IOException { graphite.send(prefix(name, "count"), format(meter.getCount()), timestamp); } private void reportHistogram(String name, Histogram histogram, long timestamp) throws IOException { final Snapshot snapshot = histogram.getSnapshot(); graphite.send(prefix(name, "count"), format(histogram.getCount()), timestamp); graphite.send(prefix(name, "max"), format(snapshot.getMax()), timestamp); graphite.send(prefix(name, "mean"), format(snapshot.getMean()), timestamp); graphite.send(prefix(name, "min"), format(snapshot.getMin()), timestamp); graphite.send(prefix(name, "stddev"), format(snapshot.getStdDev()), timestamp); graphite.send(prefix(name, "p50"), format(snapshot.getMedian()), timestamp); graphite.send(prefix(name, "p75"), format(snapshot.get75thPercentile()), timestamp); graphite.send(prefix(name, "p95"), format(snapshot.get95thPercentile()), timestamp); } private void reportCounter(String name, Counter counter, long timestamp) throws IOException { graphite.send(prefix(name, "count"), format(counter.getCount()), timestamp); } private void reportGauge(String name, Gauge gauge, long timestamp) throws IOException { final String value = format(gauge.getValue()); if (value != null) { graphite.send(prefix(name), value, timestamp); } } private String format(Object o) { if (o instanceof Float) { return format(((Float) o).doubleValue()); } else if (o instanceof Double) { return format(((Double) o).doubleValue()); } else if (o instanceof Byte) { return format(((Byte) o).longValue()); } else if (o instanceof Short) { return format(((Short) o).longValue()); } else if (o instanceof Integer) { return format(((Integer) o).longValue()); } else if (o instanceof Long) { return format(((Long) o).longValue()); } return null; } private String prefix(String... components) { return MetricRegistry.name(prefix, components); } private String format(long n) { return Long.toString(n); } private String format(double v) { // the Carbon plaintext format is pretty underspecified, but it seems like it just wants // US-formatted digits return String.format(Locale.US, "%2.2f", v); } } trapperkeeper-metrics-1.3.1/test/000077500000000000000000000000001370145725700170235ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/test/puppetlabs/000077500000000000000000000000001370145725700212025ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/test/puppetlabs/metrics_test.clj000066400000000000000000000057751370145725700244170ustar00rootroot00000000000000(ns puppetlabs.metrics-test (:import (com.codahale.metrics MetricRegistry Meter Timer) (java.util.concurrent TimeUnit)) (:require [clojure.test :refer :all] [puppetlabs.metrics :refer :all] [schema.test :as schema-test])) (use-fixtures :once schema-test/validate-schemas) (deftest test-host-metric-name (testing "host-metric-name should create a metric name" (is (= "puppetlabs.localhost.foocount" (host-metric-name "localhost" "foocount"))))) (deftest test-http-metric-name (testing "http-metric-name should create a metric name" (is (= "puppetlabs.localhost.http.foocount" (http-metric-name "localhost" "foocount"))))) (deftest test-register (testing "register should add a metric to the registry" (let [registry (MetricRegistry.)] (register registry (host-metric-name "localhost" "foo") (gauge 2)) (let [gauges (.getGauges registry)] (is (= 1 (count gauges))) (is (= "puppetlabs.localhost.foo" (first (.keySet gauges)))))))) (deftest test-mean-utils (let [timer (Timer.)] (time! timer (Thread/sleep 1)) (is (= 1 (.getCount timer))) (let [elapsed-nanos (mean timer) elapsed-millis (mean-millis timer) elapsed-in-millis (mean-in-unit timer TimeUnit/MILLISECONDS) elapsed-in-nanos (mean-in-unit timer TimeUnit/NANOSECONDS)] (is (>= elapsed-millis 1)) (is (>= elapsed-in-millis 1)) (is (<= elapsed-millis 100)) (is (<= elapsed-in-millis 100)) (is (>= elapsed-nanos (* 1000 1000))) (is (>= elapsed-in-nanos (* 1000 1000))) (is (<= elapsed-nanos (* 1000 1000 100))) (is (<= elapsed-in-nanos (* 1000 1000 100)))))) (deftest test-ratio (testing "ratio should create a ratio metric" (let [numerator (atom 4) numerator-fn (fn [] @numerator) denominator-fn (constantly 2) ratio-metric (ratio numerator-fn denominator-fn)] (is (= 2.0 (.. ratio-metric getRatio getValue))) (reset! numerator 6) (is (= 3.0 (.. ratio-metric getRatio getValue)))))) (deftest test-metered-ratio (testing "metered-ratio builds a ratio metric from two counters" (let [numerator-meter (Meter.) denominator-timer (Timer.) ratio-metric (metered-ratio numerator-meter denominator-timer)] (dotimes [_ 2] (.update denominator-timer 0 TimeUnit/MILLISECONDS)) (dotimes [_ 4] (.mark numerator-meter)) (is (= 2.0 (.. ratio-metric getRatio getValue))) (dotimes [_ 2] (.mark numerator-meter)) (is (= 3.0 (.. ratio-metric getRatio getValue)))))) (deftest test-gauge (testing "gauge creates a gauge metric with an initial value" (let [gauge-metric (gauge 42)] (is (= 42 (.getValue gauge-metric)))))) (deftest test-time! (testing "time! will time the enclosed form" (let [timer (Timer.)] (time! timer (Thread/sleep 1)) (is (= 1 (.getCount timer))) (let [elapsed (mean-millis timer)] (is (>= elapsed 1)) (is (<= elapsed 100)))))) trapperkeeper-metrics-1.3.1/test/puppetlabs/trapperkeeper/000077500000000000000000000000001370145725700240535ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/test/puppetlabs/trapperkeeper/services/000077500000000000000000000000001370145725700256765ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/test/puppetlabs/trapperkeeper/services/metrics/000077500000000000000000000000001370145725700273445ustar00rootroot00000000000000trapperkeeper-metrics-1.3.1/test/puppetlabs/trapperkeeper/services/metrics/metrics_core_test.clj000066400000000000000000000532551370145725700335650ustar00rootroot00000000000000(ns puppetlabs.trapperkeeper.services.metrics.metrics-core-test (:import (com.codahale.metrics MetricRegistry JmxReporter) (com.puppetlabs.trapperkeeper.metrics GraphiteReporter) (clojure.lang ExceptionInfo)) (:require [clojure.test :refer :all] [puppetlabs.trapperkeeper.testutils.logging :refer :all] [puppetlabs.trapperkeeper.services.metrics.metrics-core :as core] [puppetlabs.trapperkeeper.services.metrics.metrics-testutils :as utils] [schema.test :as schema-test] [puppetlabs.kitchensink.core :as ks])) (use-fixtures :once schema-test/validate-schemas) (deftest test-initialize-registry-context (testing "initializes registry and adds to context" (let [domain :my.epic.domain context (core/initialize-registry-context {} domain)] (is (instance? MetricRegistry (:registry context))) (is (nil? (:jmx-reporter context))))) (testing "enables jmx reporter if configured to do so" (let [domain :foo.bar.baz context (core/initialize-registry-context {:reporters {:jmx {:enabled true}}} domain)] (is (instance? MetricRegistry (:registry context))) (is (instance? JmxReporter (:jmx-reporter context))))) (testing "does not enable jmx reporter if configured to not do so" (let [domain :foo.bar.baz context (core/initialize-registry-context {:reporters {:jmx {:enabled false}}} domain)] (is (instance? MetricRegistry (:registry context))) (is (nil? (:jmx-reporter context)))))) (deftest test-lifecycle (testing "enables graphite reporter if configured to do so" (let [context (-> utils/test-config (assoc-in [:registries :default :reporters :graphite :enabled] true) core/create-initial-service-context core/add-graphite-reporters) default-registry (get @(:registries context) :default)] (is (instance? MetricRegistry (:registry default-registry))) (is (instance? GraphiteReporter (:graphite-reporter default-registry))) (core/stop default-registry))) (testing "does not enable graphite reporter if not configured to do so" (let [context (-> utils/test-config core/create-initial-service-context core/add-graphite-reporters) default-registry (get @(:registries context) :default)] (is (instance? MetricRegistry (:registry default-registry))) (is (nil? (:graphite-reporter default-registry)))))) (deftest get-graphite-config-test (testing "get-graphite-config function works with `reporters` section" (let [config (utils/build-config-with-registries {:no.graphite.settings {:metrics-allowed ["foo" "bar"]} :enabled.graphite {:reporters {:graphite {:enabled true}}} :new.interval.graphite {:reporters {:graphite {:enabled true :update-interval-seconds 42}}} :disabled.graphite {:reporters {:graphite {:enabled false}}}})] (testing "returns defaults if no registry settings in config" (is (= {:enabled false :host "my.graphite.server" :port 2003 :update-interval-seconds 10} (core/get-graphite-config config :not.in.the.config)))) (testing "returns disabled graphite if no registry-specific graphite settings" (is (= {:enabled false :host "my.graphite.server" :port 2003 :update-interval-seconds 10} (core/get-graphite-config config :no.graphite.settings)))) (testing "returns defaults with enabled graphite for registry with graphite enabled" (is (= {:enabled true :host "my.graphite.server" :port 2003 :update-interval-seconds 10} (core/get-graphite-config config :enabled.graphite)))) (testing "returns overrides for registry with graphite overrides" (is (= {:enabled true :host "my.graphite.server" :port 2003 :update-interval-seconds 42} (core/get-graphite-config config :new.interval.graphite)))) (testing "sets enabled to false if specified for the registry" (is (= {:enabled false :host "my.graphite.server" :port 2003 :update-interval-seconds 10} (core/get-graphite-config config :no.graphite)))) (testing "get-graphite-config function does the right thing without `reporters` section" (testing "returns nil if no graphite settings are in the config" (is (= nil (core/get-graphite-config {:server-id "localhost"} :foo)))) (testing "throws error if output does not include full Graphite config" (is (thrown? ExceptionInfo (core/get-graphite-config {:server-id "localhost" :registries {:incomplete.graphite {:reporters {:graphite {:enabled true :update-interval-seconds 10}}}}} :incomplete.graphite)))) (testing "works if output does include full Graphite config" (is (= {:enabled true :host "my.graphite.server" :port 2003 :update-interval-seconds 10} (core/get-graphite-config {:server-id "localhost" :registries {:graphite.set.in.registry {:reporters (assoc-in utils/graphite-config [:graphite :enabled] true)}}} :graphite.set.in.registry)))))))) (deftest get-or-initialize-test (let [config (utils/build-config-with-registries {:pre-existing {:reporters {:jmx {:enabled true}}}}) context (core/create-initial-service-context config)] (testing "returns existing registry if already created" (let [pre-existing-registry (:pre-existing @(:registries context))] (is (instance? MetricRegistry (:registry pre-existing-registry))) (is (instance? JmxReporter (:jmx-reporter pre-existing-registry))) (is (= pre-existing-registry (core/get-or-initialize-registry-context context :pre-existing))))) (testing "creates a new registry if it does not already exist in context" (is (nil? (:new-registry @(:registries context)))) (let [new-registry (core/get-or-initialize-registry-context context :new-registry)] (is (instance? MetricRegistry (:registry new-registry))) (testing "updates the context to have the new registry" (is (= new-registry (:new-registry @(:registries context))))))))) (deftest add-graphite-reporters-test (testing "add-graphite-reporters function" (let [config (utils/build-config-with-registries {:enabled.graphite {:reporters {:graphite {:enabled true}}} :disabled.graphite {:reporters {:graphite {:enabled false}}} :enabled.graphite.with-defaults {:reporters {:graphite {:enabled true}}} :disabled.graphite.with-defaults {:reporters {:graphite {:enabled false}}} :enabled.graphite.with.metrics-allowed {:metrics-allowed ["foo"] :reporters {:graphite {:enabled true}}} :disabled.graphite.with.metrics-allowed {:metrics-allowed ["foo"] :reporters {:graphite {:enabled false}}}}) context (core/create-initial-service-context config) get-graphite-reporter (fn [domain] (get-in @(:registries context) [domain :graphite-reporter]))] (core/update-registry-settings context :enabled.graphite.with-defaults {:default-metrics-allowed ["bar"]}) (core/update-registry-settings context :disabled.graphite.with-defaults {:default-metrics-allowed ["bar"]}) (core/update-registry-settings context :not-in-config.with-defaults {:default-metrics-allowed ["bar"]}) (core/get-or-initialize-registry-context context :not-in-config) (core/add-graphite-reporters context) (testing "adds graphite reporter for registry with graphite enabled" (is (instance? GraphiteReporter (get-graphite-reporter :enabled.graphite)))) (testing "doesn't add graphite reporter for registry with graphite disabled" (is (nil? (get-graphite-reporter :disabled.graphite)))) (testing (str "adds graphite reporter for registry with default metrics allwowed" " and graphite enabled") (is (instance? GraphiteReporter (get-graphite-reporter :enabled.graphite)))) (testing (str "doesn't add graphite reporter for registry with default metrics allowed" " but graphite disabled") (is (nil? (get-graphite-reporter :disabled.graphite.with-defaults)))) (testing (str "adds graphite reporter for registry with metrics allowed" " and graphite enabled") (is (instance? GraphiteReporter (get-graphite-reporter :enabled.graphite.with.metrics-allowed)))) (testing (str "doesn't add graphite reporter for registry with metrics allowed" " but graphite disabled") (is (nil? (get-graphite-reporter :disabled.graphite.with.metrics-allowed)))) (testing "doesn't add graphite reporter for registry not in config" (is (nil? (get-graphite-reporter :not-in-config)))) (testing "doesn't add graphite for registry with default metrics-allowed not in config" (is (nil? (get-graphite-reporter :not-in-config.with-defaults)))) (core/stop-all context)))) (deftest initialize-registries-from-config-test (testing "initialize-registries-from-config + maybe-add-default-to-config creates default registry" (is (= [:default] (keys (core/initialize-registries-from-config (core/maybe-add-default-to-config {:server-id "localhost"})))))) (testing (str "initialize-registries-from-config + maybe-add-default-to-config creates registries" " for everything in config plus default") (let [registries (core/initialize-registries-from-config (core/maybe-add-default-to-config {:server-id "localhost" :registries {:foo {:reporters {:jmx {:enabled true}} :metrics-allowed ["foo"]}}}))] (is (= #{:default :foo} (ks/keyset registries))) (is (instance? MetricRegistry (get-in registries [:default :registry]))) (is (instance? MetricRegistry (get-in registries [:foo :registry]))) (testing "adds jmx reporter if configured" (is (nil? (get-in registries [:default :jmx-reporter]))) (is (instance? JmxReporter (get-in registries [:foo :jmx-reporter])))))) (testing "initialize-registries-from-config creates default with settings from config" (let [registries (core/initialize-registries-from-config (core/maybe-add-default-to-config {:server-id "localhost" :registries {:default {:reporters {:jmx {:enabled true}}} :foo {:metrics-allowed ["foo"]}}}))] (is (= #{:default :foo} (ks/keyset registries))) (is (instance? MetricRegistry (get-in registries [:default :registry]))) (is (instance? MetricRegistry (get-in registries [:foo :registry]))) (testing "adds jmx reporter if configured" (is (instance? JmxReporter (get-in registries [:default :jmx-reporter]))) (is (nil? (get-in registries [:foo :jmx-reporter]))))))) (deftest update-registry-settings-test (let [context (core/create-initial-service-context utils/test-config)] (testing "update-registry-settings adds settings for a registry" (is (= {:foo.bar {:default-metrics-allowed ["foo.bar"]}} (core/update-registry-settings context :foo.bar {:default-metrics-allowed ["foo.bar"]})))) (testing "update-registry-settings appends settings if the settings already exist" (is (= {:foo.bar {:default-metrics-allowed ["foo.bar" "bar.baz"]}} (core/update-registry-settings context :foo.bar {:default-metrics-allowed ["bar.baz"]})))) (testing "update-registry-settings throws an error if it is called after init lifecycle phase" (is (thrown? RuntimeException (core/update-registry-settings (core/lock-registry-settings context) :nope {:default-metrics-allowed ["default"]})))) ; Make sure all the graphite reporters get shutdown, otherwise they spawn background threads (core/stop-all context))) (deftest ^:unit construct-metric-names-test (testing "prefix is correctly added" (let [metrics ["one" "two" "two"] prefixed-metrics (core/construct-metric-names "foo.bar" metrics)] (is (= #{"foo.bar.one" "foo.bar.two"} prefixed-metrics))))) (deftest get-metrics-allowed-test (testing "get-metrics-allowed function") (let [config (utils/build-config-with-registries {:with-metrics-allowed {:metrics-allowed ["foo"]} :with-metrics-allowed-and-defaults {:metrics-allowed ["foo"]} :with-prefix {:metric-prefix "prefix"} :with-prefix-and-defaults {:metric-prefix "prefix"} :with-prefix-and-metrics-allowed {:metric-prefix "prefix" :metrics-allowed ["foo"]} :with-prefix-defaults-and-metrics-allowed {:metric-prefix "prefix" :metrics-allowed ["foo"]} :without-metrics-allowed-or-prefix {:reporters {:jmx {:enabled true}}}})] (testing "works with metrics allowed in config" (is (= #{"puppetlabs.localhost.foo"} (core/get-metrics-allowed config {} :with-metrics-allowed)))) (testing "works with default metrics allowed" (is (= #{"puppetlabs.localhost.default"} (core/get-metrics-allowed config {:with-defaults {:default-metrics-allowed ["default"]}} :with-defaults)))) (testing "works with metrics allowed in config and defaults" (is (= #{"puppetlabs.localhost.foo" "puppetlabs.localhost.default"} (core/get-metrics-allowed config {:with-metrics-allowed-and-defaults {:default-metrics-allowed ["default"]}} :with-metrics-allowed-and-defaults)))) (testing "removes duplicates between metrics allowed in config and defaults" (is (= #{"puppetlabs.localhost.foo" "puppetlabs.localhost.default"} (core/get-metrics-allowed config {:with-metrics-allowed-and-defaults {:default-metrics-allowed ["default" "foo"]}} :with-metrics-allowed-and-defaults)))) (testing "returns empty set if prefix but not metrics allowed" (is (= #{} (core/get-metrics-allowed config {} :with-prefix)))) (testing "works with prefix and defaults" (is (= #{"prefix.default"} (core/get-metrics-allowed config {:with-prefix-and-defaults {:default-metrics-allowed ["default"]}} :with-prefix-and-defaults)))) (testing "works with prefix and metrics allowed" (is (= #{"prefix.foo"} (core/get-metrics-allowed config {} :with-prefix-and-metrics-allowed)))) (testing "works with prefix, defaults, and metrics allowed in config" (is (= #{"prefix.default" "prefix.foo"} (core/get-metrics-allowed config {:with-prefix-defaults-and-metrics-allowed {:default-metrics-allowed ["default"]}} :with-prefix-defaults-and-metrics-allowed)))) (testing "returns empty set without metrics allowed or prefix" (is (= #{} (core/get-metrics-allowed config {} :without-metrics-allowed-or-prefix)))) (testing "returns empty set for registry not in config" (is (= #{} (core/get-metrics-allowed config {} :not-in-config)))) (testing "returns empty set when there is no registry config" (is (= #{} (core/get-metrics-allowed {:server-id "localhost"} {} :not-in-config)))))) (deftest ^:unit allowed-names-metrics-filter-match-test ; Wrap .matches so nil doesn't have to be passed every time (let [matches (fn [metric-filter name] (.matches metric-filter name nil))] (let [metrics-allowed #{"foo" "example.domain" "example.domain.more.specific" "compiler.evaluate_resource.Class[Puppet_enterprise::Profile::Console]"} metric-filter (core/build-metric-filter metrics-allowed)] (testing "allowed strings match" (is (true? (matches metric-filter "foo"))) (is (true? (matches metric-filter "example.domain"))) (is (true? (matches metric-filter "example.domain.more.specific"))) (is (true? (matches metric-filter "compiler.evaluate_resource.Class[Puppet_enterprise::Profile::Console]")))) (testing "non allowed strings don't match" (is (false? (matches metric-filter "foo.bar"))) (is (false? (matches metric-filter "example.domain.more"))) (is (false? (matches metric-filter "example"))) (is (false? (matches metric-filter "Class[Puppet_enterprise::Profile::Console]"))))) (testing "empty allow-metrics list allows everything" (let [metric-filter (core/build-metric-filter #{})] (is (matches metric-filter "anything")))))) (deftest get-metric-prefix-test (testing "get-metric-prefix chooses the correct prefix" (testing "no metric-prefix provided" (is (= "puppetlabs.localhost" (core/get-metric-prefix utils/test-config :default)))) (testing "metric-prefix provided" (let [config (assoc-in utils/test-config [:registries :default :metric-prefix] "test-prefix")] (is (= "test-prefix" (core/get-metric-prefix config :default))))))) (deftest graphite-reporter-filter-test (let [registry-context (core/initialize-registry-context nil :default) registry (:registry registry-context) reported-metrics (atom {}) metrics-allowed #{"test-histogram" "test-meter" "test-timer"} graphite-reporter (core/build-graphite-reporter registry metrics-allowed (utils/make-graphite-sender reported-metrics :default))] ; Create a few Metric objects of the types we'd like to test (.histogram registry "test-histogram") (.histogram registry "should-not-be-reported") (.meter registry "test-meter") (.timer registry "test-timer") ; Call report once so our GraphiteSender instance can collect some data (.report graphite-reporter) (testing "Graphite reporter only sends whitelisted metrics" (is (utils/reported? @reported-metrics "test-histogram.mean")) (is (not (utils/reported? @reported-metrics "should-not-be-reported.mean")))) (testing "Graphite reporter sends desired histogram fields" (is (utils/reported? @reported-metrics "test-histogram.mean")) (is (utils/reported? @reported-metrics "test-histogram.min")) (is (utils/reported? @reported-metrics "test-histogram.max")) (is (utils/reported? @reported-metrics "test-histogram.stddev")) (is (utils/reported? @reported-metrics "test-histogram.p50")) (is (utils/reported? @reported-metrics "test-histogram.p75")) (is (utils/reported? @reported-metrics "test-histogram.p95"))) (testing "Graphite reporter doesn't send any other histogram fields" (is (not (utils/reported? @reported-metrics "test-histogram.p98"))) (is (not (utils/reported? @reported-metrics "test-histogram.p99"))) (is (not (utils/reported? @reported-metrics "test-histogram.p999")))) (testing "Graphite reporter sends desired timer fields" (is (utils/reported? @reported-metrics "test-timer.mean")) (is (utils/reported? @reported-metrics "test-timer.min")) (is (utils/reported? @reported-metrics "test-timer.max")) (is (utils/reported? @reported-metrics "test-timer.stddev")) (is (utils/reported? @reported-metrics "test-timer.p50")) (is (utils/reported? @reported-metrics "test-timer.p75")) (is (utils/reported? @reported-metrics "test-timer.p95"))) (testing "Graphite reporter doesn't send any other timer fields" (is (not (utils/reported? @reported-metrics "test-timer.p98"))) (is (not (utils/reported? @reported-metrics "test-timer.p99"))) (is (not (utils/reported? @reported-metrics "test-timer.p999")))) (testing "Graphite reporter sends desired meter fields" (is (utils/reported? @reported-metrics "test-meter.count"))) (testing "Graphite reporter doesn't send any other meter fields" (is (not (utils/reported? @reported-metrics "test-meter.mean_rate"))) (is (not (utils/reported? @reported-metrics "test-meter.m1_rate"))) (is (not (utils/reported? @reported-metrics "test-meter.m5_rate"))) (is (not (utils/reported? @reported-metrics "test-meter.m15_rate")))))) trapperkeeper-metrics-1.3.1/test/puppetlabs/trapperkeeper/services/metrics/metrics_service_test.clj000066400000000000000000000627431370145725700342770ustar00rootroot00000000000000(ns puppetlabs.trapperkeeper.services.metrics.metrics-service-test (:import (com.codahale.metrics MetricRegistry JmxReporter) (clojure.lang ExceptionInfo) (com.puppetlabs.trapperkeeper.metrics GraphiteReporter)) (:require [clojure.test :refer :all] [cheshire.core :as json] [clojure.string :as string] [ring.util.codec :as codec] [puppetlabs.http.client.sync :as http-client] [puppetlabs.metrics :as metrics] [puppetlabs.trapperkeeper.services.authorization.authorization-service :as authorization-service] [puppetlabs.trapperkeeper.services.metrics.metrics-service :refer :all] [puppetlabs.trapperkeeper.services.protocols.metrics :as metrics-protocol] [schema.test :as schema-test] [puppetlabs.trapperkeeper.services.webrouting.webrouting-service :as webrouting-service] [puppetlabs.trapperkeeper.services.webserver.jetty9-service :as jetty9-service] [puppetlabs.trapperkeeper.testutils.bootstrap :refer [with-app-with-config]] [puppetlabs.trapperkeeper.testutils.logging :refer [with-test-logging]] [puppetlabs.trapperkeeper.app :as app] [puppetlabs.kitchensink.core :as ks] [puppetlabs.trapperkeeper.testutils.logging :as logging] [puppetlabs.trapperkeeper.core :as trapperkeeper] [puppetlabs.trapperkeeper.services.metrics.metrics-testutils :as utils] [puppetlabs.trapperkeeper.services :as tk-services] [puppetlabs.trapperkeeper.services.metrics.metrics-core :as core])) (use-fixtures :once schema-test/validate-schemas) (defn parse-response ([resp] (parse-response resp false)) ([resp keywordize?] (-> resp :body slurp (json/parse-string keywordize?)))) (defn jolokia-encode "Encodes a MBean name according to the rules laid out in: https://jolokia.org/reference/html/protocol.html#escape-rules" [mbean-name] (-> mbean-name (string/escape {\/ "!/" \! "!!" \" "!\""}) codec/url-encode)) (def test-resources-dir (ks/absolute-path "./dev-resources/puppetlabs/trapperkeeper/services/metrics/metrics_service_test")) (def services [jetty9-service/jetty9-service webrouting-service/webrouting-service metrics-service metrics-webservice]) (def ssl-opts {:ssl-cert "./dev-resources/ssl/cert.pem" :ssl-key "./dev-resources/ssl/key.pem" :ssl-ca-cert "./dev-resources/ssl/ca.pem"}) (def ssl-webserver-config {:webserver (merge {:ssl-port 8180 :ssl-host "0.0.0.0"} ssl-opts)}) (def metrics-service-config {:metrics {:server-id "localhost" :registries {:pl.test.reg {:reporters {:jmx {:enabled true}}} :pl.other.reg {:reporters {:jmx {:enabled true}}}}} :webserver {:port 8180 :host "0.0.0.0"} :web-router-service {:puppetlabs.trapperkeeper.services.metrics.metrics-service/metrics-webservice "/metrics"}}) (def auth-config {:authorization {:version 1 :rules [{:match-request {:path "/metrics/v2/list" :type "path" :method ["get" "head" "post" "put"]} :allow "localhost" :sort-order 500 :name "list"} {:match-request {:path "/metrics/v2" :type "path" :method ["get" "head" "post" "put"]} :deny "localhost" :sort-order 500 :name "metrics"}]}}) (deftest test-metrics-service-error (testing "Metrics service throws an error if missing server-id" (logging/with-test-logging (is (thrown-with-msg? ExceptionInfo #"Value does not match schema: .*server-id missing-required-key.*" (with-app-with-config app services (ks/dissoc-in metrics-service-config [:metrics :server-id]))))))) (deftest test-metrics-service (testing "Can boot metrics service and access registry" (with-app-with-config app services (assoc-in metrics-service-config [:metrics :metrics-webservice :mbeans :enabled] true) (testing "metrics service functions" (let [svc (app/get-service app :MetricsService)] (testing "`get-metrics-registry` called without domain works" (is (instance? MetricRegistry (metrics-protocol/get-metrics-registry svc)))) (testing "`get-metrics-registry` called with domain works" (is (instance? MetricRegistry (metrics-protocol/get-metrics-registry svc :pl.foo.reg)))) (testing "`get-server-id` works" (is (= "localhost" (metrics-protocol/get-server-id svc)))))) (testing "returns latest status for all services" (let [resp (http-client/get "http://localhost:8180/metrics/v1/mbeans") body (parse-response resp)] (is (= 200 (:status resp))) (doseq [[metric path] body :let [resp (http-client/get (str "http://localhost:8180/metrics/v1" path))]] (is (= 200 (:status resp))))) (let [resp (http-client/get "http://localhost:8180/metrics/v2/list") body (parse-response resp)] (is (= 200 (:status resp))) (doseq [[namesp mbeans] (get body "value") mbean (keys mbeans) :let [url (str "http://localhost:8180/metrics/v2/read/" (jolokia-encode (str namesp ":" mbean)) ;; NOTE: Some memory pools intentionally don't ;; implement MBean attributes. This results ;; in an error being thrown when those ;; attributes are read and is expected. "?ignoreErrors=true") resp (http-client/get url) body (parse-response resp)]] ;; NOTE: Jolokia returns 200 OK for most responses. The actual ;; status code is in the JSON payload that makes up the body. (is (= 200 (get body "status")))))) (testing "register should add a metric to the registry with a keyword domain" (let [svc (app/get-service app :MetricsService) register-and-get-metric (fn [domain metric] (metrics/register (metrics-protocol/get-metrics-registry svc domain) (metrics/host-metric-name "localhost" metric) (metrics/gauge 2)) (http-client/get (str "http://localhost:8180/metrics/v1/mbeans/" (codec/url-encode (str (name domain) ":name=puppetlabs.localhost." metric))))) resp (register-and-get-metric :pl.test.reg "foo")] (is (= 200 (:status resp))) (is (= {"Value" 2} (parse-response resp))))) (testing "querying multiple metrics via POST should work" (let [svc (app/get-service app :MetricsService) registry (metrics-protocol/get-metrics-registry svc :pl.other.reg)] (metrics/register registry (metrics/host-metric-name "localhost" "foo") (metrics/gauge 2)) (metrics/register registry (metrics/host-metric-name "localhost" "bar") (metrics/gauge 500)) (let [resp (http-client/post "http://localhost:8180/metrics/v1/mbeans" {:body (json/generate-string ["pl.other.reg:name=puppetlabs.localhost.foo" "pl.other.reg:name=puppetlabs.localhost.bar"])}) body (parse-response resp)] (is (= 200 (:status resp))) (is (= [{"Value" 2} {"Value" 500}] body))) (let [resp (http-client/post "http://localhost:8180/metrics/v1/mbeans" {:body (json/generate-string {:foo "pl.other.reg:name=puppetlabs.localhost.foo" :bar "pl.other.reg:name=puppetlabs.localhost.bar"})}) body (parse-response resp)] (is (= 200 (:status resp))) (is (= {"foo" {"Value" 2} "bar" {"Value" 500}} body))) (let [resp (http-client/post "http://localhost:8180/metrics/v1/mbeans" {:body (json/generate-string "pl.other.reg:name=puppetlabs.localhost.foo")}) body (parse-response resp)] (is (= 200 (:status resp))) (is (= {"Value" 2} body))) (let [resp (http-client/post "http://localhost:8180/metrics/v1/mbeans" {:body "{\"malformed json"}) body (slurp (:body resp))] (is (= 400 (:status resp))) (is (re-find #"Unexpected end-of-input" body))) (let [resp (http-client/post "http://localhost:8180/metrics/v2" {:body (json/generate-string [{:type "read" :mbean "pl.other.reg:name=puppetlabs.localhost.foo"} {:type "read" :mbean "pl.other.reg:name=puppetlabs.localhost.bar"}])}) body (parse-response resp true)] (is (= [200 200] (map :status body))) (is (= [{:Value 2} {:Value 500}] (map :value body)))))) (testing "metrics/v2 should deny write requests" (with-test-logging (let [resp (http-client/get (str "http://localhost:8180/metrics/v2/write/" (jolokia-encode "java.lang:type=Memory") "/Verbose/true")) body (parse-response resp)] (is (= 403 (get body "status")))))) (testing "metrics/v2 should deny exec requests" (with-test-logging (let [resp (http-client/get (str "http://localhost:8180/metrics/v2/exec/" (jolokia-encode "java.util.logging:type=Logging") "/getLoggerLevel/root")) body (parse-response resp)] (is (= 403 (get body "status"))))))))) (deftest metrics-service-with-tk-auth (testing "tk-auth works when included in bootstrap" (with-app-with-config app (conj services authorization-service/authorization-service) (merge metrics-service-config auth-config ssl-webserver-config) (let [resp (http-client/get "https://localhost:8180/metrics/v2/list" ssl-opts)] (is (= 200 (:status resp)))) (let [resp (http-client/get "https://localhost:8180/metrics/v2" ssl-opts)] (is (= 403 (:status resp))))))) (deftest metrics-v1-endpoint-disabled-by-default (testing "metrics/v1 is disabled by default, returns 404" (with-app-with-config app [jetty9-service/jetty9-service webrouting-service/webrouting-service metrics-service metrics-webservice] metrics-service-config (let [resp (http-client/get "http://localhost:8180/metrics/v1/mbeans")] (is (= 404 (:status resp))))))) (deftest metrics-endpoint-with-jolokia-disabled-test (testing "metrics/v2 returns 404 when Jolokia is not enabled" (let [config (assoc-in metrics-service-config [:metrics :metrics-webservice :jolokia :enabled] false)] (with-app-with-config app [jetty9-service/jetty9-service webrouting-service/webrouting-service metrics-service metrics-webservice] config (let [resp (http-client/get "http://localhost:8180/metrics/v2/version")] (is (= 404 (:status resp)))))))) (deftest metrics-endpoint-with-permissive-jolokia-policy (testing "metrics/v2 allows exec requests when configured with a permissive policy" (let [config (assoc-in metrics-service-config [:metrics :metrics-webservice :jolokia :servlet-init-params :policyLocation] (str "file://" test-resources-dir "/jolokia-access-permissive.xml"))] (with-app-with-config app [jetty9-service/jetty9-service webrouting-service/webrouting-service metrics-service metrics-webservice] config (let [resp (http-client/get (str "http://localhost:8180/metrics/v2/exec/" (jolokia-encode "java.util.logging:type=Logging") "/getLoggerLevel/root")) body (parse-response resp)] (is (= 200 (get body "status")))))))) (deftest metrics-endpoint-with-jmx-disabled-test (testing "returns data for jvm even when jmx is not enabled" (let [config (-> metrics-service-config (assoc-in [:metrics :metrics-webservice :mbeans :enabled] true) (assoc :registries {:pl.no.jmx {:reporters {:jmx {:enabled false}}}}))] (with-app-with-config app [jetty9-service/jetty9-service webrouting-service/webrouting-service metrics-service metrics-webservice] config (testing "returns latest status for all services" (let [resp (http-client/get "http://localhost:8180/metrics/v1/mbeans") body (parse-response resp)] (is (= 200 (:status resp))) (is (not (empty? body))))) (testing "returns Memoory mbean information" (let [resp (http-client/get "http://localhost:8180/metrics/v1/mbeans/java.lang%3Atype%3DMemory") body (parse-response resp) heap-memory (get body "HeapMemoryUsage")] (is (= 200 (:status resp))) (is (= #{"committed" "init" "max" "used"} (ks/keyset heap-memory))) (is (every? #(< 0 %) (vals heap-memory))))))))) (deftest get-metrics-registry-service-function-test (with-app-with-config app [metrics-service] {:metrics {:server-id "localhost"}} (let [svc (app/get-service app :MetricsService)] (testing "Can access default registry" (is (instance? MetricRegistry (metrics-protocol/get-metrics-registry svc)))) (testing "Can access other registries registry" (is (instance? MetricRegistry (metrics-protocol/get-metrics-registry svc :my-domain))))))) (deftest get-server-id-service-function-test (with-app-with-config app [metrics-service] {:metrics {:server-id "foo"}} (let [svc (app/get-service app :MetricsService)] (testing "Can access server-id" (is (= "foo" (metrics-protocol/get-server-id svc))))))) (deftest update-registry-settings-service-function-test (testing "intialize-registry-settings adds settings for a registry" (let [service (trapperkeeper/service [[:MetricsService update-registry-settings]] (init [this context] (update-registry-settings :foo.bar {:default-metrics-allowed ["foo.bar"]}) context)) metrics-app (trapperkeeper/build-app [service metrics-service] {:metrics utils/test-config}) metrics-svc (app/get-service metrics-app :MetricsService)] (try (app/init metrics-app) (is (= {:foo.bar {:default-metrics-allowed ["foo.bar"]}} @(:registry-settings (tk-services/service-context metrics-svc)))) (finally (app/stop metrics-app))))) (testing "update-registry-settings throws an error if called outside `init`" (let [service (trapperkeeper/service [[:MetricsService update-registry-settings]] (start [this context] (update-registry-settings :nope {:default-metrics-allowed ["fail"]}) context)) metrics-app (trapperkeeper/build-app [service metrics-service] {:metrics utils/test-config})] (with-test-logging (try (is (thrown? RuntimeException (app/check-for-errors! (app/start metrics-app)))) (finally (app/stop metrics-app))))))) (deftest jmx-enabled-globally-deprecated-test (with-test-logging (with-app-with-config app [metrics-service] {:metrics {:server-id "localhost" :reporters {:jmx {:enabled true}}}}) (is (logged? #"Enabling JMX globally is deprecated; JMX can only be enabled per-registry.")))) (deftest jmx-works-test (with-app-with-config app [metrics-service] {:metrics {:server-id "localhost" :registries {:jmx.registry {:reporters {:jmx {:enabled true}}} :no.jmx.registry {:reporters {:jmx {:enabled false}}} :foo {:metrics-allowed ["foo"]}}}} (let [svc (app/get-service app :MetricsService) context (tk-services/service-context svc) get-jmx-reporter (fn [domain] (get-in @(:registries context) [domain :jmx-reporter]))] (testing "Registry with jmx enabled gets a jmx reporter" (metrics-protocol/get-metrics-registry svc :jmx.registry) (is (instance? JmxReporter (get-jmx-reporter :jmx.registry)))) (testing "Registry with jmx disabled does not get a jmx reporter" (metrics-protocol/get-metrics-registry svc :jmx.registry) (is (nil? (get-jmx-reporter :no.jmx.registry)))) (testing "Registry with no mention of jmx does not get a jmx reporter" (metrics-protocol/get-metrics-registry svc :foo) (is (nil? (get-jmx-reporter :foo)))) (testing "Registry not mentioned in config does not get a jmx reporter" (metrics-protocol/get-metrics-registry svc :not.in.the.config) (is (nil? (get-jmx-reporter :not.in.the.config))))))) (defn create-meters! [registries meter-names] (doseq [{:keys [registry]} (vals registries) meter meter-names] (.meter registry meter ))) (defn report-to-graphite! [registries] (doseq [graphite-reporter (map :graphite-reporter (vals registries))] (when graphite-reporter (.report graphite-reporter)))) (def graphite-enabled {:reporters {:graphite {:enabled true}}}) (def metrics-allowed {:metrics-allowed ["not-default"]}) (def default-metrics-allowed {:default-metrics-allowed ["default"]}) (deftest integration-test (let [registries-config {:graphite-enabled graphite-enabled :graphite-with-default-metrics-allowed graphite-enabled :graphite-with-metrics-allowed (merge metrics-allowed graphite-enabled) :graphite-with-defaults-and-metrics-allowed (merge metrics-allowed graphite-enabled)} config {:metrics (utils/build-config-with-registries registries-config)} service (trapperkeeper/service [[:MetricsService get-metrics-registry update-registry-settings]] (init [this context] (get-metrics-registry :graphite-enabled) (get-metrics-registry :graphite-with-default-metrics-allowed) (update-registry-settings :graphite-with-default-metrics-allowed default-metrics-allowed) (get-metrics-registry :graphite-with-metrics-allowed) ;; shouldn't matter whether `get-metrics-registry` or ;; `update-registry-settings` is called first (update-registry-settings :graphite-with-defaults-and-metrics-allowed default-metrics-allowed) (get-metrics-registry :graphite-with-defaults-and-metrics-allowed) context)) reported-metrics-atom (atom {})] (with-redefs [core/build-graphite-sender (fn [_ domain] (utils/make-graphite-sender reported-metrics-atom domain))] (let [metrics-app (trapperkeeper/build-app [metrics-service service] config) metrics-svc (app/get-service metrics-app :MetricsService) get-context (fn [] (tk-services/service-context metrics-svc))] (try (testing "init phase of lifecycle" (app/init metrics-app) (let [context (get-context) registries @(:registries context) registry-settings @(:registry-settings context)] (testing "all registries in config (plus default) get created" (is (= #{:default :graphite-enabled :graphite-with-default-metrics-allowed :graphite-with-metrics-allowed :graphite-with-defaults-and-metrics-allowed} (ks/keyset registries))) (is (every? #(instance? MetricRegistry %) (map :registry (vals registries))))) (testing "graphite reporters are not created in it" (is (every? nil? (map :graphite-reproter (vals registries))))) (testing "registry settings are initialized in init" (is (= {:graphite-with-default-metrics-allowed default-metrics-allowed :graphite-with-defaults-and-metrics-allowed default-metrics-allowed} registry-settings))))) (testing "start phase of lifecycle" (app/start metrics-app) (let [context (get-context) registries @(:registries context) registry-settings @(:registry-settings context)] (testing "graphite reporters are created in start" (is (every? #(instance? MetricRegistry %) (map :registry (vals registries)))) (is (every? #(instance? GraphiteReporter %) (map :graphite-reporter (vals (dissoc registries :default))))) (is (nil? (get-in registries [:default :graphite-reporter])))) (testing "the right metrics are reported to graphite" (create-meters! registries ["puppetlabs.localhost.default" "puppetlabs.localhost.not-default" "puppetlabs.localhost.foo"]) (report-to-graphite! registries) (let [reported-metrics @reported-metrics-atom] (is (= #{:graphite-enabled :graphite-with-default-metrics-allowed :graphite-with-metrics-allowed :graphite-with-defaults-and-metrics-allowed} (ks/keyset reported-metrics))) (testing "without any metrics filter configured all metrics are reported" (is (utils/reported? reported-metrics :graphite-enabled "puppetlabs.localhost.foo.count")) (is (utils/reported? reported-metrics :graphite-enabled "puppetlabs.localhost.default.count")) (is (utils/reported? reported-metrics :graphite-enabled "puppetlabs.localhost.not-default.count"))) (testing "default metrics are reported to graphite" (is (not (utils/reported? reported-metrics :graphite-with-default-metrics-allowed "puppetlabs.localhost.foo.count"))) (is (utils/reported? reported-metrics :graphite-with-default-metrics-allowed "puppetlabs.localhost.default.count")) (is (not (utils/reported? reported-metrics :graphite-with-default-metrics-allowed "puppetlabs.localhost.not-default.count")))) (testing "configured metrics are reported to graphite" (is (not (utils/reported? reported-metrics :graphite-with-metrics-allowed "puppetlabs.localhost.foo.count"))) (is (not (utils/reported? reported-metrics :graphite-with-metrics-allowed "puppetlabs.localhost.default.count"))) (is (utils/reported? reported-metrics :graphite-with-metrics-allowed "puppetlabs.localhost.not-default.count"))) (testing "configured metrics and default allowed metrics are reported to graphite" (is (not (utils/reported? reported-metrics :graphite-with-defaults-and-metrics-allowed "localhost.foo.count"))) (is (utils/reported? reported-metrics :graphite-with-defaults-and-metrics-allowed "puppetlabs.localhost.default.count")) (is (utils/reported? reported-metrics :graphite-with-defaults-and-metrics-allowed "puppetlabs.localhost.not-default.count"))))))) (finally (app/stop metrics-app))))))) trapperkeeper-metrics-1.3.1/test/puppetlabs/trapperkeeper/services/metrics/metrics_testutils.clj000066400000000000000000000025401370145725700336250ustar00rootroot00000000000000(ns puppetlabs.trapperkeeper.services.metrics.metrics-testutils (:import (com.codahale.metrics.graphite GraphiteSender)) (:require [schema.core :as schema])) (def graphite-config {:graphite {:host "my.graphite.server" :port 2003 :update-interval-seconds 10}}) (def test-config {:server-id "localhost" :reporters graphite-config}) (defn build-config-with-registries [registries] (assoc test-config :registries registries)) (schema/defn make-graphite-sender :- GraphiteSender "Creates a GraphiteSender object which doesn't make any network requests, but instead collects the data from each call to send into the provided atom" [reported-metrics :- (schema/atom {schema/Keyword #{schema/Str}}) domain :- schema/Keyword] (reify GraphiteSender (connect [this]) (send [this name value timestamp] (swap! reported-metrics #(update-in % [domain] clojure.set/union #{name}))) (flush [this]) (isConnected [this] true) (getFailures [this] 0) (close [this]))) (schema/defn reported? "Returns true if the given metric-name is found in reported-metrics" ([reported-metrics metric-name] (reported? reported-metrics :default metric-name)) ([reported-metrics :- {schema/Keyword #{schema/Str}} domain :- schema/Keyword metric-name :- schema/Str] (contains? (get reported-metrics domain) metric-name)))