gitlab-experiment-1.3.0/0000755000004100000410000000000015151343620015163 5ustar www-datawww-datagitlab-experiment-1.3.0/lib/0000755000004100000410000000000015151343620015731 5ustar www-datawww-datagitlab-experiment-1.3.0/lib/generators/0000755000004100000410000000000015151343617020110 5ustar www-datawww-datagitlab-experiment-1.3.0/lib/generators/gitlab/0000755000004100000410000000000015151343617021352 5ustar www-datawww-datagitlab-experiment-1.3.0/lib/generators/gitlab/experiment/0000755000004100000410000000000015151343617023532 5ustar www-datawww-datagitlab-experiment-1.3.0/lib/generators/gitlab/experiment/templates/0000755000004100000410000000000015151343617025530 5ustar www-datawww-datagitlab-experiment-1.3.0/lib/generators/gitlab/experiment/templates/experiment.rb.tt0000644000004100000410000000566115151343617030673 0ustar www-datawww-data# frozen_string_literal: true <% if namespaced? -%> require_dependency "<%= namespaced_path %>/application_experiment" <% end -%> <% module_namespacing do -%> class <%= class_name %>Experiment < ApplicationExperiment # Describe your experiment: # # The variant behaviors defined here will be called whenever the experiment # is run unless overrides are provided. <% variants.each do |variant| -%> <% if %w[control candidate].include?(variant) -%> <%= variant %> { } <% else -%> variant(:<%= variant %>) { } <% end -%> <% end -%> <% unless options[:skip_comments] -%> # You can register a `control`, `candidate`, or by naming variants directly. # All of these can be registered using blocks, or by specifying a method. # # Here's some ways you might want to register your control logic: # #control { 'class level control' } # yield this block #control :custom_control # call a private method #control # call the private `control_behavior` method # # You can register candidate logic in the same way: # #candidate { 'class level candidate' } # yield this block #candidate :custom_candidate # call a private method #candidate # call the private `candidate_behavior` method # # For named variants it's the same, but a variant name must be provided: # #variant(:example) { 'class level example variant' } #variant(:example) :example_variant #variant(:example) # call the private `example_behavior` method # # Advanced customization: # # Some additional tools are provided to exclude and segment contexts. To # exclude a given context, you can provide rules. For example, we could # exclude all old accounts and all users with a specific first name. # #exclude :old_account?, ->{ context.user.first_name == 'Richard' } # # Segmentation allows for logic to be used to determine which variant a # context will be assigned. Let's say you want to put all old accounts into a # specific variant, and all users with a specific first name in another: # #segment :old_account?, variant: :variant_two #segment(variant: :variant_one) { context.actor.first_name == 'Richard' } # # Utilizing your experiment: # # Once you've defined your experiment, you can run it elsewhere. You'll want # to specify a context (you can read more about context here), and overrides # for any or all of the variants you've registered in your experiment above. # # Here's an example of running the experiment that's sticky to current_user, # with an override for our class level candidate logic: # # experiment(:<%= file_name %>, user: current_user) do |e| # e.candidate { 'override <%= class_name %>Experiment behavior' } # end # # If you want to publish the experiment to the client without running any # code paths on the server, you can simply call publish instead of passing an # experimental block: # # experiment(:<%= file_name %>, project: project).publish # <% end -%> end <% end -%> gitlab-experiment-1.3.0/lib/generators/gitlab/experiment/install/0000755000004100000410000000000015151343617025200 5ustar www-datawww-datagitlab-experiment-1.3.0/lib/generators/gitlab/experiment/install/templates/0000755000004100000410000000000015151343617027176 5ustar www-datawww-data././@LongLink0000644000000000000000000000015000000000000011577 Lustar rootrootgitlab-experiment-1.3.0/lib/generators/gitlab/experiment/install/templates/application_experiment.rb.ttgitlab-experiment-1.3.0/lib/generators/gitlab/experiment/install/templates/application_experiment.rb0000644000004100000410000000012415151343617034263 0ustar www-datawww-data# frozen_string_literal: true class ApplicationExperiment < Gitlab::Experiment end gitlab-experiment-1.3.0/lib/generators/gitlab/experiment/install/templates/initializer.rb.tt0000644000004100000410000001445315151343617032503 0ustar www-datawww-data# frozen_string_literal: true Gitlab::Experiment.configure do |config| # Prefix all experiment names with a given string value. # Use `nil` for no prefix. config.name_prefix = nil # The logger can be used to log various details of the experiments. config.logger = Logger.new($stdout) # The base class that should be instantiated for basic experiments. # It should be a string, so we can constantize it later. config.base_class = 'ApplicationExperiment' # Require experiments to be defined in a class, with variants registered. # This will disallow any anonymous experiments that are run inline # without previously defining a class. config.strict_registration = false # The caching layer is expected to match the Rails.cache interface. # If no cache is provided some rollout strategies may behave differently. # Use `nil` for no caching. config.cache = nil # The domain to use on cookies. # # When not set, it uses the current host. If you want to provide specific # hosts, you use `:all`, or provide an array. # # Examples: # nil, :all, or ['www.gitlab.com', '.gitlab.com'] config.cookie_domain = :all # Mark experiment cookies as secure (HTTPS only). # # When set to true, cookies will have the secure flag set, meaning they # will only be sent over HTTPS connections. Defaults to true. # # Set to false in development/test environments if needed: # config.secure_cookie = Rails.env.production? # The default rollout strategy. # # The recommended default rollout strategy when not using caching would # be `Gitlab::Experiment::Rollout::Percent` as that will consistently # assign the same variant with or without caching. # # Gitlab::Experiment::Rollout::Base can be inherited to implement your # own rollout strategies. # # Each experiment can specify its own rollout strategy: # # class ExampleExperiment < ApplicationExperiment # default_rollout :random # :percent, :round_robin, or MyCustomRollout # end # # Included rollout strategies: # :percent (recommended), :round_robin, or :random config.default_rollout = :percent # Secret seed used in generating context keys. # # You'll typically want to use an environment variable or secret value # for this. # # Consider not using one that's shared with other systems, like Rails' # SECRET_KEY_BASE for instance. Generate a new secret and utilize that # instead. config.context_key_secret = nil # Bit length used by SHA2 in generating context keys. # # Using a higher bit length would require more computation time. # # Valid bit lengths: # 256, 384, or 512 config.context_key_bit_length = 256 # The default base path that the middleware (or rails engine) will be # mounted. The middleware enables an instrumentation url, that's similar # to links that can be instrumented in email campaigns. # # Use `nil` if you don't want to mount the middleware. # # Examples: # '/-/experiment', '/redirect', nil config.mount_at = '/experiment' # When using the middleware, links can be instrumented and redirected # elsewhere. This can be exploited to make a harmful url look innocuous # or that it's a valid url on your domain. To avoid this, you can provide # your own logic for what urls will be considered valid and redirected # to. # # Expected to return a boolean value. config.redirect_url_validator = lambda do |_redirect_url| true end # Tracking behavior can be implemented to link an event to an experiment. # # This block is executed within the scope of the experiment and so can # access experiment methods, like `name`, `context`, and `signature`. config.tracking_behavior = lambda do |event, args| # An example of using a generic logger to track events: config.logger.info "Gitlab::Experiment[#{name}] #{event}: #{args.merge(signature: signature)}" # Using something like snowplow to track events (in gitlab): # # Gitlab::Tracking.event(name, event, **args.merge( # context: (args[:context] || []) << SnowplowTracker::SelfDescribingJson.new( # 'iglu:com.gitlab/gitlab_experiment/jsonschema/0-2-0', signature # ) # )) end # Logic designed to respond when a given experiment is nested within # another experiment. This can be useful to identify overlaps and when a # code path leads to an experiment being nested within another. # # Reporting complexity can arise when one experiment changes rollout, and # a downstream experiment is impacted by that. # # The base_class or a custom experiment can provide a `nest_experiment` # method that implements its own logic that may allow certain experiments # to be nested within it. # # This block is executed within the scope of the experiment and so can # access experiment methods, like `name`, `context`, and `signature`. # # The default exception will include the where the experiment calls were # initiated on, so for instance: # # Gitlab::Experiment::NestingError: unable to nest level2 within level1: # level1 initiated by file_name.rb:2 # level2 initiated by file_name.rb:3 config.nested_behavior = lambda do |nested_experiment| raise Gitlab::Experiment::NestingError.new(experiment: self, nested_experiment: nested_experiment) end # Called at the end of every experiment run, with the result. # # You may want to track that you've assigned a variant to a given # context, or push the experiment into the client or publish results # elsewhere like into redis. # # This block is executed within the scope of the experiment and so can # access experiment methods, like `name`, `context`, and `signature`. config.publishing_behavior = lambda do |result| # Track the event using our own configured tracking logic. track(:assignment) # Log using our logging system, so the result (which can be large) can # be reviewed later if we want to. # # Lograge::Event.log(experiment: name, result: result, signature: signature) # Experiments that have been run during the request lifecycle can be # pushed to the client layer by injecting the published experiments # into javascript in a layout or view using something like: # # = javascript_tag(nonce: content_security_policy_nonce) do # window.experiments = #{raw Gitlab::Experiment.published_experiments.to_json}; end end gitlab-experiment-1.3.0/lib/generators/gitlab/experiment/install/templates/POST_INSTALL0000644000004100000410000000017715151343617031101 0ustar www-datawww-dataGitlab::Experiment has been installed. You may want to adjust the configuration that's been provided in the Rails initializer. gitlab-experiment-1.3.0/lib/generators/gitlab/experiment/install/install_generator.rb0000644000004100000410000000210015151343617031232 0ustar www-datawww-data# frozen_string_literal: true require 'rails/generators' module Gitlab module Generators module Experiment class InstallGenerator < Rails::Generators::Base source_root File.expand_path('templates', __dir__) desc 'Installs the Gitlab::Experiment initializer and optional ApplicationExperiment into your application.' class_option :skip_initializer, type: :boolean, default: false, desc: 'Skip the initializer with default configuration' class_option :skip_baseclass, type: :boolean, default: false, desc: 'Skip the ApplicationExperiment base class' def create_initializer return if options[:skip_initializer] template 'initializer.rb', 'config/initializers/gitlab_experiment.rb' end def create_baseclass return if options[:skip_baseclass] template 'application_experiment.rb', 'app/experiments/application_experiment.rb' end def display_post_install readme 'POST_INSTALL' if behavior == :invoke end end end end end gitlab-experiment-1.3.0/lib/generators/gitlab/experiment/experiment_generator.rb0000644000004100000410000000160715151343617030311 0ustar www-datawww-data# frozen_string_literal: true require 'rails/generators' module Gitlab module Generators class ExperimentGenerator < Rails::Generators::NamedBase source_root File.expand_path('templates/', __dir__) check_class_collision suffix: 'Experiment' argument :variants, type: :array, default: %w[control candidate], banner: 'variant variant' class_option :skip_comments, type: :boolean, default: false, desc: 'Omit helpful comments from generated files' def create_experiment template 'experiment.rb', File.join('app/experiments', class_path, "#{file_name}_experiment.rb") end hook_for :test_framework private def file_name @_file_name ||= remove_possible_suffix(super) end def remove_possible_suffix(name) name.sub(/_?exp[ei]riment$/i, "") # be somewhat forgiving with spelling end end end end gitlab-experiment-1.3.0/lib/generators/gitlab/experiment/USAGE0000644000004100000410000000140715151343617024323 0ustar www-datawww-dataDescription: Stubs out a new experiment and its variants. Pass the experiment name, either CamelCased or under_scored, and a list of variants as arguments. To create an experiment within a module, specify the experiment name as a path like 'parent_module/experiment_name'. This generates an experiment class in app/experiments and invokes feature flag, and test framework generators. Example: `rails generate gitlab:experiment NullHypothesis control candidate alt_variant` NullHypothesis experiment with default variants. Experiment: app/experiments/null_hypothesis_experiment.rb Feature Flag: config/feature_flags/experiment/null_hypothesis.yaml Test: test/experiments/null_hypothesis_experiment_test.rb gitlab-experiment-1.3.0/lib/generators/test_unit/0000755000004100000410000000000015151343620022120 5ustar www-datawww-datagitlab-experiment-1.3.0/lib/generators/test_unit/experiment/0000755000004100000410000000000015151343620024300 5ustar www-datawww-datagitlab-experiment-1.3.0/lib/generators/test_unit/experiment/templates/0000755000004100000410000000000015151343620026276 5ustar www-datawww-datagitlab-experiment-1.3.0/lib/generators/test_unit/experiment/templates/experiment_test.rb.tt0000644000004100000410000000032415151343620032467 0ustar www-datawww-data# frozen_string_literal: true require 'test_helper' <% module_namespacing do -%> class <%= class_name %>ExperimentTest < ActiveSupport::TestCase # test "the truth" do # assert true # end end <% end -%> gitlab-experiment-1.3.0/lib/generators/test_unit/experiment/experiment_generator.rb0000644000004100000410000000072415151343620031056 0ustar www-datawww-data# frozen_string_literal: true require 'rails/generators/test_unit' module TestUnit # :nodoc: module Generators # :nodoc: class ExperimentGenerator < TestUnit::Generators::Base # :nodoc: source_root File.expand_path('templates/', __dir__) check_class_collision suffix: 'Test' def create_test_file template 'experiment_test.rb', File.join('test/experiments', class_path, "#{file_name}_experiment_test.rb") end end end end gitlab-experiment-1.3.0/lib/generators/rspec/0000755000004100000410000000000015151343617021224 5ustar www-datawww-datagitlab-experiment-1.3.0/lib/generators/rspec/experiment/0000755000004100000410000000000015151343617023404 5ustar www-datawww-datagitlab-experiment-1.3.0/lib/generators/rspec/experiment/templates/0000755000004100000410000000000015151343617025402 5ustar www-datawww-datagitlab-experiment-1.3.0/lib/generators/rspec/experiment/templates/experiment_spec.rb.tt0000644000004100000410000000031115151343617031542 0ustar www-datawww-data# frozen_string_literal: true require 'spec_helper' <% module_namespacing do -%> RSpec.describe <%= class_name %>Experiment do pending "add some examples to (or delete) #{__FILE__}" end <% end -%> gitlab-experiment-1.3.0/lib/generators/rspec/experiment/experiment_generator.rb0000644000004100000410000000060015151343617030153 0ustar www-datawww-data# frozen_string_literal: true require 'generators/rspec' module Rspec module Generators class ExperimentGenerator < Rspec::Generators::Base source_root File.expand_path('templates/', __dir__) def create_experiment_spec template 'experiment_spec.rb', File.join('spec/experiments', class_path, "#{file_name}_experiment_spec.rb") end end end end gitlab-experiment-1.3.0/lib/gitlab/0000755000004100000410000000000015151343620017173 5ustar www-datawww-datagitlab-experiment-1.3.0/lib/gitlab/experiment/0000755000004100000410000000000015151343620021353 5ustar www-datawww-datagitlab-experiment-1.3.0/lib/gitlab/experiment/dsl.rb0000644000004100000410000000133115151343620022460 0ustar www-datawww-data# frozen_string_literal: true module Gitlab class Experiment module Dsl def self.include_in(klass, with_helper: false) klass.include(self).tap { |base| base.helper_method(:experiment) if with_helper } end def experiment(name, variant_name = nil, **context, &block) raise ArgumentError, 'name is required' if name.nil? context[:request] ||= request if respond_to?(:request) base = Configuration.base_class.constantize klass = base.constantize(name) || base instance = klass.new(name, variant_name, **context, &block) return instance unless block instance.context.frozen? ? instance.run : instance.tap(&:run) end end end end gitlab-experiment-1.3.0/lib/gitlab/experiment/rollout/0000755000004100000410000000000015151343620023053 5ustar www-datawww-datagitlab-experiment-1.3.0/lib/gitlab/experiment/rollout/random.rb0000644000004100000410000000166215151343620024665 0ustar www-datawww-data# frozen_string_literal: true # The random rollout strategy will randomly assign a variant when the context is determined to be within the experiment # group. # # If caching is enabled this is a predicable and consistent assignment that will eventually assign a variant (since # control isn't cached) but if caching isn't enabled, assignment will be random each time. # # Example configuration usage: # # config.default_rollout = Gitlab::Experiment::Rollout::Random.new # # Example class usage: # # class PillColorExperiment < ApplicationExperiment # control { } # variant(:red) { } # variant(:blue) { } # # # Randomize between all behaviors, with a mostly even distribution). # default_rollout :random # end # module Gitlab class Experiment module Rollout class Random < Base protected def execute_assignment behavior_names.sample # pick a random variant end end end end end gitlab-experiment-1.3.0/lib/gitlab/experiment/rollout/percent.rb0000644000004100000410000000524615151343620025047 0ustar www-datawww-data# frozen_string_literal: true require "zlib" # The percent rollout strategy is the most comprehensive included with Gitlab::Experiment. It allows specifying the # percentages per variant using an array, a hash, or will default to even distribution when no rules are provided. # # A given experiment id (context key) will always be given the same variant assignment. # # Example configuration usage: # # config.default_rollout = Gitlab::Experiment::Rollout::Percent.new # # Example class usage: # # class PillColorExperiment < ApplicationExperiment # control { } # variant(:red) { } # variant(:blue) { } # # # Even distribution between all behaviors. # default_rollout :percent # # # With specific distribution percentages. # default_rollout :percent, distribution: { control: 25, red: 30, blue: 45 } # end # module Gitlab class Experiment module Rollout class Percent < Base protected def validate! case distribution_rules when nil then nil when Array validate_distribution_rules(distribution_rules) when Hash validate_distribution_rules(distribution_rules.values) else raise InvalidRolloutRules, 'unknown distribution options type' end end def execute_assignment crc = normalized_id total = 0 case distribution_rules when Array # run through the rules until finding an acceptable one index = distribution_rules.find_index { |percent| crc % 100 < total += percent unless percent == 0 } behavior_names[index] when Hash # run through the variant names until finding an acceptable one distribution_rules.find { |_, percent| crc % 100 < total += percent unless percent == 0 }.first else # assume even distribution on no rules behavior_names.empty? ? nil : behavior_names[crc % behavior_names.length] end end private def normalized_id Zlib.crc32(id, nil) end def distribution_rules options[:distribution] end def validate_distribution_rules(distributions) if distributions.length != behavior_names.length raise InvalidRolloutRules, "the distribution rules don't match the number of behaviors defined" end if distributions.any? { |percent| percent < 0 } raise InvalidRolloutRules, "the distribution percentage cannot be negative" end return if distributions.sum == 100 raise InvalidRolloutRules, 'the distribution percentages should add up to 100' end end end end end gitlab-experiment-1.3.0/lib/gitlab/experiment/rollout/round_robin.rb0000644000004100000410000000210115151343620025712 0ustar www-datawww-data# frozen_string_literal: true # The round robin strategy will assign the next variant in the list, looping back to the first variant after all # variants have been assigned. This is useful for very small sample sizes where very even distribution can be required. # # Requires a cache to be configured. # # Keeps track of the number of assignments into the experiment group, and uses this to rotate "round robin" style # through the variants that are defined. # # Example configuration usage: # # config.default_rollout = Gitlab::Experiment::Rollout::RoundRobin.new # # Example class usage: # # class PillColorExperiment < ApplicationExperiment # control { } # variant(:red) { } # variant(:blue) { } # # # Rotate evenly between all behaviors. # default_rollout :round_robin # end # module Gitlab class Experiment module Rollout class RoundRobin < Base KEY_NAME = :last_round_robin_variant protected def execute_assignment behavior_names[(cache.attr_inc(KEY_NAME) - 1) % behavior_names.size] end end end end end gitlab-experiment-1.3.0/lib/gitlab/experiment/configuration.rb0000644000004100000410000002151515151343620024553 0ustar www-datawww-data# frozen_string_literal: true require 'singleton' require 'logger' require 'digest' module Gitlab class Experiment class Configuration include Singleton # Prefix all experiment names with a given string value. # Use `nil` for no prefix. @name_prefix = nil # The logger can be used to log various details of the experiments. @logger = Logger.new($stdout) # The base class that should be instantiated for basic experiments. # It should be a string, so we can constantize it later. @base_class = 'Gitlab::Experiment' # Require experiments to be defined in a class, with variants registered. # This will disallow any anonymous experiments that are run inline # without previously defining a class. @strict_registration = false # The caching layer is expected to match the Rails.cache interface. # If no cache is provided some rollout strategies may behave differently. # Use `nil` for no caching. @cache = nil # The domain to use on cookies. # # When not set, it uses the current host. If you want to provide specific # hosts, you use `:all`, or provide an array. # # Examples: # nil, :all, or ['www.gitlab.com', '.gitlab.com'] @cookie_domain = :all # The cookie name for an experiment. @cookie_name = lambda do |experiment| "#{experiment.name}_id" end # Allow forced variant assignment via query parameter and cookie. # # When enabled, experiments will check for a `glex_force` query # parameter and a `_glex_force` cookie to override variant assignment. # This is intended for QA/UAT testing in staging and production. # # Defaults to false (disabled). @allow_forced_assignment = false # Mark experiment cookies as secure (HTTPS only). # # When set to true, cookies will have the secure flag set, meaning they # will only be sent over HTTPS connections. Defaults to true. # # Set to false in development/test environments if needed: # config.secure_cookie = Rails.env.production? @secure_cookie = true # The default rollout strategy. # # The recommended default rollout strategy when not using caching would # be `Gitlab::Experiment::Rollout::Percent` as that will consistently # assign the same variant with or without caching. # # Gitlab::Experiment::Rollout::Base can be inherited to implement your # own rollout strategies. # # Each experiment can specify its own rollout strategy: # # class ExampleExperiment < ApplicationExperiment # default_rollout :random # :percent, :round_robin, or MyCustomRollout # end # # Included rollout strategies: # :percent, (recommended), :round_robin, or :random @default_rollout = Rollout.resolve(:percent) # Secret seed used in generating context keys. # # You'll typically want to use an environment variable or secret value # for this. # # Consider not using one that's shared with other systems, like Rails' # SECRET_KEY_BASE for instance. Generate a new secret and utilize that # instead. @context_key_secret = nil # Bit length used by SHA2 in generating context keys. # # Using a higher bit length would require more computation time. # # Valid bit lengths: # 256, 384, or 512 @context_key_bit_length = 256 # The default base path that the middleware (or rails engine) will be # mounted. The middleware enables an instrumentation url, that's similar # to links that can be instrumented in email campaigns. # # Use `nil` if you don't want to mount the middleware. # # Examples: # '/-/experiment', '/redirect', nil @mount_at = nil # When using the middleware, links can be instrumented and redirected # elsewhere. This can be exploited to make a harmful url look innocuous # or that it's a valid url on your domain. To avoid this, you can provide # your own logic for what urls will be considered valid and redirected # to. # # Expected to return a boolean value. @redirect_url_validator = lambda do |_redirect_url| true end # Tracking behavior can be implemented to link an event to an experiment. # # This block is executed within the scope of the experiment and so can # access experiment methods, like `name`, `context`, and `signature`. @tracking_behavior = lambda do |event, args| # An example of using a generic logger to track events: Configuration.logger.info("#{self.class.name}[#{name}] #{event}: #{args.merge(signature: signature)}") # Using something like snowplow to track events (in gitlab): # # Gitlab::Tracking.event(name, event, **args.merge( # context: (args[:context] || []) << SnowplowTracker::SelfDescribingJson.new( # 'iglu:com.gitlab/gitlab_experiment/jsonschema/0-2-0', signature # ) # )) end # Logic designed to respond when a given experiment is nested within # another experiment. This can be useful to identify overlaps and when a # code path leads to an experiment being nested within another. # # Reporting complexity can arise when one experiment changes rollout, and # a downstream experiment is impacted by that. # # The base_class or a custom experiment can provide a `nest_experiment` # method that implements its own logic that may allow certain experiments # to be nested within it. # # This block is executed within the scope of the experiment and so can # access experiment methods, like `name`, `context`, and `signature`. # # The default exception will include the where the experiment calls were # initiated on, so for instance: # # Gitlab::Experiment::NestingError: unable to nest level2 within level1: # level1 initiated by file_name.rb:2 # level2 initiated by file_name.rb:3 @nested_behavior = lambda do |nested_experiment| raise NestingError.new(experiment: self, nested_experiment: nested_experiment) end # Called at the end of every experiment run, with the result. # # You may want to track that you've assigned a variant to a given # context, or push the experiment into the client or publish results # elsewhere like into redis. # # This block is executed within the scope of the experiment and so can # access experiment methods, like `name`, `context`, and `signature`. @publishing_behavior = lambda do |_result| # Track the event using our own configured tracking logic. track(:assignment) # Log using our logging system, so the result (which can be large) can # be reviewed later if we want to. # # Lograge::Event.log(experiment: name, result: result, signature: signature) # Experiments that have been run during the request lifecycle can be # pushed to the client layer by injecting the published experiments # into javascript in a layout or view using something like: # # = javascript_tag(nonce: content_security_policy_nonce) do # window.experiments = #{raw Gitlab::Experiment.published_experiments.to_json}; end class << self attr_accessor( :name_prefix, :logger, :base_class, :strict_registration, :cache, :cookie_domain, :cookie_name, :secure_cookie, :allow_forced_assignment, :context_key_secret, :context_key_bit_length, :mount_at, :default_rollout, :redirect_url_validator, :tracking_behavior, :nested_behavior, :publishing_behavior ) # Attribute method overrides. def default_rollout=(args) # rubocop:disable Lint/DuplicateMethods @default_rollout = Rollout.resolve(*args) end # Internal warning helpers. def deprecated(*args, version:, stack: 0) deprecator = deprecator(version) args << args.pop.to_s.gsub('{{release}}', "#{deprecator.gem_name} #{deprecator.deprecation_horizon}") args << caller_locations(4 + stack) if args.length == 2 deprecator.warn(*args) else args[0] = "`#{args[0]}`" deprecator.deprecation_warning(*args) end end private def deprecator(version = VERSION) version = Gem::Version.new(version).bump.to_s @__dep_versions ||= {} @__dep_versions[version] ||= ActiveSupport::Deprecation.new(version, 'Gitlab::Experiment') end end end end end gitlab-experiment-1.3.0/lib/gitlab/experiment/callbacks.rb0000644000004100000410000001060115151343620023615 0ustar www-datawww-data# frozen_string_literal: true module Gitlab class Experiment module Callbacks extend ActiveSupport::Concern include ActiveSupport::Callbacks included do # Callbacks are listed in order of when they're executed when running an experiment. # Exclusion check chain: # # The :exclusion_check chain is executed when determining if the context should be excluded from the experiment. # # If any callback returns true, further chain execution is terminated, the context will be considered excluded, # and the control behavior will be provided. define_callbacks(:exclusion_check, skip_after_callbacks_if_terminated: true) # Segmentation chain: # # The :segmentation chain is executed when no variant has been explicitly provided, the experiment is enabled, # and the context hasn't been excluded. # # If the :segmentation callback chain doesn't need to be executed, the :segmentation_skipped chain will be # executed as the fallback. # # If any callback explicitly sets a variant, further chain execution is terminated. define_callbacks(:segmentation) define_callbacks(:segmentation_skipped) # Run chain: # # The :run chain is executed when the experiment is enabled, and the context hasn't been excluded. # # If the :run callback chain doesn't need to be executed, the :run_skipped chain will be executed as the # fallback. define_callbacks(:run) define_callbacks(:run_skipped) end class_methods do def registered_behavior_callbacks @_registered_behavior_callbacks ||= {} end private def build_behavior_callback(filters, variant, **options, &block) if registered_behavior_callbacks[variant] raise ExistingBehaviorError, "a behavior for the `#{variant}` variant has already been registered" end callback_behavior = "#{variant}_behavior".to_sym # Register a the behavior so we can define the block later. registered_behavior_callbacks[variant] = callback_behavior # Add our block or default behavior method. filters.push(block) if block.present? filters.unshift(callback_behavior) if filters.empty? # Define and build the callback that will set our result. define_callbacks(callback_behavior) build_callback(callback_behavior, *filters, **options) do |target, callback| target.instance_variable_set(:@_behavior_callback_result, callback.call(target, nil)) end end def build_exclude_callback(filters, **options) build_callback(:exclusion_check, *filters, **options) do |target, callback| throw(:abort) if target.instance_variable_get(:@_excluded) || callback.call(target, nil) == true end end def build_segment_callback(filters, variant, **options) build_callback(:segmentation, *filters, **options) do |target, callback| if target.instance_variable_get(:@_assigned_variant_name).nil? && callback.call(target, nil) target.assigned(variant) end end end def build_run_callback(filters, **options) set_callback(:run, *filters.compact, **options) end def build_callback(chain, *filters, **options) filters = filters.compact.map do |filter| result_lambda = ActiveSupport::Callbacks::CallTemplate.build(filter, self).make_lambda ->(target) { yield(target, result_lambda) } end raise ArgumentError, 'no filters provided' if filters.empty? set_callback(chain, *filters, **options) end end private def exclusion_callback_chain :exclusion_check end def segmentation_callback_chain return :segmentation if @_assigned_variant_name.nil? && enabled? && !excluded? :segmentation_skipped end def run_callback_chain return :run if enabled? && !excluded? :run_skipped end def registered_behavior_callbacks self.class.registered_behavior_callbacks.transform_values do |callback_behavior| -> { run_callbacks(callback_behavior) { @_behavior_callback_result } } end end end end end gitlab-experiment-1.3.0/lib/gitlab/experiment/rollout.rb0000644000004100000410000000334415151343620023404 0ustar www-datawww-data# frozen_string_literal: true module Gitlab class Experiment module Rollout autoload :Percent, 'gitlab/experiment/rollout/percent.rb' autoload :Random, 'gitlab/experiment/rollout/random.rb' autoload :RoundRobin, 'gitlab/experiment/rollout/round_robin.rb' def self.resolve(klass, options = {}) options ||= {} case klass when String Strategy.new(klass.classify.constantize, options) when Symbol Strategy.new("#{name}::#{klass.to_s.classify}".constantize, options) when Class Strategy.new(klass, options) else raise ArgumentError, "unable to resolve rollout from #{klass.inspect}" end end class Base attr_reader :experiment, :options delegate :cache, :id, to: :experiment def initialize(experiment, options = {}) raise ArgumentError, 'you must provide an experiment instance' unless experiment.class <= Gitlab::Experiment @experiment = experiment @options = options end def enabled? true end def resolve validate! # allow the rollout strategy to validate itself assignment = execute_assignment assignment == :control ? nil : assignment # avoid caching control by returning nil end private def validate! # base is always valid end def execute_assignment behavior_names.first end def behavior_names experiment.behaviors.keys end end Strategy = Struct.new(:klass, :options) do def for(experiment) klass.new(experiment, options) end end end end end gitlab-experiment-1.3.0/lib/gitlab/experiment/test_behaviors/0000755000004100000410000000000015151343620024374 5ustar www-datawww-datagitlab-experiment-1.3.0/lib/gitlab/experiment/test_behaviors/trackable.rb0000644000004100000410000000272115151343620026653 0ustar www-datawww-data# frozen_string_literal: true module Gitlab class Experiment module TestBehaviors module Trackable private def manage_nested_stack TrackedStructure.push(self) super ensure TrackedStructure.pop end end class TrackedStructure include Singleton # dependency tracking @flat = {} @stack = [] # structure tracking @tree = { name: nil, count: 0, children: {} } @node = @tree class << self def reset! # dependency tracking @flat = {} @stack = [] # structure tracking @tree = { name: nil, count: 0, children: {} } @node = @tree end def hierarchy @tree[:children] end def dependencies @flat end def push(instance) # dependency tracking @flat[instance.name] = ((@flat[instance.name] || []) + @stack.map(&:name)).uniq @stack.push(instance) # structure tracking @last = @node @node = @node[:children][instance.name] ||= { name: instance.name, count: 0, children: {} } @node[:count] += 1 end def pop # dependency tracking @stack.pop # structure tracking @node = @last end end end end end end gitlab-experiment-1.3.0/lib/gitlab/experiment/engine.rb0000644000004100000410000000300415151343620023142 0ustar www-datawww-data# frozen_string_literal: true require 'active_model' module Gitlab class Experiment include ActiveModel::Model # Used for generating routes. We've included the method and `ActiveModel::Model` here because these things don't # make sense outside of Rails environments. def self.model_name ActiveModel::Name.new(self, Gitlab) end class Engine < ::Rails::Engine isolate_namespace Experiment initializer('gitlab_experiment.include_dsl') { include_dsl } initializer('gitlab_experiment.mount_engine') { |app| mount_engine(app, Configuration.mount_at) } private def include_dsl Dsl.include_in(ActionController::API, with_helper: false) if defined?(ActionController) Dsl.include_in(ActionController::Base, with_helper: true) if defined?(ActionController) Dsl.include_in(ActionMailer::Base, with_helper: true) if defined?(ActionMailer) end def mount_engine(app, mount_at) return if mount_at.blank? engine = routes do default_url_options app.routes.default_url_options.clone.without(:script_name) resources :experiments, path: '/', only: :show end app.config.middleware.use(Middleware, mount_at) app.routes.append do mount Engine, at: mount_at, as: :experiment_engine direct(:experiment_redirect) do |ex, options| url = options[:url] "#{engine.url_helpers.experiment_url(ex)}?#{url}" end end end end end end gitlab-experiment-1.3.0/lib/gitlab/experiment/cache/0000755000004100000410000000000015151343620022416 5ustar www-datawww-datagitlab-experiment-1.3.0/lib/gitlab/experiment/cache/redis_hash_store.rb0000644000004100000410000000425515151343620026276 0ustar www-datawww-data# frozen_string_literal: true # This cache strategy is an implementation on top of the redis hash data type, that also adheres to the # ActiveSupport::Cache::Store interface. It's a good example of how to build a custom caching strategy for # Gitlab::Experiment, and is intended to be a long lived cache -- until the experiment is cleaned up. # # The data structure: # key: experiment.name # fields: context key => variant name # # Example configuration usage: # # config.cache = Gitlab::Experiment::Cache::RedisHashStore.new( # pool: ->(&block) { block.call(Redis.current) } # ) # module Gitlab class Experiment module Cache class RedisHashStore < ActiveSupport::Cache::Store # Clears the entire cache for a given experiment. Be careful with this since it would reset all resolved # variants for the entire experiment. def clear(key:) key = hkey(key)[0] # extract only the first part of the key pool do |redis| case redis.type(key) when 'hash', 'none' redis.del(key) # delete the primary experiment key redis.del("#{key}_attrs") # delete the experiment attributes key else raise ArgumentError, 'invalid call to clear a non-hash cache key' end end end def increment(key, amount = 1) pool { |redis| redis.hincrby(*hkey(key), amount) } end private def pool(&block) raise ArgumentError, 'missing block' unless block.present? @options[:pool].call(&block) end def hkey(key) key.to_s.split(':') # this assumes the default strategy in gitlab-experiment end def read_entry(key, **_options) value = pool { |redis| redis.hget(*hkey(key)) } value.nil? ? nil : ActiveSupport::Cache::Entry.new(value) end def write_entry(key, entry, **_options) return false if entry.value.blank? # don't cache any empty values pool { |redis| redis.hset(*hkey(key), entry.value) } end def delete_entry(key, **_options) pool { |redis| redis.hdel(*hkey(key)) } end end end end end gitlab-experiment-1.3.0/lib/gitlab/experiment/rspec.rb0000644000004100000410000003052315151343620023017 0ustar www-datawww-data# frozen_string_literal: true module Gitlab class Experiment module TestBehaviors autoload :Trackable, 'gitlab/experiment/test_behaviors/trackable.rb' end WrappedExperiment = Struct.new(:klass, :experiment_name, :variant_name, :expectation_chain, :blocks, :assigned) module RSpecMocks @__gitlab_experiment_receivers = {} def self.track_gitlab_experiment_receiver(method, receiver) # Leverage the `>=` method on Gitlab::Experiment to determine if the receiver is an experiment, not the other # way round -- `receiver.<=` could be mocked and we want to be extra careful. (@__gitlab_experiment_receivers[method] ||= []) << receiver if Gitlab::Experiment >= receiver rescue StandardError # again, let's just be extra careful false end def self.bind_gitlab_experiment_receiver(method) method.unbind.bind(@__gitlab_experiment_receivers[method].pop) end module MethodDouble def proxy_method_invoked(receiver, *args, &block) RSpecMocks.track_gitlab_experiment_receiver(original_method, receiver) super end ruby2_keywords :proxy_method_invoked if respond_to?(:ruby2_keywords, true) end end module RSpecHelpers def stub_experiments(experiments) experiments.each do |experiment| wrapped_experiment(experiment, remock: true) do |instance, wrapped| # Stub internal methods that will make it behave as we've instructed. allow(instance).to receive(:enabled?) { wrapped.variant_name != false } # Stub the variant resolution logic to handle true/false, and named variants. allow(instance).to receive(:resolve_variant_name).and_wrap_original { |method| # Call the original method if we specified simply `true`. wrapped.variant_name == true ? method.call : wrapped.variant_name } # Stub find_variant only if caching is not enabled unless Configuration.cache variant_return_value = wrapped.assigned ? wrapped.variant_name.to_s : nil allow(instance).to receive(:find_variant).and_return(variant_return_value) end end end wrapped_experiments end def wrapped_experiment(experiment, remock: false, &block) klass, experiment_name, variant_name, assigned = *extract_experiment_details(experiment) wrapped_experiment = wrapped_experiments[experiment_name] = (!remock && wrapped_experiments[experiment_name]) || WrappedExperiment.new(klass, experiment_name, variant_name, wrapped_experiment_chain_for(klass), [], assigned) wrapped_experiment.blocks << block if block wrapped_experiment end private def wrapped_experiments @__wrapped_experiments ||= defined?(HashWithIndifferentAccess) ? HashWithIndifferentAccess.new : {} end def wrapped_experiment_chain_for(klass) @__wrapped_experiment_chains ||= {} @__wrapped_experiment_chains[klass.name || klass.object_id] ||= begin allow(klass).to receive(:new).and_wrap_original do |method, *args, **kwargs, &original_block| RSpecMocks.bind_gitlab_experiment_receiver(method).call(*args, **kwargs).tap do |instance| wrapped = @__wrapped_experiments[instance.instance_variable_get(:@_name)] wrapped&.blocks&.each { |b| b.call(instance, wrapped) } original_block&.call(instance) end end end end def extract_experiment_details(experiment) experiment_name = nil variant_name = nil assigned = nil if experiment.is_a?(Array) # From normalize_experiments: [experiment_name, variant_name_or_config] experiment_name, variant_name = *experiment assigned = variant_name.is_a?(Hash) ? variant_name.delete(:assigned) : nil variant_name = variant_name[:variant] if variant_name.is_a?(Hash) elsif experiment.is_a?(Symbol) experiment_name = experiment end base_klass = Configuration.base_class.constantize variant_name = experiment.assigned.name if experiment.is_a?(base_klass) resolved_klass = experiment_klass(experiment) { base_klass.constantize(experiment_name) } experiment_name ||= experiment.instance_variable_get(:@_name) [resolved_klass, experiment_name.to_s, variant_name, assigned] end def experiment_klass(experiment, &block) if experiment.class.name.nil? # anonymous class instance experiment.class elsif experiment.instance_of?(Class) # class level stubbing, eg. "MyExperiment" experiment elsif block yield end end end module RSpecMatchers extend RSpec::Matchers::DSL def require_experiment(experiment, matcher, instances_only: true) klass = experiment.instance_of?(Class) ? experiment : experiment.class raise ArgumentError, "the #{matcher} matcher is limited to experiments" unless klass <= Gitlab::Experiment if instances_only && experiment == klass raise ArgumentError, "the #{matcher} matcher is limited to experiment instances" end experiment end matcher :register_behavior do |behavior_name| match do |experiment| @experiment = require_experiment(experiment, 'register_behavior') block = @experiment.behaviors[behavior_name] @return_expected = false unless block if @return_expected @actual_return = block.call @expected_return == @actual_return else block end end chain :with do |expected| @return_expected = true @expected_return = expected end failure_message do add_details("expected the #{behavior_name} behavior to be registered") end failure_message_when_negated do add_details("expected the #{behavior_name} behavior not to be registered") end def add_details(base) details = [] if @return_expected base = "#{base} with a return value" details << " expected return: #{@expected_return.inspect}\n" \ " actual return: #{@actual_return.inspect}" else details << " behaviors: #{@experiment.behaviors.keys.inspect}" end details.unshift(base).join("\n") end end matcher :exclude do |context| match do |experiment| @experiment = require_experiment(experiment, 'exclude') @experiment.context(context) @experiment.instance_variable_set(:@_excluded, nil) !@experiment.run_callbacks(:exclusion_check) { :not_excluded } end failure_message do "expected #{context} to be excluded" end failure_message_when_negated do "expected #{context} not to be excluded" end end matcher :segment do |context| match do |experiment| @experiment = require_experiment(experiment, 'segment') @experiment.context(context) @experiment.instance_variable_set(:@_assigned_variant_name, nil) @experiment.run_callbacks(:segmentation) @actual_variant = @experiment.instance_variable_get(:@_assigned_variant_name) @expected_variant ? @actual_variant == @expected_variant : @actual_variant.present? end chain :into do |expected| raise ArgumentError, 'variant name must be provided' if expected.blank? @expected_variant = expected end failure_message do add_details("expected #{context} to be segmented") end failure_message_when_negated do add_details("expected #{context} not to be segmented") end def add_details(base) details = [] if @expected_variant base = "#{base} into variant" details << " expected variant: #{@expected_variant.inspect}\n" \ " actual variant: #{@actual_variant.inspect}" end details.unshift(base).join("\n") end end matcher :track do |event, *event_args| match do |experiment| @experiment = require_experiment(experiment, 'track', instances_only: false) set_expectations(event, *event_args, negated: false) end match_when_negated do |experiment| @experiment = require_experiment(experiment, 'track', instances_only: false) set_expectations(event, *event_args, negated: true) end chain(:for) do |expected| raise ArgumentError, 'variant name must be provided' if expected.blank? @expected_variant = expected end chain(:with_context) do |expected| raise ArgumentError, 'context name must be provided' if expected.nil? @expected_context = expected end chain(:on_next_instance) { @on_next_instance = true } def set_expectations(event, *event_args, negated:) failure_message = failure_message_with_details(event, negated: negated) expectations = proc do |e| allow(e).to receive(:track).and_call_original if negated if @expected_variant || @expected_context raise ArgumentError, 'cannot specify `for` or `with_context` when negating on tracking calls' end expect(e).not_to receive(:track).with(*[event, *event_args]), failure_message else expect(e.assigned.name).to(eq(@expected_variant), failure_message) if @expected_variant expect(e.context.value).to(include(@expected_context), failure_message) if @expected_context expect(e).to receive(:track).with(*[event, *event_args]).and_call_original, failure_message end end return wrapped_experiment(@experiment, &expectations) if @on_next_instance || @experiment.instance_of?(Class) expectations.call(@experiment) end def failure_message_with_details(event, negated: false) add_details("expected #{@experiment.inspect} #{negated ? 'not to' : 'to'} have tracked #{event.inspect}") end def add_details(base) details = [] if @expected_variant base = "#{base} for variant" details << " expected variant: #{@expected_variant.inspect}\n" \ " actual variant: #{@experiment.assigned.name.inspect})" end if @expected_context base = "#{base} with context" details << " expected context: #{@expected_context.inspect}\n" \ " actual context: #{@experiment.context.value.inspect})" end details.unshift(base).join("\n") end end end end end RSpec.configure do |config| config.include Gitlab::Experiment::RSpecHelpers config.include Gitlab::Experiment::Dsl config.before(:each) do |example| if example.metadata[:experiment] == true || example.metadata[:type] == :experiment RequestStore.clear! if defined?(Gitlab::Experiment::TestBehaviors::TrackedStructure) Gitlab::Experiment::TestBehaviors::TrackedStructure.reset! end end end config.include Gitlab::Experiment::RSpecMatchers, :experiment config.include Gitlab::Experiment::RSpecMatchers, type: :experiment config.define_derived_metadata(file_path: Regexp.new('spec/experiments/')) do |metadata| metadata[:type] ||= :experiment end # We need to monkeypatch rspec-mocks because there's an issue around stubbing class methods that impacts us here. # # You can find out what the outcome is of the issues I've opened on rspec-mocks, and maybe some day this won't be # needed. # # https://github.com/rspec/rspec-mocks/issues/1452 # https://github.com/rspec/rspec-mocks/issues/1451 (closed) # # The other way I've considered patching this is inside gitlab-experiment itself, by adding an Anonymous class and # instantiating that instead of the configured base_class, and then it's less common but still possible to run into # the issue. require 'rspec/mocks/method_double' RSpec::Mocks::MethodDouble.prepend(Gitlab::Experiment::RSpecMocks::MethodDouble) end gitlab-experiment-1.3.0/lib/gitlab/experiment/force_assignment.rb0000644000004100000410000000123315151343620025225 0ustar www-datawww-data# frozen_string_literal: true module Gitlab class Experiment module ForceAssignment PARAM_NAME = 'glex_force' private def forced_variant_name return unless Configuration.allow_forced_assignment return unless enabled? param = context&.request&.params.try(:[], PARAM_NAME) return if param.blank? experiment_name, variant_name = param.split(':', 2) return if experiment_name.blank? || variant_name.blank? return unless experiment_name == name variant_sym = variant_name.to_sym return unless behaviors.key?(variant_sym) variant_sym end end end end gitlab-experiment-1.3.0/lib/gitlab/experiment/nestable.rb0000644000004100000410000000175015151343620023500 0ustar www-datawww-data# frozen_string_literal: true module Gitlab class Experiment module Nestable extend ActiveSupport::Concern included do set_callback :run, :around, :manage_nested_stack end def nest_experiment(nested_experiment) instance_exec(nested_experiment, &Configuration.nested_behavior) end private def manage_nested_stack Stack.push(self) yield ensure Stack.pop end class Stack include Singleton delegate :pop, :length, :size, :[], to: :stack class << self delegate :pop, :push, :length, :size, :[], to: :instance end def initialize @thread_key = "#{self.class};#{object_id}".to_sym end def push(instance) stack.last&.nest_experiment(instance) stack.push(instance) end private def stack Thread.current[@thread_key] ||= [] end end end end end gitlab-experiment-1.3.0/lib/gitlab/experiment/cookies.rb0000644000004100000410000000220615151343620023334 0ustar www-datawww-data# frozen_string_literal: true module Gitlab class Experiment module Cookies private def migrate_cookie(hash, cookie_name) return hash if cookie_jar.nil? resolver = [hash, :actor, cookie_name, cookie_jar.signed[cookie_name]] resolve_cookie(*resolver) || generate_cookie(*resolver) end def cookie_jar @request&.cookie_jar end def resolve_cookie(hash, key, cookie_name, cookie) return if cookie.to_s.empty? && hash[key].nil? return hash if cookie.to_s.empty? return hash.merge(key => cookie) if hash[key].nil? add_unmerged_migration(key => cookie) cookie_jar.delete(cookie_name, domain: domain) hash end def generate_cookie(hash, key, cookie_name, cookie) return hash unless hash.key?(key) cookie ||= SecureRandom.uuid cookie_jar.permanent.signed[cookie_name] = { value: cookie, secure: Configuration.secure_cookie, domain: domain, httponly: true } hash.merge(key => cookie) end def domain Configuration.cookie_domain end end end end gitlab-experiment-1.3.0/lib/gitlab/experiment/variant.rb0000644000004100000410000000034015151343620023341 0ustar www-datawww-data# frozen_string_literal: true module Gitlab class Experiment Variant = Struct.new(:name, :payload, keyword_init: true) do def group name == 'control' ? :control : :experiment end end end end gitlab-experiment-1.3.0/lib/gitlab/experiment/context.rb0000644000004100000410000000450115151343620023364 0ustar www-datawww-data# frozen_string_literal: true module Gitlab class Experiment class Context include Cookies DNT_REGEXP = /^(true|t|yes|y|1|on)$/i attr_reader :request, :only_assigned def initialize(experiment, **initial_value) @experiment = experiment @value = {} @migrations = { merged: [], unmerged: [] } value(initial_value) end def reinitialize(request) @signature = nil # clear memoization @request = request if request.respond_to?(:headers) && request.respond_to?(:cookie_jar) end def value(value = nil) return @value if value.nil? value = value.dup # dup so we don't mutate @only_assigned = value.delete(:only_assigned) reinitialize(value.delete(:request)) key(value.delete(:sticky_to)) @value.merge!(process_migrations(value)) end def key(key = nil) return @key || @experiment.key_for(value) if key.nil? @key = @experiment.key_for(key) end def trackable? !(@request && @request.headers['DNT'].to_s.match?(DNT_REGEXP)) end def freeze signature # finalize before freezing super end def signature @signature ||= { key: key, migration_keys: migration_keys }.compact end def method_missing(method_name, *) @value.include?(method_name.to_sym) ? @value[method_name.to_sym] : super end def respond_to_missing?(method_name, *) @value.include?(method_name.to_sym) ? true : super end private def process_migrations(value) add_unmerged_migration(value.delete(:migrated_from)) add_merged_migration(value.delete(:migrated_with)) migrate_cookie(value, @experiment.instance_exec(@experiment, &Configuration.cookie_name)) end def add_unmerged_migration(value = {}) @migrations[:unmerged] << value if value.is_a?(Hash) end def add_merged_migration(value = {}) @migrations[:merged] << value if value.is_a?(Hash) end def migration_keys return nil if @migrations[:unmerged].empty? && @migrations[:merged].empty? @migrations[:unmerged].map { |m| @experiment.key_for(m) } + @migrations[:merged].map { |m| @experiment.key_for(@value.merge(m)) } end end end end gitlab-experiment-1.3.0/lib/gitlab/experiment/version.rb0000644000004100000410000000014015151343620023360 0ustar www-datawww-data# frozen_string_literal: true module Gitlab class Experiment VERSION = '1.3.0' end end gitlab-experiment-1.3.0/lib/gitlab/experiment/middleware.rb0000644000004100000410000000133615151343620024020 0ustar www-datawww-data# frozen_string_literal: true module Gitlab class Experiment class Middleware def self.redirect(id, url) raise Error, 'no url to redirect to' if url.blank? experiment = Gitlab::Experiment.from_param(id) [303, { 'Location' => experiment.process_redirect_url(url) || raise(Error, 'not redirecting') }, []] end def initialize(app, base_path) @app = app @matcher = %r{^#{base_path}/(?.+)} end def call(env) return @app.call(env) if env['REQUEST_METHOD'] != 'GET' || (match = @matcher.match(env['PATH_INFO'])).nil? Middleware.redirect(match[:id], env['QUERY_STRING']) rescue Error @app.call(env) end end end end gitlab-experiment-1.3.0/lib/gitlab/experiment/errors.rb0000644000004100000410000000152515151343620023217 0ustar www-datawww-data# frozen_string_literal: true module Gitlab class Experiment Error = Class.new(StandardError) InvalidRolloutRules = Class.new(Error) UnregisteredExperiment = Class.new(Error) ExistingBehaviorError = Class.new(Error) BehaviorMissingError = Class.new(Error) class NestingError < Error def initialize(experiment:, nested_experiment:) messages = [] experiments = [nested_experiment, experiment] callers = caller_locations callers.select.with_index do |caller, index| next if caller.label != 'experiment' messages << " #{experiments[messages.length].name} initiated by #{callers[index + 1]}" end messages << ["unable to nest #{nested_experiment.name} within #{experiment.name}:"] super(messages.reverse.join("\n")) end end end end gitlab-experiment-1.3.0/lib/gitlab/experiment/cache.rb0000644000004100000410000000370015151343620022743 0ustar www-datawww-data# frozen_string_literal: true module Gitlab class Experiment module Cache autoload :RedisHashStore, 'gitlab/experiment/cache/redis_hash_store.rb' class Interface attr_reader :store, :key def initialize(experiment, store) @experiment = experiment @store = store @key = experiment.cache_key end def read store.read(key) end def write(value = nil) store.write(key, value || @experiment.assigned.name) end def delete store.delete(key) end def attr_get(name) store.read(@experiment.cache_key(name, suffix: :attrs)) end def attr_set(name, value) store.write(@experiment.cache_key(name, suffix: :attrs), value) end def attr_inc(name, amount = 1) store.increment(@experiment.cache_key(name, suffix: :attrs), amount) end end def cache @cache ||= Interface.new(self, Configuration.cache) end def cache_variant(specified = nil, &block) return (specified.presence || yield) unless cache.store result = migrated_cache_fetch(cache.store) || find_variant(&block) return result unless specified.present? cache.write(specified) if result.to_s != specified.to_s specified end def find_variant(&block) cache.store.fetch(cache_key, &block) end def cache_key(key = nil, suffix: nil) "#{[name, suffix].compact.join('_')}:#{key || context.signature[:key]}" end private def migrated_cache_fetch(store) migrations = context.signature[:migration_keys]&.map { |key| cache_key(key) } || [] migrations.find do |old_key| value = store.read(old_key) next unless value store.write(cache_key, value) store.delete(old_key) break value end end end end end gitlab-experiment-1.3.0/lib/gitlab/experiment/base_interface.rb0000644000004100000410000000644515151343620024643 0ustar www-datawww-data# frozen_string_literal: true module Gitlab class Experiment module BaseInterface extend ActiveSupport::Concern class_methods do def configure yield Configuration end def experiment_name(name = nil, suffix: true, suffix_word: 'experiment') name = (name.presence || self.name).to_s.underscore.sub(%r{(?[_/]|)#{suffix_word}$}, '') name = "#{name}#{Regexp.last_match(:char) || '_'}#{suffix_word}" suffix ? name : name.sub(/_#{suffix_word}$/, '') end def base? self == Gitlab::Experiment || name == Configuration.base_class end def constantize(name = nil) return self if name.nil? experiment_class = experiment_name(name).classify experiment_class.safe_constantize || begin return Configuration.base_class.constantize unless Configuration.strict_registration raise UnregisteredExperiment, <<~ERR No experiment registered for `#{name}`. Please register the experiment by defining a class: class #{experiment_class} < #{Configuration.base_class} control candidate { 'candidate' } end ERR end end def from_param(id) %r{/?(?.*):(?.*)$} =~ id name = CGI.unescape(name) if name constantize(name).new(name).tap { |e| e.context.key(key) } end end def initialize(name = nil, variant_name = nil, **context) raise ArgumentError, 'name is required' if name.blank? && self.class.base? @_name = self.class.experiment_name(name, suffix: false) @_context = Context.new(self, **context) @_assigned_variant_name = cache_variant(variant_name) { nil } if variant_name.present? yield self if block_given? end def inspect "#<#{self.class.name || 'AnonymousClass'}:#{format('0x%016X', __id__)} name=#{name} context=#{context.value}>" end def run(variant_name) behaviors.freeze context.freeze block = behaviors[variant_name] raise BehaviorMissingError, "the `#{variant_name}` variant hasn't been registered" if block.nil? result = block.call publish(result) if enabled? result end def id "#{name}:#{context.key}" end alias_method :to_param, :id def process_redirect_url(url) return unless Configuration.redirect_url_validator&.call(url) track('visited', url: url) url # return the url, which allows for mutation end def key_for(source, seed = name) return source if source.is_a?(String) source = source.keys + source.values if source.is_a?(Hash) ingredients = Array(source).map { |v| identify(v) } ingredients.unshift(seed).unshift(Configuration.context_key_secret) Digest::SHA2.new(Configuration.context_key_bit_length).hexdigest(ingredients.join('|')) # rubocop:disable Fips/OpenSSL end # @deprecated def variant_names Configuration.deprecated( :variant_names, 'instead use `behavior.names`, which includes :control', version: '0.8.0' ) behaviors.keys - [:control] end end end end gitlab-experiment-1.3.0/lib/gitlab/experiment.rb0000644000004100000410000001320115151343620021675 0ustar www-datawww-data# frozen_string_literal: true require 'request_store' require 'active_support' require 'active_support/core_ext/module/delegation' require 'active_support/core_ext/object/blank' require 'active_support/core_ext/string/inflections' require 'gitlab/experiment/errors' require 'gitlab/experiment/base_interface' require 'gitlab/experiment/cache' require 'gitlab/experiment/callbacks' require 'gitlab/experiment/rollout' require 'gitlab/experiment/configuration' require 'gitlab/experiment/cookies' require 'gitlab/experiment/force_assignment' require 'gitlab/experiment/context' require 'gitlab/experiment/dsl' require 'gitlab/experiment/middleware' require 'gitlab/experiment/nestable' require 'gitlab/experiment/variant' require 'gitlab/experiment/version' require 'gitlab/experiment/engine' if defined?(Rails::Engine) module Gitlab class Experiment include BaseInterface include Cache include Callbacks include ForceAssignment include Nestable class << self # Class level behavior registration methods. def control(*filter_list, **options, &block) variant(:control, *filter_list, **options, &block) end def candidate(*filter_list, **options, &block) variant(:candidate, *filter_list, **options, &block) end def variant(variant, *filter_list, **options, &block) build_behavior_callback(filter_list, variant, **options, &block) end # Class level callback registration methods. def exclude(*filter_list, **options, &block) build_exclude_callback(filter_list.unshift(block), **options) end def segment(*filter_list, variant:, **options, &block) build_segment_callback(filter_list.unshift(block), variant, **options) end def before_run(*filter_list, **options, &block) build_run_callback(filter_list.unshift(:before, block), **options) end def around_run(*filter_list, **options, &block) build_run_callback(filter_list.unshift(:around, block), **options) end def after_run(*filter_list, **options, &block) build_run_callback(filter_list.unshift(:after, block), **options) end # Class level definition methods. def default_rollout(rollout = nil, options = {}) return @_rollout ||= Configuration.default_rollout if rollout.blank? @_rollout = Rollout.resolve(rollout, options) end # Class level accessor methods. def published_experiments RequestStore.store[:published_gitlab_experiments] || {} end end def name [Configuration.name_prefix, @_name].compact.join('_') end def control(&block) variant(:control, &block) end def candidate(&block) variant(:candidate, &block) end def variant(name, &block) raise ArgumentError, 'name required' if name.blank? raise ArgumentError, 'block required' unless block.present? behaviors[name] = block end def context(value = nil) return @_context if value.blank? @_context.value(value) @_context end def assigned(value = nil) # Skip force assignment if a variant was already set (e.g., via constructor or explicit #assigned call). value ||= forced_variant_name unless @_assigned_variant_name @_assigned_variant_name = cache_variant(value) if value.present? return Variant.new(name: @_assigned_variant_name || :unresolved) if @_assigned_variant_name || @_resolving_variant if enabled? @_resolving_variant = true @_assigned_variant_name = cached_variant_resolver(@_assigned_variant_name) end run_callbacks(segmentation_callback_chain) do @_assigned_variant_name ||= :control Variant.new(name: @_assigned_variant_name) end ensure @_resolving_variant = false end def rollout(rollout = nil, options = {}) return @_rollout ||= self.class.default_rollout(nil, options).for(self) if rollout.blank? @_rollout = Rollout.resolve(rollout, options).for(self) end def exclude! @_excluded = true end def run(variant_name = nil) return @_result if context.frozen? @_result = run_callbacks(run_callback_chain) { super(assigned(variant_name).name) } end def publish(result = nil) instance_exec(result, &Configuration.publishing_behavior) (RequestStore.store[:published_gitlab_experiments] ||= {})[name] = signature.merge(excluded: excluded?) end def track(action, **event_args) return unless should_track? instance_exec(action, tracking_context(event_args).try(:compact) || {}, &Configuration.tracking_behavior) end def enabled? rollout.enabled? end def excluded? return @_excluded if defined?(@_excluded) @_excluded = !run_callbacks(exclusion_callback_chain) { :not_excluded } || only_assigned? end def only_assigned? !!context.only_assigned && find_variant.blank? end def should_track? enabled? && context.trackable? && !excluded? end def signature { variant: assigned.name.to_s, experiment: name }.merge(context.signature) end def behaviors @_behaviors ||= registered_behavior_callbacks end protected def identify(object) (object.respond_to?(:to_global_id) ? object.to_global_id : object).to_s end def cached_variant_resolver(provided_variant) return :control if excluded? result = cache_variant(provided_variant) { resolve_variant_name } result.to_sym if result.present? end def resolve_variant_name rollout.resolve end def tracking_context(event_args) {}.merge(event_args) end end end gitlab-experiment-1.3.0/LICENSE.txt0000644000004100000410000000237415151343617017022 0ustar www-datawww-dataCopyright (c) 2020-2022 GitLab B.V. With regard to the GitLab Software: Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. For all third party components incorporated into the GitLab Software, those components are licensed under the original license provided by the owner of the applicable component. gitlab-experiment-1.3.0/gitlab-experiment.gemspec0000644000004100000410000000730715151343620022157 0ustar www-datawww-data######################################################### # This file has been automatically generated by gem2tgz # ######################################################### # -*- encoding: utf-8 -*- # stub: gitlab-experiment 1.3.0 ruby lib Gem::Specification.new do |s| s.name = "gitlab-experiment".freeze s.version = "1.3.0".freeze s.required_rubygems_version = Gem::Requirement.new(">= 0".freeze) if s.respond_to? :required_rubygems_version= s.require_paths = ["lib".freeze] s.authors = ["GitLab".freeze] s.date = "2026-02-25" s.email = ["gitlab_rubygems@gitlab.com".freeze] s.files = ["LICENSE.txt".freeze, "README.md".freeze, "lib/generators/gitlab/experiment/USAGE".freeze, "lib/generators/gitlab/experiment/experiment_generator.rb".freeze, "lib/generators/gitlab/experiment/install/install_generator.rb".freeze, "lib/generators/gitlab/experiment/install/templates/POST_INSTALL".freeze, "lib/generators/gitlab/experiment/install/templates/application_experiment.rb.tt".freeze, "lib/generators/gitlab/experiment/install/templates/initializer.rb.tt".freeze, "lib/generators/gitlab/experiment/templates/experiment.rb.tt".freeze, "lib/generators/rspec/experiment/experiment_generator.rb".freeze, "lib/generators/rspec/experiment/templates/experiment_spec.rb.tt".freeze, "lib/generators/test_unit/experiment/experiment_generator.rb".freeze, "lib/generators/test_unit/experiment/templates/experiment_test.rb.tt".freeze, "lib/gitlab/experiment.rb".freeze, "lib/gitlab/experiment/base_interface.rb".freeze, "lib/gitlab/experiment/cache.rb".freeze, "lib/gitlab/experiment/cache/redis_hash_store.rb".freeze, "lib/gitlab/experiment/callbacks.rb".freeze, "lib/gitlab/experiment/configuration.rb".freeze, "lib/gitlab/experiment/context.rb".freeze, "lib/gitlab/experiment/cookies.rb".freeze, "lib/gitlab/experiment/dsl.rb".freeze, "lib/gitlab/experiment/engine.rb".freeze, "lib/gitlab/experiment/errors.rb".freeze, "lib/gitlab/experiment/force_assignment.rb".freeze, "lib/gitlab/experiment/middleware.rb".freeze, "lib/gitlab/experiment/nestable.rb".freeze, "lib/gitlab/experiment/rollout.rb".freeze, "lib/gitlab/experiment/rollout/percent.rb".freeze, "lib/gitlab/experiment/rollout/random.rb".freeze, "lib/gitlab/experiment/rollout/round_robin.rb".freeze, "lib/gitlab/experiment/rspec.rb".freeze, "lib/gitlab/experiment/test_behaviors/trackable.rb".freeze, "lib/gitlab/experiment/variant.rb".freeze, "lib/gitlab/experiment/version.rb".freeze] s.homepage = "https://gitlab.com/gitlab-org/ruby/gems/gitlab-experiment".freeze s.licenses = ["MIT".freeze] s.required_ruby_version = Gem::Requirement.new(">= 3.0".freeze) s.rubygems_version = "3.5.22".freeze s.summary = "GitLab experimentation library.".freeze s.specification_version = 4 s.add_runtime_dependency(%q.freeze, [">= 3.0".freeze]) s.add_development_dependency(%q.freeze, ["~> 0.26.2".freeze]) s.add_development_dependency(%q.freeze, ["~> 0.9.4".freeze]) s.add_development_dependency(%q.freeze, ["~> 4.1.0".freeze]) s.add_development_dependency(%q.freeze, ["~> 12.0.1".freeze]) s.add_development_dependency(%q.freeze, ["~> 1.4.7".freeze]) s.add_development_dependency(%q.freeze, ["~> 3.10.1".freeze]) s.add_runtime_dependency(%q.freeze, [">= 1.0".freeze]) s.add_development_dependency(%q.freeze, ["~> 1.0.0".freeze]) s.add_development_dependency(%q.freeze, ["~> 6.0.3".freeze]) s.add_development_dependency(%q.freeze, ["~> 2.24.0".freeze]) s.add_development_dependency(%q.freeze, ["~> 2.27.1".freeze]) s.add_development_dependency(%q.freeze, ["~> 3.1.0".freeze]) end gitlab-experiment-1.3.0/README.md0000644000004100000410000015515515151343617016464 0ustar www-datawww-dataGitLab Experiment Platform ================= experiment **A comprehensive experimentation platform for building data-driven organizations** GitLab Experiment is an enterprise-grade experimentation framework that enables teams to validate hypotheses, optimize user experiences, and make evidence-based product decisions at scale. Built on years of production experience at GitLab, this platform provides the foundation for a mature experimentation culture across your entire organization. At GitLab, we run experiments as A/B/n tests and review the data they generate. From that data, we determine the best performing code path and promote it as the new default, or revert back to the original code path. You can read our [Experiment Guide](https://docs.gitlab.com/ee/development/experiment_guide/) to learn how we use this gem internally at GitLab. [[_TOC_]] ## Why GitLab Experiment? ### Built for Scale and Reliability - **Production-tested** at GitLab scale with millions of users - **Type-safe and testable** with comprehensive RSpec support - **Framework agnostic** - works with Rails, Sinatra, or standalone Ruby applications - **Redis-backed caching** for consistent user experiences across sessions - **GDPR-compliant** with anonymous tracking and built-in DNT (Do Not Track) support ### Designed for Teams - **Developer-friendly DSL** that reads like natural language - **Organized experiment classes** that live alongside your application code - **Built-in generators** for rapid experiment creation - **Comprehensive testing support** with custom RSpec matchers - **Rails integration** with automatic middleware mounting and view helpers ### Enterprise-Ready Features - **Flexible rollout strategies** (percent-based, random, round-robin, or custom) - **Advanced segmentation** to target specific user populations - **Multi-variant testing** (A/B/n) with unlimited experimental paths - **Progressive rollouts** with the `only_assigned` feature - **Context migrations** for evolving experiments without losing data - **Integration-ready** with existing feature flag systems (Flipper, Unleash, etc.)
## Use Cases Across Your Organization ### Product Teams: Optimize User Experiences - **Onboarding flows**: Test different signup sequences to maximize activation - **UI/UX changes**: Validate design decisions with real user behavior data - **Feature rollouts**: Gradually release features to measure impact before full deployment - **Pricing experiments**: Test different pricing strategies and messaging ### Growth Teams: Drive Conversion - **Call-to-action optimization**: Test button colors, copy, and placement - **Landing page variations**: Experiment with different value propositions - **Email campaigns**: A/B test subject lines and content - **Trial conversion**: Optimize paths from trial to paid subscriptions ### Engineering Teams: Safe Deployments - **Performance optimizations**: Compare algorithm implementations under real load - **Architecture changes**: Validate new code paths before full migration - **API versions**: Run multiple API implementations side-by-side - **Infrastructure experiments**: Test different caching or database strategies ### Data Science Teams: Recommendation Systems - **Algorithm tuning**: Compare ML model variations in production - **Personalization**: Test different recommendation strategies - **Search ranking**: Optimize search results based on user engagement - **Content discovery**: Experiment with different content surfaces ## Platform Capabilities ### Core Experimentation Features **Multi-variant Testing (A/B/n)** Run experiments with any number of variants, not just A/B tests. Perfect for testing multiple approaches simultaneously. **Smart Segmentation** Route specific user populations to predetermined variants based on business rules, ensuring consistent experiences for targeted groups. **Progressive Rollouts** Use the `only_assigned` feature to show experimental features only to users already in the experiment, enabling controlled expansion. **Context Flexibility** Experiments can be sticky to users, projects, organizations, or any combination - enabling complex scenarios beyond user-centric testing. **Anonymous Tracking** Built-in privacy protection with anonymous context keys, automatic cookie migration, and GDPR compliance. **Automatic Assignment Tracking** Every experiment automatically tracks an `:assignment` event when it runs - zero configuration required. Combined with the anonymous context key, this gives your data team a complete picture of variant distribution and funnel entry without any additional instrumentation. **Client-Side Integration** Seamlessly extend experiments to the frontend with JavaScript integration, enabling full-stack experimentation. **Inline and Class-Based APIs** Define experiments inline with blocks for quick iterations, or use dedicated experiment classes for complex logic - or combine both. Class-based experiments define default behaviors that can be overridden inline at any call site, giving teams the flexibility to start simple and evolve without rewriting: ```ruby experiment(:pill_color, actor: current_user) do |e| e.control { 'control' } end ``` **Context as a Design Framework** Context is the most important design decision in any experiment. It determines stickiness, cache behavior, and how events are correlated. Choose per-user context for personalization experiments, per-project for infrastructure tests, per-group for organizational rollouts, or combine dimensions for precision targeting. This flexibility enables experimentation strategies that go far beyond simple user-centric A/B tests. **Decoupled Assignment with Publish** Surface experiment assignments to the client layer without executing server-side behavior using `publish`. This enables frontend-only experiments, pre-assignment in `before_action` hooks, and scenarios where variant data needs to be available across the stack without triggering server-side code paths: ```ruby before_action -> { experiment(:pill_color, actor: current_user).publish }, only: [:show] ``` ### Integration Ecosystem **Feature Flag Integration** Connect with existing feature flag systems like Flipper or Unleash through custom rollout strategies. **Analytics Integration** Flexible tracking callbacks integrate with any analytics platform - Snowplow, Amplitude, Mixpanel, or your data warehouse. **Monitoring and Observability** Built-in logging and callbacks for integration with APM tools and monitoring systems. **Email and Markdown** Special middleware for tracking experiments in email links and static content. ### Terminology When we discuss the platform, we use specific terms that are worth understanding: - **experiment** - Any deviation of code paths we want to test - **context** - Identifies a consistent experience (user, project, session, etc.) - **control** - The default or "original" code path - **candidate** - One experimental code path (used in A/B tests) - **variant(s)** - Multiple experimental paths (used in A/B/n tests) - **behaviors** - All possible code paths (control + all variants) - **rollout strategy** - Logic determining if an experiment is enabled and how variants are assigned - **segmentation** - Rules for routing specific contexts to predetermined variants - **exclusion** - Rules for keeping contexts out of experiments entirely
## Quick Start: From Zero to Experiment in 5 Minutes ### Installation Add the gem to your Gemfile and then `bundle install`. ```ruby gem 'gitlab-experiment' ``` If you're using Rails, install the initializer which provides basic configuration, documentation, and the base experiment class: ```shell $ rails generate gitlab:experiment:install ``` ### Your First Experiment Let's create a real-world experiment to optimize a call-to-action button. This example demonstrates the power of the platform while remaining practical. #### Step 1: Generate the experiment **Hypothesis**: A more prominent call-to-action button will increase conversion rates ```shell $ rails generate gitlab:experiment signup_cta ``` This creates `app/experiments/signup_cta_experiment.rb` with helpful inline documentation. #### Step 2: Define your experiment class ```ruby class SignupCtaExperiment < ApplicationExperiment # Define the control (current experience) control { 'btn-default' } # Define the candidate (new experience to test) candidate { 'btn-primary btn-lg' } # Optional: Exclude certain users exclude :existing_customers # Optional: Track when the experiment runs after_run :log_experiment_assignment private def existing_customers context.actor&.subscribed? end def log_experiment_assignment Rails.logger.info("User assigned to #{assigned.name} variant") end end ``` #### Step 3: Use the experiment in your view ```haml -# The experiment is sticky to the current user -# Anonymous users get a cookie-based assignment %button{ class: experiment(:signup_cta, actor: current_user).run } Start Free Trial ``` #### Step 4: Track engagement ```ruby # In your controller, track when users click the button def create_trial experiment(:signup_cta, actor: current_user).track(:signup_completed) # ... rest of your trial creation logic end ``` **That's it!** Your experiment is now running, collecting data, and providing consistent experiences to your users. ## Real-World Examples ### Example 1: Onboarding Flow Optimization **Business Context**: Product team wants to increase new user activation by testing different onboarding sequences. ```ruby class OnboardingFlowExperiment < ApplicationExperiment # Three different onboarding approaches control { :standard_tour } # Current 5-step tour variant(:quick) { :quick_start } # Streamlined 2-step flow variant(:video) { :video_guide } # Video-based walkthrough # Only show to new users who haven't completed onboarding exclude :has_completed_onboarding # Segment enterprise trial users to the standard tour segment :enterprise_trial?, variant: :control private def has_completed_onboarding context.actor&.onboarding_completed_at.present? end def enterprise_trial? context.actor&.trial_type == 'enterprise' end end # In your onboarding controller def show flow = experiment(:onboarding_flow, actor: current_user).run render_onboarding_flow(flow) end # Track completion def complete experiment(:onboarding_flow, actor: current_user).track(:completed) # ... mark user as onboarded end ``` ### Example 2: Pricing Page Experiment **Business Context**: Growth team wants to test whether showing annual savings increases annual plan selection. ```ruby class PricingDisplayExperiment < ApplicationExperiment control { :monthly_default } candidate { :annual_default_with_savings } # Only run for unauthenticated visitors exclude :authenticated_user private def authenticated_user context.actor.present? end end # In your pricing view - pricing_variant = experiment(:pricing_display, actor: current_user).run = render "pricing/#{pricing_variant}" # Track plan selections def select_plan experiment(:pricing_display, actor: current_user).track(:plan_selected, value: params[:plan_type] == 'annual' ? 1 : 0 ) end ``` ### Example 3: Algorithm Performance Test **Business Context**: Engineering team wants to compare a new search algorithm's performance before full rollout. ```ruby class SearchAlgorithmExperiment < ApplicationExperiment control { SearchEngine::Legacy } candidate { SearchEngine::Neural } # Only run for 25% of searches default_rollout :percent, distribution: { control: 75, candidate: 25 } # Exclude searches from API (higher SLA requirements) exclude :api_request # Track performance metrics after_run :record_search_timing private def api_request context.request&.path&.start_with?('/api/') end def record_search_timing # Custom metrics tracking end end # In your search service def search(query) algorithm = experiment(:search_algorithm, actor: current_user, project: current_project ).run results = algorithm.search(query) experiment(:search_algorithm, actor: current_user, project: current_project ).track(:search_completed, value: results.count) results end ``` ### Example 4: Progressive Feature Rollout **Business Context**: Launching a new AI-assisted code review feature, want to expand gradually to manage load and gather feedback. ```ruby class AiCodeReviewExperiment < ApplicationExperiment control { false } # Feature disabled candidate { true } # Feature enabled # Start with 5% rollout default_rollout :percent, distribution: { control: 95, candidate: 5 } # Segment beta program users to always get the feature segment :beta_user?, variant: :candidate # Exclude free tier (computational cost consideration) exclude :free_tier_user private def beta_user? context.actor&.beta_features_enabled? end def free_tier_user context.actor&.subscription_tier == 'free' end end # In your merge request view - if experiment(:ai_code_review, actor: current_user, project: @project).run .ai-code-review-panel = render 'ai_suggestions' # Track usage def apply_ai_suggestion experiment(:ai_code_review, actor: current_user, project: @project) .track(:suggestion_applied) end ``` ## Platform Integration Patterns ### Integration with Feature Flags (Flipper) Many organizations already use feature flag systems. GitLab Experiment integrates seamlessly: ```ruby module Gitlab::Experiment::Rollout class Flipper < Percent def enabled? ::Flipper.enabled?(experiment.name, experiment_actor) end def experiment_actor Struct.new(:flipper_id).new("Experiment;#{id}") end end end # Configure globally Gitlab::Experiment.configure do |config| config.default_rollout = Gitlab::Experiment::Rollout::Flipper.new end # Now Flipper controls your experiments Flipper.enable_percentage_of_actors(:signup_cta, 50) ``` ### Integration with Analytics Platforms Connect experiments to your analytics stack: ```ruby Gitlab::Experiment.configure do |config| config.tracking_behavior = lambda do |event_name, **data| # Snowplow SnowplowTracker.track_struct_event( category: 'experiment', action: event_name, property: data[:experiment], context: [{ schema: 'experiment_context', data: data }] ) # Amplitude (example) Amplitude.track( user_id: data[:key], # Anonymous experiment key event_type: "experiment_#{event_name}", event_properties: data ) # Custom data warehouse DataWarehouse.log_experiment_event(event_name, data) end end ``` ### Multi-Application Consistency Share experiment assignments across multiple applications: ```ruby # Shared Redis cache Gitlab::Experiment.configure do |config| config.cache = Gitlab::Experiment::Cache::RedisHashStore.new( Redis.new(url: ENV['REDIS_URL']), expires_in: 30.days ) end # Now experiments stay consistent across your web app, API, and background jobs ``` ## Advanced Features ### Multi-Variant (A/B/n) Testing Test multiple variations simultaneously to find the optimal solution: ```ruby class NotificationStyleExperiment < ApplicationExperiment # Test three different notification approaches control { :banner } # Current: banner at top variant(:toast) { :toast } # Toast notification variant(:modal) { :modal } # Modal dialog # Distribute traffic evenly across all three default_rollout :percent, distribution: { control: 34, toast: 33, modal: 33 } # Exclude mobile users (different UI constraints) exclude :mobile_user # Segment power users to toast (less intrusive) segment :power_user?, variant: :toast private def mobile_user context.request&.user_agent&.match?(/Mobile/) end def power_user? context.actor&.actions_count > 1000 end end ``` ### Exclusion Rules Keep contexts out of experiments entirely based on business rules: ```ruby class FeatureExperiment < ApplicationExperiment # Exclude existing customers (only test on prospects) exclude :existing_customer # Exclude during maintenance windows exclude -> { context.project&.under_maintenance? } # Exclude if feature is explicitly disabled exclude :feature_disabled private def existing_customer context.actor&.subscribed? end def feature_disabled !FeatureFlag.enabled?(:allow_experiment, context.actor) end end ``` **Key behaviors:** - Excluded users always receive the control experience - No tracking events are recorded for excluded users - Exclusion rules are evaluated in order, first match wins - Exclusions improve performance by exiting early **Inline exclusion** is also supported: ```ruby experiment(:feature, actor: current_user) do |e| e.exclude! unless can?(current_user, :manage, project) e.control { 'standard' } e.candidate { 'enhanced' } end ``` Note: Although tracking calls will be ignored on all exclusions, you may want to check exclusion yourself in expensive custom logic by calling the `should_track?` or `excluded?` methods. Note: When using exclusion rules it's important to understand that the control assignment is cached, which improves future experiment run performance but can be a gotcha around caching. Note: Exclusion rules aren't the best way to determine if an experiment is enabled. There's an `enabled?` method that can be overridden to have a high-level way of determining if an experiment should be running and tracking at all. This `enabled?` check should be as efficient as possible because it's the first early opt out path an experiment can implement. This can be seen in [How Experiments Work (Technical)](#how-experiments-work-technical). ### Segmentation Rules Route specific populations to predetermined variants: ```ruby class NewFeatureExperiment < ApplicationExperiment # Route VIP customers to the new feature segment :vip_customer?, variant: :candidate # Route enterprise trial users to the enhanced experience segment :enterprise_trial?, variant: :candidate # Route users from specific campaigns to specific variants segment(variant: :candidate) { context.campaign == 'product_launch_2024' } private def vip_customer? context.actor&.account_value > 100_000 end def enterprise_trial? context.actor&.trial_tier == 'enterprise' end end ``` **Key behaviors:** - Segmentation rules are evaluated in order, first match wins - Segmented assignments are cached for consistency - Perfect for gradually expanding successful experiments - Enables sophisticated population targeting ### Lifecycle Callbacks Execute custom logic at different stages of experiment execution: ```ruby class PerformanceExperiment < ApplicationExperiment # Run before the variant is determined before_run :log_experiment_start # Run after the variant is executed after_run :record_timing_metrics, :notify_analytics_team # Wrap the entire execution around_run do |experiment, block| start_time = Time.current result = block.call duration = Time.current - start_time Metrics.record("experiment.#{experiment.name}.duration", duration) result end private def log_experiment_start Rails.logger.info("Starting experiment: #{name}") end def record_timing_metrics # Custom timing logic end def notify_analytics_team # Send to analytics platform end end ``` **Use cases for callbacks:** - Performance monitoring and APM integration - Custom analytics and data warehouse updates - Experiment-specific logging and debugging - Integration with external systems ### Progressive Rollout with `only_assigned` Control experiment expansion by only showing features to users already assigned to the experiment. This is critical for managing blast radius and controlled rollouts: **The Challenge**: You launch an experiment to 10% of new signups. Later, you want to show experimental features on other pages, but only to users already in the experiment - not expand to 10% of all users across the platform. **The Solution**: Use `only_assigned: true` ```ruby # Step 1: Assign users during signup (10% of new signups) class RegistrationsController < ApplicationController def create user = User.create!(user_params) # This assigns 10% to candidate, 90% to control experiment(:onboarding_v2, actor: user).publish redirect_to dashboard_path end end # Step 2: Later, show features only to those already assigned class DashboardController < ApplicationController def show # This will NOT expand the experiment to 10% of all users # Only users assigned in Step 1 will see the experimental UI @show_new_features = experiment(:onboarding_v2, actor: current_user, only_assigned: true ).assigned.name == 'candidate' end end # Step 3: Show UI conditionally across the app - if experiment(:onboarding_v2, actor: current_user, only_assigned: true).run .new-onboarding-features = render 'enhanced_dashboard' ``` **Behavior with `only_assigned: true`:** - ✅ If user already assigned → returns their cached variant - ✅ If user not assigned → returns control, no tracking - ✅ Experiment reach stays controlled - ✅ Perfect for multi-page experimental experiences **Real-world use cases:** - **Post-signup experiences**: Assign at signup, show features throughout the app - **Gradual feature expansion**: Roll out to 5%, then add more touchpoints without expanding population - **Cleanup phases**: Maintain experience for existing participants while preventing new assignments - **A/B testing with multiple surfaces**: Test a hypothesis across multiple pages without assignment leakage ### Custom Rollout Strategies The platform supports multiple rollout strategies out of the box, and you can create custom strategies for your specific needs. **Built-in strategies:** - [`Percent`](lib/gitlab/experiment/rollout/percent.rb) - Consistent percentage-based assignment (default, recommended) - [`Random`](lib/gitlab/experiment/rollout/random.rb) - True random assignment (useful for load testing) - [`RoundRobin`](lib/gitlab/experiment/rollout/round_robin.rb) - Cycle through variants (requires caching) - [`Base`](lib/gitlab/experiment/rollout.rb) - Useful for building custom rollout strategies ```ruby class LoadTestExperiment < ApplicationExperiment # Randomly test two different caching strategies default_rollout :random control { CacheStrategy::Redis } candidate { CacheStrategy::Memcached } end class GradualRolloutExperiment < ApplicationExperiment # Start with 5% in the new experience default_rollout :percent, distribution: { control: 95, candidate: 5 } end ``` See the [Advanced: Custom Rollout Strategies](#advanced-custom-rollout-strategies) section for building your own integration with feature flag systems. ## Organizational Best Practices ### Experiment Lifecycle Management **1. Hypothesis Formation** ```ruby # Document your hypothesis in the experiment class class CheckoutFlowExperiment < ApplicationExperiment # Hypothesis: Reducing checkout steps from 3 to 2 will increase completion rate # Success metric: 5% increase in checkout completion # Target: All free trial users # Duration: 2 weeks # Owner: @growth-team control { :three_step_checkout } candidate { :two_step_checkout } exclude :existing_customer end ``` **2. Gradual Rollout** ```ruby # Week 1: 5% rollout default_rollout :percent, distribution: { control: 95, candidate: 5 } # Week 2: Increase to 25% if metrics look good default_rollout :percent, distribution: { control: 75, candidate: 25 } # Week 3: Full rollout if successful default_rollout :percent, distribution: { control: 0, candidate: 100 } ``` **3. Monitoring and Alerting** ```ruby class CriticalPathExperiment < ApplicationExperiment after_run :monitor_performance after_run :alert_on_errors private def monitor_performance Metrics.increment("experiment.#{name}.#{assigned.name}") end def alert_on_errors if context.error_rate > threshold PagerDuty.alert("High error rate in #{name}") end end end ``` **4. Experiment Cleanup** ```ruby # When experiment is conclusive, clean up: # 1. Remove the experiment code # 2. Promote winner to production # 3. Document learnings # Before cleanup, archive results: experiment(:checkout_flow).publish # Export data for historical analysis ``` ### Team Collaboration Patterns **Product + Engineering + Data Science** ```ruby class CollaborativeExperiment < ApplicationExperiment # Product defines the hypothesis and variants control { :current_flow } candidate { :new_flow } # Engineering defines segmentation and rollout segment :beta_users, variant: :candidate default_rollout :percent, distribution: { control: 90, candidate: 10 } # Data science defines tracking and metrics after_run :track_funnel_step def track_funnel_step Analytics.track_experiment_step( experiment: name, variant: assigned.name, funnel_position: context.step, user_segment: context.actor&.segment ) end end ``` ### Testing Strategy Write tests for your experiments using the included RSpec matchers: ```ruby RSpec.describe CheckoutFlowExperiment do describe 'segmentation' do it 'routes existing customers to control' do customer = create(:user, :with_subscription) expect(experiment(:checkout_flow)).to exclude(actor: customer) end it 'routes enterprise trials to candidate' do trial = create(:user, :enterprise_trial) expect(experiment(:checkout_flow)) .to segment(actor: trial).into(:candidate) end end describe 'tracking' do it 'tracks checkout completion' do expect(experiment(:checkout_flow)).to track(:completed) .on_next_instance CheckoutService.complete(user: user) end end end ``` ### Naming Conventions Establish clear naming conventions for your organization: ```ruby # Good: Descriptive experiment names class OnboardingFlowV2Experiment < ApplicationExperiment; end class PricingPageAnnualFocusExperiment < ApplicationExperiment; end class SearchAlgorithmNeuralExperiment < ApplicationExperiment; end # Avoid: Vague names class TestExperiment < ApplicationExperiment; end # What are we testing? class ExperimentOne < ApplicationExperiment; end # No context ``` ## Technical Reference ### How Experiments Work (Technical) Understanding the experiment resolution flow helps you design better experiments and debug issues: **Decision tree for variant assignment:** ```mermaid graph TD GP[General Pool/Population] --> Running?[Rollout Enabled?] Running? -->|Yes| Forced?[Forced Assignment?] Running? -->|No| Excluded[Control / No Tracking] Forced? -->|Yes / Cached| ForcedVariant[Forced Variant] Forced? -->|No| Cached?[Cached? / Pre-segmented?] Cached? -->|No| Excluded? Cached? -->|Yes| Cached[Cached Value] Excluded? -->|Yes / Cached| Excluded Excluded? -->|No| Segmented? Segmented? -->|Yes / Cached| VariantA Segmented? -->|No| Rollout[Rollout Resolve] Rollout --> Control Rollout -->|Cached| VariantA Rollout -->|Cached| VariantB Rollout -->|Cached| VariantN class ForcedVariant,VariantA,VariantB,VariantN included class Control,Excluded excluded class Cached cached ``` **Key points:** 1. Rollout must be enabled for any variant assignment (including forced assignment) 2. Forced assignment takes priority over cache/exclusion/segmentation (via `glex_force` query parameter) 3. Cache provides consistency across calls 4. Segmentation takes priority over rollout 5. `only_assigned: true` exits early if no cache hit ### Experiment Context and Stickiness Internally, experiments have what's referred to as the context "key" that represents the unique and anonymous id of a given context. This allows us to assign the same variant between different calls to the experiment, is used in caching and can be used in event data downstream. This context "key" is how an experiment remains "sticky" to a given context, and is an important aspect to understand. **Context defines stickiness** - experiments remain consistent by generating an anonymous key from the context: ```ruby # Sticky to user - same user gets same variant everywhere experiment(:feature, actor: current_user) # Sticky to project - all users on a project get the same experience experiment(:feature, project: project) # Sticky to user+project - same user gets same variant per project experiment(:feature, actor: current_user, project: project) # Custom stickiness - explicitly define what creates consistency experiment(:feature, actor: current_user, project: project, sticky_to: project) ``` **The `actor` keyword has special behavior:** - Anonymous users → temporary cookie-based assignment - Upon sign-in → cookie migrates to user ID automatically - Enables consistent experience across anonymous → authenticated journey ### Using Experiments Beyond Views By default, `Gitlab::Experiment` injects itself into the controller, view, and mailer layers. This exposes the `experiment` method application wide in those layers. Some experiments may extend outside of those layers however, so you may want to include it elsewhere. For instance in an irb session or the rails console, or in all your service objects, background jobs, or similar: ```ruby # In all background jobs class ApplicationJob < ActiveJob::Base include Gitlab::Experiment::Dsl end # In service objects class ApplicationService include Gitlab::Experiment::Dsl end # In a console session include Gitlab::Experiment::Dsl experiment(:feature, actor: User.first).run ``` ### Manual Variant Assignment
You can also specify the variant manually... Generally, defining segmentation rules is a better way to approach routing into specific variants, but it's possible to explicitly specify the variant when running an experiment. Caching: It's important to understand what this might do to your data during rollout, so use this with careful consideration. Any time a specific variant is assigned manually, or through segmentation (including `:control`) it will be cached for that context. That means that if you manually assign `:control`, that context will never be moved out of the control unless you do it programmatically elsewhere. ```ruby include Gitlab::Experiment::Dsl # Assign the candidate manually. ex = experiment(:pill_color, :red, actor: User.first) # => # # Run the experiment -- returning the result. ex.run # => "red" # If caching is enabled this will remain sticky between calls. experiment(:pill_color, actor: User.first).run # => "red" ```
### Forced Variant Assignment (QA/UAT) For testing and validation purposes, you can force a specific variant assignment via a URL query parameter. This is useful for QA testing in staging or production environments where you need to verify a specific variant's behavior. **Configuration:** Forced assignment is disabled by default. Enable it in your initializer: ```ruby Gitlab::Experiment.configure do |config| config.allow_forced_assignment = true end ``` **Usage:** Append the `glex_force` query parameter to any URL with the format `experiment_name:variant_name`: ``` https://your-app.com/signup?glex_force=myapp_signup_cta:candidate ``` The forced variant is written to the cache (Redis) on the same request, making it permanent for that context. The query parameter only needs to be provided once -- after that, the variant is persisted in the cache like any normal assignment. #### Anonymous user (nil actor) -- initial assignment This is the primary use case for QA testing signup flows and landing pages. The user is not signed in, so the actor is nil and the experiment uses a cookie-based context key. 1. Anonymous user visits `https://your-app.com/signup?glex_force=signup_cta:candidate` 2. The forced variant `:candidate` is written to Redis under the cookie-based context key 3. The user signs in -- the standard cookie migration carries the forced variant to their real identity 4. All future requests use `:candidate` from Redis, permanently This means a QA tester can force a variant before signup and have it follow the user through the entire anonymous-to-authenticated journey. #### Signed-in user -- initial assignment When a signed-in user hasn't been assigned a variant yet, the force param assigns and caches it immediately: ``` https://your-app.com/dashboard?glex_force=new_feature:candidate ``` The variant is cached under the user's context key on this request. No further query parameter is needed. #### Signed-in user -- re-assignment (overwriting an existing variant) If a user was previously assigned `:control` (by the rollout strategy or a prior force), the force param overwrites the cached value: ``` https://your-app.com/dashboard?glex_force=new_feature:candidate ``` The existing `:control` assignment in Redis is replaced with `:candidate`. This is useful when QA needs to switch a user between variants without clearing the cache manually. #### Disabled experiments and feature flags Forced assignment requires the experiment to be enabled. If the experiment is disabled (as determined by the rollout strategy's `enabled?` method), the `glex_force` parameter is ignored and normal resolution applies (which will assign control). This is intentional -- a disabled experiment may be disabled for valid reasons (incomplete implementation, known issues, compliance constraints, etc.) and force assignment should not provide a way to bypass that decision. To use forced assignment, ensure the experiment is enabled first through your rollout strategy. **Important notes:** - The experiment name in the parameter must match the full experiment name (including any configured `name_prefix`). - If the variant name doesn't match a registered behavior, the forced assignment is ignored and normal variant resolution proceeds (typically resulting in the control variant). - Forced assignment does not override a variant that was already set via the constructor or an explicit `assigned()` call within the same request. - This feature requires a `request` object with `params` to be available in the experiment context. > [!NOTE] > Because forcing the variant ignores the exclusion/segmentation process it will cover up those types of errors so if your experiment relies on these types of logic this testing method should be avoided. ### Experiment Signature The best way to understand the details of an experiment is through its signature. An example signature can be retrieved by calling the `signature` method, and looks like the following: ```ruby experiment(:example).signature # => {:variant=>"control", :experiment=>"example", :key=>"4d7aee..."} ``` An experiment signature is useful when tracking events and when using experiments on the client layer. The signature can also contain the optional `migration_keys`, and `excluded` properties. ### Return Value By default the return value of calling `experiment` is a `Gitlab::Experiment` instance, or whatever class the experiment is resolved to, which likely inherits from `Gitlab::Experiment`. In simple cases you may want only the results of running the experiment though. You can call `run` within the block to get the return value of the assigned variant. ```ruby # Normally an experiment instance. experiment(:example) do |e| e.control { 'A' } e.candidate { 'B' } end # => # # But calling `run` causes the return value to be the result. experiment(:example) do |e| e.control { 'A' } e.candidate { 'B' } e.run end # => 'A' ``` ### Context migrations There are times when we need to change context while an experiment is running. We make this possible by passing the migration data to the experiment. Take for instance, that you might be using `version: 1` in your context currently. To migrate this to `version: 2`, provide the portion of the context you wish to change using a `migrated_with` option. In providing the context migration data, we can resolve an experience and its events all the way back. This can also help in keeping our cache relevant. ```ruby # First implementation. experiment(:example, actor: current_user, version: 1) # Migrate just the `:version` portion. experiment(:example, actor: current_user, version: 2, migrated_with: { version: 1 }) ``` You can add or remove context by providing a `migrated_from` option. This approach expects a full context replacement -- i.e. what it was before you added or removed the new context key. If you wanted to introduce a `version` to your context, provide the full previous context. ```ruby # First implementation. experiment(:example, actor: current_user) # Migrate the full context of `{ actor: current_user }` to `{ actor: current_user, version: 1 }`. experiment(:example, actor: current_user, version: 1, migrated_from: { actor: current_user }) ``` When you migrate context, this information is included in the signature of the experiment. This can be used downstream in event handling and reporting to resolve a series of events back to a single experience, while also keeping everything anonymous. An example of our experiment signature when we migrate would include the `migration_keys` property: ```ruby ex = experiment(:example, version: 1) ex.signature # => {:key=>"20d69a...", ...} ex = experiment(:example, version: 2, migrated_from: { version: 1 }) ex.signature # => {:key=>"9e9d93...", :migration_keys=>["20d69a..."], ...} ``` ### Cookies and the actor keyword We use cookies to auto migrate an unknown value into a known value, often in the case of the current user. The implementation of this uses the same concept outlined above with context migrations, but will happen automatically for you if you use the `actor` keyword. When you use the `actor: current_user` pattern in your context, the nil case is handled by setting a special cookie for the experiment and then deleting the cookie, and migrating the context key to the one generated from the user when they've signed in. This cookie is a temporary, randomized uuid and isn't associated with a user. When we can finally provide an actor, the context is auto migrated from the cookie to that actor. ```ruby # The actor key is not present, so no cookie is set. experiment(:example, project: project) # The actor key is present but nil, so the cookie is set and used. experiment(:example, actor: nil, project: project) # The actor key is present and isn't nil, so the cookie value (if found) is # migrated forward and the cookie is deleted. experiment(:example, actor: current_user, project: project) ``` Note: The cookie is deleted when resolved, but can be assigned again if the `actor` is ever nil again. A good example of this scenario would be on a sign in page. When a potential user arrives, they would never be known, so a cookie would be set for them, and then resolved/removed as soon as they signed in. This process would repeat each time they arrived while not being signed in and can complicate reporting unless it's handled well in the data layers. Note: To read and write cookies, we provide the `request` from within the controller and views. The cookie migration will happen automatically if the experiment is within those layers. You'll need to provide the `request` as an option to the experiment if it's outside of the controller and views. ```ruby experiment(:example, actor: current_user, request: request) ``` Note: For edge cases, you can pass the cookie through by assigning it yourself -- e.g. `actor: request.cookie_jar.signed['example_id']`. The cookie name is the full experiment name (including any configured prefix) with `_id` appended -- e.g. `pill_color_id` for the `PillColorExperiment`. ### Client layer Experiments that have been run (or published) during the request lifecycle can be pushed into to the client layer by injecting the published experiments into javascript in a layout or view using something like: ```haml = javascript_tag(nonce: content_security_policy_nonce) do window.experiments = #{raw ApplicationExperiment.published_experiments.to_json}; ``` The `window.experiments` object can then be used in your client implementation to determine experimental behavior at that layer as well. For instance, we can now access the `window.experiments.pill_color` object to get the variant that was assigned, if the context was excluded, and to use the context key in our client side events. ## Adoption Guide for Organizations ### Phase 1: Foundation (Week 1-2) 1. **Install and configure** the gem 2. **Set up analytics integration** in the initializer 3. **Create a base experiment class** for your organization 4. **Run your first small experiment** (low-risk, high-visibility) ### Phase 2: Team Enablement (Week 3-4) 1. **Document your organization's patterns** (naming, testing, rollout) 2. **Train teams** on experiment lifecycle 3. **Establish experiment review process** (hypothesis → implementation → analysis) 4. **Run 2-3 experiments** across different teams ### Phase 3: Scale (Month 2+) 1. **Integrate with feature flag system** (if applicable) 2. **Build dashboards** for experiment monitoring 3. **Establish data review cadence** (weekly experiment reviews) 4. **Scale to 5-10 concurrent experiments** ### Common Pitfalls to Avoid **❌ Don't: Run experiments without clear success metrics** ```ruby class VagueExperiment < ApplicationExperiment # What are we trying to learn? control { :old_way } candidate { :new_way } end ``` **✅ Do: Document hypothesis and success criteria** ```ruby class CheckoutOptimizationExperiment < ApplicationExperiment # Hypothesis: Showing trust badges increases checkout completion # Success Metric: 5% increase in completion rate # Target: Free trial users # Duration: 2 weeks control { :without_badges } candidate { :with_trust_badges } end ``` **❌ Don't: Let experiments run indefinitely** - Set time bounds for every experiment - Review results at planned intervals - Make a decision: promote winner, revert, or iterate **✅ Do: Build experiment cleanup into your process** - Schedule experiment review meetings - Archive experiment results - Clean up experiment code after conclusion ## Platform Configuration The platform requires initial configuration to integrate with your analytics and infrastructure. **Basic configuration** (in `config/initializers/gitlab_experiment.rb`): ```ruby Gitlab::Experiment.configure do |config| # How experiment events are tracked config.tracking_behavior = lambda do |event_name, **data| YourAnalytics.track( user_id: data[:key], # Anonymous experiment key event: "experiment_#{event_name}", properties: data ) end # How experiments are cached (recommended: Redis) config.cache = Gitlab::Experiment::Cache::RedisHashStore.new( Redis.new(url: ENV['REDIS_URL']), expires_in: 30.days ) # Optional: Prefix all experiment names config.name_prefix = 'mycompany' # Optional: Default rollout strategy config.default_rollout = Gitlab::Experiment::Rollout::Percent.new end ``` See the [complete initializer template](lib/generators/gitlab/experiment/install/templates/initializer.rb.tt) for all configuration options. ### Advanced: Caching Configuration **Why caching matters:** - Ensures consistent user experience across sessions - Improves performance (skip rollout logic after first assignment) - Required for `only_assigned` functionality - Enables context migrations **Cache options:** ```ruby # Option 1: Use Rails cache (simple) Gitlab::Experiment.configure do |config| config.cache = Rails.cache end # Option 2: Use Redis directly (recommended for scale) Gitlab::Experiment.configure do |config| config.cache = Gitlab::Experiment::Cache::RedisHashStore.new( Redis.new(url: ENV['REDIS_URL']), expires_in: 30.days ) end # Option 3: No caching (deterministic rollout strategies only) config.cache = nil ``` The gem includes the [`RedisHashStore`](lib/gitlab/experiment/cache/redis_hash_store.rb) cache store, which is documented in its implementation. **Important:** Caching changes how rollout strategies behave. Once cached, subsequent calls return the cached value regardless of rollout strategy changes. ### Advanced: Custom Rollout Strategies Build custom integrations with your existing infrastructure: **Example: Flipper Integration** ```ruby # We put it in this module namespace so we can get easy resolution when # using `default_rollout :flipper` in our usage later. module Gitlab::Experiment::Rollout class Flipper < Percent def enabled? ::Flipper.enabled?(experiment.name, self) end def flipper_id "Experiment;#{id}" end end end ``` So, Flipper needs something that responds to `flipper_id`, and since our experiment "id" (which is also our context key) is unique and consistent, we're going to give that to Flipper to manage things like percentage of actors etc. You might want to consider something more complex here if you're using things that can be flipper actors in your experiment context. Anyway, now you can use your custom `Flipper` rollout strategy by instantiating it in configuration: ```ruby Gitlab::Experiment.configure do |config| config.default_rollout = Gitlab::Experiment::Rollout::Flipper.new end ``` Or if you don't want to make that change globally, you can use it in specific experiment classes: ```ruby class PillColorExperiment < Gitlab::Experiment # OR ApplicationExperiment # ...registered behaviors default_rollout :flipper, distribution: { control: 26, red: 37, blue: 37 } # optionally specify distribution end ``` Now, enabling or disabling the Flipper feature flag will control if the experiment is enabled or not. If the experiment is enabled, as determined by our custom rollout strategy, the standard resolution logic will be executed, and a variant (or control) will be assigned. ```ruby experiment(:pill_color).enabled? # => false experiment(:pill_color).assigned.name # => "control" # Now we can enable the feature flag to enable the experiment. Flipper.enable(:pill_color) # => true experiment(:pill_color).enabled? # => true experiment(:pill_color).assigned.name # => "red" ``` ### Middleware There are times when you'll need to do link tracking in email templates, or markdown content -- or other places you won't be able to implement tracking. For these cases a middleware layer that can redirect to a given URL while also tracking that the URL was visited has been provided. In Rails this middleware is mounted automatically, with a base path of what's been configured for `mount_at`. If this path is nil, the middleware won't be mounted at all. ```ruby Gitlab::Experiment.configure do |config| config.mount_at = '/experiment' # Only redirect on permitted domains. config.redirect_url_validator = ->(url) { (url = URI.parse(url)) && url.host == 'gitlab.com' } end ``` Once configured to be mounted, the experiment tracking redirect URLs can be generated using the Rails route helpers. ```ruby ex = experiment(:example) # Generating the path/url using the path and url helper. experiment_redirect_path(ex, url: 'https//gitlab.com/docs') # => "/experiment/example:20d69a...?https//gitlab.com/docs" experiment_redirect_url(ex, url: 'https//gitlab.com/docs') # => "https://gitlab.com/experiment/example:20d69a...?https//gitlab.com/docs" # Manually generating a url is a bit less clean, but is possible. "#{Gitlab::Experiment::Configuration.mount_at}/#{ex.to_param}?https//docs.gitlab.com/" ``` ## Testing (rspec support) This gem comes with some rspec helpers and custom matchers. To get the experiment specific rspec support, require the rspec support file: ```ruby require 'gitlab/experiment/rspec' ``` Any file in `spec/experiments` path will automatically get the experiment specific support, but it can also be included in other specs by adding the `:experiment` label: ```ruby describe MyExampleController do context "with my experiment", :experiment do # experiment helpers and matchers will be available here. end end ``` ### Stub helpers You can stub experiment variant resolution using the `stub_experiments` helper. The helper supports multiple formats for flexibility: **Simple hash format:** ```ruby it "stubs experiments using hash format" do stub_experiments(pill_color: :red) experiment(:pill_color) do |e| expect(e).to be_enabled expect(e.assigned.name).to eq('red') end end ``` **Hash format with options:** ```ruby it "stubs experiments with assigned option" do stub_experiments(pill_color: { variant: :red, assigned: true }) experiment(:pill_color) do |e| expect(e).to be_enabled expect(e.assigned.name).to eq('red') end end ``` **Mixed formats (symbols and hashes together):** ```ruby it "stubs multiple experiments with mixed formats" do stub_experiments( pill_color: :red, hippy: { variant: :free_love, assigned: true }, yuppie: :financial_success ) expect(experiment(:pill_color).assigned.name).to eq(:red) expect(experiment(:hippy).assigned.name).to eq(:free_love) expect(experiment(:yuppie).assigned.name).to eq(:financial_success) end ``` **Boolean true (allows rollout strategy to assign):** ```ruby it "stubs experiments while allowing the rollout strategy to assign the variant" do stub_experiments(pill_color: true) # only stubs enabled? experiment(:pill_color) do |e| expect(e).to be_enabled # expect(e.assigned.name).to eq([whatever the rollout strategy assigns]) end end ``` #### Testing `only_assigned` behavior When you use the `assigned: true` option in `stub_experiments`, the `find_variant` method is automatically stubbed to return the specified variant. This allows you to test the `only_assigned` behavior: ```ruby it "tests only_assigned behavior with a cached variant" do stub_experiments(pill_color: { variant: :red, assigned: true }) experiment_instance = experiment(:pill_color, actor: user, only_assigned: true) expect(experiment_instance).not_to be_excluded expect(experiment_instance.run).to eq('red') end it "tests only_assigned behavior without a cached variant" do stub_experiments(pill_color: :red) experiment_instance = experiment(:pill_color, actor: user, only_assigned: true) expect(experiment_instance).to be_excluded expect(experiment_instance.run).to eq('red') end ``` **Note:** The `assigned: true` option only works correctly when caching is disabled. When caching is enabled, `find_variant` will attempt to read from the actual cache store rather than using the stub. In this case, you can populate the cache naturally by running the experiment first to assign and cache a variant before testing with `only_assigned: true`. ### Registered behaviors matcher It's useful to test our registered behaviors, as well as their return values when we implement anything complex in them. The `register_behavior` matcher is useful for this. ```ruby it "tests our registered behaviors" do expect(experiment(:pill_color)).to register_behavior(:control) .with('grey') # with a default return value of "grey" expect(experiment(:pill_color)).to register_behavior(:red) expect(experiment(:pill_color)).to register_behavior(:blue) end ``` ### Exclusion and segmentation matchers You can also easily test your experiment classes using the `exclude`, `segment` matchers. ```ruby let(:excluded) { double(first_name: 'Richard', created_at: Time.current) } let(:segmented) { double(first_name: 'Jeremy', created_at: 3.weeks.ago) } it "tests the exclusion rules" do expect(experiment(:pill_color)).to exclude(actor: excluded) expect(experiment(:pill_color)).not_to exclude(actor: segmented) end it "tests the segmentation rules" do expect(experiment(:pill_color)).to segment(actor: segmented) .into(:red) # into a specific variant expect(experiment(:pill_color)).not_to segment(actor: excluded) end ``` ### Tracking matcher Tracking events is a major aspect of experimentation, and because of this we try to provide a flexible way to ensure your tracking calls are covered. ```ruby before do stub_experiments(pill_color: true) # stub the experiment so tracking is permitted end it "tests that we track an event on a specific instance" do expect(subject = experiment(:pill_color)).to track(:clicked) subject.track(:clicked) end ``` You can use the `on_next_instance` chain method to specify that the tracking call could happen on the next instance of the experiment. This can be useful if you're calling `experiment(:example).track` downstream and don't have access to that instance. Here's a full example of the methods that can be chained onto the `track` matcher: ```ruby it "tests that we track an event with specific details" do expect(experiment(:pill_color)).to track(:clicked, value: 1, property: '_property_') .on_next_instance # any time in the future .with_context(foo: :bar) # with the expected context .for(:red) # and assigned the correct variant experiment(:pill_color, :red, foo: :bar).track(:clicked, value: 1, property: '_property_') end ``` ## Tracking, anonymity and GDPR We generally try not to track things like user identifying values in our experimentation. What we can and do track is the "experiment experience" (a.k.a. the context key). We generate this key from the context passed to the experiment. This allows creating funnels without exposing any user information. This library attempts to be non-user-centric, in that a context can contain things like a user or a project. If you only include a user, that user would get the same experience across every project they view. If you only include the project, every user who views that project would get the same experience. Each of these approaches could be desirable given the objectives of your experiment. ## Development After cloning the repo, run `bundle install` to install dependencies. ## Running tests The test suite requires Redis to be running. [Install](https://redis.io/docs/latest/operate/oss_and_stack/install/archive/install-redis/) and start Redis (`redis-server`) before running tests. Once Redis is running, execute the tests: `bundle exec rake` You can also run `bundle exec pry` for an interactive prompt that will allow you to experiment. ## Contributing Bug reports and merge requests are welcome on GitLab at https://gitlab.com/gitlab-org/ruby/gems/gitlab-experiment. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct. Make sure to include a changelog entry in your commit message and read the [changelog entries section](https://docs.gitlab.com/ee/development/changelog.html). ## Release process Please refer to the [Release Process](docs/release_process.md). ## License The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT). ## Code of conduct Everyone interacting in the `Gitlab::Experiment` project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](CODE_OF_CONDUCT.md).