Turn up the Heat! A Tutorial for the MAGMA R-package.

Julian Urban, Markus D. Feuchter, Vsevolod Scherrer, Moritz L. Breit, and Franzis Preckel

Summary

Balancing quasi-experimental field research for effects of covariates is fundamental for drawing causal inference. Propensity Score Matching addresses this issue but current techniques are restricted to binary treatment variables. Moreover, they offer a variety of solutions without providing a comprehensive framework on choosing the best model. The MAGMA R-package addresses these restrictions by providing nearest neighbor matching for up to four groups using the Many-group-Matching (MAGMA) algorithm. It also includes the option to match data of a 2x2 design. In addition, MAGMA provides a framework for evaluating the post-matching balance. This vignette is a tutorial on MAGMA. It explains and illustrates the main MAGMA functions with an included simulated dataset. In total, this vignette includes a two-group, a three-group, and a 2x2-design, which is equivalent to a four-group example. These three grouping scenarios are illustrated in a standard matching process, as well as with exact matching, where only cases sharing the same value on a variable can be matched.

Introduction

Lack of experimental control is an issue in field research, observational, and non-randomized studies. A popular method to control for the effects of covariates post-hoc is propensity score matching (PSM; Rosenbaum & Rubin, 1983, 1985). By balancing out the effects of covariates, PSM approxiamtes causal inference. Since causal inference is fundamental in science, PSM is used in many scientific fields such as in organizational (e.g., Li, 2013), educational (e.g., Powell et al., 2020), and social science (e.g., Thoemmes & Kim, 2011) as well as in medical research (e.g., Austin, 2014).

PSM builds statistical pairs of individuals, for example from a treatment group and a control group, based on the distance on a specific variable. Propensity scores are often used in representation of said distance (for a review, see Stuart, 2010). They express the conditional probability of belonging to a group (e.g., the treatment group) given a set of covariates (Rosenbaum & Rubin, 1985). Although a plethora of matching techniques exist, a commonly applied method is nearest neighbor matching (Austin, 2014; Jacovidis, 2017). By matching a treated individual to its non-treated nearest neighbor, PSM controls for the effects of covariates.

PSM’s potential is nonetheless limited due to three reasons. First, current techniques are restricted to binary treatment variables, and hence, to two-group designs (for first approaches, see Imai & van Dyk, 2004; McCaffrey et al., 2013). Second, researchers must successively extract and compare several arbitrary matching solutions by hand, instead of having access to a comprehensive, unambiguous overview of possible matching solutions. Third, there is no comprehensive framework for evaluating the quality of matching solutions, especially when multiple groups are involved. We addressed these issues with our MAGMA package. For an extensive overview of MAGMA and its balance framework, see Feuchter at al. (2022).

The aims of this vignette are to provide a tutorial for the MAGMA R-package. This includes examples for all currently available group scenarios, namely two, three, and four groups, as well as 2x2 designs. We conducted these examples for standard and exact matching. Moreover, this tutorial contains examples on how to report matching relevant statistics. Lastly, we demonstrate how to compute and report balance of individual PSM solutions, using our newly developed framework.

The MAGMA Simulated Dataset

We used a simulated data set for all examples. This simulated dataset contains 800 cases and 14 variables. It is available after installing the MAGMA package. Associations between the variables were modeled to increase the usability and comprehensibility of this tutorial, and are, therefore, not representative of specific real-life phenomena. For information about this simulated dataset, use ?MAGMA_sim_data. Below is a brief look at the data and its structure. Note that the last three variables are propensity scores that were estimated using the twang package (Ridgeway et al., 2015). For a detailed overview and tutorial on twang, see (Ridgeway et al., 2015). Propensity scores estimated via twang serve as distance indicators for all subsequent examples.

For the two group example, we use the propensity score ps_gifted. This score expresses the conditional probability of receiving gifted_support. It was computed using the ps function of twang. The propensity scores ps_tar (three group example) and ps_2x2 (four group / 2x2 design example) are the result of the mnps function of twang. They express the conditional probability for the groups teacher_rated_ability (below average, average, above average) and the combination of receiving gifted_support and participating in afternoon enrichment respectively.

str(MAGMA_sim_data)
#> 'data.frame':    800 obs. of  17 variables:
#>  $ ID                    : int  1 2 3 4 5 6 7 8 9 10 ...
#>  $ gender                : int  1 0 1 1 1 0 0 0 1 0 ...
#>  $ gifted_support        : int  1 1 1 1 1 1 0 1 0 1 ...
#>  $ teacher_ability_rating: int  3 2 2 1 3 3 2 2 1 1 ...
#>  $ enrichment            : int  0 1 0 1 0 0 0 0 0 1 ...
#>  $ parents_academic      : int  1 1 1 0 1 0 0 1 0 0 ...
#>  $ GPA_school            : num  4.73 4.41 3.23 2.88 2.98 ...
#>  $ IQ_score              : num  111.6 97.8 120.1 97.9 74.4 ...
#>  $ Motivation            : num  5.36 4.41 3.57 4.34 3.54 ...
#>  $ college_GPA           : num  4.01 3.41 2.73 4.02 4.09 ...
#>  $ support_enrichment    : int  3 4 3 4 3 3 1 3 1 4 ...
#>  $ ps_tar                : num  0.0657 0.1085 0.1899 0.4261 0.4782 ...
#>  $ ps_2x2                : num  0.3 0.32 0.21 0.237 0.571 ...
#>  $ ps_gifted             : num  0.431 0.364 0.626 0.353 0.187 ...
#>  $ step_gifted           : num  56 140 249 117 52 231 266 304 NA 29 ...
#>  $ weight_gifted         : num  1 1 1 1 1 1 1 1 NA 1 ...
#>  $ distance_gifted       : num  4.55e-08 3.88e-06 5.06e-01 1.43e-06 3.27e-08 ...

Two-Group Example

This is a fictitious two-group example. The independent variable (i.e., the treatment) is gifted_support. We are interested in how receiving giftedness support affects college GPA. However, other variables, such as intelligence, previous achievement, and motivation are associated with both giftedness support and college GPA. This is because gifted students were chosen to participate in this support program based on their previous achievement or intelligence test results. Therefore, we aim to reduce differences on these covariates between the two groups by PSM. For this example, the covariates are high school GPA, intelligence, motivation, gender, and whether at least one parent has an academic background. Note that MAGMA can only consider metric and binary variables for balance estimation.

covariates_gifted <- c("GPA_school",
                       "IQ_score",
                       "Motivation",
                       "parents_academic",
                       "gender")

Defining all covariates beforehand (see above)nfacilitates estimations of descriptive statistics, standardized mean differences, and hence, the initial unbalance across all cases of our dataset. To do this we can use the functions MAGMA_desc and initial_unbalance. The first function – MAGMA_desc – computes sample size, mean, and standard deviation of all covariates for the overall sample and the two groups. Moreover, it estimates the standardized mean differences using the pooled standard deviation (i.e., Cohen’s d) between the two groups. This function needs the dataset (MAGMA_sim_data), the grouping variable (gifted_support), and all binary and metric covariates of interest (covariates_gifted). The function results in a table that contains these statistics for each covariate.

The second function – initial_unbalance - estimates the four balance criteria of our balance framework. These criteria are (1) Pillai’s Trace, which estimates group differences across all variables, (2) d-ratio, which expresses the percentage of how many pairwise effects are negligible, (3) mean g, which specifies the average pariwise effect using meta-analytical techniques, and (4) adjusted d-ratio, an adjusted version of (2) that respects that effects are point estimates. For more information, see Feuchter et al. (2022). The function initial_unbalance indicates the unbalance in the sample before matching. To this end, it is necessary to define the dataset (MAGMA_sim_data), the grouping variable (gifted_support), and all binary and metric covariates of interest (covariates_gifted). This results in a vector of length four - the four balance criteria.

# Estimate overall and group specific descriptive statistics and Cohen’s d 
descs_gifted_pre <- MAGMA_desc(Data = MAGMA_sim_data,
                               group = "gifted_support",
                               covariates = covariates_gifted,
                               filename = "stats_gifted_pre.docx")
descs_gifted_pre %>%
  purrr::set_names(c("Overall N", "Overall Mean", "Overall SD",
                     "No Support N", "No Support Mean", "No Support SD",
                     "Support N", "Support Mean", "Support SD",
                     "d"))
#>                  Overall N Overall Mean Overall SD No Support N No Support Mean
#> gifted_support         800         0.38       0.49          495            0.00
#> GPA_school             800         3.29       0.93          495            3.13
#> IQ_score               800       100.51      15.00          495           96.32
#> Motivation             800         3.82       0.89          495            3.81
#> parents_academic       800         0.40       0.49          495            0.35
#> gender                 800         0.50       0.50          495            0.50
#>                  No Support SD Support N Support Mean Support SD     d
#> gifted_support            0.00       305         1.00       0.00  -Inf
#> GPA_school                0.91       305         3.56       0.89 -0.48
#> IQ_score                 14.20       305       107.30      13.75 -0.78
#> Motivation                0.90       305         3.83       0.88 -0.02
#> parents_academic          0.48       305         0.48       0.50 -0.27
#> gender                    0.50       305         0.49       0.50  0.02

# Estimating the four balance criteria
unbalance_gifted <- initial_unbalance(Data = MAGMA_sim_data,
                                      group = "gifted_support",
                                      covariates = covariates_gifted)
#> Mean g was computed using random effects meta-analysis with metafor.
unbalance_gifted
#>           Pillai's Trace d-ratio Mean g adj. d-ratio
#> Unbalance           0.13     0.4   0.31         0.44

As may be expected from the descriptive statistics, the cases that received giftedness support differed substantially from the cases that did not receive giftedness support on at least some covariates. Before we start matching, we check the possibility of eligible matches. Therefore, we estimate and plot the common support in propensity score via the kernel-density (Pastore et al., 2022) and the function density_overlap. This function requires the dataset (MAGMA_sim_data), the variable for which the overlap should be computed (ps_gifted), and the grouping variable (gifted_support). The remaining arguments of the functions define aesthetics of the plot, namely the name of the variable (variable_name), the name of the group (group_name), and the labels of the respective groups (group_labels). The function returns a density plot and a value expressing the overlapping area.

# Density overlap in propensity scores for gifted before matching
Density_overlap(Data = MAGMA_sim_data,
                variable = "ps_gifted", 
                group = "gifted_support",
                variable_name = "Propensity Score",
                group_labels = c("No Support", "Support"),
                group_name = "Gifted Support")

#>        OV 
#> 0.6162483

As we can see, there is substantial common support between the two groups. After these initial checks, we can conduct the matching. This process is based on the respective propensity score, ps_gifted. We first address standard matching before introducing a case of exact matching.

Standard Matching

For standard nearest neighbor matching we use the MAGMA function. This function has four arguments. The first argument specifies the dataset (MAGMA_sim_data). The second argument specifies the name of the grouping variable as a character ("gifted_support"). The third argument specifies the name of the distance variable (in our case the propensity score) as a character ("ps_gifted"). The fourth argument is optional. Since MAGMA has some computational load, it includes the option of parallel computation. By default, only one CPU core is used. The usage of more cores may reduce the computation time of matching. While this reduction is rarely necessary in two-group matching, it becomes more helpful for many group matching. If parallel computation is desired, an integer larger than one can be specified for the cores argument. If the specified value exceeds the number of available cores, MAGMA will set this argument to the maximum number of available cores.

Applying this function results in a dataset that extends the original dataset by three variables. The first, weight, indicates whether this case was matched or not. The second, step, indicates the iterative step in which a case was matched. More specifically, the cases with the smallest Mahalanobis distance are matched first and, thus, receive 1 in the step variable. The matched cases with the second smallest distance receive the 2 and so on. The step variable is essential to compare and extract different matching solutions. The third variable, distance indicates the propensity score distance of the matched cases.

# Conducting matching for gifted support
MAGMA_sim_data_gifted <- MAGMA(Data = MAGMA_sim_data,
                               group = "gifted_support",
                               dist = "ps_gifted",
                               cores = 2)
#> 
#>  Input correctly identified.
#>  Distance computation finished. Starting matching.
#>  Matching complete!
str(MAGMA_sim_data_gifted)
#> 'data.frame':    800 obs. of  20 variables:
#>  $ ID                    : int  1 2 3 4 5 6 7 8 9 10 ...
#>  $ gender                : int  1 0 1 1 1 0 0 0 1 0 ...
#>  $ gifted_support        : int  1 1 1 1 1 1 0 1 0 1 ...
#>  $ teacher_ability_rating: int  3 2 2 1 3 3 2 2 1 1 ...
#>  $ enrichment            : int  0 1 0 1 0 0 0 0 0 1 ...
#>  $ parents_academic      : int  1 1 1 0 1 0 0 1 0 0 ...
#>  $ GPA_school            : num  4.73 4.41 3.23 2.88 2.98 ...
#>  $ IQ_score              : num  111.6 97.8 120.1 97.9 74.4 ...
#>  $ Motivation            : num  5.36 4.41 3.57 4.34 3.54 ...
#>  $ college_GPA           : num  4.01 3.41 2.73 4.02 4.09 ...
#>  $ support_enrichment    : int  3 4 3 4 3 3 1 3 1 4 ...
#>  $ ps_tar                : num  0.0657 0.1085 0.1899 0.4261 0.4782 ...
#>  $ ps_2x2                : num  0.3 0.32 0.21 0.237 0.571 ...
#>  $ ps_gifted             : num  0.431 0.364 0.626 0.353 0.187 ...
#>  $ step_gifted           : num  56 140 249 117 52 231 266 304 NA 29 ...
#>  $ weight_gifted         : num  1 1 1 1 1 1 1 1 NA 1 ...
#>  $ distance_gifted       : num  4.55e-08 3.88e-06 5.06e-01 1.43e-06 3.27e-08 ...
#>  $ step                  : num  56 140 249 117 52 231 266 304 NA 29 ...
#>  $ weight                : num  1 1 1 1 1 1 1 1 NA 1 ...
#>  $ distance              : num  4.55e-08 3.88e-06 5.06e-01 1.43e-06 3.27e-08 ...

Balance Estimation

MAGMA matches cases iteratively until all cases of the smallest group are matched. Thus, it is necessary to find the optimal sample size after matching. To do so, Balance_MAGMA estimates the four balance criteria for each possible sample size. Note that MAGMA uses a lower limit of at least n = 20 per group. To use the function, you need to specify at least your (matched) dataset (MAGMA_sim_data_gifted), the name of the grouping variable as a character ("gifted_support"), and your covariates as a character vector (covariates_gifted). Moreover, Balance_MAGMA needs the name of the variable that indicates the iterative step, in which a case was matched. Since the MAGMA function names this variable "step" this name is the default of the step argument.

Balance_MAGMA returns a list of length four. In the case of a univariate grouping variable, this list contains three numeric vectors and one list of length two. The three vectors comprise Pillai’s Trace, mean g, and adjusted d-ratio over the iteratively increased sample size. The list of length two includes one numeric vector, d-ratio, and one matrix. The matrix gives all pairwise d’s for all covariates. Both d-ratio and pairwise d’s are also estimated iteratively over the increasing sample sizes. The example below estimates the balance for the matching solution and shows the balance criteria and pairwise effects for a group sample size of n = 100.

This function is augmented by the function Balance_extract. This function extracts the balance criteria (effects = FALSE) or the pairwise effects (effects = TRUE) for a specific sample size per group (samplesize) from the results of Balance_MAGMA. It returns a vector containing the balance criteria or the pairwise effects respectively.

# Estimating the four balance criteria iteratively over possible sample sizes
Balance_gifted <- Balance_MAGMA(Data = MAGMA_sim_data_gifted,
                                group = "gifted_support",
                                covariates = covariates_gifted,
                                step = "step") 
#> 
#>  Start estimating Pillai's Trace.
#>  Pillai's Trace finished. Starting to compute d-ratio.
#>  d-ratio finished. Starting to compute mean-g. 
#> 
#>  Mean g was computed using random effects meta-analysis with metafor.
#>  Mean g finished. Starting to compute adjusted d-ratio.
#>  Adjusted d-ratio finished.
#>  Balance estimation finished.

# Extracting balance criteria for 100 cases per group
Balance_100_criteria <- Balance_extract(Balance = Balance_gifted,
                                        samplesize = 100,
                                        effects = FALSE)
Balance_100_criteria
#> Pillai's Trace        d-ratio         mean g   adj. d-ratio 
#>           0.02           1.00           0.10           0.74

# Extracting pairwise effects for 100 cases per group
Balance_100_effects <- Balance_extract(Balance = Balance_gifted,
                                       samplesize = 100,
                                       effects = TRUE)
Balance_100_effects
#>       GPA_school         IQ_score       Motivation parents_academic 
#>             0.07             0.20            -0.01            -0.12 
#>           gender 
#>             0.10

Balance Evaluation

After balance estimation, the “optimal” sample size to extract remains veiled. Which sample size is “optimal” strongly depends on the specific research aim. For example, the sample size requirements of some statistical procedures may narrow the selection, as may the need to eliminate a specific pairwise effect. We therefore encourage to adapt the selection of optimal sample size to the specific research aim.

A first indicator of such an optimal sample size is the trend of the balance criteria across the iteratively increased sample size. The function Plot_MAGMA visualizes this trend. The one mandatory argument, Balance, specifies the object of the balance estimation results. Note that this object must be the result of a Balance_MAGMA function. Per default, Plot_MAGMA creates plots for all four balance criteria. However, the criterion argument enables the plotting of specific balance criteria. Note that running this code will result in a warning. This warning is a result of the specified minimum sample size for balance estimation and can be ignored.

The trend of these plots depends strongly on the data. Nonetheless, they can be a good indicator of breakpoints in the sample size-balance associations. Plot_MAGMA uses fixed limits for the y-axis. This ensures comparability of these plots and prevents misinterpretation due to a restricted y-axis range. Both d-ratio and adjusted d-ratio vary between 0 and 1. Although Pillai’s Trace has theoretically the same range, we set the limits to 0 and 0.5. Larger values are unlikely and indicate a poor balance. A stricter limit might restrict a comprehensive trend evaluation. For mean g we set the limits to 0 and 1. The lower limit 0 corresponds to the theoretical lower limit. The rationale for the upper limit is comparable to that of Pillai’s Trace.

# Plotting balance trend over sample size
Plot_MAGMA(Balance = Balance_gifted,
           criterion = c("Pillai", "d_ratio", "mean_g", "Adj_d_ratio"))

Besides the trend over sample size, the “optimized” values for each criterion are of interest. We interpret a criterions as optimized as it takes its lowest (Pillai’s Trace and mean g) or highest (d-ratio and adjusted d-ratio) value and having the highest sample size with this value. The function Table_MAGMA extracts these values. With the optional argument filename the function can return these values in a table in an extra file. More specifically it returns a 4x5 table. In each of the four rows one balance criterion has its “optimal” value. The five columns present the sample size per group for the matching solution where the respective criterion is optimal, and the four balance criteria values for this sample size per group. In addition to creating a file with a table, the table is printed to the console, too.

Similar to Plot_MAGMA the first argument, Balance, of Table_MAGMA specifies the object of the balance estimation results. The second argument, filename, specifies the desired name of the extra file containing the table. This argument is a character.

Table_MAGMA(Balance = Balance_gifted,
            filename = "Balance_gifted.docx")
#> Balance table successfully created!
#> # A tibble: 4 × 6
#>   Criterion_optimized Pillai_Trace d_ratio mean_g adjusted_d_ratio n_per_group
#>   <chr>                      <dbl>   <dbl>  <dbl>            <dbl>       <int>
#> 1 Best Pillai                 0          1   0.03             0.97         276
#> 2 Best mean g                 0          1   0.03             0.97         276
#> 3 Best adj. d-ratio           0          1   0.03             0.97         276
#> 4 Best d-ratio                0.01       1   0.06             0.88         296

Post-Matching Statistics

After the decision for a final matching model, we suggest reporting the descriptive statistics and pairwise effects of the selected model. To do this, we can use the same function as we used for reporting pre-matching statistics (MAGMA_desc). Additional to the parameters we specified above (Data, covariates, and group), we need to name the step variable (step_var) and the desired sample size per group (step_num). As above, this function results in a table with descriptive statistics and pairwise effects (Cohen’s d) of all covariates.

# Computing descriptive statistics and pairwise effects for 100 cases per group
descs_gifted_post <- MAGMA_desc(Data = MAGMA_sim_data_gifted,
                                group = "gifted_support",
                                covariates = covariates_gifted,
                                covariates_ordinal = "teacher_ability_rating",
                               covariates_nominal = "enrichment",
                                step_num = 100,
                                step_var = "step",
                                filename = "stats_gifted_post.docx")

# Displaying the table with defined colum names
descs_gifted_post %>%
  purrr::set_names(c("Overall N", "Overall Mean", "Overall SD",
                     "No Support N", "No Support Mean", "No Support SD",
                     "Support N", "Support Mean", "Support SD",
                     "d"))
#>                        Overall N Overall Mean Overall SD No Support N
#> gifted_support               200         0.50       0.50          100
#> GPA_school                   200         3.38       0.76          100
#> IQ_score                     200       102.82      12.50          100
#> Motivation                   200         3.87       0.84          100
#> parents_academic             200         0.42       0.49          100
#> gender                       200         0.47       0.50          100
#> teacher_ability_rating       200         2.00       2.00          100
#> enrichment                   200         0.00       2.00          100
#>                        No Support Mean No Support SD Support N Support Mean
#> gifted_support                    0.00          0.00       100         1.00
#> GPA_school                        3.41          0.76       100         3.36
#> IQ_score                        104.05         12.13       100       101.59
#> Motivation                        3.87          0.84       100         3.88
#> parents_academic                  0.39          0.49       100         0.45
#> gender                            0.49          0.50       100         0.44
#> teacher_ability_rating            2.00          2.00       100         2.00
#> enrichment                        0.00          2.00       100         0.00
#>                        Support SD     d
#> gifted_support               0.00  -Inf
#> GPA_school                   0.77  0.07
#> IQ_score                    12.80  0.20
#> Motivation                   0.84 -0.01
#> parents_academic             0.50 -0.12
#> gender                       0.50  0.10
#> teacher_ability_rating       1.00 -0.13
#> enrichment                   2.00  0.08

Exact Matching

Unlike in standard matching, only cases sharing the same value on a variable can be matched in exact matching. In this example, the variable for exact matching is enrichment. This means that cases that participated in afternoon enrichment can only be matched with other cases that participated. Consequentially, non-participants can only be matched with other non-participants. Reasons for choosing exact matching instead of standard matching may be, for instance, a nested data structure. For example, students may come from different schools and only students from the same school should be matched together. Except for this preselection of possible matches, the process of matching stays the same as in the previous example. The main difference is that we need to use the function MAGMA_exact instead of MAGMA to match the data. This function has an additional argument, where the name of the exact variable must be defined as a character ("enrichment"). Note that the initial unbalance, descriptive statistics, and density overlap are the same as for standard matching.

MAGMA_sim_data_gifted_exact <- MAGMA_exact(Data = MAGMA_sim_data,
                                           group = "gifted_support",
                                           dist = "ps_gifted",
                                           exact = "enrichment",
                                           cores = 2)
#> 
#>  Input correctly identified!
#>  Distance computation finished. Starting matching
#>  Matching complete!
str(MAGMA_sim_data_gifted_exact)
#> 'data.frame':    800 obs. of  20 variables:
#>  $ ID                    : int  1 2 3 4 5 6 7 8 9 10 ...
#>  $ gender                : int  1 0 1 1 1 0 0 0 1 0 ...
#>  $ gifted_support        : int  1 1 1 1 1 1 0 1 0 1 ...
#>  $ teacher_ability_rating: int  3 2 2 1 3 3 2 2 1 1 ...
#>  $ enrichment            : int  0 1 0 1 0 0 0 0 0 1 ...
#>  $ parents_academic      : int  1 1 1 0 1 0 0 1 0 0 ...
#>  $ GPA_school            : num  4.73 4.41 3.23 2.88 2.98 ...
#>  $ IQ_score              : num  111.6 97.8 120.1 97.9 74.4 ...
#>  $ Motivation            : num  5.36 4.41 3.57 4.34 3.54 ...
#>  $ college_GPA           : num  4.01 3.41 2.73 4.02 4.09 ...
#>  $ support_enrichment    : int  3 4 3 4 3 3 1 3 1 4 ...
#>  $ ps_tar                : num  0.0657 0.1085 0.1899 0.4261 0.4782 ...
#>  $ ps_2x2                : num  0.3 0.32 0.21 0.237 0.571 ...
#>  $ ps_gifted             : num  0.431 0.364 0.626 0.353 0.187 ...
#>  $ step_gifted           : num  56 140 249 117 52 231 266 304 NA 29 ...
#>  $ weight_gifted         : num  1 1 1 1 1 1 1 1 NA 1 ...
#>  $ distance_gifted       : num  4.55e-08 3.88e-06 5.06e-01 1.43e-06 3.27e-08 ...
#>  $ step                  : num  156 136 125 167 32 230 265 303 NA 18 ...
#>  $ weight                : num  1 1 1 1 1 1 1 1 NA 1 ...
#>  $ distance              : num  3.92e-05 1.91e-05 1.10e-05 9.41e-05 3.27e-08 ...

Balance Estimation and Visualization

The steps of balance estimation, visualization, and post matching statistics are the same as for standard matching. Therefore, we only shortly summarize these steps in this example. The only change we need to consider is the different data frame that contains the result of the exact matching solution MAGMA_sim_data_gifted_exact.

Balance_gifted_exact <- Balance_MAGMA(Data = MAGMA_sim_data_gifted_exact,
                                      group = "gifted_support",
                                      covariates = covariates_gifted,
                                      step = "step") 
#> 
#>  Start estimating Pillai's Trace.
#>  Pillai's Trace finished. Starting to compute d-ratio.
#>  d-ratio finished. Starting to compute mean-g. 
#> 
#>  Mean g was computed using random effects meta-analysis with metafor.
#>  Mean g finished. Starting to compute adjusted d-ratio.
#>  Adjusted d-ratio finished.
#>  Balance estimation finished.

# Extracting balance criteria for 100 cases per group
Balance_100_criteria_exact <- Balance_extract(Balance = Balance_gifted_exact,
                                        samplesize = 100,
                                        effects = FALSE)
Balance_100_criteria_exact
#> Pillai's Trace        d-ratio         mean g   adj. d-ratio 
#>           0.02           1.00           0.12           0.70

# Extracting pairwise effects for 100 cases per group
Balance_100_effects_exact <- Balance_extract(Balance = Balance_gifted_exact,
                                       samplesize = 100,
                                       effects = TRUE)
Balance_100_effects_exact
#>       GPA_school         IQ_score       Motivation parents_academic 
#>             0.16             0.17             0.05            -0.16 
#>           gender 
#>             0.06

# Plotting trend over increasing sample size
Plot_MAGMA(Balance = Balance_gifted_exact,
           criterion = c("Pillai", "d_ratio", "mean_g", "Adj_d_ratio")) 


# Creating table
Table_MAGMA(Balance = Balance_gifted_exact,
            filename = "Balance_gifted_exact.docx")
#> Balance table successfully created!
#> # A tibble: 4 × 6
#>   Criterion_optimized Pillai_Trace d_ratio mean_g adjusted_d_ratio n_per_group
#>   <chr>                      <dbl>   <dbl>  <dbl>            <dbl>       <int>
#> 1 Best mean g                 0          1   0.03             0.97         274
#> 2 Best Pillai                 0          1   0.03             0.97         279
#> 3 Best adj. d-ratio           0          1   0.03             0.97         279
#> 4 Best d-ratio                0.01       1   0.07             0.88         296

# Computing descriptive statistics and pairwise effects for 100 cases per group
descs_gifted_post_exact <- MAGMA_desc(Data = MAGMA_sim_data_gifted_exact,
                                      group = "gifted_support",
                                      covariates = covariates_gifted,
                                      step_num = 100,
                                      step_var = "step",
                                      filename = "stats_gifted_post_exact.docx")

# Displaying the table with defined colum names
descs_gifted_post_exact %>%
  purrr::set_names(c("Overall N", "Overall Mean", "Overall SD",
                     "No Support N", "No Support Mean", "No Support SD",
                     "Support N", "Support Mean", "Support SD",
                     "d"))
#>                  Overall N Overall Mean Overall SD No Support N No Support Mean
#> gifted_support         200         0.50       0.50          100            0.00
#> GPA_school             200         3.39       0.79          100            3.45
#> IQ_score               200       101.70      13.25          100          102.79
#> Motivation             200         3.88       0.88          100            3.90
#> parents_academic       200         0.38       0.49          100            0.34
#> gender                 200         0.47       0.50          100            0.48
#>                  No Support SD Support N Support Mean Support SD     d
#> gifted_support            0.00       100         1.00       0.00  -Inf
#> GPA_school                0.79       100         3.32       0.80  0.16
#> IQ_score                 13.24       100       100.61      13.23  0.16
#> Motivation                0.92       100         3.86       0.85  0.05
#> parents_academic          0.48       100         0.42       0.50 -0.16
#> gender                    0.50       100         0.45       0.50  0.06

Three-Group Example

This is a three-group example. The independent variable (i.e., the treatment) is teacher_ability_rating. We are interested in how teacher-rated student ability, which ranged from below average (BA) to average (A) and above average (AA), affects the college GPA. While the process of matching and balance evaluation is the same as in the two-group example, the grouping variable (teacher_ability_rating), and consequently the distance variable/propensity score changes (ps_tar). Moreover, in this example we change the covariates by substituting gender with gifted_support.

covariates_tar <- c("GPA_school",
                    "IQ_score",
                    "Motivation",
                    "parents_academic",
                    "gifted_support")

Due to these changes in grouping variable and covariates, we need to re-estimate the initial unbalance, descriptive statistics, and area of common support for teacher-rated ability. Note that due to having three groups, the estimation of mean g slightly changes. Instead of estimating the mean effect using metafor (Viechtbauer, 2010) we use robumeta (Fischer et al., 2023). However, this only affects the backend computation, while the code and the display of results are not affected. For more details, see Feuchter et al. (2022).

# Computing descriptive statistics and all pairwise effects for three groups
descs_tar_pre <- MAGMA_desc(Data = MAGMA_sim_data,
                            group = "teacher_ability_rating",
                            covariates = covariates_tar,
                            filename = "stats_tar_pre.docx")

descs_tar_pre %>%
  purrr::set_names(c("Overall N", "Overall Mean", "Overall SD",
                     "BA N", "BA Support Mean", "BA Support SD",
                     "A N", "A Mean", "A SD",
                     "AA N", "AA Mean", "AA SD",
                     "d BA-A", "d BA-AA", "d A-AA"))
#>                        Overall N Overall Mean Overall SD BA N BA Support Mean
#> teacher_ability_rating       800         2.02       0.80  243            1.00
#> GPA_school                   800         3.29       0.93  243            2.76
#> IQ_score                     800       100.51      15.00  243           92.93
#> Motivation                   800         3.82       0.89  243            3.77
#> parents_academic             800         0.40       0.49  243            0.39
#> gifted_support               800         0.38       0.49  243            0.35
#>                        BA Support SD A N A Mean  A SD AA N AA Mean AA SD d BA-A
#> teacher_ability_rating          0.00 294   2.00  0.00  263    3.00  0.00   -Inf
#> GPA_school                      0.82 294   3.33  0.85  263    3.73  0.86  -0.68
#> IQ_score                       14.63 294 101.20 12.84  263  106.73 14.54  -0.60
#> Motivation                      0.88 294   3.72  0.88  263    3.98  0.90   0.06
#> parents_academic                0.49 294   0.38  0.49  263    0.42  0.49   0.02
#> gifted_support                  0.48 294   0.40  0.49  263    0.39  0.49  -0.10
#>                        d BA-AA d A-AA
#> teacher_ability_rating    -Inf   -Inf
#> GPA_school               -1.15  -0.47
#> IQ_score                 -0.95  -0.40
#> Motivation               -0.24  -0.29
#> parents_academic         -0.06  -0.08
#> gifted_support           -0.08   0.02

# Estimating and printing initial unbalance for teacher rated ability
unbalance_tar <- initial_unbalance(Data = MAGMA_sim_data,
                                   group = "teacher_ability_rating",
                                   covariates = covariates_tar)
#> Mean g was computed using robust variance meta-analysis with robumeta.
unbalance_tar
#>           Pillai's Trace d-ratio Mean g adj. d-ratio
#> Unbalance           0.24    0.47   0.35         0.47

# Estimating and plotting density overlap in teacher rated ability propensity scores
# Returns vector for each pairwise overlap
Density_overlap(Data = MAGMA_sim_data,
                variable = "ps_tar", 
                group = "teacher_ability_rating",
                variable_name = "Propensity Score",
                group_labels = c("Below Average", "Average", "Above Average"),
                group_name = "Teacher Rated Ability")

#>                                  OV       OVPairs.Below Average-Average 
#>                           0.7387731                           0.5164718 
#> OVPairs.Below Average-Above Average       OVPairs.Average-Above Average 
#>                           0.3483660                           0.7658709

Standard Matching

After this check of initial unbalance, descriptive statistics, and common support, we can conduct the standard matching. As mentioned above, we need to adapt the grouping variable and the distance variable/propensity score for this example.

MAGMA_sim_data_tar <- MAGMA(Data = MAGMA_sim_data,
                            group = "teacher_ability_rating",
                            dist = "ps_tar",
                            cores = 2)
str(MAGMA_sim_data_tar)
#> 'data.frame':    800 obs. of  17 variables:
#>  $ ID                    : int  1 2 3 4 5 6 7 8 9 10 ...
#>  $ gender                : int  1 0 1 1 1 0 0 0 1 0 ...
#>  $ gifted_support        : int  1 1 1 1 1 1 0 1 0 1 ...
#>  $ teacher_ability_rating: int  3 2 2 1 3 3 2 2 1 1 ...
#>  $ enrichment            : int  0 1 0 1 0 0 0 0 0 1 ...
#>  $ parents_academic      : int  1 1 1 0 1 0 0 1 0 0 ...
#>  $ GPA_school            : num  4.73 4.41 3.23 2.88 2.98 ...
#>  $ IQ_score              : num  111.6 97.8 120.1 97.9 74.4 ...
#>  $ Motivation            : num  5.36 4.41 3.57 4.34 3.54 ...
#>  $ college_GPA           : num  4.01 3.41 2.73 4.02 4.09 ...
#>  $ support_enrichment    : int  3 4 3 4 3 3 1 3 1 4 ...
#>  $ ps_tar                : num  0.0657 0.1085 0.1899 0.4261 0.4782 ...
#>  $ ps_2x2                : num  0.3 0.32 0.21 0.237 0.571 ...
#>  $ ps_gifted             : num  0.431 0.364 0.626 0.353 0.187 ...
#>  $ step                  : num  222 237 28 106 66 62 182 NA 236 38 ...
#>  $ weight                : num  1 1 1 1 1 1 1 NA 1 1 ...
#>  $ distance              : num  2.53 3.35 2.96e-06 1.88e-01 5.22e-05 ...

Balance Estimation and Visualization

We only need to change the specific object names, covariates, grouping variable, and the filename for the table. Otherwise, the process of balance estimation and visualization remains the same.

Balance_tar <- Balance_MAGMA(Data = MAGMA_sim_data_tar,
                             group = "teacher_ability_rating",
                             covariates = covariates_tar,
                             step = "step")
# Balance criteria for 100 cases per group
Balance_100_tar_criteria <- Balance_extract(Balance = Balance_tar,
                                            samplesize = 100,
                                            effects = FALSE)
Balance_100_tar_criteria
#> Pillai's Trace        d-ratio         mean g   adj. d-ratio 
#>           0.03           0.67           0.14           0.64

# Extracting pairwise effects for 100 cases per group
Balance_100_tar_effects <- Balance_extract(Balance = Balance_tar,
                                           samplesize = 100,
                                           effects = TRUE)
Balance_100_tar_effects
#>       GPA_school_1_2         IQ_score_1_2       Motivation_1_2 
#>                 0.32                 0.22                 0.20 
#> parents_academic_1_2   gifted_support_1_2       GPA_school_1_3 
#>                 0.04                 0.00                 0.37 
#>         IQ_score_1_3       Motivation_1_3 parents_academic_1_3 
#>                 0.23                 0.07                 0.14 
#>   gifted_support_1_3       GPA_school_2_3         IQ_score_2_3 
#>                 0.04                 0.03                 0.05 
#>       Motivation_2_3 parents_academic_2_3   gifted_support_2_3 
#>                 0.26                 0.10                 0.04

# Plotting trend over increasing sample size
Plot_MAGMA(Balance = Balance_tar,
           criterion = c("Pillai", "d_ratio", "mean_g", "Adj_d_ratio")) 


# Creating table
Table_MAGMA(Balance = Balance_tar,
            filename = "Balance_tar.docx")
#> Balance table successfully created!
#> # A tibble: 4 × 6
#>   Criterion_optimized Pillai_Trace d_ratio mean_g adjusted_d_ratio n_per_group
#>   <chr>                      <dbl>   <dbl>  <dbl>            <dbl>       <int>
#> 1 Best mean g                 0.01    0.8    0.1              0.75         116
#> 2 Best adj. d-ratio           0.01    0.87   0.1              0.75         117
#> 3 Best Pillai                 0.01    0.93   0.11             0.74         121
#> 4 Best d-ratio                0.01    0.93   0.12             0.72         123

Post-matching statistics

After selecting a final solution we can report the post-matching descriptive statistics. To this end, we need to adapt our arguments to this three-group example.

# Computing descriptive statistics and pairwise effects for 100 cases per group
descs_tar_post <- MAGMA_desc(Data = MAGMA_sim_data_tar,
                            group = "teacher_ability_rating",
                            covariates = covariates_tar,
                            step_num = 100,
                            step_var = "step",
                            filename = "stats_tar_post.docx")

# Displaying the table with defined colum names
descs_tar_post %>%
    purrr::set_names(c("Overall N", "Overall Mean", "Overall SD",
                       "BA N", "BA Support Mean", "BA Support SD",
                       "A N", "A Mean", "A SD",
                       "AA N", "AA Mean", "AA SD",
                       "d BA-A", "d BA-AA", "d A-AA"))
#>                        Overall N Overall Mean Overall SD BA N BA Support Mean
#> teacher_ability_rating       300         2.00       0.82  100            1.00
#> GPA_school                   300         3.21       0.66  100            3.36
#> IQ_score                     300        99.29      12.99  100          101.19
#> Motivation                   300         3.86       0.87  100            3.89
#> parents_academic             300         0.39       0.49  100            0.42
#> gifted_support               300         0.41       0.49  100            0.42
#>                        BA Support SD A N A Mean  A SD AA N AA Mean AA SD d BA-A
#> teacher_ability_rating          0.00 100   2.00  0.00  100    3.00  0.00   -Inf
#> GPA_school                      0.68 100   3.14  0.70  100    3.12  0.59   0.32
#> IQ_score                       11.71 100  98.67 11.74  100   98.02 15.13   0.21
#> Motivation                      0.86 100   3.73  0.82  100    3.96  0.93   0.19
#> parents_academic                0.50 100   0.40  0.49  100    0.35  0.48   0.04
#> gifted_support                  0.50 100   0.42  0.50  100    0.40  0.49   0.00
#>                        d BA-AA d A-AA
#> teacher_ability_rating    -Inf   -Inf
#> GPA_school                0.38   0.03
#> IQ_score                  0.23   0.05
#> Motivation               -0.08  -0.26
#> parents_academic          0.14   0.10
#> gifted_support            0.04   0.04

Exact Matching

We use the same covariates as we used for the standard three-group matching. The only change is the use of gender as exact variable. Thus, girls will only be matched to girls, and boys will only be matched to other boys across the three groups. Initial unbalance, descriptive statistics, and common support are the same as for standard matching. Below you find the entire matching, balance estimation, and visualization process.

MAGMA_sim_data_tar_exact <- MAGMA_exact(Data = MAGMA_sim_data,
                                        group = "teacher_ability_rating",
                                        dist = "ps_tar",
                                        exact = "gender",
                                        cores = 2)
str(MAGMA_sim_data_tar_exact)
#> 'data.frame':    800 obs. of  17 variables:
#>  $ ID                    : int  1 2 3 4 5 6 7 8 9 10 ...
#>  $ gender                : int  1 0 1 1 1 0 0 0 1 0 ...
#>  $ gifted_support        : int  1 1 1 1 1 1 0 1 0 1 ...
#>  $ teacher_ability_rating: int  3 2 2 1 3 3 2 2 1 1 ...
#>  $ enrichment            : int  0 1 0 1 0 0 0 0 0 1 ...
#>  $ parents_academic      : int  1 1 1 0 1 0 0 1 0 0 ...
#>  $ GPA_school            : num  4.73 4.41 3.23 2.88 2.98 ...
#>  $ IQ_score              : num  111.6 97.8 120.1 97.9 74.4 ...
#>  $ Motivation            : num  5.36 4.41 3.57 4.34 3.54 ...
#>  $ college_GPA           : num  4.01 3.41 2.73 4.02 4.09 ...
#>  $ support_enrichment    : int  3 4 3 4 3 3 1 3 1 4 ...
#>  $ ps_tar                : num  0.0657 0.1085 0.1899 0.4261 0.4782 ...
#>  $ ps_2x2                : num  0.3 0.32 0.21 0.237 0.571 ...
#>  $ ps_gifted             : num  0.431 0.364 0.626 0.353 0.187 ...
#>  $ step                  : num  226 NA 165 108 71 43 188 NA 241 54 ...
#>  $ weight                : num  1 NA 1 1 1 1 1 NA 1 1 ...
#>  $ distance              : num  2.787944 NA 1.040494 0.231533 0.000273 ...
Balance_tar_exact <- Balance_MAGMA(Data = MAGMA_sim_data_tar_exact,
                                   group = "teacher_ability_rating",
                                   covariates = covariates_tar,
                                   step = "step") 
# Balance criteria for 100 cases per group
Balance_100_tar_criteria_exact <- Balance_extract(Balance = Balance_tar_exact,
                                                  samplesize = 100,
                                                  effects = FALSE)
Balance_100_tar_criteria_exact
#> Pillai's Trace        d-ratio         mean g   adj. d-ratio 
#>           0.03           0.67           0.16           0.61

# Extracting pairwise effects for 100 cases per group
Balance_100_tar_effects_exact <- Balance_extract(Balance = Balance_tar_exact,
                                                 samplesize = 100,
                                                 effects = TRUE)
Balance_100_tar_effects_exact
#>       GPA_school_1_2         IQ_score_1_2       Motivation_1_2 
#>                 0.36                 0.34                 0.10 
#> parents_academic_1_2   gifted_support_1_2       GPA_school_1_3 
#>                 0.12                 0.12                 0.37 
#>         IQ_score_1_3       Motivation_1_3 parents_academic_1_3 
#>                 0.24                 0.14                 0.12 
#>   gifted_support_1_3       GPA_school_2_3         IQ_score_2_3 
#>                 0.04                 0.00                 0.05 
#>       Motivation_2_3 parents_academic_2_3   gifted_support_2_3 
#>                 0.24                 0.00                 0.08

# Plotting trend over increasing sample size
Plot_MAGMA(Balance = Balance_tar_exact,
           criterion = c("Pillai", "d_ratio", "mean_g", "Adj_d_ratio")) #Could be omitted


# Creating table
Table_MAGMA(Balance = Balance_tar_exact,
            filename = "Balance_tar_exact.docx")
#> Balance table successfully created!
#> # A tibble: 4 × 6
#>   Criterion_optimized Pillai_Trace d_ratio mean_g adjusted_d_ratio n_per_group
#>   <chr>                      <dbl>   <dbl>  <dbl>            <dbl>       <int>
#> 1 Best mean g                 0.01    0.8    0.12             0.7          113
#> 2 Best adj. d-ratio           0.01    0.8    0.12             0.7          114
#> 3 Best Pillai                 0.01    0.73   0.12             0.69         116
#> 4 Best d-ratio                0.01    0.8    0.14             0.65         127

# Computing descriptive statistics and pairwise effects for 100 cases per group
descs_tar_post_exact <- MAGMA_desc(Data = MAGMA_sim_data_tar_exact,
                                   group = "teacher_ability_rating",
                                   covariates = covariates_tar,
                                   step_num = 100,
                                   step_var = "step",
                                   filename = "stats_tar_post_exact.docx")

# Displaying the table with defined column names
descs_tar_post_exact %>%
    purrr::set_names(c("Overall N", "Overall Mean", "Overall SD",
                       "BA N", "BA Support Mean", "BA Support SD",
                       "A N", "A Mean", "A SD",
                       "AA N", "AA Mean", "AA SD",
                       "d BA-A", "d BA-AA", "d A-AA"))
#>                        Overall N Overall Mean Overall SD BA N BA Support Mean
#> teacher_ability_rating       300         2.00       0.82  100            1.00
#> GPA_school                   300         3.21       0.65  100            3.37
#> IQ_score                     300        98.95      13.11  100          101.40
#> Motivation                   300         3.83       0.91  100            3.82
#> parents_academic             300         0.38       0.49  100            0.42
#> gifted_support               300         0.39       0.49  100            0.42
#>                        BA Support SD A N A Mean  A SD AA N AA Mean AA SD d BA-A
#> teacher_ability_rating          0.00 100   2.00  0.00  100    3.00  0.00   -Inf
#> GPA_school                      0.68 100   3.13  0.64  100    3.13  0.60   0.36
#> IQ_score                       11.93 100  97.36 11.53  100   98.07 15.30   0.34
#> Motivation                      0.91 100   3.73  0.91  100    3.95  0.90   0.10
#> parents_academic                0.50 100   0.36  0.48  100    0.36  0.48   0.12
#> gifted_support                  0.50 100   0.36  0.48  100    0.40  0.49   0.12
#>                        d BA-AA d A-AA
#> teacher_ability_rating    -Inf   -Inf
#> GPA_school                0.37   0.00
#> IQ_score                  0.24  -0.05
#> Motivation               -0.14  -0.24
#> parents_academic          0.12   0.00
#> gifted_support            0.04  -0.08

2x2/four-group Example

In this 2x2-design example the variables gifted_support and enrichment are the independent (i.e., the treatment) variables. We are interested in how receiving giftedness support and the participation in afternoon enrichment affect the college GPA. The process of matching and balance evaluation is the same as in the previous examples. The grouping variable consists of two variables, so we use a character vector as input for the grouping variable (c("gifted_support", "enrichment")). Consequently, we also adapt the distance variable/propensity score (ps_2x2). Moreover, we use the same covariates as in the two-group example. Note that this 2x2 matching is largely equivalent to a 1x4 matching. Thus, using the multinominal variable support_enrichment as grouping variable would lead to similar results. In this example, we focus on the 2x2 case only.

Major distinctions between such a multinominal four-group matching and a 2x2 matching occur only in the balance estimation. Because of the changes in grouping variable and covariates, we need to estimate the initial unbalance for the two independent variables gifted_support and enrichment. The result of the initial_unbalance function changes slightly in this two-factorial design. Pillai’s Trace is estimated separately for the two main effects and the interaction. Thus, d-ratio, mean g, and adjusted d-ratio are the same for a 2x2 or a four-group matching, while changes occur only for Pillai’s Trace. In a first step, we define the covariates and estimate initial unbalance, descriptive statistics, and area of common support.

# Defining the covariates
covariates_2x2 <- c("GPA_school",
                    "IQ_score",
                    "Motivation",
                    "parents_academic",
                    "gender")

# Computing descriptive statistics and all pairwise effects
descs_2x2_pre <- MAGMA_desc(Data = MAGMA_sim_data,
                            group = c("gifted_support", "enrichment"),
                            covariates = covariates_2x2,
                            filename = "stats_2x2_pre.docx")
#> 2x2 groups are represented as 4 groups.

descs_2x2_pre %>%
  purrr::set_names(c("Overall N", "Overall Mean", "Overall SD",
                     "Sup & No En N", "Sup & No En Mean", 
                     "Sup & No En SD",
                     "Sup & En N", "Sup & En Mean", "Sup & En SD",
                     "No Sup & No En N", "No Sup & No En Mean", "No Sup & No En SD",
                     "No Sup & En N", "No Sup & En Mean", "No Sup & En SD",
                     "d YesNo-YesYes", "d YesNo-NoNo", "d YesNo-NoYes",
                     "d YesYes-NoNo", "d YesYes-YNoYes",
                     "d NoNo-NoYes"))
#>                  Overall N Overall Mean Overall SD Sup & No En N
#> group_long             800         2.63       1.07           175
#> GPA_school             800         3.29       0.93           175
#> IQ_score               800       100.51      15.00           175
#> Motivation             800         3.82       0.89           175
#> parents_academic       800         0.40       0.49           175
#> gender                 800         0.50       0.50           175
#>                  Sup & No En Mean Sup & No En SD Sup & En N Sup & En Mean
#> group_long                   1.00           0.00        130          2.00
#> GPA_school                   3.44           0.88        130          3.71
#> IQ_score                   105.09          14.09        130        110.26
#> Motivation                   3.79           0.90        130          3.89
#> parents_academic             0.43           0.50        130          0.53
#> gender                       0.49           0.50        130          0.48
#>                  Sup & En SD No Sup & No En N No Sup & No En Mean
#> group_long              0.00              309                3.00
#> GPA_school              0.88              309                3.01
#> IQ_score               12.74              309               94.27
#> Motivation              0.86              309                3.75
#> parents_academic        0.50              309                0.37
#> gender                  0.50              309                0.50
#>                  No Sup & No En SD No Sup & En N No Sup & En Mean
#> group_long                    0.00           186             4.00
#> GPA_school                    0.93           186             3.31
#> IQ_score                     13.30           186            99.74
#> Motivation                    0.89           186             3.92
#> parents_academic              0.48           186             0.32
#> gender                        0.50           186             0.51
#>                  No Sup & En SD d YesNo-YesYes d YesNo-NoNo d YesNo-NoYes
#> group_long                 0.00           -Inf         -Inf          -Inf
#> GPA_school                 0.86          -0.31         0.47          0.15
#> IQ_score                  15.01          -0.38         0.80          0.37
#> Motivation                 0.90          -0.11         0.04         -0.14
#> parents_academic           0.47          -0.20         0.12          0.23
#> gender                     0.50           0.02        -0.02         -0.04
#>                  d YesYes-NoNo d YesYes-YNoYes d NoNo-NoYes
#> group_long                -Inf            -Inf         -Inf
#> GPA_school                0.76            0.46        -0.33
#> IQ_score                  1.22            0.74        -0.39
#> Motivation                0.16           -0.03        -0.19
#> parents_academic          0.33            0.44         0.10
#> gender                   -0.04           -0.06        -0.02

# Estimating initial unbalance
unbalance_2x2 <- initial_unbalance(Data = MAGMA_sim_data,
                                   group = c("gifted_support", "enrichment"),
                                   covariates = covariates_2x2)
#> Mean g was computed using robust variance meta-analysis with robumeta.
unbalance_2x2
#>           Pillai's Trace gifted_support Pillai's Trace enrichment
#> Unbalance                          0.13                      0.05
#>           Pillai's Trace IA d-ratio Mean g adj. d-ratio
#> Unbalance              0.01    0.53   0.29         0.46

# Estimating and plotting density overlap in gifted support & enrichment propensity score
Density_overlap(Data = MAGMA_sim_data,
                variable = "ps_2x2", 
                group = c("gifted_support", "enrichment"),
                variable_name = "Propensity Score",
                group_labels = c("No Support & No Enrichment",
                                 "No Support & Enrichment",
                                 "Support & No Enrichment",
                                 "Support & Enrichment"),
                group_name = "Gifted Support & Enrichment")
#> 2x2 groups are represented as 4 groups.

#>                                                         OV 
#>                                                  0.8692445 
#> OVPairs.No Support & No Enrichment-No Support & Enrichment 
#>                                                  0.8264201 
#> OVPairs.No Support & No Enrichment-Support & No Enrichment 
#>                                                  0.6062272 
#>    OVPairs.No Support & No Enrichment-Support & Enrichment 
#>                                                  0.8951000 
#>    OVPairs.No Support & Enrichment-Support & No Enrichment 
#>                                                  0.4477958 
#>       OVPairs.No Support & Enrichment-Support & Enrichment 
#>                                                  0.7450246 
#>       OVPairs.Support & No Enrichment-Support & Enrichment 
#>                                                  0.6573381

Standard Matching

After this check of initial unbalance, descriptive statistics, and area of common support, we can conduct the matching. As mentioned above, we need to adapt the grouping variable and the distance variable/propensity score for this example. Moreover, conducting a 2x2/four-group matching has some computational load. If a RAM threshold is exceeded, MAGMA computes quasi-systematic matching. This means that the systematic matching algorithm of MAGMA is applied to random subgroups of the sample. Again, this only affects the computational backend and does not affect the user application. Note that in the case of a multifactorial matching, MAGMA returns a fourth variable. This fourth variable is a multinominal version of the two independent variables (group_long).

MAGMA_sim_data_2x2 <- MAGMA(Data = MAGMA_sim_data,
                            group = c("gifted_support", "enrichment"),
                            dist = "ps_2x2",
                            cores = 2)
str(MAGMA_sim_data_2x2)
#> 'data.frame':    800 obs. of  18 variables:
#>  $ ID                    : int  1 2 3 4 5 6 7 8 9 10 ...
#>  $ gender                : int  1 0 1 1 1 0 0 0 1 0 ...
#>  $ gifted_support        : int  1 1 1 1 1 1 0 1 0 1 ...
#>  $ teacher_ability_rating: int  3 2 2 1 3 3 2 2 1 1 ...
#>  $ enrichment            : int  0 1 0 1 0 0 0 0 0 1 ...
#>  $ parents_academic      : int  1 1 1 0 1 0 0 1 0 0 ...
#>  $ GPA_school            : num  4.73 4.41 3.23 2.88 2.98 ...
#>  $ IQ_score              : num  111.6 97.8 120.1 97.9 74.4 ...
#>  $ Motivation            : num  5.36 4.41 3.57 4.34 3.54 ...
#>  $ college_GPA           : num  4.01 3.41 2.73 4.02 4.09 ...
#>  $ support_enrichment    : int  3 4 3 4 3 3 1 3 1 4 ...
#>  $ ps_tar                : num  0.0657 0.1085 0.1899 0.4261 0.4782 ...
#>  $ ps_2x2                : num  0.3 0.32 0.21 0.237 0.571 ...
#>  $ ps_gifted             : num  0.431 0.364 0.626 0.353 0.187 ...
#>  $ group_long            : num  1 2 1 2 1 1 3 1 3 2 ...
#>  $ step                  : num  71 41 35 19 NA 12 NA NA NA 105 ...
#>  $ weight                : num  1 1 1 1 NA 1 NA NA NA 1 ...
#>  $ distance              : num  3.61e-03 2.15e-04 1.24e-04 4.69e-05 NA ...

Balance Estimation and Visualization

The only difference to the previous examples in the process is, that the grouping variable is a character vector of length two. As a result of this, Plot_MAGMA displays three Pillai’s Trace plots corresponding to the two main effects and the interaction.

Balance_2x2 <- Balance_MAGMA(Data = MAGMA_sim_data_2x2,
                             group = c("gifted_support", "enrichment"),
                             covariates = covariates_2x2,
                             step = "step")

# Balance criteria for 100 cases per group
Balance_100_2x2_criteria <- Balance_extract(Balance = Balance_2x2,
                                            samplesize = 100,
                                            effects = FALSE)
Balance_100_2x2_criteria
#> Pillai's Trace ME 1 Pillai's Trace ME 2   Pillai's Trace IA             d-ratio 
#>                0.02                0.01                0.02                0.73 
#>              mean g        adj. d-ratio 
#>                0.16                0.60

# Extracting pairwise effects for 100 cases per group
Balance_100_2x2_effects <- Balance_extract(Balance = Balance_2x2,
                                           samplesize = 100,
                                           effects = TRUE)
Balance_100_2x2_effects
#>       GPA_school_1_2         IQ_score_1_2       Motivation_1_2 
#>                 0.17                 0.12                 0.17 
#> parents_academic_1_2           gender_1_2       GPA_school_1_3 
#>                 0.22                 0.06                 0.10 
#>         IQ_score_1_3       Motivation_1_3 parents_academic_1_3 
#>                 0.03                 0.21                 0.10 
#>           gender_1_3       GPA_school_1_4         IQ_score_1_4 
#>                 0.12                 0.01                 0.27 
#>       Motivation_1_4 parents_academic_1_4           gender_1_4 
#>                 0.34                 0.19                 0.06 
#>       GPA_school_2_3         IQ_score_2_3       Motivation_2_3 
#>                 0.07                 0.09                 0.03 
#> parents_academic_2_3           gender_2_3       GPA_school_2_4 
#>                 0.12                 0.18                 0.19 
#>         IQ_score_2_4       Motivation_2_4 parents_academic_2_4 
#>                 0.38                 0.17                 0.41 
#>           gender_2_4       GPA_school_3_4         IQ_score_3_4 
#>                 0.12                 0.12                 0.29 
#>       Motivation_3_4 parents_academic_3_4           gender_3_4 
#>                 0.14                 0.29                 0.06

# Plotting trend over increasing sample size
Plot_MAGMA(Balance = Balance_2x2,
           criterion = c("Pillai", "d_ratio", "mean_g", "Adj_d_ratio")) 


# Creating table
Table_MAGMA(Balance = Balance_2x2,
            filename = "Balance_2x2.docx")
#> Balance table successfully created!
#> # A tibble: 6 × 8
#>   Criterion_optimized Pillai_Trace_ME1 Pillai_Trace_ME2 Pillai_Trace_IA d_ratio
#>   <chr>                          <dbl>            <dbl>           <dbl>   <dbl>
#> 1 Best Pillai ME 2                0.02             0               0.04    0.7 
#> 2 Best Pillai IA                  0.02             0.02            0.01    0.8 
#> 3 Best mean g                     0.02             0.02            0.01    0.8 
#> 4 Best d-ratio                    0.02             0.01            0.02    0.83
#> 5 Best Pillai ME 1                0.01             0.01            0.02    0.8 
#> 6 Best adj. d-ratio               0.02             0.01            0.02    0.8 
#> # ℹ 3 more variables: mean_g <dbl>, adjusted_d_ratio <dbl>, n_per_group <int>

# Computing descriptive statistics and all pairwise effects after matching
descs_2x2_post <- MAGMA_desc(Data = MAGMA_sim_data_2x2,
                             group = c("gifted_support", "enrichment"),
                             covariates = covariates_2x2,
                             step_num = 100,
                             step_var = "step",
                             filename = "stats_post_pre.docx")
#> 2x2 groups are represented as 4 groups.

descs_2x2_post %>%
  purrr::set_names(c("Overall N", "Overall Mean", "Overall SD",
                     "Sup & No En N", "Sup & No En Mean", 
                     "Sup & No En SD",
                     "Sup & En N", "Sup & En Mean", "Sup & En SD",
                     "No Sup & No En N", "No Sup & No En Mean", "No Sup & No En SD",
                     "No Sup & En N", "No Sup & En Mean", "No Sup & En SD",
                     "d YesNo-YesYes", "d YesNo-NoNo", "d YesNo-NoYes",
                     "d YesYes-NoNo", "d YesYes-YNoYes",
                     "d NoNo-NoYes"))
#>                  Overall N Overall Mean Overall SD Sup & No En N
#> group_long             400         2.50       1.12           100
#> GPA_school             400         3.52       0.86           100
#> IQ_score               400       104.15      11.20           100
#> Motivation             400         3.90       0.85           100
#> parents_academic       400         0.44       0.50           100
#> gender                 400         0.52       0.50           100
#>                  Sup & No En Mean Sup & No En SD Sup & En N Sup & En Mean
#> group_long                   1.00           0.00        100          2.00
#> GPA_school                   3.46           0.90        100          3.61
#> IQ_score                   104.55          10.59        100        105.81
#> Motivation                   3.74           0.88        100          3.89
#> parents_academic             0.42           0.50        100          0.53
#> gender                       0.50           0.50        100          0.47
#>                  Sup & En SD No Sup & No En N No Sup & No En Mean
#> group_long              0.00              100                3.00
#> GPA_school              0.86              100                3.55
#> IQ_score               10.29              100              104.85
#> Motivation              0.85              100                3.92
#> parents_academic        0.50              100                0.47
#> gender                  0.50              100                0.56
#>                  No Sup & No En SD No Sup & En N No Sup & En Mean
#> group_long                    0.00           100             4.00
#> GPA_school                    0.89           100             3.46
#> IQ_score                     10.79           100           101.40
#> Motivation                    0.83           100             4.03
#> parents_academic              0.50           100             0.33
#> gender                        0.50           100             0.53
#>                  No Sup & En SD d YesNo-YesYes d YesNo-NoNo d YesNo-NoYes
#> group_long                 0.00           -Inf         -Inf          -Inf
#> GPA_school                 0.78          -0.17        -0.10          0.00
#> IQ_score                  12.66          -0.12        -0.03          0.27
#> Motivation                 0.85          -0.17        -0.21         -0.34
#> parents_academic           0.47          -0.22        -0.10          0.19
#> gender                     0.50           0.06        -0.12         -0.06
#>                  d YesYes-NoNo d YesYes-YNoYes d NoNo-NoYes
#> group_long                -Inf            -Inf         -Inf
#> GPA_school                0.07            0.18         0.11
#> IQ_score                  0.09            0.38         0.29
#> Motivation               -0.04           -0.16        -0.13
#> parents_academic          0.12            0.41         0.29
#> gender                   -0.18           -0.12         0.06

Exact Matching

We use the same covariates as we used for the standard 2x2 matching. The only change constitutes the use of teacher_ability_rating as an exact variable. This means that only students with same teacher rated ability can be matched (e.g., above average with above average). Below is the entire matching, balance estimation, and visualization process. Initial unbalance, descriptive statistics, and area of common support are the same as for standard matching.

MAGMA_sim_data_2x2_exact <- MAGMA_exact(Data = MAGMA_sim_data,
                                        group = c("gifted_support", "enrichment"),
                                        dist = "ps_2x2",
                                        exact = "teacher_ability_rating",
                                        cores = 2)
str(MAGMA_sim_data_2x2_exact)
#> 'data.frame':    520 obs. of  18 variables:
#>  $ ID                    : int  1 2 3 4 5 6 10 13 14 16 ...
#>  $ gender                : int  1 0 1 1 1 0 0 1 0 1 ...
#>  $ gifted_support        : int  1 1 1 1 1 1 1 0 0 0 ...
#>  $ teacher_ability_rating: int  3 2 2 1 3 3 1 3 2 1 ...
#>  $ enrichment            : int  0 1 0 1 0 0 1 0 0 1 ...
#>  $ parents_academic      : int  1 1 1 0 1 0 0 0 0 0 ...
#>  $ GPA_school            : num  4.73 4.41 3.23 2.88 2.98 ...
#>  $ IQ_score              : num  111.6 97.8 120.1 97.9 74.4 ...
#>  $ Motivation            : num  5.36 4.41 3.57 4.34 3.54 ...
#>  $ college_GPA           : num  4.01 3.41 2.73 4.02 4.09 ...
#>  $ support_enrichment    : int  3 4 3 4 3 3 4 1 1 2 ...
#>  $ ps_tar                : num  0.0657 0.1085 0.1899 0.4261 0.4782 ...
#>  $ ps_2x2                : num  0.3 0.32 0.21 0.237 0.571 ...
#>  $ ps_gifted             : num  0.431 0.364 0.626 0.353 0.187 ...
#>  $ group_long            : num  1 2 1 2 1 1 2 3 3 4 ...
#>  $ step                  : num  58 7 116 88 48 19 62 101 82 114 ...
#>  $ weight                : num  1 1 1 1 1 1 1 1 1 1 ...
#>  $ distance              : num  1.50e-03 1.76e-05 5.02e-01 9.73e-02 6.57e-04 ...
# Estimating Balance
Balance_2x2_exact <- Balance_MAGMA(Data = MAGMA_sim_data_2x2_exact,
                                   group = c("gifted_support", "enrichment"),
                                   covariates = covariates_2x2,
                                   step = "step") #Not necessary to define here
# Balance criteria for 100 cases per group
Balance_100_2x2_criteria_exact <- Balance_extract(Balance = Balance_2x2_exact,
                                                  samplesize = 100,
                                                  effects = FALSE)
Balance_100_2x2_criteria_exact
#> Pillai's Trace ME 1 Pillai's Trace ME 2   Pillai's Trace IA             d-ratio 
#>                0.01                0.01                0.02                0.77 
#>              mean g        adj. d-ratio 
#>                0.12                0.69

# Extracting pairwise effects for 100 cases per group
Balance_100_2x2_effects_exact <- Balance_extract(Balance = Balance_2x2_exact,
                                                 samplesize = 100,
                                                 effects = TRUE)
Balance_100_2x2_effects_exact
#>       GPA_school_1_2         IQ_score_1_2       Motivation_1_2 
#>                 0.12                 0.16                 0.17 
#> parents_academic_1_2           gender_1_2       GPA_school_1_3 
#>                 0.04                 0.10                 0.10 
#>         IQ_score_1_3       Motivation_1_3 parents_academic_1_3 
#>                 0.06                 0.23                 0.08 
#>           gender_1_3       GPA_school_1_4         IQ_score_1_4 
#>                 0.02                 0.01                 0.18 
#>       Motivation_1_4 parents_academic_1_4           gender_1_4 
#>                 0.21                 0.22                 0.02 
#>       GPA_school_2_3         IQ_score_2_3       Motivation_2_3 
#>                 0.02                 0.10                 0.05 
#> parents_academic_2_3           gender_2_3       GPA_school_2_4 
#>                 0.04                 0.12                 0.14 
#>         IQ_score_2_4       Motivation_2_4 parents_academic_2_4 
#>                 0.34                 0.04                 0.26 
#>           gender_2_4       GPA_school_3_4         IQ_score_3_4 
#>                 0.12                 0.12                 0.24 
#>       Motivation_3_4 parents_academic_3_4           gender_3_4 
#>                 0.01                 0.30                 0.00

# Plotting trend over increasing sample size
Plot_MAGMA(Balance = Balance_2x2_exact,
           criterion = c("Pillai", "d_ratio", "mean_g", "Adj_d_ratio")) #Could be omitted


# Creating table
Table_MAGMA(Balance = Balance_2x2_exact,
            filename = "Balance_2x2_exact.docx")
#> Balance table successfully created!
#> # A tibble: 6 × 8
#>   Criterion_optimized Pillai_Trace_ME1 Pillai_Trace_ME2 Pillai_Trace_IA d_ratio
#>   <chr>                          <dbl>            <dbl>           <dbl>   <dbl>
#> 1 Best Pillai ME 1                0                0.04            0.07    0.6 
#> 2 Best Pillai IA                  0.01             0.01            0.02    0.83
#> 3 Best mean g                     0.01             0.01            0.02    0.83
#> 4 Best d-ratio                    0.01             0               0.02    0.87
#> 5 Best adj. d-ratio               0.01             0               0.02    0.8 
#> 6 Best Pillai ME 2                0.01             0               0.02    0.8 
#> # ℹ 3 more variables: mean_g <dbl>, adjusted_d_ratio <dbl>, n_per_group <int>

# Computing descriptive statistics and all pairwise effects post matching
descs_2x2_post <- MAGMA_desc(Data = MAGMA_sim_data_2x2_exact,
                             group = c("gifted_support", "enrichment"),
                             covariates = covariates_2x2,
                             step_num = 100,
                             step_var = "step",
                             filename = "stats_2x2_post.docx")
#> 2x2 groups are represented as 4 groups.

descs_2x2_post %>%
  purrr::set_names(c("Overall N", "Overall Mean", "Overall SD",
                     "Sup & No En N", "Sup & No En Mean", 
                     "Sup & No En SD",
                     "Sup & En N", "Sup & En Mean", "Sup & En SD",
                     "No Sup & No En N", "No Sup & No En Mean", "No Sup & No En SD",
                     "No Sup & En N", "No Sup & En Mean", "No Sup & En SD",
                     "d YesNo-YesYes", "d YesNo-NoNo", "d YesNo-NoYes",
                     "d YesYes-NoNo", "d YesYes-YNoYes",
                     "d NoNo-NoYes"))
#>                  Overall N Overall Mean Overall SD Sup & No En N
#> group_long             400         2.50       1.12           100
#> GPA_school             400         3.54       0.86           100
#> IQ_score               400       104.14      11.18           100
#> Motivation             400         3.90       0.85           100
#> parents_academic       400         0.46       0.50           100
#> gender                 400         0.50       0.50           100
#>                  Sup & No En Mean Sup & No En SD Sup & En N Sup & En Mean
#> group_long                   1.00           0.00        100          2.00
#> GPA_school                   3.50           0.92        100          3.60
#> IQ_score                   104.05          11.57        100        105.81
#> Motivation                   3.77           0.86        100          3.92
#> parents_academic             0.47           0.50        100          0.49
#> gender                       0.51           0.50        100          0.46
#>                  Sup & En SD No Sup & No En N No Sup & No En Mean
#> group_long              0.00              100                3.00
#> GPA_school              0.86              100                3.59
#> IQ_score               10.35              100              104.74
#> Motivation              0.85              100                3.96
#> parents_academic        0.50              100                0.51
#> gender                  0.50              100                0.52
#>                  No Sup & No En SD No Sup & En N No Sup & En Mean
#> group_long                    0.00           100             4.00
#> GPA_school                    0.89           100             3.49
#> IQ_score                     10.56           100           101.97
#> Motivation                    0.84           100             3.95
#> parents_academic              0.50           100             0.36
#> gender                        0.50           100             0.52
#>                  No Sup & En SD d YesNo-YesYes d YesNo-NoNo d YesNo-NoYes
#> group_long                 0.00           -Inf         -Inf          -Inf
#> GPA_school                 0.76          -0.11        -0.10          0.01
#> IQ_score                  11.99          -0.16        -0.06          0.18
#> Motivation                 0.85          -0.18        -0.22         -0.21
#> parents_academic           0.48          -0.04        -0.08          0.22
#> gender                     0.50           0.10        -0.02         -0.02
#>                  d YesYes-NoNo d YesYes-YNoYes d NoNo-NoYes
#> group_long                -Inf            -Inf         -Inf
#> GPA_school                0.01            0.14         0.12
#> IQ_score                  0.10            0.34         0.25
#> Motivation               -0.05           -0.04         0.01
#> parents_academic         -0.04            0.27         0.31
#> gender                   -0.12           -0.12         0.00

References

Austin, P. C. (2014). A comparison of 12 algorithms for matching on the propensity score. Statistics in Medicine, 33(6), 1057–1069. https://doi.org/10.1002/sim.6004

Feuchter, M. D., Urban, J., Scherrer, V., Breit, M. L., & Preckel, F. (2022). Introduction and Demonstration of the Many-Group Matching (MAGMA)-Algorithm: Matching Solutions for Two or More Groups. https://doi.org/10.17605/OSF.IO/AEDXB

Fisher, Z., Tipton, E., & Zhipeng, H. (2023). robumeta: Robust Variance Meta-Regression. R package version 2.1, https://CRAN.R-project.org/package=robumeta.

Imai, K., & van Dyk, D. A. (2004). Causal Inference With General Treatment Regimes. Journal of the American Statistical Association, 99(467), 854–866. https://doi.org/10.1198/016214504000001187

Jacovidis, J. N. (2017). Evaluating the performance of propensity score matching methods: A simulation study [Doctoral Dissertation]. James Madison University.

Li, M. (2013). Using the Propensity Score Method to Estimate Causal Effects. Organizational Research Methods, 16(2), 188–226. https://doi.org/10.1177/1094428112447816

McCaffrey, D. F., Griffin, B. A., Almirall, D., Slaughter, M. E., Ramchand, R., & Burgette, L. F. (2013). A tutorial on propensity score estimation for multiple treatments using generalized boosted models. Statistics in Medicine, 32(19), 3388–3414. https://doi.org/10.1002/sim.5753

Pastore, M., Loro, P.A.D., Mingione, M., Calcagni, A. (2022). overlapping: Estimation of Overlapping in Empirical Distributions. R package version 2.1. https://CRAN.R-project.org/package=overlapping

Powell, M. G., Hull, D. M., & Beaujean, A. A. (2020). Propensity Score Matching for Education Data: Worked Examples. The Journal of Experimental Education, 88(1), 145–164. https://doi.org/10.1080/00220973.2018.1541850

Ridgeway, G., McCaffrey, D. F., Morral, A. R., Cefalu, M., Burgette, L. F., Pane, J. D., & Griffin, B. A. (2015). Toolkit for Weighting and Analysis of Nonequivalent Groups: A tutorial for the twang package. Rand.

Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55. https://doi.org/10.1093/biomet/70.1.41

Rosenbaum, P. R., & Rubin, D. B. (1985). Constructing a Control Group Using Multivariate Matched Sampling Methods That Incorporate the Propensity Score. The American Statistician, 39(1), 33–38.

Stuart, E. A. (2010). Matching methods for causal inference: A review and a look forward. Statistical Science : A Review Journal of the Institute of Mathematical Statistics, 25(1), 1–21. https://doi.org/10.1214/09-STS313

Thoemmes, F. J., & Kim, E. S. (2011). A Systematic Review of Propensity Score Methods in the Social Sciences. Multivariate Behavioral Research, 46(1), 90–118. https://doi.org/10.1080/00273171.2011.540475

Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3), 1-48. https://doi.org/10.18637/jss.v036.i03