Longitudinal microbiome analysis#

In this section of the tutorial we’ll perform several analyses using QIIME 2’s q2-longitudinal [BDZ+18] plugin. These will allow us to track microbiome changes across time on a per-subject basis - something that was harder to do in the ordination plots that we viewed earlier in this tutorial.

Preparing our feature table for longitudinal analysis#

Before applying these analyses, we’re going to perform some additional operations on the feature table that will make these analyses run quicker and make the results more interpretable.

First, we’re going to use the taxonomic information that we generated earlier to redefine our features as microbial genera. To do this, we group (or collapse) ASV features based on their taxonomic assignments through the genus level. This is achieved using the q2-taxa plugin’s collapse action.

genus_table, = taxa_actions.collapse(
    table=filtered_table_4,
    taxonomy=taxonomy,
    level=6,
)

action_results <- taxa_actions$collapse(
    table=filtered_table_4,
    taxonomy=taxonomy,
    level=6L,
)
genus_table <- action_results$collapsed_table

qiime taxa collapse \
  --i-table filtered-table-4.qza \
  --i-taxonomy taxonomy.qza \
  --p-level 6 \
  --o-collapsed-table genus-table.qza

genus_table, = use.action(
    use.UsageAction(plugin_id='taxa', action_id='collapse'),
    use.UsageInputs(table=filtered_table_4, taxonomy=taxonomy, level=6),
    use.UsageOutputNames(collapsed_table='genus_table')
)
Using the qiime2 taxa collapse tool:

  1. Set “table” to #: filtered-table-4.qza

  2. Set “taxonomy” to #: taxonomy.qza

  3. Set “level” to 6

  4. Press the Execute button.

Once completed, for the new entry in your history, use the Edit button to set the name as follows:

(Renaming is optional, but it will make any subsequent steps easier to complete.)

History Name

“Name” to set (be sure to press Save)

#: qiime2 taxa collapse [...] : collapsed_table.qza

genus-table.qza

Then, to focus on the genera that are likely to display the most interesting patterns over time (and to reduce the runtime of the steps that come next), we will perform even more filtering. This time we’ll apply prevalence and abundance based filtering. Specifically, we’ll require that a genus’s abundance is at least 1% in at least 10% of the samples.

filtered_genus_table, = feature_table_actions.filter_features_conditionally(
    table=genus_table,
    prevalence=0.1,
    abundance=0.01,
)

action_results <- feature_table_actions$filter_features_conditionally(
    table=genus_table,
    prevalence=0.1,
    abundance=0.01,
)
filtered_genus_table <- action_results$filtered_table

qiime feature-table filter-features-conditionally \
  --i-table genus-table.qza \
  --p-prevalence 0.1 \
  --p-abundance 0.01 \
  --o-filtered-table filtered-genus-table.qza

filtered_genus_table, = use.action(
    use.UsageAction(plugin_id='feature_table', action_id='filter_features_conditionally'),
    use.UsageInputs(table=genus_table, prevalence=0.1, abundance=0.01),
    use.UsageOutputNames(filtered_table='filtered_genus_table')
)
Using the qiime2 feature-table filter-features-conditionally tool:

  1. Set “table” to #: genus-table.qza

  2. Set “abundance” to 0.01

  3. Set “prevalence” to 0.1

  4. Press the Execute button.

Once completed, for the new entry in your history, use the Edit button to set the name as follows:

(Renaming is optional, but it will make any subsequent steps easier to complete.)

History Name

“Name” to set (be sure to press Save)

#: qiime2 feature-table filter-features-conditionally [...] : filtered_table.qza

filtered-genus-table.qza

Finally, we’ll convert the counts in our feature table to relative frequencies. This is required for some of the analyses that we’re about to perform.

genus_rf_table, = feature_table_actions.relative_frequency(
    table=filtered_genus_table,
)

action_results <- feature_table_actions$relative_frequency(
    table=filtered_genus_table,
)
genus_rf_table <- action_results$relative_frequency_table

qiime feature-table relative-frequency \
  --i-table filtered-genus-table.qza \
  --o-relative-frequency-table genus-rf-table.qza

genus_rf_table, = use.action(
    use.UsageAction(plugin_id='feature_table', action_id='relative_frequency'),
    use.UsageInputs(table=filtered_genus_table),
    use.UsageOutputNames(relative_frequency_table='genus_rf_table')
)
Using the qiime2 feature-table relative-frequency tool:

  1. Set “table” to #: filtered-genus-table.qza

  2. Press the Execute button.

Once completed, for the new entry in your history, use the Edit button to set the name as follows:

(Renaming is optional, but it will make any subsequent steps easier to complete.)

History Name

“Name” to set (be sure to press Save)

#: qiime2 feature-table relative-frequency [...] : relative_frequency_table.qza

genus-rf-table.qza

Volatility plots#

The first plots we’ll generate are volatility plots. We’ll generate these using two different time variables. First, we’ll plot based on week-relative-to-hct.

volatility_plot_1_viz, = longitudinal_actions.volatility(
    table=genus_rf_table,
    state_column='week-relative-to-hct',
    metadata=expanded_sample_metadata_md,
    individual_id_column='PatientID',
    default_group_column='autoFmtGroup',
)

action_results <- longitudinal_actions$volatility(
    table=genus_rf_table,
    state_column='week-relative-to-hct',
    metadata=expanded_sample_metadata_md,
    individual_id_column='PatientID',
    default_group_column='autoFmtGroup',
)
volatility_plot_1_viz <- action_results$visualization

qiime longitudinal volatility \
  --i-table genus-rf-table.qza \
  --p-state-column week-relative-to-hct \
  --m-metadata-file sample-metadata.tsv uu-umap.qza diversity-core-metrics-phylogenetic/faith_pd_vector.qza diversity-core-metrics-phylogenetic/evenness_vector.qza diversity-core-metrics-phylogenetic/shannon_vector.qza \
  --p-individual-id-column PatientID \
  --p-default-group-column autoFmtGroup \
  --o-visualization volatility-plot-1.qzv

use.action(
    use.UsageAction(plugin_id='longitudinal', action_id='volatility'),
    use.UsageInputs(table=genus_rf_table, state_column='week-relative-to-hct',
                    metadata=expanded_sample_metadata, individual_id_column='PatientID',
                    default_group_column='autoFmtGroup'),
    use.UsageOutputNames(visualization='volatility_plot_1'),
)
Using the qiime2 longitudinal volatility tool:

  1. For “metadata”:

    1. Perform the following steps.

      1. Leave as Metadata from TSV

      2. Set “Metadata Source” to sample-metadata.tsv

    2. Press the + Insert metadata button to set up the next steps.

      1. Change to Metadata from Artifact

      2. Set “Metadata Source” to uu-umap.qza

    3. Press the + Insert metadata button to set up the next steps.

      1. Change to Metadata from Artifact

      2. Set “Metadata Source” to qiime2 diversity core-metrics-phylogenetic [...] : faith_pd_vector.qza

    4. Press the + Insert metadata button to set up the next steps.

      1. Change to Metadata from Artifact

      2. Set “Metadata Source” to qiime2 diversity core-metrics-phylogenetic [...] : evenness_vector.qza

    5. Press the + Insert metadata button to set up the next steps.

      1. Change to Metadata from Artifact

      2. Set “Metadata Source” to qiime2 diversity core-metrics-phylogenetic [...] : shannon_vector.qza

  2. Set “state_column” to week-relative-to-hct

  3. Expand the additional options section

    1. Set “table” to #: genus-rf-table.qza

    2. Set “individual_id_column” to PatientID

    3. Set “default_group_column” to autoFmtGroup

  4. Press the Execute button.

Once completed, for the new entry in your history, use the Edit button to set the name as follows:

(Renaming is optional, but it will make any subsequent steps easier to complete.)

History Name

“Name” to set (be sure to press Save)

#: qiime2 longitudinal volatility [...] : visualization.qzv

volatility-plot-1.qzv

Next, we’ll plot based on week-relative-to-fmt.

volatility_plot_2_viz, = longitudinal_actions.volatility(
    table=genus_rf_table,
    state_column='week-relative-to-fmt',
    metadata=expanded_sample_metadata_md,
    individual_id_column='PatientID',
    default_group_column='autoFmtGroup',
)

action_results <- longitudinal_actions$volatility(
    table=genus_rf_table,
    state_column='week-relative-to-fmt',
    metadata=expanded_sample_metadata_md,
    individual_id_column='PatientID',
    default_group_column='autoFmtGroup',
)
volatility_plot_2_viz <- action_results$visualization

qiime longitudinal volatility \
  --i-table genus-rf-table.qza \
  --p-state-column week-relative-to-fmt \
  --m-metadata-file sample-metadata.tsv uu-umap.qza diversity-core-metrics-phylogenetic/faith_pd_vector.qza diversity-core-metrics-phylogenetic/evenness_vector.qza diversity-core-metrics-phylogenetic/shannon_vector.qza \
  --p-individual-id-column PatientID \
  --p-default-group-column autoFmtGroup \
  --o-visualization volatility-plot-2.qzv

use.action(
    use.UsageAction(plugin_id='longitudinal', action_id='volatility'),
    use.UsageInputs(table=genus_rf_table, state_column='week-relative-to-fmt',
                    metadata=expanded_sample_metadata, individual_id_column='PatientID',
                    default_group_column='autoFmtGroup'),
    use.UsageOutputNames(visualization='volatility_plot_2'),
)
Using the qiime2 longitudinal volatility tool:

  1. For “metadata”:

    1. Perform the following steps.

      1. Leave as Metadata from TSV

      2. Set “Metadata Source” to sample-metadata.tsv

    2. Press the + Insert metadata button to set up the next steps.

      1. Change to Metadata from Artifact

      2. Set “Metadata Source” to uu-umap.qza

    3. Press the + Insert metadata button to set up the next steps.

      1. Change to Metadata from Artifact

      2. Set “Metadata Source” to qiime2 diversity core-metrics-phylogenetic [...] : faith_pd_vector.qza

    4. Press the + Insert metadata button to set up the next steps.

      1. Change to Metadata from Artifact

      2. Set “Metadata Source” to qiime2 diversity core-metrics-phylogenetic [...] : evenness_vector.qza

    5. Press the + Insert metadata button to set up the next steps.

      1. Change to Metadata from Artifact

      2. Set “Metadata Source” to qiime2 diversity core-metrics-phylogenetic [...] : shannon_vector.qza

  2. Set “state_column” to week-relative-to-fmt

  3. Expand the additional options section

    1. Set “table” to #: genus-rf-table.qza

    2. Set “individual_id_column” to PatientID

    3. Set “default_group_column” to autoFmtGroup

  4. Press the Execute button.

Once completed, for the new entry in your history, use the Edit button to set the name as follows:

(Renaming is optional, but it will make any subsequent steps easier to complete.)

History Name

“Name” to set (be sure to press Save)

#: qiime2 longitudinal volatility [...] : visualization.qzv

volatility-plot-2.qzv

Feature volatility#

The last plots we’ll generate in this section will come from a QIIME 2 pipeline called feature-volatility. These use supervised regression to identify features that are most associated with changes over time, and add plotting of those features to a volatility control chart.

Again, we’ll generate the same plots but using two different time variables on the x-axes. First, we’ll plot based on week-relative-to-hct.

action_results = longitudinal_actions.feature_volatility(
    table=filtered_genus_table,
    metadata=expanded_sample_metadata_md,
    state_column='week-relative-to-hct',
    individual_id_column='PatientID',
)
important_genera_table_1 = action_results.filtered_table
genus_importances_1 = action_results.feature_importance
genus_volatility_plot_1_viz = action_results.volatility_plot
accuracy_results_1_viz = action_results.accuracy_results
sample_estimator_1 = action_results.sample_estimator

action_results <- longitudinal_actions$feature_volatility(
    table=filtered_genus_table,
    metadata=expanded_sample_metadata_md,
    state_column='week-relative-to-hct',
    individual_id_column='PatientID',
)
important_genera_table_1 <- action_results$filtered_table
genus_importances_1 <- action_results$feature_importance
genus_volatility_plot_1_viz <- action_results$volatility_plot
accuracy_results_1_viz <- action_results$accuracy_results
sample_estimator_1 <- action_results$sample_estimator

qiime longitudinal feature-volatility \
  --i-table filtered-genus-table.qza \
  --m-metadata-file sample-metadata.tsv uu-umap.qza diversity-core-metrics-phylogenetic/faith_pd_vector.qza diversity-core-metrics-phylogenetic/evenness_vector.qza diversity-core-metrics-phylogenetic/shannon_vector.qza \
  --p-state-column week-relative-to-hct \
  --p-individual-id-column PatientID \
  --output-dir longitudinal-feature-volatility

use.action(
    use.UsageAction(plugin_id='longitudinal', action_id='feature_volatility'),
    use.UsageInputs(table=filtered_genus_table, metadata=expanded_sample_metadata,
                    state_column='week-relative-to-hct', individual_id_column='PatientID'),
    use.UsageOutputNames(filtered_table='important_genera_table_1',
                         feature_importance='genus_importances_1',
                         volatility_plot='genus_volatility_plot_1',
                         accuracy_results='accuracy_results_1',
                         sample_estimator='sample_estimator_1')
)
Using the qiime2 longitudinal feature-volatility tool:

  1. Set “table” to #: filtered-genus-table.qza

  2. For “metadata”:

    1. Perform the following steps.

      1. Leave as Metadata from TSV

      2. Set “Metadata Source” to sample-metadata.tsv

    2. Press the + Insert metadata button to set up the next steps.

      1. Change to Metadata from Artifact

      2. Set “Metadata Source” to uu-umap.qza

    3. Press the + Insert metadata button to set up the next steps.

      1. Change to Metadata from Artifact

      2. Set “Metadata Source” to qiime2 diversity core-metrics-phylogenetic [...] : faith_pd_vector.qza

    4. Press the + Insert metadata button to set up the next steps.

      1. Change to Metadata from Artifact

      2. Set “Metadata Source” to qiime2 diversity core-metrics-phylogenetic [...] : evenness_vector.qza

    5. Press the + Insert metadata button to set up the next steps.

      1. Change to Metadata from Artifact

      2. Set “Metadata Source” to qiime2 diversity core-metrics-phylogenetic [...] : shannon_vector.qza

  3. Set “state_column” to week-relative-to-hct

  4. Expand the additional options section

    • Set “individual_id_column” to PatientID

  5. Press the Execute button.

Once completed, for each new entry in your history, use the Edit button to set the name as follows:

(Renaming is optional, but it will make any subsequent steps easier to complete.)

History Name

“Name” to set (be sure to press Save)

#: qiime2 longitudinal feature-volatility [...] : filtered_table.qza

important-genera-table-1.qza

#: qiime2 longitudinal feature-volatility [...] : feature_importance.qza

genus-importances-1.qza

#: qiime2 longitudinal feature-volatility [...] : volatility_plot.qzv

genus-volatility-plot-1.qzv

#: qiime2 longitudinal feature-volatility [...] : accuracy_results.qzv

accuracy-results-1.qzv

#: qiime2 longitudinal feature-volatility [...] : sample_estimator.qza

sample-estimator-1.qza

  • longitudinal-feature-volatility/filtered_table.qza | view | download

  • longitudinal-feature-volatility/feature_importance.qza | view | download

  • longitudinal-feature-volatility/volatility_plot.qzv | view | download

  • longitudinal-feature-volatility/accuracy_results.qzv | view | download

  • longitudinal-feature-volatility/sample_estimator.qza | view | download

Next, we’ll plot based on week-relative-to-fmt.

action_results = longitudinal_actions.feature_volatility(
    table=filtered_genus_table,
    metadata=expanded_sample_metadata_md,
    state_column='week-relative-to-fmt',
    individual_id_column='PatientID',
)
important_genera_table_2 = action_results.filtered_table
genus_importances_2 = action_results.feature_importance
genus_volatility_plot_2_viz = action_results.volatility_plot
accuracy_results_2_viz = action_results.accuracy_results
sample_estimator_2 = action_results.sample_estimator

action_results <- longitudinal_actions$feature_volatility(
    table=filtered_genus_table,
    metadata=expanded_sample_metadata_md,
    state_column='week-relative-to-fmt',
    individual_id_column='PatientID',
)
important_genera_table_2 <- action_results$filtered_table
genus_importances_2 <- action_results$feature_importance
genus_volatility_plot_2_viz <- action_results$volatility_plot
accuracy_results_2_viz <- action_results$accuracy_results
sample_estimator_2 <- action_results$sample_estimator

qiime longitudinal feature-volatility \
  --i-table filtered-genus-table.qza \
  --m-metadata-file sample-metadata.tsv uu-umap.qza diversity-core-metrics-phylogenetic/faith_pd_vector.qza diversity-core-metrics-phylogenetic/evenness_vector.qza diversity-core-metrics-phylogenetic/shannon_vector.qza \
  --p-state-column week-relative-to-fmt \
  --p-individual-id-column PatientID \
  --output-dir longitudinal-feature-volatility-2

use.action(
    use.UsageAction(plugin_id='longitudinal', action_id='feature_volatility'),
    use.UsageInputs(table=filtered_genus_table, metadata=expanded_sample_metadata,
                    state_column='week-relative-to-fmt', individual_id_column='PatientID'),
    use.UsageOutputNames(filtered_table='important_genera_table_2',
                         feature_importance='genus_importances_2',
                         volatility_plot='genus_volatility_plot_2',
                         accuracy_results='accuracy_results_2',
                         sample_estimator='sample_estimator_2')
)
Using the qiime2 longitudinal feature-volatility tool:

  1. Set “table” to #: filtered-genus-table.qza

  2. For “metadata”:

    1. Perform the following steps.

      1. Leave as Metadata from TSV

      2. Set “Metadata Source” to sample-metadata.tsv

    2. Press the + Insert metadata button to set up the next steps.

      1. Change to Metadata from Artifact

      2. Set “Metadata Source” to uu-umap.qza

    3. Press the + Insert metadata button to set up the next steps.

      1. Change to Metadata from Artifact

      2. Set “Metadata Source” to qiime2 diversity core-metrics-phylogenetic [...] : faith_pd_vector.qza

    4. Press the + Insert metadata button to set up the next steps.

      1. Change to Metadata from Artifact

      2. Set “Metadata Source” to qiime2 diversity core-metrics-phylogenetic [...] : evenness_vector.qza

    5. Press the + Insert metadata button to set up the next steps.

      1. Change to Metadata from Artifact

      2. Set “Metadata Source” to qiime2 diversity core-metrics-phylogenetic [...] : shannon_vector.qza

  3. Set “state_column” to week-relative-to-fmt

  4. Expand the additional options section

    • Set “individual_id_column” to PatientID

  5. Press the Execute button.

Once completed, for each new entry in your history, use the Edit button to set the name as follows:

(Renaming is optional, but it will make any subsequent steps easier to complete.)

History Name

“Name” to set (be sure to press Save)

#: qiime2 longitudinal feature-volatility [...] : filtered_table.qza

important-genera-table-2.qza

#: qiime2 longitudinal feature-volatility [...] : feature_importance.qza

genus-importances-2.qza

#: qiime2 longitudinal feature-volatility [...] : volatility_plot.qzv

genus-volatility-plot-2.qzv

#: qiime2 longitudinal feature-volatility [...] : accuracy_results.qzv

accuracy-results-2.qzv

#: qiime2 longitudinal feature-volatility [...] : sample_estimator.qza

sample-estimator-2.qza

  • longitudinal-feature-volatility-2/filtered_table.qza | view | download

  • longitudinal-feature-volatility-2/feature_importance.qza | view | download

  • longitudinal-feature-volatility-2/volatility_plot.qzv | view | download

  • longitudinal-feature-volatility-2/accuracy_results.qzv | view | download

  • longitudinal-feature-volatility-2/sample_estimator.qza | view | download

Wrapping up: what did you do five days ago?#

Exercise 6

We’re now getting close to the end of our analysis. Over the course of the tutorial we’ve applied multiple filtering steps to our feature table, and if we were writing this work up for publication it would be essential to accurately describe the filtering steps that we applied, the order in which we applied them, and the parameters that were used for each filter.

Without referring back to the earlier sections of the tutorial, but rather using your memory or information from the results that you’ve generated, describe all filtering steps that were applied to the feature table in the order that we applied them.