Skip to content

Commit

Permalink
Merge pull request #1470 from phac-nml/release/23.01
Browse files Browse the repository at this point in the history
Release/23.01
  • Loading branch information
deepsidhu85 committed Mar 2, 2023
2 parents 54f829c + 54dc607 commit 661ed89
Show file tree
Hide file tree
Showing 490 changed files with 15,937 additions and 13,062 deletions.
12 changes: 7 additions & 5 deletions .github/workflows/ci-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,17 +7,18 @@ name: Integration Tests
on:
pull_request: # Run on all pull requests
push:
branches: # Run on any push to development or master
branches: # Run on any push to development or main
- development
- master
schedule: # Run daily on development and weekly on master
- main
schedule: # Run daily on development and weekly on main
- cron: 0 1 * * *
branches: development
- cron: 0 2 * * 0
branches: master
branches: main

jobs:
gradle-build: # Gradle integration tests
timeout-minutes: 60
runs-on: ubuntu-20.04 #See pre-installed software at https://github.com/actions/virtual-environments/blob/main/images/linux/Ubuntu2004-Readme.md
env:
MYSQL_PORT: 3800
Expand All @@ -38,6 +39,7 @@ jobs:
"galaxy_testing",
"galaxy_pipeline_testing",
"open_api_testing",
"file_system_testing",
]

steps:
Expand Down Expand Up @@ -137,4 +139,4 @@ jobs:
sleep 1
done
- name: Build Documentation Site
run: ./gradlew clean buildDocsSite -Dliquibase.update.database.schema=true -Dspring.datasource.url=jdbc:mysql://${{env.MYSQL_HOST}}:${{env.MYSQL_PORT}}/${{env.MYSQL_DATABASE}} -Dspring.datasource.dbcp2.max-wait=10000
run: ./gradlew clean buildDocsSite -Dliquibase.update.database.schema=true -Dspring.datasource.url=jdbc:mysql://${{env.MYSQL_HOST}}:${{env.MYSQL_PORT}}/${{env.MYSQL_DATABASE}} -Dspring.datasource.dbcp2.max-wait=10000
2 changes: 1 addition & 1 deletion .github/workflows/codeql-analysis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ name: "CodeQL"

on:
push:
branches: [ development, master ]
branches: [ development, main ]
pull_request:
# The branches below must be a subset of the branches above
branches: [ development ]
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/grype-scan.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,14 @@ name: Grype WAR security scan
on:
pull_request: # Run on all pull requests
push:
branches: # Run on any push to development or master
branches: # Run on any push to development or main
- development
- master
schedule: # Run daily on development and weekly on master
- main
schedule: # Run daily on development and weekly on main
- cron: 0 1 * * *
branches: development
- cron: 0 2 * * 0
branches: master
branches: main

jobs:
war-build: # Gradle integration tests
Expand Down
66 changes: 66 additions & 0 deletions .github/workflows/linting.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
name: Linting

on:
pull_request: # Run on all pull requests
push:
branches: # Run on any push to development or main
- development
- main

jobs:
eslint: # Lint changed JS/TS files
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout@v3 #Checkout the project from git
- name: Get pnpm store directory
id: pnpm-cache
run: |
echo "::set-output name=pnpm_cache_dir::$(./gradlew pnpmCacheDir -q)"
- name: Setup pnpm cache
uses: actions/cache@v3
with:
path: ${{ steps.pnpm-cache.outputs.pnpm_cache_dir }}
key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }}
restore-keys: |
${{ runner.os }}-pnpm-store-
- name: Set up JDK 11 # Installs java 11
uses: actions/setup-java@v1
with:
java-version: 11
- name: Validate Gradle wrapper
uses: gradle/wrapper-validation-action@v1
- name: Setup Gradle
uses: gradle/gradle-build-action@v2
- name: Install dependencies
run: ./gradlew pnpmInstall
- name: Run eslint on changed files
uses: tj-actions/eslint-changed-files@v18
with:
config_path: "src/main/webapp/.eslintrc.js"
extra_args: "--max-warnings=0"
file_extensions: |
src/main/webapp/resources/js/**/*.ts
src/main/webapp/resources/js/**/*.tsx
src/main/webapp/resources/js/**/*.js
src/main/webapp/resources/js/**/*.jsx
checkstyle: # Lint changed Java files
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout@v3 #Checkout the project from git
- name: Set up JDK 11 # Installs java 11
uses: actions/setup-java@v1
with:
java-version: 11
- name: Validate Gradle wrapper
uses: gradle/wrapper-validation-action@v1
- name: Setup Gradle
uses: gradle/gradle-build-action@v2
- name: Run checkstyle
uses: dbelyaev/[email protected]
with:
github_token: ${{ secrets.github_token }}
reporter: github-pr-review
fail_on_error: true
level: error
checkstyle_config: './checkstyle.xml'
4 changes: 3 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -55,4 +55,6 @@ src/main/webapp/node_modules
### MISC ###
/.theia/launch.json
*.swp
/src/main/resources/application-local.properties
/src/main/resources/application-local.properties
java_pid*.hprof
gradle.properties
22 changes: 21 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,24 @@

# Changelog

## [22.09.7] - 2022/01/24
## [23.01] - 2023/02/28
* [UI/Developer]: Updated `react-router` to the version 6.4.3. See[PR 1405](https://github.com/phac-nml/irida/pull/1405)
* [Developer] Updated developer setup documentation, ignore java_pid\*.hprof files, and added quality of life file `gradle.properties`. See [PR 1415](https://github.com/phac-nml/irida/pull/1415).
* [Documentation]: Updated broken links in developer documentation. See [PR 1418](https://github.com/phac-nml/irida/pull/1418).
* [Developer] Updated developer setup documentation, ignore java_pid*.hprof files, and added quality of life file `gradle.properties`
* [UI]: Refreshed global search page to use Ant Design. See [PR 1409](https://github.com/phac-nml/irida/pull/1409)
* [UI/Developer]: Removed old notification system hack and updated to use only Ant Design notifications. See [PR 1447](https://github.com/phac-nml/irida/pull/1447)
* [UI]: Fixed bug where the `User` column was named `User Group` on the admin User Groups page. [See PR 1450](https://github.com/phac-nml/irida/pull/1450)
* [Developer]: Replaced Apache OLTU with Nimbusds for performing OAuth2 authentication flow during syncing and Galaxy exporting. See [PR 1432](https://github.com/phac-nml/irida/pull/1432)
* [Developer/UI]: Performance enhancements to the metadata uploader. See [PR 1445](https://github.com/phac-nml/irida/pull/1445).
* [Developer/UI]: Fix for updating sample modified date when metadata is deleted. See [PR 1457](https://github.com/phac-nml/irida/pull/1457).
* [Developer]: Added support for cloud based storage. Currently, Microsoft Azure Blob and Amazon AWS S3 are supported. [See PR 1194](https://github.com/phac-nml/irida/pull/1194)
* [Developer]: Updated metadata uploader to use react-router-dom Outlet. [See PR 1464](https://github.com/phac-nml/irida/pull/1464)
* [Developer]: Deprecated "/api/projects/{projectId}/samples/bySequencerId/{seqeuncerId}" in favour of "/api/projects/{projectId}/samples/bySampleName", which accepts a json property "sampleName"
* [Developer]: Fixed bug in setting a `default_sequencing_object and default_genome_assembly to `NULL` for a sample when the default sequencing object or genome assembly were removed. [See PR 1466](https://github.com/phac-nml/irida/pull/1466)
* [Developer]: Fixed bug preventing a `sample` with an analysis submission from being deleted. [See PR 1467](https://github.com/phac-nml/irida/pull/1467)

## [22.09.7] - 2023/01/24
* [UI]: Fixed bugs on NCBI Export page preventing the NCBI `submission.xml` file from being properly written. See [PR 1451](https://github.com/phac-nml/irida/pull/1451)

## [22.09.6] - 2022/12/21
Expand Down Expand Up @@ -159,6 +176,9 @@

## [...previous](https://github.com/phac-nml/irida/blob/21.09.2/CHANGELOG.md)

[Unreleased]: https://github.com/phac-nml/irida/compare/23.01...HEAD

[23.01]: https://github.com/phac-nml/irida/compare/22.09.7...23.01
[22.09.7]: https://github.com/phac-nml/irida/compare/22.09.6...22.09.7
[22.09.6]: https://github.com/phac-nml/irida/compare/22.09.5...22.09.6
[22.09.5]: https://github.com/phac-nml/irida/compare/22.09.4...22.09.5
Expand Down
35 changes: 26 additions & 9 deletions build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ plugins {
}

group = "ca.corefacility.bioinformatics"
version = "22.09.7"
version = "23.01"
description = "irida"

java {
Expand Down Expand Up @@ -123,9 +123,7 @@ dependencies {
implementation("org.springframework.boot:spring-boot-starter-security")
implementation("org.springframework.security:spring-security-oauth2-authorization-server:0.3.1")
implementation("org.springframework.security:spring-security-oauth2-resource-server:5.7.3")
implementation("org.apache.oltu.oauth2:org.apache.oltu.oauth2.client:1.0.0") {
exclude(group = "org.slf4j")
}
implementation("com.nimbusds:oauth2-oidc-sdk:10.1")
implementation("org.springframework.boot:spring-boot-starter-data-jpa")
implementation("org.springframework.data:spring-data-envers") {
exclude(group = "org.slf4j")
Expand Down Expand Up @@ -176,6 +174,17 @@ dependencies {
exclude(group = "jakarta.validation", module = "jakarta.validation-api")
}

// Microsoft Azure
implementation("com.azure:azure-storage-blob:12.18.0") {
exclude(group = "jakarta.xml.bind", module = "jakarta.xml.bind-api")
exclude(group = "jakarta.activation", module = "jakarta.activation-api")
}

// Amazon AWS
implementation("com.amazonaws:aws-java-sdk-s3:1.12.326") {
exclude(group = "commons-logging", module = "commons-logging")
}

// Customized fastqc
implementation(files("${projectDir}/lib/jbzip2-0.9.jar"))
implementation(files("${projectDir}/lib/sam-1.103.jar"))
Expand Down Expand Up @@ -304,15 +313,19 @@ tasks.register<PnpmTask>("cleanWebapp") {
}

tasks.register<PnpmTask>("buildWebapp") {
dependsOn(":cleanWebapp")
pnpmCommand.set(listOf("build"))
inputs.dir("${project.projectDir}/src/main/webapp/resources")
inputs.files(fileTree("${project.projectDir}/src/main/webapp/resources"))
inputs.file("${project.projectDir}/src/main/webapp/package.json")
inputs.file("${project.projectDir}/src/main/webapp/pnpm-lock.yaml")

outputs.dir("${project.projectDir}/src/main/webapp/dist")

dependsOn(":pnpmInstall")
pnpmCommand.set(listOf("clean", "build"))
}

tasks.register<PnpmTask>("startWebapp") {
dependsOn(":cleanWebapp")
pnpmCommand.set(listOf("start"))
dependsOn(":pnpmInstall")
pnpmCommand.set(listOf("clean", "start"))
inputs.dir("${project.projectDir}/src/main/webapp/resources")
outputs.dir("${project.projectDir}/src/main/webapp/dist")
}
Expand Down Expand Up @@ -399,6 +412,10 @@ val integrationTestsMap = mapOf(
"tags" to "IntegrationTest & Galaxy & Pipeline",
"excludeListeners" to "ca.corefacility.bioinformatics.irida.junit5.listeners.*"
),
"fileSystem" to mapOf(
"tags" to "IntegrationTest & FileSystem",
"excludeListeners" to "ca.corefacility.bioinformatics.irida.junit5.listeners.*"
),
)

integrationTestsMap.forEach {
Expand Down
85 changes: 73 additions & 12 deletions checkstyle.xml
Original file line number Diff line number Diff line change
@@ -1,22 +1,83 @@
<?xml version="1.0"?>
<!DOCTYPE module PUBLIC
"-//Puppy Crawl//DTD Check Configuration 1.2//EN"
"http://www.puppycrawl.com/dtds/configuration_1_2.dtd">

<!DOCTYPE module PUBLIC "-//Checkstyle//DTD Checkstyle Configuration 1.3//EN" "https://checkstyle.org/dtds/configuration_1_3.dtd">
<module name="Checker">
<property name="severity" value="error" />

<property name="fileExtensions" value="java, properties, xml" />

<module name="TreeWalker">
<module name="EmptyBlock" />
<module name="RegexpSinglelineJava">
<property name="format" value="^\t* " />
<property name="message" value="Indent must use tab characters" />
<property name="ignoreComments" value="true" />
</module>

<!-- Checks for Javadoc comments. -->
<!-- See https://checkstyle.org/config_javadoc.html -->
<module name="InvalidJavadocPosition" />
<module name="JavadocMethod">
<property name="scope" value="protected"/>
<property name="allowMissingPropertyJavadoc" value="true"/>
<property name="allowedAnnotations" value="Override, Bean"/>
<property name="tokens" value="METHOD_DEF"/>
<property name="ignoreMethodNamesRegex" value="^main$"/>
<property name="allowedAnnotations" value="Override, Bean" />
<property name="tokens" value="METHOD_DEF" />
</module>
<module name="JavadocType">
<property name="scope" value="protected"/>
<property name="scope" value="protected" />
</module>

<!-- Checks for Naming Conventions. -->
<!-- See https://checkstyle.org/config_naming.html -->
<module name="PackageName" />

<!-- Checks for imports -->
<!-- See https://checkstyle.org/config_imports.html -->
<module name="IllegalImport" /> <!-- defaults to sun.* packages -->
<module name="RedundantImport" />
<module name="UnusedImports">
<property name="processJavadoc" value="true"/>
<property name="processJavadoc" value="true" />
</module>

<!-- Checks for whitespace -->
<!-- See https://checkstyle.org/config_whitespace.html -->
<module name="EmptyForIteratorPad" />
<module name="GenericWhitespace" />
<module name="MethodParamPad" />
<module name="NoWhitespaceBefore" />
<module name="TypecastParenPad" />
<module name="WhitespaceAfter" />

<!-- Modifier Checks -->
<!-- See https://checkstyle.org/config_modifier.html -->
<module name="ModifierOrder" />

<!-- Checks for blocks. You know, those {}'s -->
<!-- See https://checkstyle.org/config_blocks.html -->
<module name="AvoidNestedBlocks" />
<module name="EmptyBlock" />

<!-- Checks for common coding problems -->
<!-- See https://checkstyle.org/config_coding.html -->
<module name="EmptyStatement" />
<module name="IllegalInstantiation" />
<module name="InnerAssignment" />
<module name="SimplifyBooleanExpression" />
<module name="SimplifyBooleanReturn" />

<!-- Checks for class design -->
<!-- See https://checkstyle.org/config_design.html -->
<module name="FinalClass" />
<module name="InterfaceIsType" />

<!-- Miscellaneous other checks. -->
<!-- See https://checkstyle.org/config_misc.html -->
<module name="ArrayTypeStyle" />
<module name="TodoComment" />
<module name="UpperEll" />

<!-- https://checkstyle.org/config_filters.html#SuppressionXpathFilter -->
<module name="SuppressionXpathFilter">
<property name="file" value="${org.checkstyle.sun.suppressionxpathfilter.config}"
default="checkstyle-xpath-suppressions.xml" />
<property name="optional" value="true" />
</module>
</module>
</module>
</module>
22 changes: 22 additions & 0 deletions doc/administrator/galaxy/cleanup/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,3 +66,25 @@ Once this script is installed, it can be scheduled to run periodically by adding
This will clean up any **deleted** files every day at 2:00 am. Log files will be stored in `galaxy/galaxy_cleanup.log` and `galaxy/cleanup_datasets/*.log`.

For more information please see the [Purging Histories and Datasets](https://galaxyproject.org/admin/config/performance/purge-histories-and-datasets/) document. ***Note: the metadata about each analysis will still be stored and available in Galaxy, but the data file contents will be permanently removed.***

# Cleaning up temporary files

When using Galaxy with an IRIDA instance which is using cloud based storage (Azure, AWS, etc) for example, files are uploaded from IRIDA instead of linking to them since the files are stored in the cloud and not on a shared filesystem. Since these files are uploaded to Galaxy it is a good idea to clean these files up. An example script that can be used to clean these files up is provided below:

```bash
#!/bin/bash

GALAXY_ROOT_DIR=/path/to/galaxy-dist
CLEANUP_LOG=$GALAXY_ROOT_DIR/irida_galaxy_tmp_files_cleanup.log
TMP_FILES_DIR=$GALAXY_ROOT_DIR/databases/tmp/
NUMBER_OF_DAYS_OLD=30

source $CONDA_ROOT/bin/activate galaxy

echo -e "\nBegin temporary file cleanup at `date`" >> $CLEANUP_LOG
find $TMP_FILES_DIR -mindepth 1 -mtime +$NUMBER_OF_DAYS_OLD -delete

echo -e "\nEnd temporary file cleanup at `date`" >> $CLEANUP_LOG
```

This can be added as a cleanup script which can be scheduled to run using cron.
4 changes: 2 additions & 2 deletions doc/administrator/galaxy/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,9 +59,9 @@ The overall architecture of IRIDA and Galaxy is as follows:

![irida-galaxy.jpg][]

1. IRIDA manages all input files for a workflow. This includes sequencing reads, reference files, and the Galaxy workflow definition file. On execution of a workflow, references to these files are sent to a Galaxy instance using the [Galaxy API][]. It is assumed that these files exist on a file system shared between IRIDA and Galaxy.
1. IRIDA manages all input files for a workflow. This includes sequencing reads, reference files, and the Galaxy workflow definition file. On execution of a workflow, if using cloud based storage the files are uploaded to a Galaxy instance, otherwise references to these files are sent to a Galaxy instance, using the [Galaxy API][]. If using IRIDA with cloud based storage (Azure, AWS, etc) the files will be downloaded to the IRIDA server, then uploaded to Galaxy, otherwise it is assumed that these files exist on a file system shared between IRIDA and Galaxy.
2. All tools used by a workflow are assumed to have been installed in Galaxy during the setup of IRIDA. The Galaxy workflow is uploaded to Galaxy and the necessary tools are executed by Galaxy. Galaxy can be setup to either execute tools on a local machine, or submit jobs to a cluster.
3. Once the workflow execution is complete, a copy of the results are downloaded into IRIDA and stored in the shared filesystem.
3. Once the workflow execution is complete, a copy of the results are downloaded into IRIDA and stored in the shared filesystem or uploaded to the cloud based storage being used by IRIDA.

[Docker]: https://www.docker.com/
[irida-galaxy.jpg]: images/irida-galaxy.jpg
Expand Down
2 changes: 1 addition & 1 deletion doc/administrator/galaxy/setup/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ This document describes the necessary steps for installing and integrating [Gala
The following must be set up before proceeding with the installation.

1. A machine that has been set up to install Galaxy. This could be the same machine as the IRIDA web interface, or (recommended) a separate machine.
2. A shared filesystem has been set up between IRIDA and Galaxy. If Galaxy will be submitting to a compute cluster this filesystem must also be shared with the cluster.
2. A shared filesystem has been set up between IRIDA and Galaxy if using a local filesystem and not cloud based storage. If Galaxy will be submitting to a compute cluster this filesystem must also be shared with the cluster.

* this comment becomes the table of contents.
{:toc}
Expand Down
Loading

0 comments on commit 661ed89

Please sign in to comment.