CPB Wiki - User contributions [en]

Computing Resources

2025-07-18T19:12:33Z

Giusti:

{| class="wikitable" style="margin:auto"
|+ CPB Storage Requirements in TB (terabytes)
|-
! Type of Data !! Project !! Lab !! 2021 || 2022 || 2023 || 2024 || 2025
|-
| Raw MRD (MiSeq) || Routine || Molecular Biology || style="text-align:right;" | || style="text-align:right;" | || style="text-align:right;" | || style="text-align:right;" | || style="text-align:right;" |
|-
| Processed MRD (Vidjil) || Routine || Molecular Biology || style="text-align:right;" |0.05 || style="text-align:right;" |0.07 || style="text-align:right;" |0.20 || style="text-align:right;" |0.30 || style="text-align:right;" |0.40
|-
| Exome || Precision Medicine || Epigenomics ||
|-
| ctDNA || Circular Tumoral DNA || Molecular Biology ||
|-
| Methylome || Precision Medicine || Epigenomics ||
|-
| Mass spectra || DESI || Leukemia Biology / Pathology || style="text-align:right;" |4 || style="text-align:right;" |- || style="text-align:right;" |- || style="text-align:right;" |- || style="text-align:right;" |-
|-
| Imaging || Drug Screening || Leukemia Biology || || || style="text-align:right;" |15 || style="text-align:right;" |25 ||
|-
| Panel || Drug Screening || Leukemia Biology || || || || style="text-align:right;" |0.4 ||
|-
| Methylome || Drug Screening || Leukemia Biology || || || || style="text-align:right;" |0.03 ||
|-
| RNASeq || Drug Screening || Leukemia Biology || || || || style="text-align:right;" |0.4 ||
|-
| Methylome || FINEP || All || || || || || style="text-align:right;" |0.5
|-
| RNASeq || FINEP || All || || || || || style="text-align:right;" |4
|-
| Exome || FINEP || All || || || || || style="text-align:right;" |8
|-
| Total || || || style="text-align:right;" |4.05 || style="text-align:right;" |0.07 || style="text-align:right;" |15.2 || style="text-align:right;" |26.3 || style="text-align:right;" |12.9
|-
|}

{| class="wikitable" style="margin:auto"
|+ Virtual Machines for Host: Machine 60
|-
! Name !! IP !! RAM (GB) !! Disk (TB) !! Processors
|-
| Vidjil || 200.136.108.242 || 12 || 1.0 || 16
|-
| RNASeq || 10.10.70.114 || 64 || 1.0 || 16
|-
| RStudio || 10.10.70.57 || 65 || 4.0 || 1
|-
| Bioterium || 192.168.200.241 || 12 || 0.1 || 4
|-
| Total || || 224 || 12.0 || 24
|-
|}

{| class="wikitable" style="margin:auto"
|+ CPB Servers
|-
! Name !! Acquired !! IP !! RAM (GB) !! Disk (TB) !! Cores !! Current Use !! Planned, Additional Use
|-
| Pronon201942 || 2023 || 10.10.70.56 || 128 || 33 || 112 || PRONON Epigenomics ||
|-
| Node1 || 2024 || 10.10.70.59 || 256 || 7 || 48 || Migo, NextSeq Join || RNA Panel analysis
|-
| QNAP Storage || 2024 || 10.10.70.60 || 4 || 113 || 4 || Backup ||
|-
| Node2 || 2024 || 10.10.70.61 || 256 || 7 || 48 || Drug Screening Dashboards & Database ||
|-
| Node3 || 2024 || 10.10.70.62 || 256 || 7 || 48 || BRB Seq analysis, VEP ||
|-
| Node4 || 2024 || 10.10.70.63 || 256 || 7 || 48 || Drug Screening Processing || CPB Proxy
|-
| Tirana || 2024 || 10.10.70.64 || 128 || 65 || 80 || Genomic analyses || Wiki, Vidjil Background
|-
| Pesquisa-113 || Long ago || 10.10.50.113 || 4 || 0.5 || 4 || Wiki, MySQL ||
|-
| CPB (oem-desktop) || Loan || 200.136.108.111 || 4 || 0.5 || 4 || CPB Reverse Proxy || Will be returned
|-
| Caragua || 2019 || 200.136.108.250 || 16 || 1 || 8 || REDCap ||
|-
|}

Making entire 7 TB disk available on node1-4:

sudo lvm

lvm> vgs
VG #PV #LV #SN Attr VSize VFree
fedora 1 1 0 wz--n- 6.98t 6.97t

lvm> lvcreate -L 6.97t fedora
Rounding up size to full physical extent 6.97 TiB
Logical volume "lvol0" created.

Quit lvm

sudo mkdir -vp /data

sudo mkfs.ext4 /dev/fedora/lvol0
mke2fs 1.47.0 (5-Feb-2023)
Discarding device blocks: done
Creating filesystem with 1870995456 4k blocks and 233877504 inodes
Filesystem UUID: ac2a59ab-e038-4013-aeb6-bb3889b6720e
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000, 214990848, 512000000, 550731776, 644972544
Allocating group tables: done
Writing inode tables: done
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done

Add the following line to /etc/fstab

/dev/fedora/lvol0 /data ext4 defaults 0 2

Test with

sudo mount -a

Services Backup

2024-01-31T14:56:07Z

Giusti:

{| class="wikitable" style="margin:auto"
|+ Summary of Backup Information for Services
|-
! Service !! What to Backup !! Location !! Periodicity !! Responsible
|-
| Vidjil || [[Vidjil backup]] || VM (242) in server 60 || Weekly on Mondays || Julia
|-
| Wiki || SQLite database || Server 113 || ||
|-
| CEUA Migo || SQLite database || Server 113 || ||
|-
| Drug Screening DB || MySQL database || Server 113 || ||
|-
| ELDA || Post-processing plate files || Server 110 || ||
|}

Services Backup

2024-01-31T14:13:56Z

Giusti: Creating Services Backup

Main Page

2024-01-31T13:49:27Z

Giusti: Link to services backup page

CPB Wiki

[[CPB Meetings]]

[[CPB Meetings 2023]]

[[Vidjil]]

[[MiXCR for RNA-Seq and WES]]

[[Services Backup]]

Restarting Vidjil

2023-12-06T12:37:10Z

Giusti: formatting

Sometimes a Vidjil instance needs to be restarted, such as when the application is down or when you need to reload it for some reason. Another important aspect of performing a restart is whether or not the Vidjil instance in question is the production server. As such, below are presented two sets of instructions on how to perform this task, one for the production server and one for test servers in general.

== Restarting the production Vidjil instance ==
* send an email to the users to let them know Vidjil will be undergoing a brief maintenance. User emails can be obtained by going to the users tab in the web application (https://vidjil.boldrini.org.br/).
* access via <code>ssh</code> the Vidjil production server (<code>vidjil.boldrini.org.br</code>).
* go to the Vidjil Docker directory:
<blockquote>
<code>cd ~vidjil/code/2020-vidjil/docker</code>
</blockquote>
* check the status of the Docker containers. Vidjil is composed by 5 containers, and when the application is down, usually at least one of the containers will also be down.
<blockquote>
<code>docker-compose ps</code>
</blockquote>
* shut down the 5 containers:
<blockquote>
<code>docker-compose -f docker-compose-wrapper.yml down</code>
</blockquote>
* after the shut down is finished, restart the containers:
<blockquote>
<code>docker-compose -f docker-compose-wrapper.yml up -d</code>
</blockquote>
* after the containers finish restarting, check if the service is indeed back online both by running <code>docker-compose ps</code> (all containers must be up) and by accessing the web application.
* send another email to the users, letting them now that the maintenance is finished and that Vidjil is back online.

== Restarting a generic Vidjil instance ==
* access your Vidjil server.
* go to the Vidjil Docker directory.
<blockquote>
<code>cd <path/to/vidjil>/docker</code>
</blockquote>
* check the status of the Docker containers. Vidjil is composed by 5 containers, and when the application is down, usually at least one of the containers will also be down.
<blockquote>
<code>docker-compose ps</code>
</blockquote>
* shut down the 5 containers:
<blockquote>
<code>docker-compose -f docker-compose-wrapper.yml down</code>
</blockquote>
* after the shut down is finished, restart the containers:
<blockquote>
<code>docker-compose -f docker-compose-wrapper.yml up -d</code>
</blockquote>
* after the containers finish restarting, check if the service is indeed back online both by running <code>docker-compose ps</docker> (all containers must be up) and by accessing the web application.

Restarting Vidjil

2023-12-06T12:31:48Z

Giusti: fix code tag

Sometimes a Vidjil instance needs to be restarted, such as when the application is down or when you need to reload it for some reason. Another important aspect of performing a restart is whether or not the Vidjil instance in question is the production server. As such, below are presented two sets of instructions on how to perform this task, one for the production server and one for test servers in general.

To restart the production Vidjil instance (https://vidjil.boldrini.org.br/):
* send an email to the users to let them know Vidjil will be undergoing a brief maintenance. User emails can be obtained by going to the users tab in the web application.
* access via <code>ssh</code> the Vidjil production server (<code>vidjil.boldrini.org.br</code>).
* go to the Vidjil Docker directory:
<code>cd ~vidjil/code/2020-vidjil/docker</code>
* check the status of the Docker containers. Vidjil is composed by 5 containers, and when the application is down, usually at least one of the containers will also be down.
<code>docker-compose ps</code>
* shut down the 5 containers:
<code>docker-compose -f docker-compose-wrapper.yml down</code>
* after the shut down is finished, restart the containers:
<code>docker-compose -f docker-compose-wrapper.yml up -d</code>
* after the containers finish restarting, check if the service is indeed back online both by running <code>docker-compose ps</code> (all containers must be up) and by accessing the web application.
* send another email to the users, letting them now that the maintenance is finished and that Vidjil is back online.

To restart a generic Vidjil instance:
* access your Vidjil server.
* go to the Vidjil Docker directory.
<code>cd <path/to/vidjil>/docker</code>
* check the status of the Docker containers. Vidjil is composed by 5 containers, and when the application is down, usually at least one of the containers will also be down.
<code>docker-compose ps</code>
* shut down the 5 containers:
<code>docker-compose -f docker-compose-wrapper.yml down</code>
* after the shut down is finished, restart the containers:
<code>docker-compose -f docker-compose-wrapper.yml up -d</code>
* after the containers finish restarting, check if the service is indeed back online both by running <code>docker-compose ps</docker> (all containers must be up) and by accessing the web application.

Restarting Vidjil

2023-12-06T12:30:02Z

Giusti: no blockquote

Sometimes a Vidjil instance needs to be restarted, such as when the application is down or when you need to reload it for some reason. Another important aspect of performing a restart is whether or not the Vidjil instance in question is the production server. As such, below are presented two sets of instructions on how to perform this task, one for the production server and one for test servers in general.

To restart the production Vidjil instance (https://vidjil.boldrini.org.br/):
* send an email to the users to let them know Vidjil will be undergoing a brief maintenance. User emails can be obtained by going to the users tab in the web application.
* access via <code>ssh</code> the Vidjil production server (<code>vidjil.boldrini.org.br</code>).
* go to the Vidjil Docker directory:
<code>cd ~vidjil/code/2020-vidjil/docker</code>
* check the status of the Docker containers. Vidjil is composed by 5 containers, and when the application is down, usually at least one of the containers will also be down.
<code>docker-compose ps</code>
* shut down the 5 containers:
<code>docker-compose -f docker-compose-wrapper.yml down</code>
* after the shut down is finished, restart the containers:
<code>docker-compose -f docker-compose-wrapper.yml up -d</code>
* after the containers finish restarting, check if the service is indeed back online both by running <code>docker-compose ps</docker> (all containers must be up) and by accessing the web application.
* send another email to the users, letting them now that the maintenance is finished and that Vidjil is back online.

To restart a generic Vidjil instance:
* access your Vidjil server.
* go to the Vidjil Docker directory.
<code>cd <path/to/vidjil>/docker</code>
* check the status of the Docker containers. Vidjil is composed by 5 containers, and when the application is down, usually at least one of the containers will also be down.
<code>docker-compose ps</code>
* shut down the 5 containers:
<code>docker-compose -f docker-compose-wrapper.yml down</code>
* after the shut down is finished, restart the containers:
<code>docker-compose -f docker-compose-wrapper.yml up -d</code>
* after the containers finish restarting, check if the service is indeed back online both by running <code>docker-compose ps</docker> (all containers must be up) and by accessing the web application.

Restarting Vidjil

2023-12-06T12:19:42Z

Giusti: vidjil restart first commit

Sometimes a Vidjil instance needs to be restarted, such as when the application is down or when you need to reload it for some reason. Another important aspect of performing a restart is whether or not the Vidjil instance in question is the production server. As such, below are presented two sets of instructions on how to perform this task, one for the production server and one for test servers in general.

To restart the production Vidjil instance (https://vidjil.boldrini.org.br/):
* send an email to the users to let them know Vidjil will be undergoing a brief maintenance. User emails can be obtained by going to the users tab in the web application.
* access via <code>ssh</code> the Vidjil production server (<code>vidjil.boldrini.org.br</code>).
* go to the Vidjil Docker directory:
<blockquote>
<code>cd ~vidjil/code/2020-vidjil/docker</code>
</blockquote>
* check the status of the Docker containers. Vidjil is composed by 5 containers, and when the application is down, usually at least one of the containers will also be down.
<blockquote>
<code>docker-compose ps</code>
</blockquote>
* shut down the 5 containers:
<blockquote>
<code>docker-compose -f docker-compose-wrapper.yml down</code>
</blockquote>
* after the shut down is finished, restart the containers:
<blockquote>
<code>docker-compose -f docker-compose-wrapper.yml up -d</code>
</blockquote>
* after the containers finish restarting, check if the service is indeed back online both by running <code>docker-compose ps</docker> (all containers must be up) and by accessing the web application.
* send another email to the users, letting them now that the maintenance is finished and that Vidjil is back online.

To restart a generic Vidjil instance:
* access your Vidjil server.
* go to the Vidjil Docker directory.
<blockquote>
<code>cd <path/to/vidjil>/docker</code>
</blockquote>
* check the status of the Docker containers. Vidjil is composed by 5 containers, and when the application is down, usually at least one of the containers will also be down.
<blockquote>
<code>docker-compose ps</code>
</blockquote>
* shut down the 5 containers:
<blockquote>
<code>docker-compose -f docker-compose-wrapper.yml down</code>
</blockquote>
* after the shut down is finished, restart the containers:
<blockquote>
<code>docker-compose -f docker-compose-wrapper.yml up -d</code>
</blockquote>
* after the containers finish restarting, check if the service is indeed back online both by running <code>docker-compose ps</docker> (all containers must be up) and by accessing the web application.

Vidjil

2023-12-06T11:26:23Z

Giusti:

Vidjil is a software developed in Lille, France devoted to the analysis of V(D)J rearrangements. It is freely available and we have installed a production version and a number of test versions at the Boldrini Research Center (BRC). The production instance runs in a dedicated server.

Maintenance tasks:

* [[Vidjil backup|perform backup]]
* [[Vidjil backup|restore backup]]
* [[Restarting Vidjil|restart Vidjil]]
* renew SSL certificate (Let's Encrypt)
* keep it up-to-date with the Lille version
* develop specific addons for our own use

To renew the certificates:

* this operation must be done between 1 and 8 days before expiration.
* certified machine name: <code>vidjil.boldrini.org.br</code>
* go to the <docker-dir> and impersonate a super-user: <code>sudo su joao.m</code> (<docker-dir> is currently <code>/home/vidjil/code/2020-vidjil/docker</code>).
* do <code>docker-compose -f docker-compose-wrapper.yml down</code> to prevent Vidjil from disturbing standalone verification.
* renew the certificate:

<blockquote>
<code>sudo certbot certonly --standalone -d vidjil.boldrini.org.br</code>
</blockquote>

* issue the extra commands below to make sure the renewal is seen by Vidjil:

<blockquote>
<code>sudo cp /etc/letsencrypt/live/vidjil.boldrini.org.br/fullchain.pem /home/vidjil/code/2020-vidjil/docker/vidjil-client/ssl/fullchain.pem</code>

<code>sudo cp /etc/letsencrypt/live/vidjil.boldrini.org.br/privkey.pem /home/vidjil/code/2020-vidjil/docker/vidjil-client/ssl/privkey.pem</code>
</blockquote>
* make sure file privkey.pem has permissions 0600 and owner root
* do <code>docker-compose -f docker-compose-wrapper.yml up -d</code> to restart Vidjil.
* visit <code>vidjil.boldrini.org.br</code> to confirm the new certificate in on.
* mark on your calendar a date for the next renewal.

MiXCR for RNA-Seq and WES

2023-08-21T15:35:36Z

Giusti: Removing gitlab text

Collection of tools to run MiXCR on shotgun (RNA-Seq and WES) samples and analyze them. Analysis tools for clonality, heavy/light chain ratio (pro-B signature) and VDJ gene usage heatmaps.

Git repository: https://gitlab.com/guilhermenngiusti/mixcr_shotgun

MiXCR for RNA-Seq and WES

2023-08-21T15:33:55Z

Giusti: MiXCR for RNA-Seq and WES

# MiXCR Shotgun Analysis

Collection of tools to run MiXCR on shotgun (RNA-Seq and WES) samples and analyze them. Analysis tools for clonality, heavy/light chain ratio (pro-B signature) and VDJ gene usage heatmaps.

Git repository: https://gitlab.com/guilhermenngiusti/mixcr_shotgun

## How to use

### Running MiXCR

MiXCR docker container can be obtained using the following command:

`docker pull milaboratory/mixcr`

MiXCR analysis of RNA-Seq or WES can bu run using the following command:

`docker run -it --rm -v $(pwd)/VOLUME_DIR:/work milaboratory/mixcr:latest mixcr analyze shotgun -s mmu --starting-material MATERIAL R1_FASTQ R2_FASTQ WORK_DIR`

### Clonality analysis

The Python script to perform clonality analysis can be run from the top level directory using the following command:

`python3 tools/clonality.py -i SAMPLE_DIR -o OUTPUT_DIR -m TARGET_MARKER`

Argument description:

- `-i` Directory where MiXCR clonotypes files are stored;
- `-o` Directory for output files;
- `-m` Target marker. Options: `IGH`, `IGK`, `IGL`, `TRA`, `TRB`, `TRD` and `TRG`;
- `-t` (Optional) Translation table used to rename samples. Table must be a `tsv` file with the columns `original_name` and `translated_name`.

Each wedge of the resulting plot will only be labeled if its clonotype presents frequency ≥ 5%.

Gini-Simpson diversity index presented at the center of the plot.

This script also produces a `count.txt` file containing the total number of reads for the analyzed marker.

### Pro-B cell signature analysis

The Python script to perform pro-B cell signature analysis can be run from the top level directory using the following command:

`python3 tools/pro-B_signature.py -i SAMPLE_DIR -O OUTPUT_DIR -n ANALYSIS_TITLE`

Argument description:

- `-i` Directory where MiXCR clonotypes files are stored. Each sample contained in this directory must be stored in its own subdirectory;
- `-o` Directory for output files;
- `-n` Title for the analysis. Used as the name for the `jpg` file generated;
- `-t` (Optional) Translation table used to rename samples. Table must be a `tsv` file with the columns `original_name` and `translated_name`;
- `--yscale` (Optional) Max value for the ordinates axis. Default `10`.

### VDJ genes usage analysis

The Python script to perform VDJ genes usage analysis can be run from the top level directory using the following command:

`python3 tools/vdj_heatmap.py -i SAMPLE_DIRS -o OUTPUT_DIR -m MARKER -g GENE_FAMILY`

Argument description:

- `-i` Directory where MiXCR clonotypes files are stored. Can receive multiple directories;
- `-o` Directory for output files;
- `-m` Target marker. Options: `IGH` or `IGK`;
- `-g` VDJ gene family. Options: `V`, `D` and `J`;
- `--xsize` (Optional) Heatmap width;
- `--ysize` (Optional) Heatmap length;
- `-t` (Optional) Translation table used to rename samples. Table must be a `tsv` file with the columns `original_name` and `translated_name`;
- `-l` (Optional) Draws horizontal lines separating sample groups in heatmap. Integer representing after how many sample the line should be drawn. Can receive multiple values;
- `--linewidth` (Optional) Sets width for horizonal lines separating sample groups in heatmap;
- `-s` (Optional) Sample to be represented on a secondary heatmap. Can receive multiple values. If a translation table is used, the provided sample name should be the translated one;
- `-S` (Optional) Group name for separate samples, used for secondary heatmap output file;
- `--secxsize` (Optional) Secondary heatmap width;
- `--secysize` (Optional) Secondary heatmap length.

The heatmaps generated by this script always represent only one marker (e.g. IGH) and VDJ gene family (e.g. V) at a time.

**This script currently only supports the IGH and IGK markers from mice.** In order to perform analysis on other markers, complete genome coordinate files should be added to `tools/coordinates`, using the same format as the `tsv` files already available. In order to analyze human (or other species) markers, replace the genome coordinate mice files for ones from your species.

Genomic coordinates files can be generated at https://www.ncbi.nlm.nih.gov/datasets/gene/.

## Credits

Created by Guilherme Giusti at Boldrini Research Center, 2023.

Main Page

2023-08-21T15:22:13Z

Giusti:

CPB Wiki

[[CPB Meetings]]

[[Vidjil]]

[[MiXCR for RNA-Seq and WES]]

Main Page

2023-08-21T15:18:29Z

Giusti: MiXCR for RNA-Seq

CPB Wiki

[[CPB Meetings]]

[[Vidjil]]

[[MiXCR for RNA-Seq]]