Downloading: s3://fsdx-algo-644019535899-us-west-2-an/references/hg38.fa
Downloading: s3://natera-rnd-fsdx-dev-nextflow-scratch-01/work/3a/4166b53ccbbc1b57083076e7cd06d2/.command.sh
Downloading: s3://fsdx-algo-644019535899-us-west-2-an/references/hg38.fa.0123
Downloading: s3://natera-rnd-fsdx-dev-nextflow-scratch-01/work/eb/b80a0d01494be53db10c27c8c18c0f/nf_gui_test_one.1.fastp.fq.gz
Downloading: s3://natera-rnd-fsdx-dev-nextflow-scratch-01/work/eb/b80a0d01494be53db10c27c8c18c0f/nf_gui_test_one.2.fastp.fq.gz
Downloading: s3://fsdx-algo-644019535899-us-west-2-an/references/hg38.fa.fai
Downloading: s3://fsdx-algo-644019535899-us-west-2-an/references/hg38.fa.bwt.2bit.64
Downloading: s3://fsdx-algo-644019535899-us-west-2-an/references/hg38.fa.pac
Downloading: s3://fsdx-algo-644019535899-us-west-2-an/references/hg38.fa.ann
Downloading: s3://natera-rnd-fsdx-dev-nextflow-scratch-01/work/3a/4166b53ccbbc1b57083076e7cd06d2/.command.run
Downloading: s3://fsdx-algo-644019535899-us-west-2-an/references/hg38.fa.amb
==> STAGING COMPLETE (11 inputs)
Looking to launch executable "/opt/conda/bin/bwa-mem2.avx512bw", simd = .avx512bw
Launching executable "/opt/conda/bin/bwa-mem2.avx512bw"
-----------------------------
Executing in AVX512 mode!!
-----------------------------
* SA compression enabled with xfactor: 8
* Ref file: hg38.fa
* Entering FMI_search
* Index file found. Loading index from hg38.fa.bwt.2bit.64
* Reference seq len for bi-index = 6418572211
* sentinel-index: 2729492284
* Count:
0, 1
1, 1879238230
2, 3209286106
3, 4539333982
4, 6418572211
* Reading other elements of the index from files hg38.fa
* Index prefix: hg38.fa
* Read 0 ALT contigs
* Done reading Index!!
* Reading reference genome..
* Binary seq file = hg38.fa.0123
* Reference genome size: 6418572210 bp
* Done reading reference genome !!
------------------------------------------
1. Memory pre-allocation for Chaining: 696.7080 MB
2. Memory pre-allocation for BSW: 958.4681 MB
3. Memory pre-allocation for BWT: 309.2567 MB
------------------------------------------
* Threads used (compute): 4
* No. of pipeline threads: 2
[0000] read_chunk: 50000000, work_chunk_size: 50000154, nseq: 379600
[0000][ M::kt_pipeline] read 379600 sequences (50000154 bp)...
[0000] Reallocating initial memory allocations!!
[0000] Calling mem_process_seqs.., task: 0
[0000] 1. Calling kt_for - worker_bwt
[0000] read_chunk: 50000000, work_chunk_size: 50000038, nseq: 379768
[0000][ M::kt_pipeline] read 379768 sequences (50000038 bp)...
[0000] 2. Calling kt_for - worker_aln
[0000] Inferring insert size distribution of PE reads from data, l_pac: 3209286105, n: 379600
[0000][PE] # candidate unique pairs for (FF, FR, RF, RR): (0, 188812, 0, 0)
[0000][PE] skip orientation FF as there are not enough pairs
[0000][PE] analyzing insert size distribution for orientation FR...
[0000][PE] (25, 50, 75) percentile: (137, 169, 202)
[0000][PE] low and high boundaries for computing mean and std.dev: (7, 332)
[0000][PE] mean and std.dev: (169.59, 48.59)
[0000][PE] low and high boundaries for proper pairs: (1, 397)
[0000][PE] skip orientation RF as there are not enough pairs
[0000][PE] skip orientation RR as there are not enough pairs
[0000] 3. Calling kt_for - worker_sam
[0000][ M::mem_process_seqs] Processed 379600 reads in 30.722 CPU sec, 7.606 real sec
[0000] Reallocating initial memory allocations!!
[W::sam_hrecs_update_hashes] PG line with multiple ID tags. The first encountered was preferred - ID:bwa-mem2
[0000] Calling mem_process_seqs.., task: 1
[0000] 1. Calling kt_for - worker_bwt
[0000] read_chunk: 50000000, work_chunk_size: 50000096, nseq: 380632
[0000][ M::kt_pipeline] read 380632 sequences (50000096 bp)...
[0000] 2. Calling kt_for - worker_aln
[0000] Inferring insert size distribution of PE reads from data, l_pac: 3209286105, n: 379768
[0000][PE] # candidate unique pairs for (FF, FR, RF, RR): (0, 188899, 0, 0)
[0000][PE] skip orientation FF as there are not enough pairs
[0000][PE] analyzing insert size distribution for orientation FR...
[0000][PE] (25, 50, 75) percentile: (136, 169, 203)
[0000][PE] low and high boundaries for computing mean and std.dev: (2, 337)
[0000][PE] mean and std.dev: (169.56, 48.50)
[0000][PE] low and high boundaries for proper pairs: (1, 404)
[0000][PE] skip orientation RF as there are not enough pairs
[0000][PE] skip orientation RR as there are not enough pairs
[0000] 3. Calling kt_for - worker_sam
[0000][ M::mem_process_seqs] Processed 379768 reads in 32.415 CPU sec, 7.963 real sec
[0000] Reallocating initial memory allocations!!
[0000] Calling mem_process_seqs.., task: 2
[0000] 1. Calling kt_for - worker_bwt
[0000] read_chunk: 50000000, work_chunk_size: 50000080, nseq: 381286
[0000][ M::kt_pipeline] read 381286 sequences (50000080 bp)...
[0000] 2. Calling kt_for - worker_aln
[0000] Inferring insert size distribution of PE reads from data, l_pac: 3209286105, n: 380632
[0000][PE] # candidate unique pairs for (FF, FR, RF, RR): (0, 189221, 0, 0)
[0000][PE] skip orientation FF as there are not enough pairs
[0000][PE] analyzing insert size distribution for orientation FR...
[0000][PE] (25, 50, 75) percentile: (135, 169, 202)
[0000][PE] low and high boundaries for computing mean and std.dev: (1, 336)
[0000][PE] mean and std.dev: (169.02, 48.67)
[0000][PE] low and high boundaries for proper pairs: (1, 403)
[0000][PE] skip orientation RF as there are not enough pairs
[0000][PE] skip orientation RR as there are not enough pairs
[0000] 3. Calling kt_for - worker_sam
[0000][ M::mem_process_seqs] Processed 380632 reads in 33.856 CPU sec, 8.310 real sec
[0000] Reallocating initial memory allocations!!
[0000] Calling mem_process_seqs.., task: 3
[0000] 1. Calling kt_for - worker_bwt
[0000] read_chunk: 50000000, work_chunk_size: 31334196, nseq: 238858
[0000][ M::kt_pipeline] read 238858 sequences (31334196 bp)...
[0000] 2. Calling kt_for - worker_aln
[0000] Inferring insert size distribution of PE reads from data, l_pac: 3209286105, n: 381286
[0000][PE] # candidate unique pairs for (FF, FR, RF, RR): (0, 189301, 0, 0)
[0000][PE] skip orientation FF as there are not enough pairs
[0000][PE] analyzing insert size distribution for orientation FR...
[0000][PE] (25, 50, 75) percentile: (135, 167, 202)
[0000][PE] low and high boundaries for computing mean and std.dev: (1, 336)
[0000][PE] mean and std.dev: (168.63, 49.15)
[0000][PE] low and high boundaries for proper pairs: (1, 403)
[0000][PE] skip orientation RF as there are not enough pairs
[0000][PE] skip orientation RR as there are not enough pairs
[0000] 3. Calling kt_for - worker_sam
[0000][ M::mem_process_seqs] Processed 381286 reads in 32.896 CPU sec, 8.144 real sec
[0000] Calling mem_process_seqs.., task: 4
[0000] 1. Calling kt_for - worker_bwt
[0000] read_chunk: 50000000, work_chunk_size: 0, nseq: 0
[0000] 2. Calling kt_for - worker_aln
[0000] Inferring insert size distribution of PE reads from data, l_pac: 3209286105, n: 238858
[0000][PE] # candidate unique pairs for (FF, FR, RF, RR): (0, 118632, 0, 0)
[0000][PE] skip orientation FF as there are not enough pairs
[0000][PE] analyzing insert size distribution for orientation FR...
[0000][PE] (25, 50, 75) percentile: (135, 170, 204)
[0000][PE] low and high boundaries for computing mean and std.dev: (1, 342)
[0000][PE] mean and std.dev: (169.94, 49.54)
[0000][PE] low and high boundaries for proper pairs: (1, 411)
[0000][PE] skip orientation RF as there are not enough pairs
[0000][PE] skip orientation RR as there are not enough pairs
[0000] 3. Calling kt_for - worker_sam
[0000][ M::mem_process_seqs] Processed 238858 reads in 19.937 CPU sec, 4.973 real sec
[0000] read_chunk: 50000000, work_chunk_size: 0, nseq: 0
[0000] Computation ends..
No. of OMP threads: 4
Processor is running @2700.246754 MHz
Runtime profile:
Time taken for main_mem function: 45.74 sec
IO times (sec) :
Reading IO time (reads) avg: 2.99, (2.99, 2.99)
Writing IO time (SAM) avg: 1.13, (1.13, 1.13)
Reading IO time (Reference Genome) avg: 2.95, (2.95, 2.95)
Index read time avg: 4.42, (4.42, 4.42)
Overall time (sec) (Excluding Index reading time):
PROCESS() (Total compute time + (read + SAM) IO time) : 37.95
MEM_PROCESS_SEQ() (Total compute time (Kernel + SAM)), avg: 36.99, (36.99, 36.99)
SAM Processing time (sec):
--WORKER_SAM avg: 7.81, (7.81, 7.81)
Kernels' compute time (sec):
Total kernel (smem+sal+bsw) time avg: 29.04, (29.04, 29.04)
SMEM compute avg: 20.06, (20.07, 20.04)
SAL compute avg: 2.24, (2.25, 2.23)
MEM_SA avg: 1.43, (1.45, 1.42)
BSW time, avg: 6.30, (6.31, 6.30)
Important parameter settings:
BATCH_SIZE: 512
MAX_SEQ_LEN_REF: 256
MAX_SEQ_LEN_QER: 128
MAX_SEQ_LEN8: 128
SEEDS_PER_READ: 500
SIMD_WIDTH8 X: 64
SIMD_WIDTH16 X: 32
AVG_SEEDS_PER_READ: 64
[bam_sort_core] merging from 0 files and 4 in-memory blocks...