Downloading: s3://natera-rnd-fsdx-dev-nextflow-scratch-01/work/57/468548ec591867da26dc364c29651e/nf_gui_test_three.1.fastp.fq.gz Downloading: s3://fsdx-algo-644019535899-us-west-2-an/references/hg38.fa Downloading: s3://natera-rnd-fsdx-dev-nextflow-scratch-01/work/90/f1d6080fa9eb9eece2cee0f5c7e792/.command.sh Downloading: s3://fsdx-algo-644019535899-us-west-2-an/references/hg38.fa.0123 Downloading: s3://fsdx-algo-644019535899-us-west-2-an/references/hg38.fa.fai Downloading: s3://fsdx-algo-644019535899-us-west-2-an/references/hg38.fa.bwt.2bit.64 Downloading: s3://fsdx-algo-644019535899-us-west-2-an/references/hg38.fa.pac Downloading: s3://fsdx-algo-644019535899-us-west-2-an/references/hg38.fa.ann Downloading: s3://natera-rnd-fsdx-dev-nextflow-scratch-01/work/90/f1d6080fa9eb9eece2cee0f5c7e792/.command.run Downloading: s3://natera-rnd-fsdx-dev-nextflow-scratch-01/work/57/468548ec591867da26dc364c29651e/nf_gui_test_three.2.fastp.fq.gz Downloading: s3://fsdx-algo-644019535899-us-west-2-an/references/hg38.fa.amb ==> STAGING COMPLETE (11 inputs) Looking to launch executable "/opt/conda/bin/bwa-mem2.avx512bw", simd = .avx512bw Launching executable "/opt/conda/bin/bwa-mem2.avx512bw" ----------------------------- Executing in AVX512 mode!! ----------------------------- * SA compression enabled with xfactor: 8 * Ref file: hg38.fa * Entering FMI_search * Index file found. Loading index from hg38.fa.bwt.2bit.64 * Reference seq len for bi-index = 6418572211 * sentinel-index: 2729492284 * Count: 0, 1 1, 1879238230 2, 3209286106 3, 4539333982 4, 6418572211 * Reading other elements of the index from files hg38.fa * Index prefix: hg38.fa * Read 0 ALT contigs * Done reading Index!! * Reading reference genome.. * Binary seq file = hg38.fa.0123 * Reference genome size: 6418572210 bp * Done reading reference genome !! ------------------------------------------ 1. Memory pre-allocation for Chaining: 696.7080 MB 2. Memory pre-allocation for BSW: 958.4681 MB 3. Memory pre-allocation for BWT: 309.2567 MB ------------------------------------------ * Threads used (compute): 4 * No. of pipeline threads: 2 [0000] read_chunk: 50000000, work_chunk_size: 50000224, nseq: 380990 [0000][ M::kt_pipeline] read 380990 sequences (50000224 bp)... [0000] Reallocating initial memory allocations!! [0000] Calling mem_process_seqs.., task: 0 [0000] 1. Calling kt_for - worker_bwt [0000] read_chunk: 50000000, work_chunk_size: 50000039, nseq: 381094 [0000][ M::kt_pipeline] read 381094 sequences (50000039 bp)... [0000] 2. Calling kt_for - worker_aln [0000] Inferring insert size distribution of PE reads from data, l_pac: 3209286105, n: 380990 [0000][PE] # candidate unique pairs for (FF, FR, RF, RR): (0, 189493, 0, 0) [0000][PE] skip orientation FF as there are not enough pairs [0000][PE] analyzing insert size distribution for orientation FR... [0000][PE] (25, 50, 75) percentile: (135, 169, 203) [0000][PE] low and high boundaries for computing mean and std.dev: (1, 339) [0000][PE] mean and std.dev: (168.85, 49.14) [0000][PE] low and high boundaries for proper pairs: (1, 407) [0000][PE] skip orientation RF as there are not enough pairs [0000][PE] skip orientation RR as there are not enough pairs [0000] 3. Calling kt_for - worker_sam [0000][ M::mem_process_seqs] Processed 380990 reads in 32.050 CPU sec, 7.937 real sec [0000] Reallocating initial memory allocations!! [W::sam_hrecs_update_hashes] PG line with multiple ID tags. The first encountered was preferred - ID:bwa-mem2 [0000] Calling mem_process_seqs.., task: 1 [0000] 1. Calling kt_for - worker_bwt [0000] read_chunk: 50000000, work_chunk_size: 50000276, nseq: 381426 [0000][ M::kt_pipeline] read 381426 sequences (50000276 bp)... [0000] 2. Calling kt_for - worker_aln [0000] Inferring insert size distribution of PE reads from data, l_pac: 3209286105, n: 381094 [0000][PE] # candidate unique pairs for (FF, FR, RF, RR): (0, 189660, 0, 0) [0000][PE] skip orientation FF as there are not enough pairs [0000][PE] analyzing insert size distribution for orientation FR... [0000][PE] (25, 50, 75) percentile: (134, 169, 202) [0000][PE] low and high boundaries for computing mean and std.dev: (1, 338) [0000][PE] mean and std.dev: (168.88, 49.53) [0000][PE] low and high boundaries for proper pairs: (1, 406) [0000][PE] skip orientation RF as there are not enough pairs [0000][PE] skip orientation RR as there are not enough pairs [0000] 3. Calling kt_for - worker_sam [0000][ M::mem_process_seqs] Processed 381094 reads in 32.905 CPU sec, 8.080 real sec [0000] Reallocating initial memory allocations!! [0000] Calling mem_process_seqs.., task: 2 [0000] 1. Calling kt_for - worker_bwt [0000] read_chunk: 50000000, work_chunk_size: 50000158, nseq: 380668 [0000][ M::kt_pipeline] read 380668 sequences (50000158 bp)... [0000] 2. Calling kt_for - worker_aln [0000] Inferring insert size distribution of PE reads from data, l_pac: 3209286105, n: 381426 [0000][PE] # candidate unique pairs for (FF, FR, RF, RR): (0, 189868, 0, 0) [0000][PE] skip orientation FF as there are not enough pairs [0000][PE] analyzing insert size distribution for orientation FR... [0000][PE] (25, 50, 75) percentile: (134, 168, 202) [0000][PE] low and high boundaries for computing mean and std.dev: (1, 338) [0000][PE] mean and std.dev: (168.43, 49.30) [0000][PE] low and high boundaries for proper pairs: (1, 406) [0000][PE] skip orientation RF as there are not enough pairs [0000][PE] skip orientation RR as there are not enough pairs [0000] 3. Calling kt_for - worker_sam [0000][ M::mem_process_seqs] Processed 381426 reads in 32.254 CPU sec, 7.969 real sec [0000] Calling mem_process_seqs.., task: 3 [0000] 1. Calling kt_for - worker_bwt [0000] read_chunk: 50000000, work_chunk_size: 28728407, nseq: 219430 [0000][ M::kt_pipeline] read 219430 sequences (28728407 bp)... [0000] 2. Calling kt_for - worker_aln [0000] Inferring insert size distribution of PE reads from data, l_pac: 3209286105, n: 380668 [0000][PE] # candidate unique pairs for (FF, FR, RF, RR): (0, 189299, 0, 0) [0000][PE] skip orientation FF as there are not enough pairs [0000][PE] analyzing insert size distribution for orientation FR... [0000][PE] (25, 50, 75) percentile: (136, 169, 203) [0000][PE] low and high boundaries for computing mean and std.dev: (2, 337) [0000][PE] mean and std.dev: (169.57, 49.50) [0000][PE] low and high boundaries for proper pairs: (1, 404) [0000][PE] skip orientation RF as there are not enough pairs [0000][PE] skip orientation RR as there are not enough pairs [0000] 3. Calling kt_for - worker_sam [0000][ M::mem_process_seqs] Processed 380668 reads in 31.881 CPU sec, 7.909 real sec [0000] Calling mem_process_seqs.., task: 4 [0000] 1. Calling kt_for - worker_bwt [0000] read_chunk: 50000000, work_chunk_size: 0, nseq: 0 [0000] 2. Calling kt_for - worker_aln [0000] Inferring insert size distribution of PE reads from data, l_pac: 3209286105, n: 219430 [0000][PE] # candidate unique pairs for (FF, FR, RF, RR): (0, 109171, 0, 0) [0000][PE] skip orientation FF as there are not enough pairs [0000][PE] analyzing insert size distribution for orientation FR... [0000][PE] (25, 50, 75) percentile: (134, 168, 201) [0000][PE] low and high boundaries for computing mean and std.dev: (1, 335) [0000][PE] mean and std.dev: (167.89, 49.18) [0000][PE] low and high boundaries for proper pairs: (1, 402) [0000][PE] skip orientation RF as there are not enough pairs [0000][PE] skip orientation RR as there are not enough pairs [0000] 3. Calling kt_for - worker_sam [0000][ M::mem_process_seqs] Processed 219430 reads in 17.607 CPU sec, 4.393 real sec [0000] read_chunk: 50000000, work_chunk_size: 0, nseq: 0 [0000] Computation ends.. No. of OMP threads: 4 Processor is running @2700.238122 MHz Runtime profile: Time taken for main_mem function: 45.09 sec IO times (sec) : Reading IO time (reads) avg: 2.92, (2.92, 2.92) Writing IO time (SAM) avg: 1.11, (1.11, 1.11) Reading IO time (Reference Genome) avg: 3.00, (3.00, 3.00) Index read time avg: 4.44, (4.44, 4.44) Overall time (sec) (Excluding Index reading time): PROCESS() (Total compute time + (read + SAM) IO time) : 37.23 MEM_PROCESS_SEQ() (Total compute time (Kernel + SAM)), avg: 36.28, (36.28, 36.28) SAM Processing time (sec): --WORKER_SAM avg: 7.44, (7.44, 7.44) Kernels' compute time (sec): Total kernel (smem+sal+bsw) time avg: 28.67, (28.67, 28.67) SMEM compute avg: 19.79, (19.80, 19.78) SAL compute avg: 2.20, (2.22, 2.19) MEM_SA avg: 1.43, (1.44, 1.41) BSW time, avg: 6.25, (6.25, 6.24) Important parameter settings: BATCH_SIZE: 512 MAX_SEQ_LEN_REF: 256 MAX_SEQ_LEN_QER: 128 MAX_SEQ_LEN8: 128 SEEDS_PER_READ: 500 SIMD_WIDTH8 X: 64 SIMD_WIDTH16 X: 32 AVG_SEEDS_PER_READ: 64 [bam_sort_core] merging from 0 files and 4 in-memory blocks...