.command.out - Workdir - NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_GERMLINE_ALL:BAM_VARIANT_CALLING_SENTIEON_HAPLOTYPER_RF:DV_RESCUE (HCC1395_BL;germline_rescue_targets)

File Info

Filename: .command.out
Full Path: s3://natera-rnd-pltf-dev-nextflow-scratch-01/work/fa/50a2c2adad2650974467bb86bc424c/.command.out
Size: 14.6 KB
Task: NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_GERMLINE_ALL:BAM_VARIANT_CALLING_SENTIEON_HAPLOTYPER_RF:DV_RESCUE (HCC1395_BL;germline_rescue_targets)
Attempt
Content

***** Intermediate results will be written to tmp in docker. ****


***** Running the command:*****
time seq 0 3 | parallel -q --halt 2 --line-buffer /opt/deepvariant/bin/make_examples --mode calling --ref "Homo_sapiens_assembly38.fasta" --reads "HCC1395_BL.recalibrated.bam" --examples "tmp/make_examples.tfrecord@4.gz" --channels "insert_size" --gvcf "tmp/gvcf.tfrecord@4.gz" --regions "germline_rescue_targets.bed" --task {}

[W::hts_idx_load3] The index file is older than the data file: HCC1395_BL.recalibrated.bam.bai
I0528 22:22:11.508785 140220478957376 genomics_reader.py:222] Reading HCC1395_BL.recalibrated.bam with NativeSamReader
I0528 22:22:11.527513 140220478957376 make_examples_core.py:301] Task 2/4: Preparing inputs
[W::hts_idx_load3] The index file is older than the data file: HCC1395_BL.recalibrated.bam.bai
I0528 22:22:11.541012 140220478957376 genomics_reader.py:222] Reading HCC1395_BL.recalibrated.bam with NativeSamReader
[W::hts_idx_load3] The index file is older than the data file: HCC1395_BL.recalibrated.bam.bai
I0528 22:22:11.628685 140325694629696 genomics_reader.py:222] Reading HCC1395_BL.recalibrated.bam with NativeSamReader
I0528 22:22:11.645868 140325694629696 make_examples_core.py:301] Task 1/4: Preparing inputs
[W::hts_idx_load3] The index file is older than the data file: HCC1395_BL.recalibrated.bam.bai
I0528 22:22:11.659989 140325694629696 genomics_reader.py:222] Reading HCC1395_BL.recalibrated.bam with NativeSamReader
[W::hts_idx_load3] The index file is older than the data file: HCC1395_BL.recalibrated.bam.bai
I0528 22:22:11.515243 139843678230336 genomics_reader.py:222] Reading HCC1395_BL.recalibrated.bam with NativeSamReader
I0528 22:22:11.532741 139843678230336 make_examples_core.py:301] Task 3/4: Preparing inputs
[W::hts_idx_load3] The index file is older than the data file: HCC1395_BL.recalibrated.bam.bai
I0528 22:22:11.546486 139843678230336 genomics_reader.py:222] Reading HCC1395_BL.recalibrated.bam with NativeSamReader
[W::hts_idx_load3] The index file is older than the data file: HCC1395_BL.recalibrated.bam.bai
I0528 22:22:11.835753 140168426760000 genomics_reader.py:222] Reading HCC1395_BL.recalibrated.bam with NativeSamReader
I0528 22:22:11.852941 140168426760000 make_examples_core.py:301] Task 0/4: Preparing inputs
[W::hts_idx_load3] The index file is older than the data file: HCC1395_BL.recalibrated.bam.bai
I0528 22:22:11.866791 140168426760000 genomics_reader.py:222] Reading HCC1395_BL.recalibrated.bam with NativeSamReader
I0528 22:22:12.042623 140220478957376 make_examples_core.py:301] Task 2/4: Common contigs are ['chr1', 'chr2', 'chr3', 'chr4', 'chr5', 'chr6', 'chr7', 'chr8', 'chr9', 'chr10', 'chr11', 'chr12', 'chr13', 'chr14', 'chr15', 'chr16', 'chr17', 'chr18', 'chr19', 'chr20', 'chr21', 'chr22', 'chrX', 'chrY', 'chrM']
I0528 22:22:12.155968 140325694629696 make_examples_core.py:301] Task 1/4: Common contigs are ['chr1', 'chr2', 'chr3', 'chr4', 'chr5', 'chr6', 'chr7', 'chr8', 'chr9', 'chr10', 'chr11', 'chr12', 'chr13', 'chr14', 'chr15', 'chr16', 'chr17', 'chr18', 'chr19', 'chr20', 'chr21', 'chr22', 'chrX', 'chrY', 'chrM']
I0528 22:22:12.050936 139843678230336 make_examples_core.py:301] Task 3/4: Common contigs are ['chr1', 'chr2', 'chr3', 'chr4', 'chr5', 'chr6', 'chr7', 'chr8', 'chr9', 'chr10', 'chr11', 'chr12', 'chr13', 'chr14', 'chr15', 'chr16', 'chr17', 'chr18', 'chr19', 'chr20', 'chr21', 'chr22', 'chrX', 'chrY', 'chrM']
I0528 22:22:12.360259 140220478957376 genomics_reader.py:222] Reading germline_rescue_targets.bed with NativeBedReader
I0528 22:22:12.362723 140168426760000 make_examples_core.py:301] Task 0/4: Common contigs are ['chr1', 'chr2', 'chr3', 'chr4', 'chr5', 'chr6', 'chr7', 'chr8', 'chr9', 'chr10', 'chr11', 'chr12', 'chr13', 'chr14', 'chr15', 'chr16', 'chr17', 'chr18', 'chr19', 'chr20', 'chr21', 'chr22', 'chrX', 'chrY', 'chrM']
I0528 22:22:12.449129 139843678230336 genomics_reader.py:222] Reading germline_rescue_targets.bed with NativeBedReader
I0528 22:22:12.768704 140220478957376 make_examples_core.py:301] Task 2/4: Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
[W::hts_idx_load3] The index file is older than the data file: HCC1395_BL.recalibrated.bam.bai
I0528 22:22:12.847968 140220478957376 genomics_reader.py:222] Reading HCC1395_BL.recalibrated.bam with NativeSamReader
[W::hts_idx_load3] The index file is older than the data file: HCC1395_BL.recalibrated.bam.bai
I0528 22:22:12.775619 140168426760000 genomics_reader.py:222] Reading germline_rescue_targets.bed with NativeBedReader
I0528 22:22:12.572284 140325694629696 genomics_reader.py:222] Reading germline_rescue_targets.bed with NativeBedReader
I0528 22:22:12.912973 140220478957376 genomics_reader.py:222] Reading HCC1395_BL.recalibrated.bam with NativeSamReader
I0528 22:22:12.914052 140220478957376 make_examples_core.py:301] Task 2/4: Writing gvcf records to tmp/gvcf.tfrecord-00002-of-00004.gz
I0528 22:22:12.914654 140220478957376 make_examples_core.py:301] Task 2/4: Writing examples to tmp/make_examples.tfrecord-00002-of-00004.gz
I0528 22:22:12.914733 140220478957376 make_examples_core.py:301] Task 2/4: Overhead for preparing inputs: 1 seconds
I0528 22:22:12.918418 140220478957376 make_examples_core.py:301] Task 2/4: 0 candidates (0 examples) [0.00s elapsed]
I0528 22:22:13.223295 140168426760000 make_examples_core.py:301] Task 0/4: Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
I0528 22:22:13.011872 140325694629696 make_examples_core.py:301] Task 1/4: Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
[W::hts_idx_load3] The index file is older than the data file: HCC1395_BL.recalibrated.bam.bai
I0528 22:22:13.094909 140325694629696 genomics_reader.py:222] Reading HCC1395_BL.recalibrated.bam with NativeSamReader
[W::hts_idx_load3] The index file is older than the data file: HCC1395_BL.recalibrated.bam.bai
I0528 22:22:13.170313 140325694629696 genomics_reader.py:222] Reading HCC1395_BL.recalibrated.bam with NativeSamReader
I0528 22:22:13.172105 140325694629696 make_examples_core.py:301] Task 1/4: Writing gvcf records to tmp/gvcf.tfrecord-00001-of-00004.gz
I0528 22:22:13.172917 140325694629696 make_examples_core.py:301] Task 1/4: Writing examples to tmp/make_examples.tfrecord-00001-of-00004.gz
I0528 22:22:13.173011 140325694629696 make_examples_core.py:301] Task 1/4: Overhead for preparing inputs: 1 seconds
I0528 22:22:13.177045 140325694629696 make_examples_core.py:301] Task 1/4: 0 candidates (0 examples) [0.00s elapsed]
I0528 22:22:12.916491 139843678230336 make_examples_core.py:301] Task 3/4: Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
[W::hts_idx_load3] The index file is older than the data file: HCC1395_BL.recalibrated.bam.bai
I0528 22:22:13.003350 139843678230336 genomics_reader.py:222] Reading HCC1395_BL.recalibrated.bam with NativeSamReader
[W::hts_idx_load3] The index file is older than the data file: HCC1395_BL.recalibrated.bam.bai
I0528 22:22:13.074184 139843678230336 genomics_reader.py:222] Reading HCC1395_BL.recalibrated.bam with NativeSamReader
I0528 22:22:13.075235 139843678230336 make_examples_core.py:301] Task 3/4: Writing gvcf records to tmp/gvcf.tfrecord-00003-of-00004.gz
I0528 22:22:13.075998 139843678230336 make_examples_core.py:301] Task 3/4: Writing examples to tmp/make_examples.tfrecord-00003-of-00004.gz
I0528 22:22:13.076104 139843678230336 make_examples_core.py:301] Task 3/4: Overhead for preparing inputs: 1 seconds
I0528 22:22:13.079541 139843678230336 make_examples_core.py:301] Task 3/4: 0 candidates (0 examples) [0.00s elapsed]
[W::hts_idx_load3] The index file is older than the data file: HCC1395_BL.recalibrated.bam.bai
I0528 22:22:13.308367 140168426760000 genomics_reader.py:222] Reading HCC1395_BL.recalibrated.bam with NativeSamReader
[W::hts_idx_load3] The index file is older than the data file: HCC1395_BL.recalibrated.bam.bai
I0528 22:22:13.378443 140168426760000 genomics_reader.py:222] Reading HCC1395_BL.recalibrated.bam with NativeSamReader
I0528 22:22:13.379496 140168426760000 make_examples_core.py:301] Task 0/4: Writing gvcf records to tmp/gvcf.tfrecord-00000-of-00004.gz
I0528 22:22:13.380251 140168426760000 make_examples_core.py:301] Task 0/4: Writing examples to tmp/make_examples.tfrecord-00000-of-00004.gz
I0528 22:22:13.380352 140168426760000 make_examples_core.py:301] Task 0/4: Overhead for preparing inputs: 1 seconds
I0528 22:22:13.385137 140168426760000 make_examples_core.py:301] Task 0/4: 0 candidates (0 examples) [0.00s elapsed]
I0528 22:22:16.401721 140220478957376 make_examples_core.py:301] Task 2/4: Writing example info to tmp/make_examples.tfrecord-00002-of-00004.gz.example_info.json
I0528 22:22:16.402154 140220478957376 make_examples_core.py:2958] example_shape = [100, 221, 7]
I0528 22:22:16.402902 140220478957376 make_examples_core.py:2959] example_channels = [1, 2, 3, 4, 5, 6, 19]
I0528 22:22:16.403719 140220478957376 make_examples_core.py:301] Task 2/4: Found 6 candidate variants
I0528 22:22:16.403779 140220478957376 make_examples_core.py:301] Task 2/4: Created 6 examples
I0528 22:22:16.055581 140168426760000 make_examples_core.py:301] Task 0/4: Writing example info to tmp/make_examples.tfrecord-00000-of-00004.gz.example_info.json
I0528 22:22:16.055858 140168426760000 make_examples_core.py:2958] example_shape = [100, 221, 7]
I0528 22:22:16.056520 140168426760000 make_examples_core.py:2959] example_channels = [1, 2, 3, 4, 5, 6, 19]
I0528 22:22:16.057465 140168426760000 make_examples_core.py:301] Task 0/4: Found 3 candidate variants
I0528 22:22:16.057574 140168426760000 make_examples_core.py:301] Task 0/4: Created 5 examples
I0528 22:22:16.409767 140325694629696 make_examples_core.py:301] Task 1/4: Writing example info to tmp/make_examples.tfrecord-00001-of-00004.gz.example_info.json
I0528 22:22:16.410038 140325694629696 make_examples_core.py:2958] example_shape = [100, 221, 7]
I0528 22:22:16.410839 140325694629696 make_examples_core.py:2959] example_channels = [1, 2, 3, 4, 5, 6, 19]
I0528 22:22:16.411734 140325694629696 make_examples_core.py:301] Task 1/4: Found 1 candidate variants
I0528 22:22:16.411824 140325694629696 make_examples_core.py:301] Task 1/4: Created 1 examples
I0528 22:22:15.883314 139843678230336 make_examples_core.py:301] Task 3/4: Writing example info to tmp/make_examples.tfrecord-00003-of-00004.gz.example_info.json
I0528 22:22:15.883605 139843678230336 make_examples_core.py:2958] example_shape = [100, 221, 7]
I0528 22:22:15.884193 139843678230336 make_examples_core.py:2959] example_channels = [1, 2, 3, 4, 5, 6, 19]
I0528 22:22:15.884809 139843678230336 make_examples_core.py:301] Task 3/4: Found 3 candidate variants
I0528 22:22:15.884874 139843678230336 make_examples_core.py:301] Task 3/4: Created 5 examples

real	0m8.005s
user	0m31.568s
sys	0m21.534s

***** Running the command:*****
time /opt/deepvariant/bin/call_variants --outfile "tmp/call_variants_output.tfrecord.gz" --examples "tmp/make_examples.tfrecord@4.gz" --checkpoint "/opt/models/wes"

/usr/local/lib/python3.8/dist-packages/tensorflow_addons/utils/tfa_eol_msg.py:23: UserWarning: 

TensorFlow Addons (TFA) has ended development and introduction of new features.
TFA has entered a minimal maintenance and release mode until a planned end of life in May 2024.
Please modify downstream libraries to take dependencies from other repositories in our TensorFlow community (e.g. Keras, Keras-CV, and Keras-NLP). 

For more information see: https://github.com/tensorflow/addons/issues/2807 

  warnings.warn(
I0528 22:22:19.103289 140211816433472 call_variants.py:563] Total 1 writing processes started.
I0528 22:22:19.104772 140211816433472 dv_utils.py:370] From tmp/make_examples.tfrecord-00000-of-00004.gz.example_info.json: Shape of input examples: [100, 221, 7], Channels of input examples: [1, 2, 3, 4, 5, 6, 19].
I0528 22:22:19.104842 140211816433472 call_variants.py:588] Shape of input examples: [100, 221, 7]
I0528 22:22:19.105078 140211816433472 call_variants.py:592] Use saved model: True
I0528 22:22:24.358438 140211816433472 dv_utils.py:370] From /opt/models/wes/example_info.json: Shape of input examples: [100, 221, 7], Channels of input examples: [1, 2, 3, 4, 5, 6, 19].
I0528 22:22:24.358690 140211816433472 dv_utils.py:370] From tmp/make_examples.tfrecord-00000-of-00004.gz.example_info.json: Shape of input examples: [100, 221, 7], Channels of input examples: [1, 2, 3, 4, 5, 6, 19].
I0528 22:22:25.863098 140211816433472 call_variants.py:716] Predicted 17 examples in 1 batches [8.074 sec per 100].
I0528 22:22:25.939682 140211816433472 call_variants.py:779] Complete: call_variants.

real	0m9.749s
user	0m10.566s
sys	0m10.547s

***** Running the command:*****
time /opt/deepvariant/bin/postprocess_variants --ref "Homo_sapiens_assembly38.fasta" --infile "tmp/call_variants_output.tfrecord.gz" --outfile "HCC1395_BL.dv_rescue.vcf.gz" --cpus "4" --gvcf_outfile "HCC1395_BL.dv_rescue.g.vcf.gz" --nonvariant_site_tfrecord_path "tmp/gvcf.tfrecord@4.gz"

I0528 22:22:29.072639 139907783436096 postprocess_variants.py:1211] Using sample name from call_variants output. Sample name: HCC1395_BL
2026-05-28 22:22:29.080131: I deepvariant/postprocess_variants.cc:94] Read from: tmp/call_variants_output-00000-of-00001.tfrecord.gz
2026-05-28 22:22:29.080357: I deepvariant/postprocess_variants.cc:109] Total #entries in single_site_calls = 17
I0528 22:22:29.082077 139907783436096 postprocess_variants.py:1313] CVO sorting took 0.00013818740844726563 minutes
I0528 22:22:29.082357 139907783436096 postprocess_variants.py:1316] Transforming call_variants_output to variants.
I0528 22:22:29.082408 139907783436096 postprocess_variants.py:1318] Using 4 CPUs for parallelization of variant transformation.
I0528 22:22:29.113452 139907783436096 postprocess_variants.py:1211] Using sample name from call_variants output. Sample name: HCC1395_BL
I0528 22:22:29.120674 139907783436096 postprocess_variants.py:1386] Processing variants (and writing to temporary file) took 5.064805348714193e-05 minutes
I0528 22:22:29.273286 139907783436096 postprocess_variants.py:1407] Finished writing VCF and gVCF in 0.0025387008984883628 minutes.

real	0m3.020s
user	0m4.624s
sys	0m8.858s

***** Running the command:*****
time /opt/deepvariant/bin/vcf_stats_report --input_vcf "HCC1395_BL.dv_rescue.vcf.gz" --outfile_base "HCC1395_BL.dv_rescue"

I0528 22:22:32.174937 140051736102720 genomics_reader.py:222] Reading HCC1395_BL.dv_rescue.vcf.gz with NativeVcfReader

real	0m2.975s
user	0m4.599s
sys	0m8.980s