PacBIO SmrtAnalysis

Q&A
,
1

I am currently trying to set up SmrtAnalysis V2.2.0 for a client. This software appears to be very intertwined. It is clear that it was designed to be run as a server that forwards the jobs out to the scheduler. Especially since they completely re-did the software and jumped from V2.3 to V5.0 where they re-branded the name to smrtlink. I have tried running both V2.2 and V2.3 and they both have the same result. Here is the guide to installing the software. I have followed these a few times and I get the same results everytime.

https://smrt-analysis.readthedocs.io/en/latest/SMRT-Analysis-Software-Installation-v2.2.0/

I am not sure if this is allowed, but being that it is meant to be run as a server the output uses an extension not allowed (.html). Here is a public dropbox link to the data.

The main thing that stands out to me is in the smrtanalysis/outputs/log/master.log. There is a line:

[DEBUG] 2018-08-01 16:15:09,960 [SMRTpipe.pbpy.io.MetaAnalysisXml load 114] No header found in /home/brandon/projects/dilay/tests/outputs/input.xml. Unable to load jobId

This is particularly confusing since this xml is generated using a command built into the software and it does parse the xml for the three files successfully.

[DEBUG] 2018-08-01 16:15:09,960 [SMRTpipe.pbpy.io.MetaAnalysisXml _parseRef 587] Successfully parsed run:0000000-0000 to run 0000000-0000
[DEBUG] 2018-08-01 16:15:09,960 [SMRTpipe.pbpy.io.MetaAnalysisXml _parseRef 587] Successfully parsed run:0000000-0001 to run 0000000-0001
[DEBUG] 2018-08-01 16:15:09,960 [SMRTpipe.pbpy.io.MetaAnalysisXml _parseRef 587] Successfully parsed run:0000000-0002 to run 0000000-0002

This appears to be the outcome of some other issue causing it, but even the log doesn’t have any history of another issue. There aren’t any errors and the program thinks it runs successfully, however it doesn’t appear to do any arithmetic due to the jobid missing.

4

If you’re intent on using the 2 series of the pacbio smrt SW, good luck. It’s awful. Here’s the module file + embedded installation notes for the 2.3 version

http://moo.nac.uci.edu/~hjm/smrtanalysis-2.3.0p5-install.notes

For the 5 series of smrtanalysis, it’s hugely easier, if not transparently so. Here’s my notes on the installation:

## The installation is quite a bit smoother now.  The installation 
## package that PacBio supplies is esentially complete, albeit bloated with
## all kinds of code duplications that we already have. However, it's not worth
## the space saving to take it apart and symlink our own bits in.
## Some notes: 
## 1 - Install AS the smrtanalysis user, after having checked that the dir 
##      permissions still allow you to do so.
## 2 - Download the latest versio from <http://www.pacb.com/devnet/>
## 3 - make sure the dirs "data_root" and "jobs_root" already exist.
## 4 - export SMRT_ROOT='/data/apps/smrtanalysis/XXXXX.XXXX' (wherever you installed it)
## 4 - run the installer.

We use the environment modules system so if you want our entire module file, be happy to supply it, tho we don’t generally make them public since some of them reference internal information.

2

Not sure if this us useful, but

Maybe there are some PacBIO command line experts on ask.cyberinfrastructure.org who can provide more suggestions?

3

I definitely recommend switching to smrt-link. If you really need to run smrt-analysis for some temporary data conversion or something, here’s a link to an ansible playbook I used to set it up at the time we were running it, that may give some clues about a difference during the install procedure:

5

At IU we have SMRT link v.5.0 installed, we haven’t worked with PacBio data as much as we’d like so we haven’t run into the error you mentioned.

If you are looking for an alternative, once the Pacbio sequences format is converted to fastq using SMRT link, you can use Canu (https://canu.readthedocs.io/en/latest/quick-start.html) for trimming, correcting and assembing the reads.

6

UPDATE

Sorry, it has been a while. I have made progress. It appears that the grunt of the issue before was the RS_Modification_Detection.1.xml. I recieved a new one from PacBio but running into a new issue altogether. It runs for roughly 15 minutes and then errors out. Here is a drop box with the inputs and outputs much like before, and the terminal output below if anyone has seen this error or knows another thing to try.

https://www.dropbox.com/s/vayfzmj8h0zyk6e/smrtanalysisattempt.zip?dl=0

Question for those running smrtlink 5.0 in an HPC environment : How would you suggest your users go about using it given environment modules. Would they open an interactive session, run smrtlink in a singularity container, and than forward the port, or open a browser using x11 forwarding?

brandon@node0:dilay/inputs $ ./test.sh 

[INFO] 2018-09-17 09:38:59,059 Successfully found 3 files in xml:/home/brandon/projects/dilay/tests/outputs/input.xml

[INFO] 2018-09-17 09:38:59,441 smrtpipe.py running on node0

[INFO] 2018-09-17 09:38:59,441 SMRT Analysis 2.3.0 / SMRTpipe 1.87.139483

[INFO] 2018-09-17 09:39:00,086 Process id 7466

[INFO] 2018-09-17 09:39:05,601 Starting task://Anonymous/P_Fetch/toFofn

[INFO] 2018-09-17 09:39:05,602 task P_Fetch.toFofn successfully completed (0.00 sec (0.00 min)). freed 1 slots, using 0 slots.

[INFO] 2018-09-17 09:39:05,602 Workflow Completion Status 1/44 in (            2%) tasks completed.

[INFO] 2018-09-17 09:39:10,608 Starting task://Anonymous/P_GenomicConsensus/writeContigList

[INFO] 2018-09-17 09:39:10,608 Starting task://Anonymous/P_ModificationDetection/writeContigList

[INFO] 2018-09-17 09:39:10,608 Starting task://Anonymous/P_Fetch/adapterRpt

[INFO] 2018-09-17 09:39:10,608 Starting task://Anonymous/P_Filter/subreads.plsFofn.Scatter

[INFO] 2018-09-17 09:39:10,608 Starting task://Anonymous/P_Fetch/overviewRpt

[INFO] 2018-09-17 09:39:10,609 task P_GenomicConsensus.writeContigList successfully completed (0.00 sec (0.00 min)). freed 1 slots, using 4 slots.

[INFO] 2018-09-17 09:39:10,609 Workflow Completion Status 2/44 in (            4%) tasks completed.

[INFO] 2018-09-17 09:39:10,609 task P_ModificationDetection.writeContigList successfully completed (0.00 sec (0.00 min)). freed 1 slots, using 3 slots.

[INFO] 2018-09-17 09:39:10,609 Workflow Completion Status 3/44 in (            6%) tasks completed.

[INFO] 2018-09-17 09:39:10,610 task P_Filter.subreads.plsFofn.Scatter successfully completed (0.00 sec (0.00 min)). freed 1 slots, using 2 slots.

[INFO] 2018-09-17 09:39:10,610 Workflow Completion Status 4/44 in (            9%) tasks completed.

[INFO] 2018-09-17 09:39:11,263 task P_Fetch.overviewRpt successfully completed (1.00 sec (0.02 min)). freed 1 slots, using 1 slots.

[INFO] 2018-09-17 09:39:11,263 Workflow Completion Status 5/44 in (         . 11%) tasks completed.

[INFO] 2018-09-17 09:39:16,267 Starting task://Anonymous/P_Filter/filter_001of003

[INFO] 2018-09-17 09:39:16,267 Starting task://Anonymous/P_Filter/filter_002of003

[INFO] 2018-09-17 09:39:16,267 Starting task://Anonymous/P_Filter/filter_003of003

[INFO] 2018-09-17 09:39:22,001 task P_Fetch.adapterRpt successfully completed (15.00 sec (0.25 min)). freed 1 slots, using 6 slots.

[INFO] 2018-09-17 09:39:22,001 Workflow Completion Status 6/44 in (         . 13%) tasks completed.

[INFO] 2018-09-17 09:39:32,009 task P_Filter.filter_001of003 successfully completed (20.00 sec (0.33 min)). freed 2 slots, using 4 slots.

[INFO] 2018-09-17 09:39:32,010 Workflow Completion Status 7/44 in (         . 15%) tasks completed.

[INFO] 2018-09-17 09:39:37,015 Starting task://Anonymous/P_Filter/subreads_001of003

[INFO] 2018-09-17 09:39:37,016 task P_Filter.filter_003of003 successfully completed (21.00 sec (0.35 min)). freed 2 slots, using 3 slots.

[INFO] 2018-09-17 09:39:37,016 Workflow Completion Status 8/44 in (         . 18%) tasks completed.

[INFO] 2018-09-17 09:39:37,017 task P_Filter.filter_002of003 successfully completed (21.00 sec (0.35 min)). freed 2 slots, using 1 slots.

[INFO] 2018-09-17 09:39:37,017 Workflow Completion Status 9/44 in (        .. 20%) tasks completed.

[INFO] 2018-09-17 09:39:42,025 Starting task://Anonymous/P_Filter/subreads_002of003

[INFO] 2018-09-17 09:39:42,026 Starting task://Anonymous/P_Filter/filter.summary.Gather

[INFO] 2018-09-17 09:39:42,026 Starting task://Anonymous/P_Filter/filter.rgnFofn.Gather

[INFO] 2018-09-17 09:39:42,026 Starting task://Anonymous/P_Filter/subreads_003of003

[INFO] 2018-09-17 09:39:42,842 task P_Filter.filter.summary.Gather successfully completed (0.00 sec (0.00 min)). freed 1 slots, using 4 slots.

[INFO] 2018-09-17 09:39:42,842 Workflow Completion Status 10/44 in (        .. 22%) tasks completed.

[INFO] 2018-09-17 09:39:42,843 task P_Filter.filter.rgnFofn.Gather successfully completed (0.00 sec (0.00 min)). freed 1 slots, using 3 slots.

[INFO] 2018-09-17 09:39:42,843 Workflow Completion Status 11/44 in (        .. 25%) tasks completed.

[INFO] 2018-09-17 09:39:47,852 Starting task://Anonymous/P_FilterReports/statsRpt

[INFO] 2018-09-17 09:39:47,852 Starting task://Anonymous/P_FilterReports/loadingRpt

[INFO] 2018-09-17 09:39:47,852 Starting task://Anonymous/P_Control/align

[INFO] 2018-09-17 09:39:47,853 Starting task://Anonymous/P_Filter/subreadSummary

[INFO] 2018-09-17 09:39:54,334 task P_FilterReports.loadingRpt successfully completed (5.00 sec (0.08 min)). freed 1 slots, using 13 slots.

[INFO] 2018-09-17 09:39:54,334 Workflow Completion Status 12/44 in (        .. 27%) tasks completed.

[INFO] 2018-09-17 09:40:01,351 task P_FilterReports.statsRpt successfully completed (15.00 sec (0.25 min)). freed 1 slots, using 12 slots.

[INFO] 2018-09-17 09:40:01,351 Workflow Completion Status 13/44 in (        .. 29%) tasks completed.

[INFO] 2018-09-17 09:40:43,308 task P_Filter.subreadSummary successfully completed (55.00 sec (0.92 min)). freed 1 slots, using 11 slots.

[INFO] 2018-09-17 09:40:43,309 Workflow Completion Status 14/44 in (       ... 31%) tasks completed.

[INFO] 2018-09-17 09:40:48,316 Starting task://Anonymous/P_FilterReports/subreadRpt

[INFO] 2018-09-17 09:40:55,025 task P_FilterReports.subreadRpt successfully completed (7.00 sec (0.12 min)). freed 1 slots, using 11 slots.

[INFO] 2018-09-17 09:40:55,025 Workflow Completion Status 15/44 in (       ... 34%) tasks completed.

[INFO] 2018-09-17 09:41:10,504 task P_Filter.subreads_003of003 successfully completed (90.00 sec (1.50 min)). freed 1 slots, using 10 slots.

[INFO] 2018-09-17 09:41:10,504 Workflow Completion Status 16/44 in (       ... 36%) tasks completed.

[INFO] 2018-09-17 09:41:20,514 task P_Filter.subreads_001of003 successfully completed (104.00 sec (1.73 min)). freed 1 slots, using 9 slots.

[INFO] 2018-09-17 09:41:20,514 Workflow Completion Status 17/44 in (       ... 38%) tasks completed.

[INFO] 2018-09-17 09:41:25,520 task P_Filter.subreads_002of003 successfully completed (106.00 sec (1.77 min)). freed 1 slots, using 8 slots.

[INFO] 2018-09-17 09:41:25,520 Workflow Completion Status 18/44 in (      .... 40%) tasks completed.

[INFO] 2018-09-17 09:41:30,532 Starting task://Anonymous/P_Filter/subreads.subreads.Gather

[INFO] 2018-09-17 09:41:30,533 Starting task://Anonymous/P_Filter/subreads.subreadFastq.Gather

[INFO] 2018-09-17 09:41:31,631 task P_Filter.subreads.subreads.Gather successfully completed (3.00 sec (0.05 min)). freed 1 slots, using 9 slots.

[INFO] 2018-09-17 09:41:31,632 Workflow Completion Status 19/44 in (      .... 43%) tasks completed.

[INFO] 2018-09-17 09:41:37,645 task P_Filter.subreads.subreadFastq.Gather successfully completed (7.00 sec (0.12 min)). freed 1 slots, using 8 slots.

[INFO] 2018-09-17 09:41:37,646 Workflow Completion Status 20/44 in (      .... 45%) tasks completed.

[INFO] 2018-09-17 09:46:35,807 task P_Control.align successfully completed (409.00 sec (6.82 min)). freed 8 slots, using 0 slots.

[INFO] 2018-09-17 09:46:35,807 Workflow Completion Status 21/44 in (      .... 47%) tasks completed.

[INFO] 2018-09-17 09:46:40,810 Starting task://Anonymous/P_Control/summaryCSV

[INFO] 2018-09-17 09:46:40,811 Starting task://Anonymous/P_ControlReports/statsJsonReport

[INFO] 2018-09-17 09:46:40,811 Starting task://Anonymous/P_Control/updateRgn

[INFO] 2018-09-17 09:46:41,645 task P_Control.summaryCSV successfully completed (1.00 sec (0.02 min)). freed 1 slots, using 2 slots.

[INFO] 2018-09-17 09:46:41,645 Workflow Completion Status 22/44 in (     ..... 50%) tasks completed.

[INFO] 2018-09-17 09:46:42,274 task P_ControlReports.statsJsonReport successfully completed (2.00 sec (0.03 min)). freed 1 slots, using 1 slots.

[INFO] 2018-09-17 09:46:42,274 Workflow Completion Status 23/44 in (     ..... 52%) tasks completed.

[INFO] 2018-09-17 09:46:42,275 task P_Control.updateRgn successfully completed (4.00 sec (0.07 min)). freed 1 slots, using 0 slots.

[INFO] 2018-09-17 09:46:42,275 Workflow Completion Status 24/44 in (     ..... 54%) tasks completed.

[INFO] 2018-09-17 09:46:47,282 Starting task://Anonymous/P_Mapping/align

[INFO] 2018-09-17 09:46:47,282 Starting task://Anonymous/P_Control/noControlSubreads

[INFO] 2018-09-17 09:47:58,759 task P_Control.noControlSubreads successfully completed (73.00 sec (1.22 min)). freed 1 slots, using 8 slots.

[INFO] 2018-09-17 09:47:58,760 Workflow Completion Status 25/44 in (     ..... 56%) tasks completed.

[INFO] 2018-09-17 09:52:21,299 task P_Mapping.align successfully completed (333.00 sec (5.55 min)). freed 8 slots, using 0 slots.

[INFO] 2018-09-17 09:52:21,299 Workflow Completion Status 26/44 in (     ..... 59%) tasks completed.

[INFO] 2018-09-17 09:52:26,303 Starting task://Anonymous/P_Mapping/sort

[INFO] 2018-09-17 09:52:27,043 task P_Mapping.sort successfully completed (0.00 sec (0.00 min)). freed 8 slots, using 0 slots.

[INFO] 2018-09-17 09:52:27,044 Workflow Completion Status 27/44 in (    ...... 61%) tasks completed.

[INFO] 2018-09-17 09:52:32,049 Starting task://Anonymous/P_Mapping/repack

[INFO] 2018-09-17 09:52:32,708 task P_Mapping.repack successfully completed (0.00 sec (0.00 min)). freed 8 slots, using 0 slots.

[INFO] 2018-09-17 09:52:32,709 Workflow Completion Status 28/44 in (    ...... 63%) tasks completed.

[INFO] 2018-09-17 09:52:37,717 Starting task://Anonymous/P_MappingReports/statsJsonReport

[INFO] 2018-09-17 09:52:37,717 Starting task://Anonymous/P_ModificationDetection/computeModifications

[INFO] 2018-09-17 09:52:37,717 Starting task://Anonymous/P_Mapping/samBam

[INFO] 2018-09-17 09:52:37,717 Starting task://Anonymous/P_Mapping/covGFF

[INFO] 2018-09-17 09:52:37,717 Starting task://Anonymous/P_GenomicConsensus/callVariantsWithConsensus

[INFO] 2018-09-17 09:52:37,717 Starting task://Anonymous/P_Mapping/unmapped

[ERROR] 2018-09-17 09:52:37,718 *** Failed task task://Anonymous/P_Mapping/samBam

[ERROR] 2018-09-17 09:52:37,718 *** Failed task task://Anonymous/P_Mapping/covGFF

[ERROR] 2018-09-17 09:52:37,718 *** Failed task task://Anonymous/P_GenomicConsensus/callVariantsWithConsensus

[ERROR] 2018-09-17 09:52:37,718 *** Failed task task://Anonymous/P_ModificationDetection/computeModifications

[ERROR] 2018-09-17 09:52:37,718 *** Failed task task://Anonymous/P_MappingReports/statsJsonReport

[INFO] 2018-09-17 09:52:44,080 Found 5 failed tasks.

[INFO] 2018-09-17 09:52:44,080 task samBam FAILED 

[INFO] 2018-09-17 09:52:44,080 task computeModifications FAILED 

[INFO] 2018-09-17 09:52:44,080 task covGFF FAILED 

[INFO] 2018-09-17 09:52:44,081 task callVariantsWithConsensus FAILED 

[INFO] 2018-09-17 09:52:44,081 task statsJsonReport FAILED 

Traceback (most recent call last):

  File "/home/brandon/module/programs/smrtanalysis/2.3.0/install/smrtanalysis_2.3.0.140936/redist/python2.7/lib/python2.7/site-packages/SMRTpipe/engine/SmrtPipeWorkflow.py", line 634, in execute

    raise WorkflowError(_e_msg)

WorkflowError: task://Anonymous/P_Mapping/samBam Failed task://Anonymous/P_ModificationDetection/computeModifications Failed task://Anonymous/P_Mapping/covGFF Failed task://Anonymous/P_GenomicConsensus/callVariantsWithConsensus Failed task://Anonymous/P_MappingReports/statsJsonReport Failed

Failed smrtpipe version v1.87.139483 with exit code -1 in 826.66 seconds (13.78 minutes)

brandon@node0:dilay/inputs $