-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathFirecloud.intro.html
371 lines (318 loc) · 15.5 KB
/
Firecloud.intro.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta charset="utf-8">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="pandoc" />
<meta name="author" content="Steve Tsang" />
<meta name="date" content="2016-06-26" />
<title>Firecloud-Introduction</title>
<script src="site_libs/jquery-1.11.3/jquery.min.js"></script>
<meta name="viewport" content="width=device-width, initial-scale=1" />
<link href="site_libs/bootstrap-3.3.5/css/cosmo.min.css" rel="stylesheet" />
<script src="site_libs/bootstrap-3.3.5/js/bootstrap.min.js"></script>
<script src="site_libs/bootstrap-3.3.5/shim/html5shiv.min.js"></script>
<script src="site_libs/bootstrap-3.3.5/shim/respond.min.js"></script>
<style type="text/css">
h1 {
font-size: 34px;
}
h1.title {
font-size: 38px;
}
h2 {
font-size: 30px;
}
h3 {
font-size: 24px;
}
h4 {
font-size: 18px;
}
h5 {
font-size: 16px;
}
h6 {
font-size: 12px;
}
.table th:not([align]) {
text-align: left;
}
</style>
</head>
<body>
<style type = "text/css">
.main-container {
max-width: 940px;
margin-left: auto;
margin-right: auto;
}
code {
color: inherit;
background-color: rgba(0, 0, 0, 0.04);
}
img {
max-width:100%;
height: auto;
}
.tabbed-pane {
padding-top: 12px;
}
button.code-folding-btn:focus {
outline: none;
}
</style>
<style type="text/css">
/* padding for bootstrap navbar */
body {
padding-top: 51px;
padding-bottom: 40px;
}
/* offset scroll position for anchor links (for fixed navbar) */
.section h1 {
padding-top: 56px;
margin-top: -56px;
}
.section h2 {
padding-top: 56px;
margin-top: -56px;
}
.section h3 {
padding-top: 56px;
margin-top: -56px;
}
.section h4 {
padding-top: 56px;
margin-top: -56px;
}
.section h5 {
padding-top: 56px;
margin-top: -56px;
}
.section h6 {
padding-top: 56px;
margin-top: -56px;
}
</style>
<script>
// manage active state of menu based on current page
$(document).ready(function () {
// active menu anchor
href = window.location.pathname
href = href.substr(href.lastIndexOf('/') + 1)
if (href === "")
href = "index.html";
var menuAnchor = $('a[href="' + href + '"]');
// mark it active
menuAnchor.parent().addClass('active');
// if it's got a parent navbar menu mark it active as well
menuAnchor.closest('li.dropdown').addClass('active');
});
</script>
<div class="container-fluid main-container">
<!-- tabsets -->
<script src="site_libs/navigation-1.1/tabsets.js"></script>
<script>
$(document).ready(function () {
window.buildTabsets("TOC");
});
</script>
<!-- code folding -->
<div class="navbar navbar-default navbar-fixed-top" role="navigation">
<div class="container">
<div class="navbar-header">
<button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="index.html">TeamCGC</a>
</div>
<div id="navbar" class="navbar-collapse collapse">
<ul class="nav navbar-nav">
</ul>
<ul class="nav navbar-nav navbar-right">
<li>
<a href="index.html">Home</a>
</li>
<li>
<a href="about.html">About</a>
</li>
<li>
<a href="tutorials.html">Tutorials</a>
</li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">
cwl pipelines
<span class="caret"></span>
</a>
<ul class="dropdown-menu" role="menu">
<li>
<a href="cgpbox.html">cgpbox</a>
</li>
</ul>
</li>
</ul>
</div><!--/.nav-collapse -->
</div><!--/.container -->
</div><!--/.navbar -->
<div class="fluid-row" id="header">
<h1 class="title toc-ignore">Firecloud-Introduction</h1>
<h4 class="author"><em>Steve Tsang</em></h4>
<h4 class="date"><em>June 26, 2016</em></h4>
</div>
<p><br> <br> FireCloud, developed and maintained by the Broad Institute, is one of the three National Cancer Institute’s Cancer Genomics Cloud Pilots. <a href="https://software.broadinstitute.org/firecloud/" class="uri">https://software.broadinstitute.org/firecloud/</a>. This documment provides step-by-step description of creating a project on the FireCloud and upload RNA-seq (BAM) data. Before completing this tutorial, please set up an user account and a billing account with the instructions on the Firecloud’s website.</p>
<div id="account-registration" class="section level4">
<h4>Account registration</h4>
<p><a href="https://software.broadinstitute.org/firecloud/guide/topic?name=firecloud-registration" class="uri">https://software.broadinstitute.org/firecloud/guide/topic?name=firecloud-registration</a></p>
<pre><code>- Go to <Portal.FireCloud.org>
- Click "Register"
- Sign In with a Gmail or Google Apps Account (e.g., [email protected]). When asked to allow FireCloud to view your email address and basic profile info, etc., please click Allow. An explanation of these requests is posted in the Help Forum. You will be prompted to enter New User Registration information.
- Please enter a Contact Email if it differs from your Gmail or Google Apps Account.
- Click Register
(After registering, you will see a message indicating your FireCloud account is inactive. A FireCloud administrator will activate your account within 24 hours.)
</code></pre>
</div>
<div id="billing" class="section level4">
<h4>Billing</h4>
<p><a href="https://software.broadinstitute.org/firecloud/guide/topic?name=firecloud-google" class="uri">https://software.broadinstitute.org/firecloud/guide/topic?name=firecloud-google</a></p>
<p>Here is some information on estimating costs in FireCloud from the development team.</p>
<p>FireCloud runs on the Google Cloud Platform (GCP). All FireCloud costs (cloud storage, compute, data egress etc.) are ultimately billed via Google Billing Accounts. In order to be compatible with multiple cloud environments, institutional payment systems, and security requirements, the FireCloud interface does not directly display any part of the Google Billing Account interface.</p>
<p>Instead, FireCloud will connect a Google Billing Account to an associated FireCloud Billing Project. When you are using FireCloud, it is these FireCloud Billing Projects that you will see in the interface, and to which FireCloud will charge your usage costs.</p>
<p>In order to create or clone a new workspace in FireCloud, you must have access to at least one FireCloud Billing Project. To get started with a FireCloud Billing Project or Google Billing Account, you can check out this [Help Topic] <a href="https://software.broadinstitute.org/firecloud/guide/topic?name=firecloud-google" class="uri">https://software.broadinstitute.org/firecloud/guide/topic?name=firecloud-google</a> in the FireCloud User Guide.</p>
<p><strong>Google Cloud Platform (GCP) Pricing Structure</strong></p>
<ul>
<li>Data upload to GCP is free.</li>
<li>Data download from GCP is $120/TB with a 30% discount if over 10TB.<br />
</li>
<li>Compute costs about $0.05/core hour if less than 6.5GB RAM/core. More RAM will require running with more cores and will cost more.</li>
<li>Disk space (used only while VM is running) costs $40/TB/Month of total quota reserved.</li>
<li>General storage costs $26/TB/month.</li>
</ul>
<p><strong>Example</strong> - FireCloud testers ran a MuTect analysis on 1089 tumor/normal pairs - <strong>Compute</strong>: At $0.05/core hour and 23,305.4 core hours, the cost was $1,165.27 ($1.07 dollars/run) - <strong>Storage</strong>: There were 1089 output files with a total size of 0.001141 TB. At $26/TB/month, the storage/month was $0.03. Note: In this example, the pairs referred to BAMs that resided in the TCGA Open Access bucket, so no storage costs were incurred by the testers. - <strong>Download</strong>: To download these files, at $120/TB, the cost is $0.14. - Note: if the testers wanted to estimate costs, they could first run the analysis on a few pairs and calculate the core hours per run.</p>
<p><strong>For more information, you can check out these links:</strong> - <a href="https://cloud.google.com/compute/pricing">Google Compute Pricing</a> - <a href="https://cloud.google.com/storage/pricing">Google Storage Pricing</a> - <a href="https://software.broadinstitute.org/firecloud/guide/topic?name=firecloud-google">FireCloud Projects and Billing Accounts</a></p>
</div>
<div id="firecloud-basics" class="section level4">
<h4>FireCloud Basics</h4>
<p><a href="https://software.broadinstitute.org/firecloud/guide/topic?name=firecloud-basics" class="uri">https://software.broadinstitute.org/firecloud/guide/topic?name=firecloud-basics</a></p>
</div>
<div id="login" class="section level4">
<h4>Login</h4>
<p><a href="https://portal.firecloud.org/" class="uri">https://portal.firecloud.org/</a><br />
This tutorial is written using the Google Chrome as browser.</p>
<pre><code>Click "Sign In"
Sign in with your google credential
(If you have multiple google accounts, log in in an incognito window by clicking File -> New Incognito Window)</code></pre>
</div>
<div id="tcga-access" class="section level4">
<h4>TCGA access</h4>
<p>After you log in, your profile is displayed.<br />
If you have an NIH account, link your NIH account.</p>
<pre><code>Click "Log-In to NIH to re-link your account"
Log in with your NIH credential
(You will need to renew/re-link your account periodically)</code></pre>
</div>
<div id="create-and-share-new-workspace" class="section level4">
<h4>Create and Share New Workspace</h4>
<p><a href="https://software.broadinstitute.org/firecloud/guide/topic?name=firecloud-basics" class="uri">https://software.broadinstitute.org/firecloud/guide/topic?name=firecloud-basics</a> The FireCloud world is organized into workspaces.</p>
<p>A workspace is a computational sandbox where you can organize genomic data and tools, and run analyses. Users can create, share, and clone workspaces. Workspaces hold:</p>
<p>Data: pre-loaded or user-uploaded, open access or controlled access<br />
Workflows: pre-loaded or user-created<br />
Tools: pre-loaded or user-created<br />
Results: from all runs, captured with provenance</p>
<p>After you log in, your workspaces will be displayed. Workspaces that are TCGA access-controlled are labeled “Restricted.”</p>
<p>To create a new workspace</p>
<pre><code>Click on the blue botton on the upper-right-hand-corner "Create New Workspace... +"
Make sure it is in the correct Google Project linked to the billing account (Refer to the Billing section above)
Enter workspace name "IRPworkshop"
Enter workspace description, if any.
If this project will contain TCGA control data, check the box "Workspace intended to contain NIH protected data." If not, leave it unchecked.
Click "Create Workspace"</code></pre>
<p>Once a workspace is created, you can share with other users. Under “Summary” tab, under “Workspace Owner”, you should see your gmail account listed. To share this workspace</p>
<pre><code>Click "(Sharing...)" next to the gmail account
A pop-up window will appear
Click "Add new +"
Enter user gmail address
Set access level for the user (OWNER, WRITER, READER, NO ACCESS)
Click "Save"</code></pre>
</div>
<div id="upload-my-data-to-the-firecloud" class="section level4">
<h4>Upload my data to the FireCloud</h4>
<p>To upload file,</p>
<pre><code>Click on the line under "Google Bucket" which opens the Google Cloud Storage browser for this workspace
You will be taken to the Google Cloud Platform
Click "Upload Files"
Select all the .bam and .bai files
Click "Open"
(Multiple file tends to only upload 3 files simoutaneously, repeat this process as necessary until all bam/bai files are uploaded and can be seen in the Google Console)</code></pre>
<p>Once the files are uploaded to the Google console, you need to add them to the workspace.<br />
To do this, you need two TSV files that describes the bam/bai files.<br />
The first one, called “participants.tsv” has information about the participants and disease type.</p>
<p>participant_id/sample_id fields are set arbitrarily and should be unique.<br />
<br> participant.tsv</p>
<pre><code>entity:participant_id disease
G28029 BRCA
G41676 BRCA
G41659 LCLL
G41707 LCLL
</code></pre>
<p>The second one, called “sample.tsv” has information about the bam/bai files. sample.tsv has four columns, “sample_id”, “BAM_index”, “participant_id”, and “BAM.”<br />
“sample_id” is arbitrarily set and should be unique.<br />
“pariticpant_id” should match the participant_id in “participant.tsv” “BAM_index”/“BAM” are addresses where the bam/bai files are in the Google Cloud Storage.<br />
The address should have the format “gs://[Google_Bucket]/[file_name]”<br />
If your Google Bucket location is “fc-974b342c-b6f2-45e7-b901-xxxxxxxxxxxx” and file name is “NGS.bam”, your address in that column should be “gs://c-974b342c-b6f2-45e7-b901-xxxxxxxxxxxx/NGS.bam” For clarity, full file address is not listed below.</p>
<p>sample.tsv</p>
<pre><code>entity:sample_id BAM_index participant_id BAM
G28029_sample bai_address1 G28029 bam_address1
G41676_sample bai_address2 G41676 bam_address2
G41659_sample bai_address3 G41659 bam_address3
G41707_sample bai_address4 G41707 bam_address4
</code></pre>
<p>Once your have the two TSV files created, go back to the workspace.</p>
<pre><code>Click on "Data" tab
Click "Import Data..."
Click "Importa from file"
Click "Choose file..."
Select "participant.tsv" that you created above
Click "Open"
Click "Upload" and you should see a "Upload Successful" message
Click "Import Data..."
Click "Importa from file"
Click "Choose file..."
Select "sample.tsv" that you created above
Click "Open"
Click "Upload" and you should see a "Upload Successful" message
</code></pre>
<p>That’s it. You have uploaded the files to the workspace.</p>
<p>When you click on the address for the bam files, you will see</p>
<pre><code>Object: G28029_pe.Aligned.sortedByCoord.out.sorted.bam
File size: 974.40 MBOpen (right-click to download)Warning: Downloading this file may incur a large data egress charge</code></pre>
<p>Please note that uploading to the Firecloud is free, but you will be charged for downloading files.</p>
<p>More to come later on running analysis with those uploaded files.</p>
</div>
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','https://www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-82652476-1', 'auto');
ga('send', 'pageview');
</script>
</div>
<script>
// add bootstrap table styles to pandoc tables
$(document).ready(function () {
$('tr.header').parent('thead').parent('table').addClass('table table-condensed');
});
</script>
<!-- dynamically load mathjax for compatibility with self-contained -->
<script>
(function () {
var script = document.createElement("script");
script.type = "text/javascript";
script.src = "https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML";
document.getElementsByTagName("head")[0].appendChild(script);
})();
</script>
</body>
</html>