公开数据库汇总及下载(1)-TCGA

  本文的内容介绍、代码下载主要参考了网上多个文件汇总而成，本文仅作汇总，学习请翻阅原文。
1. 常用数据库

  近几年在生物医药圈不停有脱钩的传闻，随着24年美国大选临近，对于trump上台后的预期政策担忧也愈演愈烈。在本身吓本身的前提下，对各类数据库进行了整理，并逐步进行下载。

2. TCGA数据下载

  TCGA（The Cancer Genome Atlas）数据库是一个由美国国家癌症研究所（NCI）和国家人类基因组研究所（NHGRI）合作创建的大规模癌症基因组学数据库。该数据库中网络了来自多个癌症类型的临床数据、基因组数据和表达数据。
2.1 TCGA介绍

  2005.12.13日，美国国家癌症研究所（NCI）和国家人类基因组研究所（NHGRI）团结启动肿瘤基因组图谱（TCGA）项目。
  TCGA是一项以基因组为底子的大科学研究计划，以人类基因组计划的成果为底子，研究癌症中基因组的变化。与HGP专注于疾病的遗传因素不同，TCGA更关心人类出生后细胞中的基因变化。

  33种癌症类型的关键基因组变化的多维图谱。超过2PB的基因组数据公开。超过1万名病人提供配对的癌症和对照组织。7种数据类型。

  组织布局美满，由8种构成，形成数据闭环。

  由TSS网络样本，交由BCR进行数据整理，由GSC、GCC进行测序、数据鉴定，CGHub、GDAC进行把关，在DCC进行数据最后的整合，最后由NCI、NHGRI项目组发布。

  对样本的纳入有严格的标准，患者是原发性种类，有配对样本。肿瘤组织需冷冻保存、且大小适宜。肿瘤细胞含量在80%以上。
2.2 数据类型详解

  临床数据：TCGA数据库中包含了多种癌症类型的患者的临床信息，如年岁、性别、病理分期、生存期等。这些数据有助于研究人员了解癌症的发展和预后。

1. indexed clinical: 使用 XML 文件创建的精炼临床数据
2. XML: 原始临床数据
3.     处理 XML 格式的临床数据分为两步：
4.         使用 GDCquery 和 GDCDownload 来查询和下载 Biospecimen 或 Clinical XML 文件
5.         使用 GDCprepare_clinic 来解析文件
6. BCR Biotab: 解析 XML 文件之后的 tsv 文件

复制代码

  基因组数据：TCGA数据库中的基因组数据包罗了DNA测序和突变数据。这些数据可以资助研究人员分析癌症基因组的变化，探求与癌症发展相关的基因变异。
  表达数据：TCGA数据库中的表达数据主要是通过RNA测序获得的。这些数据可以反映癌症细胞内基因的表达水平，资助研究人员发现与癌症发展和治疗相关的基因。
  甲基化数据：TCGA数据库中的甲基化数据可以资助研究人员了解癌症细胞中DNA甲基化的模式和变化。甲基化是一种常见的基因组调控方式，与癌症的发生和发展密切相关。
  卵白质组数据：TCGA数据库中的卵白质组数据包罗了癌症细胞中卵白质的表达水平和翻译后修饰的信息。这些数据对于研究癌症细胞的功能和信号通路具有紧张意义。
2.3 TCGA数据等级

  受控数据主要是采集样本的患者隐私，测序原始数据（BAM/FASTQ），SNP6、EXON芯片的1、2级数据，生信分析的VCF、MAF等中间结果。

  从上表可知，1、2级数据为病人原始结果，涉及医学伦理学题目，一样平常处于受控状态。3、4级数据是加工处置惩罚后的数据，由TCGA整合发表。，3级部门开放，4级数据完全开放。 仅有少量的数据是open状态。
  具体可阅读说明文档GDC_Introduction
2.4 TCGA网页筛选页面介绍

2.5 各文件每个字段的寄义

  可看官网的说明文档。
  各表格的字段寄义
2.6 biospecimen IDs介绍(样本ID)

  TCGA_barcode号介绍

  目前UUID成为主要标识符，下载数据时以UUID代指一个样本。
2.7 下载代码

1. # 0. 起始 -------------------------------------------------------------------
2. library(TCGAbiolinks)
3. projects <- TCGAbiolinks::getGDCprojects()$project_id
4. projects <- projects[grepl('^TCGA', projects, perl=TRUE)]
6. # 1. Clinical -------------------------------------------------------------
8. ## 1.1 clinical ------------------------------------------------------------
10. ### XML -------------------------------------------------------------------------
11. # 使用 GDCquery 和 GDCDownload 来查询和下载 Biospecimen 或 Clinical XML 文件
12. # 使用 GDCprepare_clinic 来解析文件
14. ### 下载所有临床数据, 并将结果汇总在一个文件中 ########
16. ### TCGA-READ、TCGA-LGG、仅下载,未合并,因为有重复,需针对性处理
18. library(TCGAbiolinks)
19. library(data.table)
20. library(dplyr)
21. library(regexPipes)
23. # 获取所有索引信息
24. clinical <- TCGAbiolinks:::getGDCprojects()$project_id %>%
25.   regexPipes::grep("TCGA", value = TRUE) %>%
26.   sort %>%
27.   plyr::alply(1, GDCquery_clinic, .progress = "text") %>%
28.   rbindlist(fill = TRUE)
29. readr::write_csv(clinical, file = "all_clin_indexed.csv")
31. # 解析 XML 文件并信息获取对应的信息
32. getclinical <- function(proj) {
33.   message(proj)
34.   result <- NULL
35.   attempt <- 1
36.   max_attempts <- 5  # 设置最大尝试次数
37.   
38.   while(attempt <= max_attempts) {
39.     result <- tryCatch({
40.       query <- GDCquery(project = proj, data.category = "Clinical", data.format = "bcr xml")
41.       GDCdownload(query)
42.       clinical <- GDCprepare_clinic(query, clinical.info = "patient")
43.       
44.       clinical_data <- list(clinical)
45.       for(i in c("admin", "radiation", "follow_up", "drug", "new_tumor_event")){
46.         message(i)
47.         aux <- GDCprepare_clinic(query, clinical.info = i)
48.         if(is.null(aux) || nrow(aux) == 0) next
49.         
50.         # 处理重复的列名
51.         replicated <- which(grep("bcr_patient_barcode", colnames(aux), value = TRUE, invert = TRUE) %in% colnames(clinical))
52.         colnames(aux)[replicated] <- paste0(colnames(aux)[replicated], ".", i)
53.         
54.         if(!is.null(aux)) clinical <- merge(clinical, aux, by = "bcr_patient_barcode", all = TRUE)
55.       }
56.       
57.       # 保存临床数据到csv文件
58.       readr::write_csv(clinical, path = paste0("TCGA_alldata/", proj, "_clinical_from_XML.csv"))
59.       return(clinical)
60.     }, error = function(e) {
61.       message(paste0("Error clinical: ", proj, " Attempt: ", attempt))
62.       attempt <<- attempt + 1  # 增加尝试次数
63.       NULL
64.     })
65.    
66.     # 如果成功获取数据，则跳出循环
67.     if (!is.null(result)) break
68.   }
69.   
70.   # 如果多次尝试后仍然失败，返回NULL并发出警告
71.   if (is.null(result)) {
72.     warning(paste0("Failed to get clinical data for project: ", proj, " after ", max_attempts, " attempts."))
73.   }
74.   
75.   return(result)
76. }
78. # 患者信息
79. # 如果内存溢出,可分批次下载, 或单独下载某个数据集
80. clinical <- TCGAbiolinks:::getGDCprojects()$project_id %>%
81.   regexPipes::grep("TCGA", value = T) %>%
82.   sort %>%
83.   plyr::alply(1, getclinical, .progress = "text") %>%  
84.   rbindlist(fill = TRUE) %>%
85.   setDF %>%
86.   subset(!duplicated(clinical))
88. readr::write_csv(clinical, path = "TCGA_alldata/all_clin_XML.csv")
91. ## 1.2  supplement ---------------------------------------------------------
92. ### 1.2.1 clinical-supplement-bcr biotab, 解析 XML 文件之后的 tsv 文件-------------------------------------------------------------------------
94. sapply(projects, function(project){
95.   cat("Processing project:", project, "\n")
96.   
97.   query <- GDCquery(
98.     project = project,
99.     data.category = "Clinical",
100.     data.type = "Clinical Supplement",
101.     data.format = "BCR Biotab",
102.     access = "open"
103.   )
104.   GDCdownload(query, method = "api", files.per.chunk = 100)
105.   
106.   prepared_data <- GDCprepare(query, save = TRUE, save.filename = file.path(paste0(project, "_clinical_supplement_bcr_biotab.Rdata")))  
107.   
108. })
111. ### 1.2.2 Biospecimen 获取采样信息 -----------------------------------------------------------------------
112. sapply(projects, function(project){
113.   cat("Processing project:", project, "\n")
114.   
115.   query <- GDCquery(
116.     project = project,
117.     data.category = "Biospecimen",
118.     data.type = "Biospecimen Supplement",
119.     data.format = "BCR Biotab",
120.     access = "open"
121.   )
122.   GDCdownload(query, method = "api", files.per.chunk = 100)
123.   
124.   prepared_data <- GDCprepare(query, save = TRUE, save.filename = file.path(paste0(project, "_biospecimen_supplement_bcr_biotab.Rdata")))  
125.   
126. })
129. ### 1.2.3 Indexed 使用 XML 文件创建的精炼临床数据 -------------------------------------------------------------------------
130. clinical <- GDCquery_clinic(project = "TCGA-BRCA", type = "clinical")
131. clinical <- GDCquery_clinic(project = "TARGET-RT", type = "clinical")
134. ### 1.2.4 诊断切片（SVS 格式）-------------------------------------------------------------------------
135. # 单个数据集50-60G,暂不下载
136. sapply(projects, function(project){
137.   cat("Processing project:", project, "\n")
138.   
139.   query <- GDCquery(
140.     project = project,
141.     data.category = "Biospecimen",
142.     data.type = 'Slide Image',
143.     experimental.strategy = "Diagnostic Slide",  # 实验策略有两种,Tissue Slide和Diagnostic Slide(患者的)
144.     # barcode = c("TCGA-RU-A8FL", "TCGA-AA-3972")
145.     access = "open"
146.   )
147.   GDCdownload(query, method = "api", files.per.chunk = 100)
148.   
149.   prepared_data <- GDCprepare(query, save = TRUE, save.filename = file.path(paste0(project, "_biospecimen_slide_images.Rdata")))  
150.   
151. })
154. # 2. RNA -------------------------------------------------------------------------
156. ## 2.1 mRNA-------------------------------------------------------------------------
158. sapply(projects, function(project){
159.   # 查询
160.   query <-  GDCquery(
161.     project = project,
162.     data.category = "Transcriptome Profiling",
163.     data.type = "Gene Expression Quantification",
164.     workflow.type = "STAR - Counts"
165.   )
166.   
167.   # 下载
168.   GDCdownload(query, method = "api", files.per.chunk = 100)
169.   
170.   # 整理
171.   GDCprepare(query, save = T, save.filename = file.path(paste0(project, "_mRNA.Rdata")))
172.   
173. }
174. )
177. ## 2.2 microRNA-------------------------------------------------------------------------
179. sapply(projects, function(project){
180.   query <- GDCquery(project = project,
181.                     data.category = "Transcriptome Profiling",
182.                     data.type = "miRNA Expression Quantification"
183.   )
184.   
185.   GDCdownload(query)
186.   
187.   GDCprepare(query, save = T, save.filename = file.path(paste0(project, "_miRNA.Rdata")))
188.   
189. })
193. ## 2.3 isoform -------------------------------------------------------------------------
195. sapply(projects, function(project){
196.   query.isoform <- GDCquery(
197.     project = project,
198.     experimental.strategy = "miRNA-Seq",
199.     data.category = "Transcriptome Profiling",
200.     data.type = "Isoform Expression Quantification"
201.   )
202.   
203.   GDCdownload(query.isoform, method = "api", files.per.chunk = 100)
204.   GDCprepare(query.isoform, save = T, save.filename = file.path(paste0(project, "_mirna-isoform.Rdata")))
205.   
206. })
209. # 3.SNP-------------------------------------------------------------------------
211. # Can't combine `..17$Tumor_Seq_Allele2` <character> and `..18$Tumor_Seq_Allele2` <logical>. 部分数据库的样本字段不一致,导致合并出错概率高, 先按癌种-样本下载好
213. sapply(projects, function(project){
214.   query <- GDCquery(
215.     project = project,
216.     data.category = "Simple Nucleotide Variation",
217.     data.type = "Masked Somatic Mutation",
218.     access = "open"
219.   )
220.   
221.   GDCdownload(query, method = "api", files.per.chunk = 100)
222.   
223.   # GDCprepare(query, save = T,save.filename = file.path(project_dir,  paste0(project, "_SNP.Rdata")))
224.   
225. })
228. # 4.Protein-------------------------------------------------------------------------
230. # "TCGA-LAML-fail"  "TCGA-THCA-fail", 这两个数据集没有蛋白结果
232. sapply(projects, function(project){
234.   query <- GDCquery(
235.     project = project,
236.     data.category = "Proteome Profiling",
237.     data.type = "Protein Expression Quantification"
238.   )
239.   
240.   GDCdownload(query, method = "api", files.per.chunk = 100)
241.   
242.   GDCprepare(query, save = T, save.filename = file.path(paste0(project, "_protein.Rdata")))
243.   
244. })
247. # 5.methy-------------------------------------------------------------------------
249. # Beta值数据 分3个平台, Illumina Human Methylation 27、 Illumina Human Methylation 450、 Illumina Methylation Epic
250. # 每个癌种下,不一定含有全部3种平台的数据
251. # 本次先下载Illumina Human Methylation 27的
252. # IDAT是原始的荧光信号强度数据，而Beta值是这些原始数据的标准化表示, 本次下载优先BEta值
255. ## 5.1 Illumina Human Methylation 27---------------------------------------------------------------------
257. ### Methylation Beta Value --------------------------------------------------------------------
259. # "TCGA-BRCA" "TCGA-SARC—fail,数据集无此结果" "TCGA-ACC-fail"  "TCGA-UCEC" "TCGA-KIRC" "TCGA-LAML" "TCGA-SKCM-fail" "TCGA-PAAD-fail" "TCGA-TGCT-fail" "TCGA-CESC-fail" "TCGA-ESCA-fail" "TCGA-THCA-fail" "TCGA-LIHC-fail" "TCGA-PRAD-fail" "TCGA-READ"
260. # "TCGA-OV"   "TCGA-UVM-fail"  "TCGA-BLCA-fail" "TCGA-CHOL-fail" "TCGA-GBM"  "TCGA-UCS-fail"  "TCGA-PCPG-fail" "TCGA-MESO-fail" "TCGA-DLBC-fail" "TCGA-COAD" "TCGA-STAD" "TCGA-KIRP" "TCGA-THYM-fail" "TCGA-KICH-fail" "TCGA-LGG-fail"
261. # "TCGA-LUSC" "TCGA-LUAD" "TCGA-HNSC-fail"
263. projects <- c("TCGA-BRCA", "TCGA-UCEC", "TCGA-KIRC", "TCGA-LAML", "TCGA-READ", "TCGA-OV", "TCGA-GBM", "TCGA-COAD", "TCGA-STAD", "TCGA-KIRP", "TCGA-LUSC", "TCGA-LUAD")
265. projects <- c( "TCGA-LUAD")
266. sapply(projects, function(project){
267.   
268.   coad_methy <- GDCquery(
269.     project = project,
270.     data.category = "DNA Methylation",
271.     data.type = "Methylation Beta Value",
272.     platform = "Illumina Human Methylation 27"
273.   )
274.   
275.   GDCdownload(coad_methy, method = "api", files.per.chunk = 100)
276.   GDCprepare(coad_methy,save = T, save.filename= file.path(paste0(project, "_METHY_beta_27.Rdata")))
277.   
278. })
280. ### Masked Intensities --------------------------------------------------------------------
282. # "TCGA-BRCA" "TCGA-SARC—fail" "TCGA-ACC-fail"  "TCGA-UCEC" "TCGA-KIRC" "TCGA-LAML" "TCGA-SKCM-fail" "TCGA-PAAD-fail" "TCGA-TGCT-fail" "TCGA-CESC-fail" "TCGA-ESCA-fail" "TCGA-THCA-fail" "TCGA-LIHC-fail" "TCGA-PRAD-fail" "TCGA-READ"
283. # "TCGA-OV"   "TCGA-UVM-fail"  "TCGA-BLCA-fail" "TCGA-CHOL-fail" "TCGA-GBM"  "TCGA-UCS-fail"  "TCGA-PCPG-fail" "TCGA-MESO-fail" "TCGA-DLBC-fail" "TCGA-COAD" "TCGA-STAD" "TCGA-KIRP" "TCGA-THYM-fail" "TCGA-KICH-fail" "TCGA-LGG-fail"
284. # "TCGA-LUSC" "TCGA-LUAD" "TCGA-HNSC-fail"
286. projects <- c("TCGA-BRCA", "TCGA-UCEC", "TCGA-KIRC", "TCGA-LAML", "TCGA-READ", "TCGA-OV", "TCGA-GBM", "TCGA-COAD", "TCGA-STAD", "TCGA-KIRP", "TCGA-LUSC", "TCGA-LUAD")
288. sapply(projects, function(project){
289.   
290.   coad_methy <- GDCquery(
291.     project = project,
292.     data.category = "DNA Methylation",
293.     data.type = "Masked Intensities",
294.     platform = "Illumina Human Methylation 27"
295.   )
296.   
297.   GDCdownload(coad_methy, method = "api", files.per.chunk = 50)
298. })
301. ## 5.2 Illumina Human Methylation 450 -----------------------------------------------------------------
303. ### Methylation Beta Value --------------------------------------------------------------------
305. sapply(projects, function(project){
306.   
307.   coad_methy <- GDCquery(
308.     project = project,
309.     data.category = "DNA Methylation",
310.     data.type = "Methylation Beta Value",
311.     platform = "Illumina Human Methylation 450" # Illumina Human Methylation 450
312.   )
313.   
314.   GDCdownload(coad_methy, method = "api", files.per.chunk = 30)
315. })
317. ### Masked Intensities --------------------------------------------------------------------
318. sapply(projects, function(project){
319.   
320.   coad_methy <- GDCquery(
321.     project = project,
322.     data.category = "DNA Methylation",
323.     data.type = "Masked Intensities",
324.     platform = "Illumina Human Methylation 450" # Illumina Human Methylation 450
325.   )
326.   
327.   GDCdownload(coad_methy, method = "api", files.per.chunk = 30)
328. })
331. # 6. CNV----------------------------------------------------------------------
333. ## 6.1 Masked Copy Number Segment-------------------------------------------------------------------------
335. sapply(projects, function(project){
336.   
337.   query <- GDCquery(
338.     project = project,
339.     data.category = "Copy Number Variation",
340.     data.type = "Masked Copy Number Segment",              
341.     access = "open"
342.   )
343.   
344.   GDCdownload(query, method = "api", files.per.chunk = 100)
345.   GDCprepare(query, save = T,save.filename = file.path(paste0(project, "_CNV.Rdata")))  
346.   
347. })
350. ## 6.2 Copy Number Segment -------------------------------------------------------------------
352. sapply(projects, function(project){
353.   
354.   query <- GDCquery(
355.     project = project,
356.     data.category = "Copy Number Variation",
357.     data.type = "Copy Number Segment",              
358.     access = "open"
359.   )
360.   
361.   GDCdownload(query, method = "api", files.per.chunk = 300)
362. })
365. ## 6.3 Allele-specific Copy Number Segment ------------------------------------------------------------------
367. # There are more than one file for the same case. Please verify query results. You can use the command View(getResults(query)) in rstudio
369. sapply(projects, function(project){
370.   
371.   query <- GDCquery(
372.     project = project,
373.     data.category = "Copy Number Variation",
374.     data.type = "Allele-specific Copy Number Segment",              
375.     access = "open"
376.   )
377.   
378.   GDCdownload(query, method = "api", files.per.chunk = 300)
379. })
382. ## 6.4 Gene Level Copy Number -------------------------------------------------------------------------
383. # Warning: There are more than one file for the same case. Please verify query results. You can use the command View(getResults(query)) in rstudio
385. projects <- c("TCGA-STAD", "TCGA-KIRP", "TCGA-THYM", "TCGA-KICH", "TCGA-LGG", "TCGA-LUSC", "TCGA-LUAD", "TCGA-HNSC")
387. sapply(projects, function(project){
388.   
389.   query <- GDCquery(
390.     project = project,
391.     data.category = "Copy Number Variation",
392.     data.type = "Gene Level Copy Number",              
393.     access = "open"
394.   )
395.   
396.   GDCdownload(query, method = "api", files.per.chunk = 30)
397. })

复制代码

 
参考文件

（1）全网最全！2021最新常用肿瘤生信数据库收藏级汇总！（1）
（2）全网最全！2021最新常用肿瘤生信数据库收藏级汇总！（2）
（3）收藏：常用医学公共数据库（含临床数据库，生信数据库和机器学习数据库）
（4）6大药敏性分析数据库大汇总，助力肿瘤相关生信分析！/SCI论文/科研/研究生/生信分析热
（5）新版TCGA数据库学习：批量下载新版TCGA数据
（6）TCGA 数据下载 —— TCGAbiolinks 简单使用
（7）4年新版TCGA GDC data portal 2.0界面介绍及数据下载教程
（8）TCGA 数据下载 —— TCGAbiolinks 数据分析
（9）【TCGA数据库介绍及应用】
（10）【TCGA数据库数据分析】

免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！更多信息从访问主页：qidao123.com:ToB企服之家，中国第一个企服评测及商务社交产业平台。