GAF "filtered in order to reduce redundancy" meaning

[haerangl]

I downloaded the dataset goa_human.gaf from Gene Ontology Consortium. The description in the file and on the website say that this dataset has been "filtered in order to reduce redundancy". What does that mean?

My best guess
I found the release pipeline and the Annotation QC checks: http://wiki.geneontology.org/index.php/Release_Pipeline#Annotation_QC_checks
And this pointed me to the GO rules: https://github.com/geneontology/go-site/blob/master/metadata/rules/README.md

There are so many rules here that I don't fully understand these. However, these look more or less like quality assurance filters to make sure the data is clean and usable. What I might be worried about is a systematic removal of GO terms that I did not consider/assume. My goal is to use these GO to find protein functional similarity (based on information content and jaccard similarity - see Funsim measures section of this paper). I'd like to ensure that a chunk of the data wasn't just removed for a totally different goal/purpose (e.g., certain level of frequency that isn't useful for a specific and different task such as gene set enrichment analysis).

Any insights?

More info

Website description (http://current.geneontology.org/products/pages/downloads.html)

Filtered Files

These files are taxon-specific and reflect the work of specific projects, primarily the model organisms database groups, to provide comprehensive, non-redundant annotation files for their organism. All the files in this table have been filtered using the annotation file QC pipeline. A major component to the filtering is the requirement that particular taxon IDs can only be included within the association files provided by specific projects; the current list of authoritative groups and major model organisms can be found below.

Data downloaded:

Homo sapiens
EBI Gene Ontology Annotation Database (goa) 	protein 	543477 	goa_human.gaf (gzip)

Data dictionary: http://geneontology.org/docs/go-annotation-file-gaf-format-2.1/

Data header:

!gaf-version: 2.2
!
!generated-by: GOC
!
!date-generated: 2021-10-27T15:09
!
!Header from source association file:
!=================================
!
!generated-by: GOC
!
!date-generated: 2021-10-27T04:08
!
!Header from goa_human source association file:
!=================================
!
!The set of protein accessions included in this file is based on UniProt reference proteomes, which provide one protein per gene.
!They include the protein sequences annotated in Swiss-Prot or the longest TrEMBL transcript if there is no Swiss-Prot record.
!If a particular protein accession is not annotated with GO, then it will not appear in this file.
!
!Note that the annotation set in this file is filtered in order to reduce redundancy; the full, unfiltered set can be found in
!ftp://ftp.ebi.ac.uk/pub/databases/GO/goa/UNIPROT/goa_uniprot_all.gz
!
!date-generated: 2021-06-16 11:28
!generated-by: UniProt
!go-version: http://purl.obolibrary.org/obo/go/releases/2021-06-06/extensions/go-plus.owl
!
!=================================
!
!Header copied from paint_goa_human_valid.gaf
!=================================
!Created on Wed Sep  8 16:26:55 2021.
!generated-by: PANTHER
!date-generated: 2021-09-08
!PANTHER version: v.15.0.
!GO version: 2021-08-18.
!
!=================================
!
!Documentation about this header can be found here: https://github.com/geneontology/go-site/blob/master/docs/gaf_validation.md

[suzialeksander]

Hi @haerangl,
The 'filtering for redundancy' refers to removal or non-reference proteome entries - see https://www.uniprot.org/help/reference_proteome. We don't remove terms, or groups of terms, for reasons other than those listed in the GO_RULES, such as annotations to obsolete terms, or annotations missing supporting information.

The http://current.geneontology.org/annotations/goa_uniprot_all.gaf.gz file has also gone through the QA for completeness of annotations, but you are free to use that & filter yourself on taxon if you have any concerns- this is more useful for species we don't produce a file for e.g. cucumber.