](quickstart.md)
+
+[
](setup_root.md)
+
+For more sysadmin details also check:
### HOW-TO GUIDES:
-* [How to define data models: for developers](setup_data_scheme.md). Detailed technical specifications on how to define data models for Zendro, aimed at software developers and system administrators.
-* [How to define data models: for non-developers](non-developer_documentation.md). A brief, illustrated guide of data model specifications, data formatting and data uploading options, aimed at data modelers or managers to facilitate collaboration with developers.
-* [How to setup a distributed cloud of zendro nodes](ddm.md). A brief guide, aimed at software developers and system administrators, on how to use Zendros distributed-data-models.
* [How to use Zendro command line interface (CLI)](zendro_cli.md). A tutorial of Zendro CLI, aimed at software developers.
-* [How to query and extract data](fromGraphQlToR.html). A concise guide on how to use the Zendro API from R to extract data and perform queries, aimed at data managers or data scientists.
* [How to setup Authentication / Authorization](oauth.md). A concise guide on how to use and setup the Zendro authorization / authentication services.
-* [API documentation](api_root.md).
+* [How to setup a distributed cloud of Zendro nodes](ddm.md). A brief guide, aimed at software developers and system administrators, on how to use Zendros distributed-data-models.
+* [API documentation](api_root.md). A summary of how Zendro backend generator implements a CRUD API that can be accessed through GraphQL query language.
-### REPOSITORIES:
+### Defining data models
-* [GraphQL server](https://github.com/ScienceDb/graphql-server)
-* [GraphQL server model generator](https://github.com/ScienceDb/graphql-server-model-codegen)
-* [Single page application](https://github.com/ScienceDb/single-page-app)
+* [How to define data models: for developers](setup_data_scheme.md). Detailed technical specifications on how to define data models for Zendro, aimed at software developers and system administrators.
+* [How to define data models: for non-developers](what_are_data_models.md). A brief, illustrated guide of data model specifications, data formatting and data uploading options, aimed at data modelers or managers to facilitate collaboration with developers.
+
+### Using Zendro day to day
+
+* [How to use Zendro's graphical interface](SPA_howto.md). A full guide on how to use Zendro's graphical point and click interface. Aimed to general users and featuring lots of screenshots.
+* [Introduction to GraphQL and querying the API](GraphQL_intro.md). A friendly intro to how to perform GraphQL queries and use GraphiQL documentation.
+* [How to query and extract data from R](fromGraphQlToR.html). A concise guide on how to use the Zendro API from R to extract data and perform queries, aimed at data managers or data
+* [How to use the Zendro API with python to make changes to the data](Zendro_requests_with_python.md). A concise guide on how to access the API using your user credentials to make CRUD operations on the data using python.
+
+## Zendro users profiles
-### CONTRIBUTIONS
+We designed Zendro to be useful for research teams and institutions that include users with different areas of expertise, needs and type of activities. The table below summarizes how we envision that different users will use Zendro:
+
+
+| Profile | Background | Expected use |
+|-------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| General user / scientist | GraphQL outputs the resutls of a query in batches of max 1,000
+elements. So if the data you want to download is larger than that, then
+you need to paginate, i.e. to get the data in batches.
+pagination is is an argument within GraphQL queries that
+could be done by:
Limit-offset: indicating the first element to get
+(offset, default 0) and the number of elements to get
+(limit). The limit can’t be larger than
+1000.
Cursor-based: indicating the unique ID
+(cursor) of the element to get first, and a number of
+elements to get after.
Zendro uses the limit-offset pagination with the syntaxis:
+pagination:{limit:[integer], offset:[integer]}
See GraphQL +documentation and this tutorial on GraphQL +pagination for more details.
+In the previous examples we downloaded only 10 elements
+(pagination:{limit:10})) from the rivers type, but the
+dataset is larger. (Remember, data in GraphQL is organised in
+types and fields within those types.
+When thinking about your structured data, you can think of types as the
+names of tables, and fields as the columns of those tables. In the
+example above rivers is a type and the fields are
+river_id, name, length among
+others.)
To know how many elements does a type has we can make a query with
+the function count, if it is available for the type we are
+interested on. You can check this in the Docs at the top
+right menu of the GraphiQL interface.
For example, rivers has the function
+countRivers so with the query {countRivers} we
+can get the total number of rivers.
Similar to how we got data before, you can use this very simple query
+in the function get_from_graphQL to get the number of
+rivers into R:
# query API with count function
+no_records<-get_from_graphQL(query="{countRivers}", url="https://zendro.conabio.gob.mx/api/graphql")
+
+# change to vector, we don't need a df
+no_records<-no_records[1,1]
+no_records## [1] 50In this case we have 50. Technically we could download all the data +in a single batch because it is <1000, but for demostration purposes +we will download it in batches of 10.
+The following code calculates the number of pages needed to get a
+given number of records assuming a desired limit (size of each batch).
+Then it runs get_from_graphQL() within a loop for each page
+until getting the total number of records desired.
# Define desired number of records and limit. Number of pages and offset will be estimated based on the number of records to download
+no_records<- no_records # this was estimated above with a query to count the total number of records, but you can also manually change it to a custom desired number
+my_limit<-10 # max 1000.
+no_pages<-ceiling(no_records/my_limit)
+
+## Define offseet.
+# You can use the following loop:
+# to calculate the offset automatically based on
+# on the number of pages needed.
+my_offset<-0 # start in 0. Leave like this
+for(i in 1:no_pages){ # loop to
+ my_offset<-c(my_offset, my_limit*i)
+}
+
+# Or you can define the offset manually
+# uncommenting the following line
+# and commenting the loop above:
+# my_offset<-c(#manually define your vector)
+
+## create object where to store downloaded data. Leave empty
+data<-character()
+
+##
+## Loop to download the data from GraphQL using pagination
+##
+
+for(i in c(1:length(my_offset))){
+
+# Define pagination
+pagination <- paste0("limit:", my_limit, ", offset:", my_offset[i])
+
+# Define query looping through desired pagination:
+my_query<- paste0("{
+ rivers(pagination:{", pagination, "}){
+ river_id
+ name
+ length
+ }
+ }
+ ")
+
+
+
+# Get data and add it to the already created df
+data<-rbind(data, get_from_graphQL(query=my_query, url="https://zendro.conabio.gob.mx/api/graphql"))
+
+#end of loop
+}As a result you will get all the data in a single df:
+head(data)summary(data)## river_id name length
+## Length:50 Length:50 Min. : 65.0
+## Class :character Class :character 1st Qu.: 150.0
+## Mode :character Mode :character Median : 283.0
+## Mean : 347.1
+## 3rd Qu.: 402.5
+## Max. :1521.0
+## NA's :6get_from_graphQL() explained step by stepThe following is a step-by-step example explaining with more detail
+how does the function get_from_graphQL() that we used above
+works.
First, once you have a GraphQL query working, you’ll need to save it +to an R object as a character vector:
+my_query<- "{
+rivers(pagination:{limit:10, offset:0}){
+ river_id
+ name
+ length
+ }
}
-' # write the query as a string
-result <- POST(url, body = list(query=accessions_query)) # fetch dataThe result that we are getting is the http response. We still need to extract the data in order to be able to manipulate it. If everything went well, the http response will contain an attribute data which will itself contain an attribute named as the query, in this case accessions.
jsonResult <- content(result, as = "text")
-readableText <- fromJSON(jsonResult)
-readableText$data$accessionsNext, define as another character vector the url of the API, which is +the same of the GraphiQL web interface you explored above:
+url<-"https://zendro.conabio.gob.mx/api/graphql"Now we can a query to the API by using a POST request:
+# query server
+result <- POST(url, body = list(query=my_query), encode = c("json"))The result that we are getting is the http response.
+Before checking if we got the data, it is good practice to verify if the
+connection was successful by checking the status code. A
+200 means that all went well. Any other code means
+problems. See this.
# check server response
+result$status_code## [1] 200We now need to extract the data in order to be able to manipulate it.
+If everything went well, the http response will contain an
+attribute data which will itself contain an attribute named as the
+query, in this case rivers.
result## Response [https://zendro.conabio.gob.mx/api/graphql]
+## Date: 2022-07-27 23:15
+## Status: 200
+## Content-Type: application/json; charset=utf-8
+## Size: 983 B
+## {
+## "data": {
+## "rivers": [
+## {
+## "river_id": "1",
+## "name": "Acaponeta",
+## "length": 233
+## },
+## {
+## "river_id": "10",
+## ...If the query is not written properly or if there is any other error,
+the attribute data won’t exist and instead we will get the
+attribute erros listing the errors found.
If all wen’t well we can proceed to extract the content of the +results with:
+# get data from query result
+jsonResult <- content(result, as = "text") The result will be in json format, which we can convert into an
+Robjet (list). In this list the results are within each type used in the
+query. The argumment flatten is used to collapse the list
+into a single data frame the data from different types.
# transform to json
+readableResult <- fromJSON(jsonResult,
+ flatten = T)Extract data:
+# get data
+data<-as.data.frame(readableResult$data[1])
+head(data)With the above code we were able to visualize the data fetched from zendro server. Next we will put this data in a table form in order to be able to manipulate the data.
-dataTable <- lapply(readableText[[1]], as.data.table)
-dataTable$accessionsBy default, the name of each type will be added a the beggining of +each column name:
+colnames(data)## [1] "rivers.river_id" "rivers.name" "rivers.length"To keep only the name of the variable as it is in the original +data:
+x<-str_match(colnames(data), "\\w*$")[,1] # matches word characters (ie not the ".") at the end of the string
+colnames(data)<-x # assing new colnames So finally we have the data in a single nice looking data frame:
+head(data) Notice that you will get a dataframe like teh one above only for one +to one associations, but than in other cases you still will get +variables that are a list, which you can process in a separate step.