Getting Started with datagovindia

Setup

library(datagovindia)

API Discovery

The APIs from the portal are scraped every week to update a list of all APIs and the information attached to them like sector, source, field names etc. The website data.gov.in provides a search functionality through string searches and drop down menus but these are very limited. The functions in this package allows one to have more robust string based searches.
A user can search by API title, description, organization type, organization (ministry), sector and sources. Briefly there are two types of functions here, the first lets the user get a list of all available and unique organization type, organization (ministry), sector and sources and the other lets one “search” by these criteria and more.

Here is a demonstration of the former (getting only the first few values)

###List of organizations (or ministries)
get_list_of_organizations() %>% 
  head
#> [1] "Ministry of Agriculture and Farmers Welfare"               
#> [2] "Department of Agriculture, Cooperation and Farmers Welfare"
#> [3] "Directorate of Marketing and Inspection (DMI)"             
#> [4] "Ministry of Environment, Forest and Climate Change"        
#> [5] "Central Pollution Control Board"                           
#> [6] "Ministry of Micro, Small and Medium Enterprises"

###List of sectors 
get_list_of_sectors() %>% 
  head
#> [1] "Agriculture"               "Agricultural Marketing"   
#> [3] "Environment and Forest"    "Vehicular Air Pollution"  
#> [5] "Residential Air Pollution" "Industrial Air Pollution"

##Single Criteria
search_api_by_title(title_contains = "pollution") %>% head(2)
#> Warning in as.POSIXlt.POSIXct(x, tz): unknown timezone 'Asia/Caclutta'

#> Warning in as.POSIXlt.POSIXct(x, tz): unknown timezone 'Asia/Caclutta'

##Multiple Criteria
dplyr::intersect(search_api_by_title(title_contains = "pollution"),
                 search_api_by_organization(organization_name_contains = "pollution"))
#> Warning in as.POSIXlt.POSIXct(x, tz): unknown timezone 'Asia/Caclutta'

#> Warning in as.POSIXlt.POSIXct(x, tz): unknown timezone 'Asia/Caclutta'

Getting more information about a chosen API

There are two functions in this section, one to get API information, the other to get a available “field” names and types of the chosen API (using it’s index_name obtained above).

get_api_info(api_index = "0579cf1f-7e3b-4b15-b29a-87cf7b7c7a08")
#> Warning in as.POSIXlt.POSIXct(x, tz): unknown timezone 'Asia/Caclutta'

#> Warning in as.POSIXlt.POSIXct(x, tz): unknown timezone 'Asia/Caclutta'

get_api_fields(api_index = "0579cf1f-7e3b-4b15-b29a-87cf7b7c7a08")

Querying the chosen API

The function get_api_data is really the powerhouse in this package which allows one to do things over and above a manually constructed API query can do by utilizing the data.frame structure of the underlying data. It allows the user to filter, sort, select variables and to decide how much of the data to extract. The website can itself filter on only one field with one value at a time but one command through the wrapper can make multiple requests and append the results from these requests at the same time.

But before we dive into data extraction, we first need to validate our API key relieved from data.gov.in. To get the key, you need to register first register and then get the key from your “My Account” page after logging in. More instruction can be found on this official guide. Once you get your API key, you can validate it as follows (only need to do this once per session) :

##Using a sample key
register_api_key("579b464db66ec23bdd000001cdd3946e44ce4aad7209ff7b23ac571b")
#> Connected to the internet
#> The server is online
#> The API key is valid and you won't have to set it again

Once you have your key registered, you are ready to extract data from a chosen API. Here is what each argument means :

• api_index : index_name of the chosen API (found by using search functions)
• results_per_req : Results per request sent to the server ; can take integer values or the string “all” to get all of the available data
• filter_by : A named character vector of field id (not the name) - value(s) pairs ; can take multiple fields as well as multiple comma separated values
• field_select : A character vector of fields to select only a subset of variables in the final data.frame
• sort_by : Sort by one or multiple fields

To recap, first find the API you want using the search functions, get the index_name of the API from the results, optionally take a look at the fields present in the data of the API and then use the get_api_data function to extract the data. Suppose we choose the API “Real time Air Quality Index from various location” with index_ name 3b01bcb8-0b14-4abf-b6f2-c1bfd384ba69. First we will look at which fields are available to construct the right query.
Suppose We want to get the data from only 2 cities Chandigarh and Gurugram and pollutants PM10 and NO2. We will let all fields to be returned (dataset columns).

We will use a sample key from the website for this demonstration.

register_api_key("579b464db66ec23bdd0000019fc84f43ca52437351b43702f5998234")
#> Connected to the internet
#> The server is online
#> The API key is valid and you won't have to set it again

We now look at the fields available to play with.

get_api_fields("3b01bcb8-0b14-4abf-b6f2-c1bfd384ba69")

We accordingly select the city and pollution_id fields for constructing our query. Note that we use only field id to finally query the data.

get_api_data(api_index="3b01bcb8-0b14-4abf-b6f2-c1bfd384ba69",
             results_per_req=10,filter_by=c(city="Gurugram,Chandigarh",
                                            polutant_id="PM10,NO2"),
             field_select=c(),
             sort_by=c('state','city'))
#> Connected to the internet
#> The server is online
#> url-https://api.data.gov.in/resource/3b01bcb8-0b14-4abf-b6f2-c1bfd384ba69?api-key=579b464db66ec23bdd0000019fc84f43ca52437351b43702f5998234&format=json&offset=0&limit=10&filters[city]=Gurugram&filters[polutant_id]=PM10
#> gave the API a rest
#> url-https://api.data.gov.in/resource/3b01bcb8-0b14-4abf-b6f2-c1bfd384ba69?api-key=579b464db66ec23bdd0000019fc84f43ca52437351b43702f5998234&format=json&offset=0&limit=10&filters[city]=Chandigarh&filters[polutant_id]=PM10
#> gave the API a rest
#> url-https://api.data.gov.in/resource/3b01bcb8-0b14-4abf-b6f2-c1bfd384ba69?api-key=579b464db66ec23bdd0000019fc84f43ca52437351b43702f5998234&format=json&offset=0&limit=10&filters[city]=Gurugram&filters[polutant_id]=NO2
#> gave the API a rest
#> url-https://api.data.gov.in/resource/3b01bcb8-0b14-4abf-b6f2-c1bfd384ba69?api-key=579b464db66ec23bdd0000019fc84f43ca52437351b43702f5998234&format=json&offset=0&limit=10&filters[city]=Chandigarh&filters[polutant_id]=NO2
#> gave the API a rest
#> No results returned - check your api_index