{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Data reading example 4 - FAOstat #\n", "To run this example the file `Emissions_Agriculture_Cultivated_Organic_Soils_E_All_Data_(Normalized).csv` must be placed in the same folder as this notebook.\n", "The data is available from the [FAOstat](http://www.fao.org/faostat/en/#data/GV/metadata)." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import primap2 as pm2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Dataset Specifications ##\n", "Here we define which columns of the csv file contain the dimensions." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "file = \"Emissions_Agriculture_Cultivated_Organic_Soils_E_All_Data_(Normalized).csv\"\n", "coords_cols = {\n", " \"unit\": \"Unit\",\n", " \"entity\": \"Element\",\n", " \"area\": \"Area\",\n", " \"category\": \"Item\",\n", " \"data\": \"Value\",\n", " \"time\": \"Year\",\n", "}\n", "coords_defaults = {\n", " \"source\": \"FAOstat\",\n", "}\n", "coords_terminologies = {\n", " \"area\": \"FAOstat\",\n", " \"category\": \"FAOstat\",\n", "}\n", "\n", "# TODO: proper mapping of the area to ISO3\n", "coords_value_mapping = {\n", " \"unit\": {\"gigagrams\": \"Gg N2O / year\"},\n", " \"entity\": {\"Emissions (N2O) (Cultivation of organic soils)\": \"N2O\"},\n", "}\n", "\n", "filter_keep = {\n", " \"f1\": {\n", " \"Element\": \"Emissions (N2O) (Cultivation of organic soils)\",\n", " },\n", "}\n", "\n", "meta_data = {\n", " \"references\": \"http://www.fao.org/faostat/en/#data/GV/metadata\"\n", "}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Reading the data to interchange format ##" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": " source area (FAOstat) entity unit \\\n0 FAOstat Africa N2O Gg N2O / year \n1 FAOstat Africa N2O Gg N2O / year \n2 FAOstat Africa N2O Gg N2O / year \n3 FAOstat Albania N2O Gg N2O / year \n4 FAOstat Albania N2O Gg N2O / year \n\n category (FAOstat) 1990 1991 1992 1993 \\\n0 Cropland and grassland organic soils 34.3224 34.3224 34.3224 34.3224 \n1 Cropland organic soils 17.4597 17.4597 17.4597 17.4597 \n2 Grassland organic soils 16.8627 16.8627 16.8627 16.8627 \n3 Cropland and grassland organic soils 0.0471 0.0471 0.0471 0.0471 \n4 Cropland organic soils 0.0378 0.0378 0.0378 0.0378 \n\n 1994 ... 2012 2013 2014 2015 2016 2017 \\\n0 34.3211 ... 35.2262 35.2203 35.1451 35.1434 35.1327 35.1277 \n1 17.4588 ... 18.1675 18.1546 18.1318 18.1307 18.1014 18.0904 \n2 16.8623 ... 17.0586 17.0657 17.0133 17.0127 17.0313 17.0373 \n3 0.0471 ... 0.0470 0.0470 0.0470 0.0470 0.0468 0.0468 \n4 0.0378 ... 0.0377 0.0377 0.0377 0.0377 0.0376 0.0376 \n\n 2018 2019 2030 2050 \n0 35.0546 35.0546 35.0546 35.0546 \n1 18.0273 18.0273 18.0273 18.0273 \n2 17.0272 17.0272 17.0272 17.0272 \n3 0.0460 0.0460 0.0460 0.0460 \n4 0.0369 0.0369 0.0369 0.0369 \n\n[5 rows x 37 columns]", "text/html": "
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
sourcearea (FAOstat)entityunitcategory (FAOstat)19901991199219931994...2012201320142015201620172018201920302050
0FAOstatAfricaN2OGg N2O / yearCropland and grassland organic soils34.322434.322434.322434.322434.3211...35.226235.220335.145135.143435.132735.127735.054635.054635.054635.0546
1FAOstatAfricaN2OGg N2O / yearCropland organic soils17.459717.459717.459717.459717.4588...18.167518.154618.131818.130718.101418.090418.027318.027318.027318.0273
2FAOstatAfricaN2OGg N2O / yearGrassland organic soils16.862716.862716.862716.862716.8623...17.058617.065717.013317.012717.031317.037317.027217.027217.027217.0272
3FAOstatAlbaniaN2OGg N2O / yearCropland and grassland organic soils0.04710.04710.04710.04710.0471...0.04700.04700.04700.04700.04680.04680.04600.04600.04600.0460
4FAOstatAlbaniaN2OGg N2O / yearCropland organic soils0.03780.03780.03780.03780.0378...0.03770.03770.03770.03770.03760.03760.03690.03690.03690.0369
\n

5 rows × 37 columns

\n
" }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "AgN2O_if = pm2.pm2io.read_long_csv_file_if(\n", " file,\n", " coords_cols=coords_cols,\n", " coords_defaults=coords_defaults,\n", " coords_terminologies=coords_terminologies,\n", " coords_value_mapping=coords_value_mapping,\n", " filter_keep=filter_keep,\n", " meta_data=meta_data,\n", " time_format=\"%Y\",\n", ")\n", "AgN2O_if.head()" ] }, { "cell_type": "code", "execution_count": 4, "outputs": [ { "data": { "text/plain": "{'attrs': {'references': 'http://www.fao.org/faostat/en/#data/GV/metadata',\n 'area': 'area (FAOstat)',\n 'cat': 'category (FAOstat)'},\n 'time_format': '%Y',\n 'dimensions': {'*': ['source',\n 'area (FAOstat)',\n 'entity',\n 'unit',\n 'category (FAOstat)']}}" }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "AgN2O_if.attrs" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Transformation to PRIMAP2 xarray format ##\n", "The transformation to PRIMAP2 xarray format is done using the function `from_interchange_format` which takes an interchange format DataFrame.\n", "The resulting xr Dataset is already quantified, thus the variables are pint arrays which include a unit." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2021-04-13 17:03:06.126 | DEBUG | primap2.pm2io._interchange_format:from_interchange_format:266 - Expected array shapes: [[1, 147, 1, 3]], resulting in size 441.\n", "2021-04-13 17:03:06.145 | INFO | primap2._data_format:ensure_valid_attributes:245 - Reference information is not a DOI: 'http://www.fao.org/faostat/en/#data/GV/metadata'\n" ] }, { "data": { "text/plain": "\nDimensions: (area (FAOstat): 147, category (FAOstat): 3, source: 1, time: 32)\nCoordinates:\n * time (time) datetime64[ns] 1990-01-01 ... 2050-01-01\n * source (source) object 'FAOstat'\n * category (FAOstat) (category (FAOstat)) object 'Cropland and grassland o...\n * area (FAOstat) (area (FAOstat)) object 'Africa' 'Albania' ... 'Zambia'\nData variables:\n N2O (time, source, category (FAOstat), area (FAOstat)) float64 [Gg·N2O/a] ...\nAttributes:\n references: http://www.fao.org/faostat/en/#data/GV/metadata\n area: area (FAOstat)\n cat: category (FAOstat)", "text/html": "
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
<xarray.Dataset>\nDimensions:             (area (FAOstat): 147, category (FAOstat): 3, source: 1, time: 32)\nCoordinates:\n  * time                (time) datetime64[ns] 1990-01-01 ... 2050-01-01\n  * source              (source) object 'FAOstat'\n  * category (FAOstat)  (category (FAOstat)) object 'Cropland and grassland o...\n  * area (FAOstat)      (area (FAOstat)) object 'Africa' 'Albania' ... 'Zambia'\nData variables:\n    N2O                 (time, source, category (FAOstat), area (FAOstat)) float64 [Gg·N2O/a] ...\nAttributes:\n    references:  http://www.fao.org/faostat/en/#data/GV/metadata\n    area:        area (FAOstat)\n    cat:         category (FAOstat)
" }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "AgN2O = pm2.pm2io.from_interchange_format(AgN2O_if)\n", "AgN2O" ] }, { "cell_type": "code", "execution_count": 5, "outputs": [], "source": [], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.5" } }, "nbformat": 4, "nbformat_minor": 4 }