Interchange format

In PRIMAP2, data is internally handled in xarray datasets with defined coordinates and metadata. On disk this structure is stored as a netcdf file. Because the netcdf file format was developed for the exchange of multi-dimensional datasets with a varying number of dimensions for different entities and rich meta data, we recommend that consumers of datasets published by us use the provided netcdf files.

However, we recognise that many existing workflows rely on tools that handle tabular data exclusively and therefore also publish in the PRIMAP2 Interchange Format which is a tabular wide format with additional meta data. Users of the interchange format have to integrate the given meta data carefully into their workflows to ensure correct results.

Logical format

In the interchange format all dimensions and time points are represented by columns in a two-dimensional array. Values of the time columns are data while values of the other columns are coordinates. To store metadata, including the information contained in the attrs dict in the PRIMAP2 xarray format, we use an additional dictionary. See sections In-memory representation and on-disk representation below for information on the storage of these structures.

The requirements for the data, columns, and coordinates follow the requirements in the standard PRIMAP2 data format. Dimensions area and source, which are mandatory in the xarray format, are mandatory columns in the tabular data in the interchange format. The time dimension is included in the horizontal dimension of the tabular interchange format. Additionally, we have unit and entity as mandatory columns with the restriction that each entity can have only one unit.

All optional dimensions (see Data format details) can be added as optional columns. Secondary categories are columns with free format names. They are listed as secondary columns in the metadata dict.

Column names correspond to the dimension key of the xarray format, i.e. they contain the terminology in parentheses (e.g. area (ISO3)).

Additional columns are currently not possible, but the option will be added in a future release (#25).

The metadata dict has an attrs entry, which corresponds to the attrs dict of the xarray format (see Data format details). Additionally, the metadata dict contains information on the dimensions of the data for each entity, on the time_format of the data columns and (if stored on disk) on the name of the data_file (see Interchange format details).

Use

The interchange format is intended for use mainly in two settings.

• To publish data processed using PRIMAP2 in a way that is easy to read by others but also keeps the internal structure and metadata. The format will be used by future data publications by the PRIMAP team including PRIMAP-hist.

• To have a common intermediate format for reading data from original sources (mostly xls or csv files in different formats) to simplify data reading functions and to enable use of our data reading functionality by other projects. All data is first read into the interchange format and subsequently converted into the native PRIMAP2 format. This enables using our data reading routines in other software packages.

In-memory representation

The in-memory representation of the interchange format is using a pandas DataFrame to store the data, and a dict to store the additional metadata. Pandas DataFrames have the capability to store the metadata on their attrs, however this function is still experimental and subject to change without notice, so care has to be taken not to lose the data if processing is done on the DataFrame. For an example see Examples section below.

On-disk representation

On disk the dataset is represented by a csv file containing the array, and a yaml file containing the additional metadata as a dict. Both files should have the same name except for the ending. On disk, the key data_file is added to the metadata dict, which contains the name of the csv file. Thus, a function reading interchange format data just needs the yaml file name to read the data.

Examples

Here we show a few examples of the interchange format.

# import all the used libraries
import primap2 as pm2

Reading csv data

The PRIMAP2 data reading procedures first convert data into the interchange format. For explanations of the used parameters see the Data reading example. A more complex dataset is read in Data reading PRIMAP-hist.

file = "data_reading_writing_examples/test_csv_data_sec_cat.csv"
coords_cols = {
    "unit": "unit",
    "entity": "gas",
    "area": "country",
    "category": "category",
    "sec_cats__Class": "classification",
}
coords_defaults = {
    "source": "TESTcsv2021",
    "sec_cats__Type": "fugitive",
    "scenario": "HISTORY",
}
coords_terminologies = {
    "area": "ISO3",
    "category": "IPCC2006",
    "sec_cats__Type": "type",
    "sec_cats__Class": "class",
    "scenario": "general",
}
coords_value_mapping = {
    "category": "PRIMAP1",
    "entity": "PRIMAP1",
    "unit": "PRIMAP1",
}
data_if = pm2.pm2io.read_wide_csv_file_if(
    file,
    coords_cols=coords_cols,
    coords_defaults=coords_defaults,
    coords_terminologies=coords_terminologies,
    coords_value_mapping=coords_value_mapping,
)
data_if.head()

	source	scenario (general)	area (ISO3)	entity	unit	category (IPCC2006)	Class (class)	Type (type)	1991	2000	2010
0	TESTcsv2021	HISTORY	AUS	CO2	Gg CO2 / yr	1	TOTAL	fugitive	4000.00	5000.00	6000.00
1	TESTcsv2021	HISTORY	AUS	KYOTOGHG (SARGWP100)	Mt CO2 / yr	0	TOTAL	fugitive	8.00	9.00	10.00
2	TESTcsv2021	HISTORY	FRA	CH4	Gg CH4 / yr	2	TOTAL	fugitive	7.00	8.00	9.00
3	TESTcsv2021	HISTORY	FRA	CO2	Gg CO2 / yr	2	TOTAL	fugitive	12.00	13.00	14.00
4	TESTcsv2021	HISTORY	FRA	KYOTOGHG (SARGWP100)	Mt CO2 / yr	0	TOTAL	fugitive	0.03	0.02	0.04

Writing interchange format data

Data is written using the pm2io.write_interchange_format function which takes a filename and path (str or pathlib.Path), an interchange format dataframe (pandas.DataFrame) and optionally an attribute dict as inputs. If the filename has an ending, it will be ignored. The function writes a yaml file and a csv file.

file_if = "data_reading_writing_examples/test_csv_data_sec_cat_if"
pm2.pm2io.write_interchange_format(file_if, data_if)

Reading data from disk

To read interchange format data from disk the function pm2io.read_interchange_format is used. It just takes a filename and path as input (str or pathlib.Path) and returns a pandas.DataFrame containing the data and metadata. The filename and path has to point to the yaml file. the csv file will be read from the filename contained in the yaml file.

data_if_read = pm2.pm2io.read_interchange_format(file_if)
data_if_read.head()

	source	scenario (general)	area (ISO3)	entity	unit	category (IPCC2006)	Class (class)	Type (type)	1991	2000	2010
0	TESTcsv2021	HISTORY	AUS	CO2	Gg CO2 / yr	1	TOTAL	fugitive	4000.0000000000005	5000.000000000001	6000.000000000001
1	TESTcsv2021	HISTORY	AUS	KYOTOGHG (SARGWP100)	Mt CO2 / yr	0	TOTAL	fugitive	8.0	9.0	10.0
2	TESTcsv2021	HISTORY	FRA	CH4	Gg CH4 / yr	2	TOTAL	fugitive	7.0	8.0	9.0
3	TESTcsv2021	HISTORY	FRA	CO2	Gg CO2 / yr	2	TOTAL	fugitive	12.0	13.0	14.0
4	TESTcsv2021	HISTORY	FRA	KYOTOGHG (SARGWP100)	Mt CO2 / yr	0	TOTAL	fugitive	0.03	0.02	0.04

Converting to and from standard PRIMAP2 format

Data in the standard, xarray-based PRIMAP2 format can be converted to and from the interchange format with the corresponding functions:

ds_minimal = pm2.open_dataset("minimal_ds.nc")

if_minimal = ds_minimal.pr.to_interchange_format()

if_minimal.head()

	source	area (ISO3)	entity	unit	2000	2001	2002	2003	2004	2005	...	2011	2012	2013	2014	2015	2016	2017	2018	2019	2020
0	RAND2020	ARG	CH4	CH4 * gigagram / year	0.368532	0.088137	0.298864	0.678659	0.862995	0.131253	...	0.867758	0.874488	0.845115	0.510803	0.108674	0.822922	0.767920	0.071981	0.025005	0.089894
1	RAND2020	ARG	CO2	CO2 * gigagram / year	0.916832	0.682339	0.250728	0.852096	0.881125	0.147748	...	0.612935	0.966494	0.376384	0.487788	0.710766	0.945482	0.131772	0.938776	0.658307	0.629569
2	RAND2020	ARG	SF6	SF6 * gigagram / year	0.505571	0.526975	0.618358	0.497167	0.559072	0.943795	...	0.569415	0.044608	0.587069	0.657630	0.027363	0.711145	0.018133	0.997718	0.268657	0.761415
3	RAND2020	ARG	SF6 (SARGWP100)	CO2 * gigagram / year	12083.153938	12594.698995	14778.752239	11882.298866	13361.818016	22556.691567	...	13609.030141	1066.140102	14030.947344	15717.359528	653.987580	16996.374472	433.377608	23845.461667	6420.891957	18197.817082
4	RAND2020	BOL	CH4	CH4 * gigagram / year	0.565378	0.036782	0.752872	0.247971	0.305199	0.094644	...	0.374838	0.406868	0.352221	0.965541	0.037689	0.402788	0.173916	0.409820	0.093950	0.553759

5 rows × 25 columns

ds_minimal_re = pm2.pm2io.from_interchange_format(if_minimal)

ds_minimal_re

2023-05-10 09:53:21.886 | DEBUG    | primap2.pm2io._interchange_format:from_interchange_format:320 - Expected array shapes: [[21, 4, 1, 4], [21, 4, 1, 4], [21, 4, 1, 4], [21, 4, 1, 4]], resulting in size 1,344.

<xarray.Dataset>
Dimensions:          (time: 21, source: 1, area (ISO3): 4)
Coordinates:
  * source           (source) object 'RAND2020'
  * area (ISO3)      (area (ISO3)) object 'ARG' 'BOL' 'COL' 'MEX'
  * time             (time) datetime64[ns] 2000-01-01 2001-01-01 ... 2020-01-01
Data variables:
    CH4              (time, source, area (ISO3)) float64 [CH4·Gg/a] 0.3685 .....
    CO2              (time, source, area (ISO3)) float64 [CO2·Gg/a] 0.9168 .....
    SF6              (time, source, area (ISO3)) float64 [Gg·SF6/a] 0.5056 .....
    SF6 (SARGWP100)  (time, source, area (ISO3)) float64 [CO2·Gg/a] 1.208e+04...
Attributes:
    area:     area (ISO3)

xarray.Dataset

• Dimensions:
  • time: 21
  • source: 1
  • area (ISO3): 4
• Coordinates: (3)
  • source
    (source)
    object
    'RAND2020'

    array(['RAND2020'], dtype=object)

  • area (ISO3)
    (area (ISO3))
    object
    'ARG' 'BOL' 'COL' 'MEX'

    array(['ARG', 'BOL', 'COL', 'MEX'], dtype=object)

  • time
    (time)
    datetime64[ns]
    2000-01-01 ... 2020-01-01

    array(['2000-01-01T00:00:00.000000000', '2001-01-01T00:00:00.000000000',
           '2002-01-01T00:00:00.000000000', '2003-01-01T00:00:00.000000000',
           '2004-01-01T00:00:00.000000000', '2005-01-01T00:00:00.000000000',
           '2006-01-01T00:00:00.000000000', '2007-01-01T00:00:00.000000000',
           '2008-01-01T00:00:00.000000000', '2009-01-01T00:00:00.000000000',
           '2010-01-01T00:00:00.000000000', '2011-01-01T00:00:00.000000000',
           '2012-01-01T00:00:00.000000000', '2013-01-01T00:00:00.000000000',
           '2014-01-01T00:00:00.000000000', '2015-01-01T00:00:00.000000000',
           '2016-01-01T00:00:00.000000000', '2017-01-01T00:00:00.000000000',
           '2018-01-01T00:00:00.000000000', '2019-01-01T00:00:00.000000000',
           '2020-01-01T00:00:00.000000000'], dtype='datetime64[ns]')

• Data variables: (4)
  • CH4
    (time, source, area (ISO3))
    float64
    [CH4·Gg/a] 0.3685 ... 0.02305

    entity :

    CH4

    Magnitude

    [[[0.3685320940353728 0.5653780221221959 0.7542549558511776 0.7369296426004138]] [[0.08813720839288863 0.03678244010434961 0.8364808087479632 0.6269624635226865]] [[0.29886414437019804 0.752872424535261 0.3264265671890325 0.433428122526182]] [[0.6786592295407908 0.2479712511975457 0.99013758070399 0.724861880203653]] [[0.8629953227625237 0.3051994326528302 0.9168491852422332 0.6589798491488441]] [[0.13125277522126044 0.09464381146488332 0.363954789171993 0.5317324546827301]] [[0.9200030821096901 0.38356553316313813 0.37219028327011106 0.5765632290541318]] [[0.6988319605873081 0.29813804128060784 0.21173423093184363 0.07685377058063037]] [[0.9449887033172708 0.691475104643898 0.8256908816307077 0.029675864516146455]] [[0.7364362832821014 0.7673430557143591 0.5508098263229796 0.7567733852678931]] [[0.8079833097460629 0.9945274253910127 0.48842796850443415 0.6463223733753957]] [[0.8677580617075967 0.37483753711409995 0.12629758359095455 0.27030619541411705]] [[0.87448806091195 0.4068677527163894 0.3156350508768655 0.8384328875893682]] [[0.8451145147260058 0.3522207334108044 0.01971608147418813 0.6842069975800996]] [[0.5108025262787018 0.9655410083920338 0.23412165329055146 0.45759346753578756]] [[0.10867412804182908 0.03768909532940734 0.18459977602749444 0.669245140924568]] [[0.8229215128954231 0.40278785991644206 0.8286652785948826 0.25918844188963486]] [[0.7679201489444207 0.17391590049126393 0.8267430216430773 0.1588857548770053]] [[0.07198107823046473 0.40981960314708143 0.08214070760255643 0.020714404278233012]] [[0.025004948745150135 0.09394984504371917 0.9703624792874707 0.7211716718406348]] [[0.08989404965510184 0.5537587740490154 0.5485158893136658 0.02304916212005137]]]

    Units

    CH4 gigagram/year

  • CO2
    (time, source, area (ISO3))
    float64
    [CO2·Gg/a] 0.9168 0.6142 ... 0.1863

    entity :

    CO2

    Magnitude

    [[[0.9168318265574015 0.6141634063871825 0.6635201782042437 0.10638744826946733]] [[0.682338632666392 0.7905772609283975 0.3382253292557216 0.7395385058805823]] [[0.25072806564646855 0.4087390697622033 0.11409100697947938 0.8504489730663606]] [[0.8520959700808077 0.08031264625923029 0.801477073222692 0.7152128934482338]] [[0.8811252096269836 0.111051447212553 0.6101888801109169 0.5550000928935479]] [[0.14774847859299522 0.13586945518522808 0.812456122446051 0.14072428619992494]] [[0.9547228137993204 0.5212602891531959 0.09237210162677412 0.7905119579747139]] [[0.26581566767403764 0.43650759225482505 0.6546441668962568 0.9689908809264166]] [[0.5975266186962794 0.5551216597262866 0.6612736041974472 0.785099419487452]] [[0.05037719350908554 0.6517388086838728 0.33259990823381236 0.3369210750624343]] [[0.9106197882021331 0.5275309558025338 0.5522010519840608 0.8234802866938398]] [[0.6129352532104692 0.8143255189588381 0.547885968428845 0.7602238439362549]] [[0.9664940577619263 0.33780248501934773 0.5705158101627505 0.2635155897055561]] [[0.3763835226478044 0.34866229530317505 0.14997833554506934 0.1647368812064165]] [[0.4877882315623534 0.9163693640320614 0.4634965654407963 0.037042328100195476]] [[0.7107660047838966 0.40996965629779725 0.6455829370274393 0.4195285063193993]] [[0.9454818882042824 0.6809799063677471 0.4758651094425167 0.03006484586084257]] [[0.13177214894370792 0.07862391306443517 0.009667561812883863 0.43758244853013717]] [[0.9387762091651058 0.44603773344001674 0.24073322602911418 0.6674664205953101]] [[0.6583069409300996 0.7549426352132018 0.12764283003356602 0.19115895273354766]] [[0.6295689201632558 0.056990272829468 0.5548269020548341 0.1863119137604362]]]

    Units

    CO2 gigagram/year

  • SF6
    (time, source, area (ISO3))
    float64
    [Gg·SF6/a] 0.5056 0.1236 ... 0.2628

    entity :

    SF6

    Magnitude

    [[[0.5055712944941024 0.1235696696610431 0.0017999283398781873 0.6764281356992125]] [[0.5269748533552134 0.8096243456920031 0.4777477627293901 0.8016230618391313]] [[0.6183578342728786 0.3085480136136496 0.5331652257091747 0.23948967174374636]] [[0.49716731657668833 0.6836916441599878 0.5323686911928804 0.4340997424406753]] [[0.5590718835247671 0.33905171176763205 0.6672625868689808 0.9382487089014679]] [[0.9437946262290234 0.6703211769514555 0.1810063661773531 0.7570997700454333]] [[0.07422128273206374 0.3392561054061216 0.08218973282583775 0.8080948951738635]] [[0.6427237374065423 0.8051477668558303 0.11494516659459275 0.6470840960951291]] [[0.5204792846286578 0.621097578936615 0.15393918278957064 0.0889369331164156]] [[0.1699100337618633 0.39183340587122273 0.8996772186819172 0.6274569096295286]] [[0.23459696651127115 0.7568653740660694 0.9947633418025775 0.9471595930522158]] [[0.5694154870543 0.7174046976355728 0.11666812059542109 0.2972097217084779]] [[0.04460837247920335 0.9040802399068613 0.9083554803854986 0.19494181397753896]] [[0.5870689265407956 0.5804577179083541 0.9194928308313104 0.47963603827269163]] [[0.6576301057548541 0.5584057498489272 0.7811415855268452 0.5261474587632892]] [[0.027363497052910413 0.050565352755786 0.9128417649366534 0.9751450575411759]] [[0.7111453753978566 0.5350717635605707 0.6708354594464743 0.6874428941931122]] [[0.018132954296472215 0.5883767927200321 0.5912540921962742 0.011587687621276155]] [[0.9977180613859968 0.8229456655620809 0.5540699967566246 0.1775038540227457]] [[0.26865656723726417 0.1666851588620759 0.9649063147146625 0.9595548786166266]] [[0.7614149406593684 0.04207827124056718 0.9537394170806007 0.2628449737891362]]]

    Units

    SF6 gigagram/year

  • SF6 (SARGWP100)
    (time, source, area (ISO3))
    float64
    [CO2·Gg/a] 1.208e+04 ... 6.282e+03

    gwp_context :

    SARGWP100

    entity :

    SF6

    Magnitude

    [[[12083.153938409048 2953.3151048989303 43.01828732308868 16166.632443211178]] [[12594.698995189603 19350.021862038877 11418.171529232424 19158.791177955238]] [[14778.752239121799 7374.297525366225 12742.648894449276 5723.803154675538]] [[11882.298866182851 16340.23029542371 12723.611719509843 10374.98384433214]] [[13361.818016241936 8103.335911246406 15947.575826168642 22424.144142745085]] [[22556.691566873662 16020.676129139789 4326.052151638739 18094.684504085857]] [[1773.8886572963236 8108.220919206307 1964.3346145375222 19313.467994655337]] [[15361.097324016362 19243.031627854347 2747.189481610767 15465.309896673587]] [[12439.454902624922 14844.232136585098 3679.146468670738 2125.592701482333]] [[4060.8498069085335 9364.818400322223 21502.285526497824 14996.220140145735]] [[5606.86749961938 18089.08244017906 23774.843869081604 22637.11427394796]] [[13609.03014059777 17145.97227349019 2788.368082230564 7103.312348832623]] [[1066.14010225296 21607.517733773984 21709.69598121342 4659.109354063181]] [[14030.947344325015 13872.939458009665 21975.87865686832 11463.301314717331]] [[15717.359527541017 13345.89742138936 18669.2838940916 12574.924264442612]] [[653.9875795645589 1208.5119308632854 21816.918181986017 23305.966875234106]] [[16996.374472008774 12788.215149097641 16032.967480770736 16429.885171215385]] [[433.3776076856859 14062.205346008766 14130.972803490955 276.94573414850015]] [[23845.461667125324 19668.401406933735 13242.27292248333 4242.342111143623]] [[6420.891956970614 3983.7752968036143 23061.260921680434 22933.36159893738]] [[18197.81708175891 1005.6706826495556 22794.37206822636 6281.994873560355]]]

    Units

    CO2 gigagram/year

• Indexes: (3)
  • source
    PandasIndex

    PandasIndex(Index(['RAND2020'], dtype='object', name='source'))

  • area (ISO3)
    PandasIndex

    PandasIndex(Index(['ARG', 'BOL', 'COL', 'MEX'], dtype='object', name='area (ISO3)'))

  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2000-01-01', '2001-01-01', '2002-01-01', '2003-01-01',
                   '2004-01-01', '2005-01-01', '2006-01-01', '2007-01-01',
                   '2008-01-01', '2009-01-01', '2010-01-01', '2011-01-01',
                   '2012-01-01', '2013-01-01', '2014-01-01', '2015-01-01',
                   '2016-01-01', '2017-01-01', '2018-01-01', '2019-01-01',
                   '2020-01-01'],
                  dtype='datetime64[ns]', name='time', freq=None))

• Attributes: (1)

  area :

  area (ISO3)