Interchange format

In PRIMAP2, data is internally handled in xarray datasets with defined coordinates and metadata. On disk this structure is stored as a netcdf file. Because the netcdf file format was developed for the exchange of multi-dimensional datasets with a varying number of dimensions for different entities and rich meta data, we recommend that consumers of datasets published by us use the provided netcdf files.

However, we recognise that many existing workflows rely on tools that handle tabular data exclusively and therefore also publish in the PRIMAP2 Interchange Format which is a tabular wide format with additional meta data. Users of the interchange format have to integrate the given meta data carefully into their workflows to ensure correct results.

Logical format

In the interchange format all dimensions and time points are represented by columns in a two-dimensional array. Values of the time columns are data while values of the other columns are coordinates. To store metadata, including the information contained in the attrs dict in the PRIMAP2 xarray format, we use an additional dictionary. See sections In-memory representation and on-disk representation below for information on the storage of these structures.

The requirements for the data, columns, and coordinates follow the requirements in the standard PRIMAP2 data format. Dimensions area and source, which are mandatory in the xarray format, are mandatory columns in the tabular data in the interchange format. The time dimension is included in the horizontal dimension of the tabular interchange format. Additionally, we have unit and entity as mandatory columns with the restriction that each entity can have only one unit.

All optional dimensions (see Data format details) can be added as optional columns. Secondary categories are columns with free format names. They are listed as secondary columns in the metadata dict.

Column names correspond to the dimension key of the xarray format, i.e. they contain the terminology in parentheses (e.g. area (ISO3)).

Additional columns are currently not possible, but the option will be added in a future release (#25).

The metadata dict has an attrs entry, which corresponds to the attrs dict of the xarray format (see Data format details). Additionally, the metadata dict contains information on the dimensions of the data for each entity, on the time_format of the data columns and (if stored on disk) on the name of the data_file (see Interchange format details).

Use

The interchange format is intended for use mainly in two settings.

• To publish data processed using PRIMAP2 in a way that is easy to read by others but also keeps the internal structure and metadata. The format will be used by future data publications by the PRIMAP team including PRIMAP-hist.

• To have a common intermediate format for reading data from original sources (mostly xls or csv files in different formats) to simplify data reading functions and to enable use of our data reading functionality by other projects. All data is first read into the interchange format and subsequently converted into the native PRIMAP2 format. This enables using our data reading routines in other software packages.

In-memory representation

The in-memory representation of the interchange format is using a pandas DataFrame to store the data, and a dict to store the additional metadata. Pandas DataFrames have the capability to store the metadata on their attrs, however this function is still experimental and subject to change without notice, so care has to be taken not to lose the data if processing is done on the DataFrame. For an example see Examples section below.

On-disk representation

On disk the dataset is represented by a csv file containing the array, and a yaml file containing the additional metadata as a dict. Both files should have the same name except for the ending. On disk, the key data_file is added to the metadata dict, which contains the name of the csv file. Thus, a function reading interchange format data just needs the yaml file name to read the data.

Examples

Here we show a few examples of the interchange format.

# import all the used libraries
import primap2 as pm2

Reading csv data

The PRIMAP2 data reading procedures first convert data into the interchange format. For explanations of the used parameters see the Data reading example. A more complex dataset is read in Data reading PRIMAP-hist.

file = "../data_reading/test_csv_data_sec_cat.csv"
coords_cols = {
    "unit": "unit",
    "entity": "gas",
    "area": "country",
    "category": "category",
    "sec_cats__Class": "classification",
}
coords_defaults = {
    "source": "TESTcsv2021",
    "sec_cats__Type": "fugitive",
    "scenario": "HISTORY",
}
coords_terminologies = {
    "area": "ISO3",
    "category": "IPCC2006",
    "sec_cats__Type": "type",
    "sec_cats__Class": "class",
    "scenario": "general",
}
coords_value_mapping = {
    "category": "PRIMAP1",
    "entity": "PRIMAP1",
    "unit": "PRIMAP1",
}
data_if = pm2.pm2io.read_wide_csv_file_if(
    file,
    coords_cols=coords_cols,
    coords_defaults=coords_defaults,
    coords_terminologies=coords_terminologies,
    coords_value_mapping=coords_value_mapping,
)
data_if.head()

	source	scenario (general)	area (ISO3)	entity	unit	category (IPCC2006)	Class (class)	Type (type)	1991	2000	2010
0	TESTcsv2021	HISTORY	AUS	CO2	Gg CO2 / yr	1	TOTAL	fugitive	4000.00	5000.00	6000.00
1	TESTcsv2021	HISTORY	AUS	KYOTOGHG (SARGWP100)	Mt CO2 / yr	0	TOTAL	fugitive	8.00	9.00	10.00
2	TESTcsv2021	HISTORY	FRA	CH4	Gg CH4 / yr	2	TOTAL	fugitive	7.00	8.00	9.00
3	TESTcsv2021	HISTORY	FRA	CO2	Gg CO2 / yr	2	TOTAL	fugitive	12.00	13.00	14.00
4	TESTcsv2021	HISTORY	FRA	KYOTOGHG (SARGWP100)	Mt CO2 / yr	0	TOTAL	fugitive	0.03	0.02	0.04

Writing interchange format data

Data is written using the primap2.pm2io.write_interchange_format() function which takes a filename and path (str or pathlib.Path), an interchange format dataframe (pandas.DataFrame) and optionally an attribute dict as inputs. If the filename has an ending, it will be ignored. The function writes a yaml file and a csv file.

file_if = "../data_reading/test_csv_data_sec_cat_if"
pm2.pm2io.write_interchange_format(file_if, data_if)

Reading data from disk

To read interchange format data from disk the function primap2.pm2io.read_interchange_format() is used. It just takes a filename and path as input (str or pathlib.Path) and returns a pandas.DataFrame containing the data and metadata. The filename and path has to point to the yaml file. the csv file will be read from the filename contained in the yaml file.

data_if_read = pm2.pm2io.read_interchange_format(file_if)
data_if_read.head()

	source	scenario (general)	area (ISO3)	entity	unit	category (IPCC2006)	Class (class)	Type (type)	1991	2000	2010
0	TESTcsv2021	HISTORY	AUS	CO2	Gg CO2 / yr	1	TOTAL	fugitive	4000.0000000000005	5000.000000000001	6000.000000000001
1	TESTcsv2021	HISTORY	AUS	KYOTOGHG (SARGWP100)	Mt CO2 / yr	0	TOTAL	fugitive	8.0	9.0	10.0
2	TESTcsv2021	HISTORY	FRA	CH4	Gg CH4 / yr	2	TOTAL	fugitive	7.0	8.0	9.0
3	TESTcsv2021	HISTORY	FRA	CO2	Gg CO2 / yr	2	TOTAL	fugitive	12.0	13.0	14.0
4	TESTcsv2021	HISTORY	FRA	KYOTOGHG (SARGWP100)	Mt CO2 / yr	0	TOTAL	fugitive	0.03	0.02	0.04

Converting to and from standard PRIMAP2 format

Data in the standard, xarray-based PRIMAP2 format can be converted to and from the interchange format with the corresponding functions:

ds_minimal = pm2.open_dataset("../minimal_ds.nc")

if_minimal = ds_minimal.pr.to_interchange_format()

if_minimal.head()

	source	area (ISO3)	entity	unit	2000	2001	2002	2003	2004	2005	...	2011	2012	2013	2014	2015	2016	2017	2018	2019	2020
0	RAND2020	ARG	CH4	CH4 * gigagram / year	0.368532	0.088137	0.298864	0.678659	0.862995	0.131253	...	0.867758	0.874488	0.845115	0.510803	0.108674	0.822922	0.767920	0.071981	0.025005	0.089894
1	RAND2020	ARG	CO2	CO2 * gigagram / year	0.916832	0.682339	0.250728	0.852096	0.881125	0.147748	...	0.612935	0.966494	0.376384	0.487788	0.710766	0.945482	0.131772	0.938776	0.658307	0.629569
2	RAND2020	ARG	SF6	SF6 * gigagram / year	0.505571	0.526975	0.618358	0.497167	0.559072	0.943795	...	0.569415	0.044608	0.587069	0.657630	0.027363	0.711145	0.018133	0.997718	0.268657	0.761415
3	RAND2020	ARG	SF6 (SARGWP100)	CO2 * gigagram / year	12083.153938	12594.698995	14778.752239	11882.298866	13361.818016	22556.691567	...	13609.030141	1066.140102	14030.947344	15717.359528	653.987580	16996.374472	433.377608	23845.461667	6420.891957	18197.817082
4	RAND2020	BOL	CH4	CH4 * gigagram / year	0.565378	0.036782	0.752872	0.247971	0.305199	0.094644	...	0.374838	0.406868	0.352221	0.965541	0.037689	0.402788	0.173916	0.409820	0.093950	0.553759

5 rows × 25 columns

ds_minimal_re = pm2.pm2io.from_interchange_format(if_minimal)

ds_minimal_re

2024-10-07 16:18:28.966 | DEBUG    | primap2.pm2io._interchange_format:from_interchange_format:319 - Expected array shapes: [[21, 4, 1, 4], [21, 4, 1, 4], [21, 4, 1, 4], [21, 4, 1, 4]], resulting in size 1,344.

<xarray.Dataset> Size: 3kB
Dimensions:          (time: 21, area (ISO3): 4, source: 1)
Coordinates:
  * area (ISO3)      (area (ISO3)) object 32B 'ARG' 'BOL' 'COL' 'MEX'
  * source           (source) object 8B 'RAND2020'
  * time             (time) datetime64[ns] 168B 2000-01-01 ... 2020-01-01
Data variables:
    CH4              (time, area (ISO3), source) float64 672B [CH4·Gg/a] 0.36...
    CO2              (time, area (ISO3), source) float64 672B [CO2·Gg/a] 0.91...
    SF6              (time, area (ISO3), source) float64 672B [SF6·Gg/a] 0.50...
    SF6 (SARGWP100)  (time, area (ISO3), source) float64 672B [CO2·Gg/a] 1.20...
Attributes:
    area:     area (ISO3)

xarray.Dataset

• Dimensions:
  • time: 21
  • area (ISO3): 4
  • source: 1
• Coordinates: (3)
  • area (ISO3)
    (area (ISO3))
    object
    'ARG' 'BOL' 'COL' 'MEX'

    array(['ARG', 'BOL', 'COL', 'MEX'], dtype=object)

  • source
    (source)
    object
    'RAND2020'

    array(['RAND2020'], dtype=object)

  • time
    (time)
    datetime64[ns]
    2000-01-01 ... 2020-01-01

    array(['2000-01-01T00:00:00.000000000', '2001-01-01T00:00:00.000000000',
           '2002-01-01T00:00:00.000000000', '2003-01-01T00:00:00.000000000',
           '2004-01-01T00:00:00.000000000', '2005-01-01T00:00:00.000000000',
           '2006-01-01T00:00:00.000000000', '2007-01-01T00:00:00.000000000',
           '2008-01-01T00:00:00.000000000', '2009-01-01T00:00:00.000000000',
           '2010-01-01T00:00:00.000000000', '2011-01-01T00:00:00.000000000',
           '2012-01-01T00:00:00.000000000', '2013-01-01T00:00:00.000000000',
           '2014-01-01T00:00:00.000000000', '2015-01-01T00:00:00.000000000',
           '2016-01-01T00:00:00.000000000', '2017-01-01T00:00:00.000000000',
           '2018-01-01T00:00:00.000000000', '2019-01-01T00:00:00.000000000',
           '2020-01-01T00:00:00.000000000'], dtype='datetime64[ns]')

• Data variables: (4)
  • CH4
    (time, area (ISO3), source)
    float64
    [CH4·Gg/a] 0.3685 ... 0.02305

    entity :

    CH4

    Magnitude

    [[[0.3685320940353728] [0.5653780221221959] [0.7542549558511776] [0.7369296426004138]] [[0.08813720839288863] [0.03678244010434961] [0.8364808087479632] [0.6269624635226865]] [[0.29886414437019804] [0.752872424535261] [0.3264265671890325] [0.433428122526182]] [[0.6786592295407908] [0.2479712511975457] [0.99013758070399] [0.724861880203653]] [[0.8629953227625237] [0.3051994326528302] [0.9168491852422332] [0.6589798491488441]] [[0.13125277522126044] [0.09464381146488332] [0.363954789171993] [0.5317324546827301]] [[0.9200030821096901] [0.38356553316313813] [0.37219028327011106] [0.5765632290541318]] [[0.6988319605873081] [0.29813804128060784] [0.21173423093184363] [0.07685377058063037]] [[0.9449887033172708] [0.691475104643898] [0.8256908816307077] [0.029675864516146455]] [[0.7364362832821014] [0.7673430557143591] [0.5508098263229796] [0.7567733852678931]] [[0.8079833097460629] [0.9945274253910127] [0.48842796850443415] [0.6463223733753957]] [[0.8677580617075967] [0.37483753711409995] [0.12629758359095455] [0.27030619541411705]] [[0.87448806091195] [0.4068677527163894] [0.3156350508768655] [0.8384328875893682]] [[0.8451145147260058] [0.3522207334108044] [0.01971608147418813] [0.6842069975800996]] [[0.5108025262787018] [0.9655410083920338] [0.23412165329055146] [0.45759346753578756]] [[0.10867412804182908] [0.03768909532940734] [0.18459977602749444] [0.669245140924568]] [[0.8229215128954231] [0.40278785991644206] [0.8286652785948826] [0.25918844188963486]] [[0.7679201489444207] [0.17391590049126393] [0.8267430216430773] [0.1588857548770053]] [[0.07198107823046473] [0.40981960314708143] [0.08214070760255643] [0.020714404278233012]] [[0.025004948745150135] [0.09394984504371917] [0.9703624792874707] [0.7211716718406348]] [[0.08989404965510184] [0.5537587740490154] [0.5485158893136658] [0.02304916212005137]]]

    Units

    CH4 gigagram/year

  • CO2
    (time, area (ISO3), source)
    float64
    [CO2·Gg/a] 0.9168 0.6142 ... 0.1863

    entity :

    CO2

    Magnitude

    [[[0.9168318265574015] [0.6141634063871825] [0.6635201782042437] [0.10638744826946733]] [[0.682338632666392] [0.7905772609283975] [0.3382253292557216] [0.7395385058805823]] [[0.25072806564646855] [0.4087390697622033] [0.11409100697947938] [0.8504489730663606]] [[0.8520959700808077] [0.08031264625923029] [0.801477073222692] [0.7152128934482338]] [[0.8811252096269836] [0.111051447212553] [0.6101888801109169] [0.5550000928935479]] [[0.14774847859299522] [0.13586945518522808] [0.812456122446051] [0.14072428619992494]] [[0.9547228137993204] [0.5212602891531959] [0.09237210162677412] [0.7905119579747139]] [[0.26581566767403764] [0.43650759225482505] [0.6546441668962568] [0.9689908809264166]] [[0.5975266186962794] [0.5551216597262866] [0.6612736041974472] [0.785099419487452]] [[0.05037719350908554] [0.6517388086838728] [0.33259990823381236] [0.3369210750624343]] [[0.9106197882021331] [0.5275309558025338] [0.5522010519840608] [0.8234802866938398]] [[0.6129352532104692] [0.8143255189588381] [0.547885968428845] [0.7602238439362549]] [[0.9664940577619263] [0.33780248501934773] [0.5705158101627505] [0.2635155897055561]] [[0.3763835226478044] [0.34866229530317505] [0.14997833554506934] [0.1647368812064165]] [[0.4877882315623534] [0.9163693640320614] [0.4634965654407963] [0.037042328100195476]] [[0.7107660047838966] [0.40996965629779725] [0.6455829370274393] [0.4195285063193993]] [[0.9454818882042824] [0.6809799063677471] [0.4758651094425167] [0.03006484586084257]] [[0.13177214894370792] [0.07862391306443517] [0.009667561812883863] [0.43758244853013717]] [[0.9387762091651058] [0.44603773344001674] [0.24073322602911418] [0.6674664205953101]] [[0.6583069409300996] [0.7549426352132018] [0.12764283003356602] [0.19115895273354766]] [[0.6295689201632558] [0.056990272829468] [0.5548269020548341] [0.1863119137604362]]]

    Units

    CO2 gigagram/year

  • SF6
    (time, area (ISO3), source)
    float64
    [SF6·Gg/a] 0.5056 0.1236 ... 0.2628

    entity :

    SF6

    Magnitude

    [[[0.5055712944941024] [0.1235696696610431] [0.0017999283398781873] [0.6764281356992125]] [[0.5269748533552134] [0.8096243456920031] [0.4777477627293901] [0.8016230618391313]] [[0.6183578342728786] [0.3085480136136496] [0.5331652257091747] [0.23948967174374636]] [[0.49716731657668833] [0.6836916441599878] [0.5323686911928804] [0.4340997424406753]] [[0.5590718835247671] [0.33905171176763205] [0.6672625868689808] [0.9382487089014679]] [[0.9437946262290234] [0.6703211769514555] [0.1810063661773531] [0.7570997700454333]] [[0.07422128273206374] [0.3392561054061216] [0.08218973282583775] [0.8080948951738635]] [[0.6427237374065423] [0.8051477668558303] [0.11494516659459275] [0.6470840960951291]] [[0.5204792846286578] [0.621097578936615] [0.15393918278957064] [0.0889369331164156]] [[0.1699100337618633] [0.39183340587122273] [0.8996772186819172] [0.6274569096295286]] [[0.23459696651127115] [0.7568653740660694] [0.9947633418025775] [0.9471595930522158]] [[0.5694154870543] [0.7174046976355728] [0.11666812059542109] [0.2972097217084779]] [[0.04460837247920335] [0.9040802399068613] [0.9083554803854986] [0.19494181397753896]] [[0.5870689265407956] [0.5804577179083541] [0.9194928308313104] [0.47963603827269163]] [[0.6576301057548541] [0.5584057498489272] [0.7811415855268452] [0.5261474587632892]] [[0.027363497052910413] [0.050565352755786] [0.9128417649366534] [0.9751450575411759]] [[0.7111453753978566] [0.5350717635605707] [0.6708354594464743] [0.6874428941931122]] [[0.018132954296472215] [0.5883767927200321] [0.5912540921962742] [0.011587687621276155]] [[0.9977180613859968] [0.8229456655620809] [0.5540699967566246] [0.1775038540227457]] [[0.26865656723726417] [0.1666851588620759] [0.9649063147146625] [0.9595548786166266]] [[0.7614149406593684] [0.04207827124056718] [0.9537394170806007] [0.2628449737891362]]]

    Units

    SF6 gigagram/year

  • SF6 (SARGWP100)
    (time, area (ISO3), source)
    float64
    [CO2·Gg/a] 1.208e+04 ... 6.282e+03

    gwp_context :

    SARGWP100

    entity :

    SF6

    Magnitude

    [[[12083.153938409048] [2953.3151048989303] [43.01828732308868] [16166.632443211178]] [[12594.698995189603] [19350.021862038877] [11418.171529232424] [19158.791177955238]] [[14778.752239121799] [7374.297525366225] [12742.648894449276] [5723.803154675538]] [[11882.298866182851] [16340.23029542371] [12723.611719509843] [10374.98384433214]] [[13361.818016241936] [8103.335911246406] [15947.575826168642] [22424.144142745085]] [[22556.691566873662] [16020.676129139789] [4326.052151638739] [18094.684504085857]] [[1773.8886572963236] [8108.220919206307] [1964.3346145375222] [19313.467994655337]] [[15361.097324016362] [19243.031627854347] [2747.189481610767] [15465.309896673587]] [[12439.454902624922] [14844.232136585098] [3679.146468670738] [2125.592701482333]] [[4060.8498069085335] [9364.818400322223] [21502.285526497824] [14996.220140145735]] [[5606.86749961938] [18089.08244017906] [23774.843869081604] [22637.11427394796]] [[13609.03014059777] [17145.97227349019] [2788.368082230564] [7103.312348832623]] [[1066.14010225296] [21607.517733773984] [21709.69598121342] [4659.109354063181]] [[14030.947344325015] [13872.939458009665] [21975.87865686832] [11463.301314717331]] [[15717.359527541017] [13345.89742138936] [18669.2838940916] [12574.924264442612]] [[653.9875795645589] [1208.5119308632854] [21816.918181986017] [23305.966875234106]] [[16996.374472008774] [12788.215149097641] [16032.967480770736] [16429.885171215385]] [[433.3776076856859] [14062.205346008766] [14130.972803490955] [276.94573414850015]] [[23845.461667125324] [19668.401406933735] [13242.27292248333] [4242.342111143623]] [[6420.891956970614] [3983.7752968036143] [23061.260921680434] [22933.36159893738]] [[18197.81708175891] [1005.6706826495556] [22794.37206822636] [6281.994873560355]]]

    Units

    CO2 gigagram/year

• Indexes: (3)
  • area (ISO3)
    PandasIndex

    PandasIndex(Index(['ARG', 'BOL', 'COL', 'MEX'], dtype='object', name='area (ISO3)'))

  • source
    PandasIndex

    PandasIndex(Index(['RAND2020'], dtype='object', name='source'))

  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2000-01-01', '2001-01-01', '2002-01-01', '2003-01-01',
                   '2004-01-01', '2005-01-01', '2006-01-01', '2007-01-01',
                   '2008-01-01', '2009-01-01', '2010-01-01', '2011-01-01',
                   '2012-01-01', '2013-01-01', '2014-01-01', '2015-01-01',
                   '2016-01-01', '2017-01-01', '2018-01-01', '2019-01-01',
                   '2020-01-01'],
                  dtype='datetime64[ns]', name='time', freq=None))

• Attributes: (1)

  area :

  area (ISO3)