diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 69b2e8dc5..e36ea174e 100755 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -458,6 +458,8 @@ Changed `#882 `_ * Insert crossboarding gas pipeline with Germany in eGon100RE `#881 `_ +* Harmonize H2 carrier names in eGon100RE + `#929 `_ * Rename noflex to lowflex scenario for motorized individual travel `#921 `_ * Update creation of heat demand timeseries @@ -477,6 +479,8 @@ Changed `#519 `_ * Add missing VOM costs for heat sector components `#942 `_ +* Add sanity checks for gas sector in eGon2035 + `#864 `_ * Desaggregate industry demands to OSM areas and industrial sites `#1001 `_ * Add gas generator in Norway @@ -486,6 +490,8 @@ Changed created for a single process. This fixes issue `#799`_. * Insert rural heat per supply technology `#1026 `_ +* Insert lifetime for components from p-e-s in eGon100RE + `#1073 `_ * Change hgv data source to use database `#1086 `_ * Rename eMob MIT carrier names (use underscores) @@ -731,6 +737,8 @@ Bug Fixes * Fix URL of TYNDP scenario dataset * Automatically generated tasks now get unique :code:`task_id`\s. Fixes issue `#985`_ via PR `#986`_. +* Adjust capcities of German CH4 stores + `#1096 `_ * Fix faulty DSM time series `#1088 `_ * Set upper limit on commissioning date for units from MaStR @@ -738,6 +746,8 @@ Bug Fixes `#1098 `_ * Fix conversion factor for CH4 loads abroad in eGon2035 `#1104 `_ +* Change structure of documentation in rtd + `#11126 `_ .. _PR #692: https://github.com/openego/eGon-data/pull/692 .. _#343: https://github.com/openego/eGon-data/issues/343 diff --git a/docs/about.rst b/docs/about.rst new file mode 100644 index 000000000..28f7d3525 --- /dev/null +++ b/docs/about.rst @@ -0,0 +1,111 @@ +*************** +About eGon-data +*************** + +Project background +================== + +egon-data provides a transparent and reproducible open data-based data processing pipeline for generating data models suitable for energy system modeling. The data is customized for the requirements of the research project eGo_n. The research project aims to develop tools for open and cross-sectoral planning of transmission and distribution grids. For further information please visit the `eGo_n project website `_. +egon-data is a further development of the `Data processing `_ developed in the former research project `open_eGo `_. It aims to extend the data models as well as improve the replicability and manageability of the data preparation and processing. +The resulting data set serves as an input for the optimization tools `eTraGo `_, `ding0 `_ and `eDisGo `_ and delivers, for example, data on grid topologies, demands/demand curves and generation capacities in a high spatial resolution. The outputs of egon-data are published under open-source and open-data licenses. + + +Objectives of the project +========================= + +Driven by the expansion of renewable generation capacity and the progressing electrification of other energy sectors, the electrical grid increasingly faces new challenges: fluctuating supply of renewable energy and simultaneously a changing demand pattern caused by sector coupling. However, the integration of non-electric sectors such as gas, heat, and e-mobility enables more flexibility options. The eGo_n project aims to investigate the effects of sector coupling on the electrical grid and the benefits of new flexibility options. This requires the creation of a spatially and temporally highly resolved database for all sectors considered. + +Project consortium and funding +================================== + +The following universities and research institutes were involved in the creation of eGon-data: + +* University of Applied Sciences Flensburg +* Reiner Lemoine Institut +* Otto von Guericke University Magdeburg +* DLR Institute of Networked Energy Systems +* Europa-Universität Flensburg + +The eGo_n project (FKZ: 03EI1002) is supported by the Federal Ministry for Economic Affairs and Climate Action (BMWK) on the basis of a decision by the German Bundestag. + +.. image:: images/Logos_Projektpartner_egon_data.png + :width: 400 + :alt: Logos of project partners + + +eGon-data as one element of the eGo-Toolchain +============================================= + +In the eGo_n project different tools were developed, which are in exchange with each other and have to serve the respective requirements on data scope, resolution, and format. The results of the data model creation have to be especially adapted to the requirements of the tools eTraGo and eDisGo for power grid optimization on different grid levels. +A PostgreSQL database serves as an interface between the data model creation and the optimization tools. +The figure below visualizes the interdependencies between the different tools. + +.. image:: images/Toolchain_web_desktop.svg + :width: 800 + :alt: eGon-data tool chain + +.. _concept-and-scenarios-ref: + +Modeling concept and scenarios +=============================== + +eGon-data provides a data model suitable for calculations and optimizations with the tools eTraGo, eDisGo and eGo and therefore aims to satisfy all requirements regarding the scope and temporal as well as spatial granularity of the resulting data model. +The following image visualizes the different components considered in scenario ``eGon2035``. + +.. image:: images/egon-modell-szenario-egon2035.png + :width: 800 + :alt: Components of the data models + +eGon-data aims to create different scenarios, which differ in terms of RE penetration or the availability of flexibility options. Currently, the following scenarios are available or in progress. + +* ``eGon2035`` Mid-termin scenario based on assumptions from the German network expansion plan 'scenario C2035', version 2021 and TYNDP +* ``eGon2035_lowflex`` Mid-termin scenario similar to 'eGon2035', but with a limited availability of flexibility options +* ``eGon100RE`` Long-term scenario with a 100% RE penetration, based on optimization results with PyPSA-Eur-Sec and additional data inputs (work-in-progress) + +.. list-table:: Installed capacities of German power park in scenario ``eGon2035`` and ``eGon2035_lowflex`` + :widths: 50 50 + :header-rows: 1 + + * - carrier + - Installed capacities + * - gas + - 46.7 GW + * - oil + - 1.3 GW + * - pumped hydro + - 10.2 GW + * - wind onshore + - 90.9 GW + * - wind offshore + - 34.0 GW + * - solar + - 120.1 GW + * - biomass + - 8.7 GW + * - others + - 5.4 GW + + +.. list-table:: German energy demands in scenarios ``eGon2035`` and ``eGon2035_lowflex`` + :widths: 50 50 + :header-rows: 1 + + * - Demand sector + - Energy demand + * - MIT transport + - 41.4 TWh :subscript:`el` + * - central heat + - 68.9 TWh :subscript:`th` + * - rural heat + - 423.2 TWh :subscript:`th` + * - electricity + - 498.1 TWh :subscript:`el` + * - Methane industry + - 196.0 TWh :subscript:`CH4` + * - Hydrogen industry + - 16.1 TWh :subscript:`H2` + * - Hydrogen transport + - 26.5 TWh :subscript:`H2` + + + diff --git a/docs/conf.py b/docs/conf.py index 975e216b4..3a1c2fa84 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -47,3 +47,8 @@ add_module_names = False modindex_common_prefix = ["egon.data.", "egon.data.datasets."] + +autodoc_type_aliases = { + "Dependencies": "egon.data.datasets.Dependencies", + "Tasks": "egon.data.datasets.Tasks" +} diff --git a/docs/data.rst b/docs/data.rst index e4e3cf6b9..4bbf47357 100644 --- a/docs/data.rst +++ b/docs/data.rst @@ -1,21 +1,124 @@ **** Data **** +The description of the methods, input data and results of the eGon-data pipeline is given in the following section. +References to datasets and functions are integrated if more detailed information is required. + +Main input data and their processing +==================================== + +All methods in the eGon-data workflow rely on public and freely available data from different external sources. The most important data sources +and their processing within the eGon-data pipeline are described here. + +.. include:: data/input_data.rst + +Grid models +=========== + +Power grid models of different voltage levels form a central part of the eGon data model, which is required for cross-grid-level optimization. +In addition, sector coupling necessitates the representation of the gas grid infrastructure, which is also described in this section. + +Electricity grid +---------------- + +.. include:: data/electricity_grids.rst + +Gas grid +-------- + +.. include:: data/gas_grids.rst + +Demand +====== + +Electricity, heat and gas demands from different consumption sectors are taken into account in eGon-data. The related methods to distribute and +process the demand data are described in the following chapters for the different consumption sectors separately. + +.. _electricity-demand-ref: + +Electricity +----------- + +.. include:: data/electricity_demand.rst + +Heat +---- + +.. include:: data/heat_demand.rst + +Gas +--- + +.. include:: data/gas_demand.rst + +.. _mobility-demand-ref: + +Mobility +-------- + +.. include:: data/mobility_demand.rst + + +Supply +====== + +The distribution and assignment of supply capacities or potentials are carried out technology-specific. The different methods are described in the +following chapters. + +Electricity +----------- + +.. include:: data/electricity_supply.rst + +Heat +---- + +.. include:: data/heat_supply.rst + +Gas +--- +.. include:: data/gas_supply.rst + + +Flexibility options +=================== + +Different flexibility options are part of the model and can be utilized in the optimization of the energy system. Therefore detailed information about +flexibility potentials and their distribution are needed. The considered technologies described in the following chapters range from different storage units, +through dynamic line rating to Demand-Side-Management measures. + +Demand-Side-Management +---------------------- + +.. include:: data/DSM.rst + +Dynamic line rating +------------------- + +.. include:: data/DLR.rst + +E-Mobility +---------- + +.. include:: data/e-mobility.rst + +Battery stores +---------------- + +.. include:: data/batteries.rst + +Gas stores +----------------- + +.. include:: data/gas_stores.rst + +Heat stores +------------- + +.. include:: data/heat_stores.rst -Scenarios -========= Published data ============== -Data bundle ------------ -The data bundle is published on -`zenodo `_. It contains several data -sets, which serve as a basis for egon-data. One such data set is the geocoding -for the `MaStR data set `_ which is -used for eGon-data as well. Whenever the MaStR data set is updated it is -necessary to redo the geocoding with the new data set and update the data -bundle accordingly. The geocoding can be done based on the -`mastr-geocoding repository `_. diff --git a/docs/data/DLR.rst b/docs/data/DLR.rst new file mode 100644 index 000000000..5414db22a --- /dev/null +++ b/docs/data/DLR.rst @@ -0,0 +1,47 @@ +==================================================== +Methods to include dynamic line rating in our model +==================================================== + +To calculate the transmission capacity of each transmission line in the model, +the procedure suggested in the **Principles for the Expansion Planning of the +German Transmission Network** [NEP2021] where used: + +1. Import the temperature and wind temporal raster layers from ERA-5. Hourly +resolution data from the year 2011 was used. Raster resolution +latitude-longitude grids at 0.25° x 0.25°. + +2. Import shape file for the 9 regions proposed by the Principles for +the Expansion Planning. See Figure 1. + +.. image:: images/regions_DLR.png + :width: 400 + :alt: regions DLR + +Figure 1: Representative regions in Germany for DLR analysis [NEP2021] + +3. Find the lowest wind speed in each region. To perform this, for each +independent region, the wind speed of every cell in the raster layer should be +extracted and compared. This procedure is repeated for each hour in the +year 2011. The results are the 8760 lowest wind speed per region. + +4. Find the highest temperature in each region. To perform this, for each +independent region, the temperature of every cell in the raster layer should +be extracted and compared. This procedure is repeated for each hour in the +year 2011. The results are the 8760 maximum temperature per region. + +5. Calculate the maximum capacity for each region using the parameters shown in +Figure 2. + +.. image:: images/table_max_capacity_DLR.png + :width: 400 + :alt: table_max_capacity_DLR + +Figure 2: transmission capacity based on max temperature and min wind speed [NEP2021] + +6. Assign the maximum capacity of the corresponding region to each transmission +line inside each one of them. Crossborder lines and underground lines receive +no values. It means that their capacities are static and equal to their nominal +values. Lines that cross borders between regions receive the lowest +capacity per hour of the regions containing the line. + +.. [NEP2021] Principles for the Expansion Planning of the German Transmission Network https://www.netzentwicklungsplan.de/ \ No newline at end of file diff --git a/docs/data/DSM.rst b/docs/data/DSM.rst new file mode 100644 index 000000000..79d4a96ef --- /dev/null +++ b/docs/data/DSM.rst @@ -0,0 +1 @@ +How did we implement DSM? Results etc. diff --git a/docs/data/batteries.rst b/docs/data/batteries.rst new file mode 100644 index 000000000..2fcb79d46 --- /dev/null +++ b/docs/data/batteries.rst @@ -0,0 +1,37 @@ +Battery storage units comprise home batteries and larger, grid-supportive batteries. National capacities for home batteries arise from external sources, e.g. the Grid Development Plan for the scenario ``eGon2035``, whereas the capacities of large-scale batteries are a result of the grid optimization tool `eTraGo `_. + +Home battery capacities are first distributed to medium-voltage grid districts (MVGD) and based on that further disaggregated to single buildings. The distribution on MVGD level is done proportional to the installed capacities of solar rooftop power plants, assuming that they are used as solar home storage. + +Potential large-scale batteries are included in the data model at every substation. The data model includes technical and economic parameters, such as efficiencies and investment costs. The energy-to-power ratio is set to a fixed value of 6 hours. Other central parameters are given in the following table + +.. list-table:: Parameters of batteries for scenario eGon2035 + :widths: 40 30 30 + :header-rows: 1 + + * - + - Value + - Sources + + * - Efficiency store + - 98 % + - [DAE_store]_ + + * - Efficiency dispatch + - 98 % + - [DAE_store]_ + + * - Standing loss + - 0 % + - [DAE_store]_ + + * - Investment costs + - 838 €/kW + - [DAE_store]_ + + * - Home storage units + - 16.8 GW + - [NEP2021]_ + + +On transmission grid level, distinguishing between home batteries and large-scale batteries was not possible. Therefore, the capacities of home batteries were set as a lower boundary of the large-scale battery capacities. +This is implemented in the dataset :py:class:`StorageEtrago `, the data for batteries in the transmission grid is stored in the database table :py:class:`grid.egon_etrago_storage `. diff --git a/docs/data/e-mobility.rst b/docs/data/e-mobility.rst new file mode 100644 index 000000000..0f4be98c9 --- /dev/null +++ b/docs/data/e-mobility.rst @@ -0,0 +1 @@ +What flexibilities does e-mobility provide to the system. How did we implement it? diff --git a/docs/data/electricity_demand.rst b/docs/data/electricity_demand.rst new file mode 100644 index 000000000..d375e954c --- /dev/null +++ b/docs/data/electricity_demand.rst @@ -0,0 +1 @@ +Information about electricity demands and their spatial and temporal aggregation diff --git a/docs/data/electricity_grids.rst b/docs/data/electricity_grids.rst new file mode 100644 index 000000000..0d288e836 --- /dev/null +++ b/docs/data/electricity_grids.rst @@ -0,0 +1,125 @@ +.. _ehv-hv-grids: + +High and extra-high voltage grids +++++++++++++++++++++++++++++++++++ + +The model of the German extra-high (eHV) and high voltage (HV) grid is based +on data retrieved from OpenStreetMap (status January 2021) [OSM]_ and additional +parameters for standard transmission lines from [Brakelmann2004]_. To gather all +required information, such as line topology, voltage level, substation locations, +and electrical parameters, to create a calculable power system model, the `osmTGmod +tool `_ was used. The corresponding dataset +:py:class:`Osmtgmod ` executes osmTGmod +and writes the resulting data to the database. + +The resulting grid model includes the voltage levels 380, 220 and 110 kV and +all substations interconnecting these grid levels. For further information on the +generation of the grid topology please refer to [Mueller2018]_. + +.. _ding0-grids: + +Medium and low-voltage grids +++++++++++++++++++++++++++++ + +Medium (MV) and low (LV) voltage grid topologies for entire Germany are generated using +the python tool ding0 `ding0 `_. +ding0 generates synthetic grid topologies based on high-resolution geodata and routing +algorithms as well as typical network planning principles. +The generation of the +grid topologies is not part of eGon_data, but ding0 solely uses data generated with eGon_data, +such as locations of HV/MV stations (see :ref:`ehv-hv-grids`), locations and peak demands +of buildings in the grid (see :ref:`building-data-ref` respectively :ref:`electricity-demand-ref`), +as well as locations of generators from MaStR (see :ref:`mastr-ref`). A full list +of tables used in ding0 can be found in its `config `_. +An exemplary MV grid with one underlying LV grid is shown in figure :ref:`ding0-mv-grid`. +The grid data of all over 3.800 MV grids is published on `zenodo `_. + +.. figure:: /images/ding0_mv_lv_grid.png + :name: ding0-mv-grid + :width: 600 + + Exemplary synthetic medium-voltage grid with underlying low-voltage grid generated with ding0 + +Besides data on buildings and generators, ding0 requires data on the supplied areas +by each grid. This is as well done in eGon_data and described in the following. + +.. _mv-grid-districts: + +MV grid districts +~~~~~~~~~~~~~~~~~~ + +Medium-voltage (MV) grid districts describe the area supplied by one MV grid. +They are defined by one polygon that represents the +supply area. Each MV grid district is connected to the HV grid via a single +substation. An exemplary MV grid district is shown in figure :ref:`ding0-mv-grid` (orange line). + +The MV grid districts are generated in the dataset +:class:`MvGridDistricts`. +The methods used for identifying the MV grid districts are heavily inspired +by Hülk et al. (2017) [Huelk2017]_ +(section 2.3), but the implementation differs in detail. +The main difference is that direct adjacency is preferred over proximity. +For polygons of municipalities +without a substation inside, it is iteratively checked for direct adjacent +other polygons that have a substation inside. Speaking visually, a MV grid +district grows around a polygon with a substation inside. + +The grid districts are identified using three data sources + +1. Polygons of municipalities (:class:`Vg250GemClean`) +2. Locations of HV-MV substations (:class:`EgonHvmvSubstation`) +3. HV-MV substation voronoi polygons (:class:`EgonHvmvSubstationVoronoi`) + +Fundamentally, it is assumed that grid districts (supply areas) often go +along borders of administrative units, in particular along the borders of +municipalities due to the concession levy. +Furthermore, it is assumed that one grid district is supplied via a single +substation and that locations of substations and grid districts are designed +for aiming least lengths of grid line and cables. + +With these assumptions, the three data sources from above are processed as +follows: + +* Find the number of substations inside each municipality +* Split municipalities with more than one substation inside + + * Cut polygons of municipalities with voronoi polygons of respective + substations + * Assign resulting municipality polygon fragments to nearest substation +* Assign municipalities without a single substation to nearest substation in + the neighborhood +* Merge all municipality polygons and parts of municipality polygons to a + single polygon grouped by the assigned substation + +For finding the nearest substation, as already said, direct adjacency is +preferred over closest distance. This means, the nearest substation does not +necessarily have to be the closest substation in the sense of beeline distance. +But it is the substation definitely located in a neighboring polygon. This +prevents the algorithm to find solutions where a MV grid districts consists of +multi-polygons with some space in between. +Nevertheless, beeline distance still plays an important role, as the algorithm +acts in two steps + +1. Iteratively look for neighboring polygons until there are no further + polygons +2. Find a polygon to assign to by minimum beeline distance + +The second step is required in order to cover edge cases, such as islands. + +For understanding how this is implemented into separate functions, please +see :func:`define_mv_grid_districts`. + +.. _load-areas-ref: + +Load areas +~~~~~~~~~~~~ + +Load areas (LAs) are defined as geographic clusters where electricity is consumed. +They are used in ding0 to determine the extent and number of LV grids. Thus, within +each LA there are one or multiple MV-LV substations, each supplying one LV grid. +Exemplary load areas are shown in figure :ref:`ding0-mv-grid` (grey and orange areas). + +The load areas are set up in the +:class:`LoadArea` dataset. +The methods used for identifying the load areas are heavily inspired +by Hülk et al. (2017) [Huelk2017]_ (section 2.4). diff --git a/docs/data/electricity_supply.rst b/docs/data/electricity_supply.rst new file mode 100644 index 000000000..fffd7379c --- /dev/null +++ b/docs/data/electricity_supply.rst @@ -0,0 +1,88 @@ +The electrical power plants park, including data on geolocations, installed capacities, etc. +for the different scenarios is set up in the dataset +:class:`PowerPlants`. + +Main inputs into the dataset are target capacities per technology and federal state +in each scenario (see :ref:`concept-and-scenarios-ref`) as well as the MaStR (see :ref:`mastr-ref`), +OpenStreetMap (see :ref:`osm-ref`) and potential areas (provided through the data bundle, +see :ref:`data-bundle-ref`) to distribute the generator capacities within each federal state region. +The approach taken to distribute the target capacities within each federal state differs for +the different technologies and is described in the following. +The final distribution in the eGon2035 scenario is shown in figure :ref:`generator-park-egon-2035`. + +.. figure:: /images/Erzeugerpark.png + :name: generator-park-egon-2035 + :width: 400 + + Generator park in the eGon2035 scenario + +Onshore wind ++++++++++++++ + +Offshore wind +++++++++++++++ + +PV ground mounted +++++++++++++++++++ + +.. _pv-rooftop-ref: + +PV rooftop ++++++++++++ + +In a first step, the target capacity in the eGon2035 and eGon100RE scenarios is distributed +to all MV grid districts linear to the residential and CTS electricity demands in the +grid district (done in function +:func:`pv_rooftop_per_mv_grid`). + +Afterwards, the PV rooftop capacity per MV grid district is disaggregated +to individual buildings inside the grid district (done in function +:func:`pv_rooftop_to_buildings`). +The basis for this is data from the MaStR, which is first cleaned and missing information +inferred, and then allocated to specific buildings. New PV plants are in a last step +added based on the capacity distribution from MaStR. +These steps are in more detail described in the following. + +MaStR data cleaning and inference: + +* Drop duplicates and entries with missing critical data. +* Determine most plausible capacity from multiple values given in MaStR data. +* Drop generators that don't have a plausible capacity (23.5 MW > P > 0.1 kW). +* Randomly and weighted add a start-up date if it is missing. +* Extract zip and municipality from 'site' given in MaStR data. +* Geocode unique zip and municipality combinations with Nominatim (1 sec + delay). Drop generators for which geocoding failed or which are located + outside the municipalities of Germany. +* Add some visual sanity checks for cleaned data. + +Allocation of MaStR plants to buildings: + +* Allocate each generator to an existing building from OSM or a synthetic building + (see :ref:`building-data-ref`). +* Determine the quantile each generator and building is in depending on the + capacity of the generator and the area of the polygon of the building. +* Randomly distribute generators within each municipality preferably within + the same building area quantile as the generators are capacity wise. +* If not enough buildings exist within a municipality and quantile additional + buildings from other quantiles are chosen randomly. + +Disaggregation of PV rooftop scenario capacities: + +* The scenario data per federal state is linearly distributed to the MV grid + districts according to the PV rooftop potential per MV grid district. +* The rooftop potential is estimated from the building area given from the OSM + buildings. +* Grid districts, which are located in several federal states, are allocated + PV capacity according to their respective roof potential in the individual + federal states. +* The disaggregation of PV plants within a grid district respects existing + plants from MaStR, which did not reach their end of life. +* New PV plants are randomly and weighted generated using the capacity distribution of + PV rooftop plants from MaStR. +* Plant metadata (e.g. plant orientation) is also added randomly and weighted + using MaStR data as basis. + +Hydro +++++++ + + diff --git a/docs/data/gas_demand.rst b/docs/data/gas_demand.rst new file mode 100644 index 000000000..072c66394 --- /dev/null +++ b/docs/data/gas_demand.rst @@ -0,0 +1 @@ +Information about gas demands and their spatial and temporal aggregation, including hydrogen and methane demands diff --git a/docs/data/gas_grids.rst b/docs/data/gas_grids.rst new file mode 100644 index 000000000..f3df18ea2 --- /dev/null +++ b/docs/data/gas_grids.rst @@ -0,0 +1,7 @@ +Information about the gas grids and how they were created + +Methane grid +++++++++++++++ + +Hydrogen grid +++++++++++++++ diff --git a/docs/data/gas_stores.rst b/docs/data/gas_stores.rst new file mode 100644 index 000000000..35c10d070 --- /dev/null +++ b/docs/data/gas_stores.rst @@ -0,0 +1,7 @@ +Description of methods and assumptions to include potential h2 stores in the system + +Hydrogen stores +--------------- + +Methane stores +-------------- diff --git a/docs/data/gas_supply.rst b/docs/data/gas_supply.rst new file mode 100644 index 000000000..e5cc586f9 --- /dev/null +++ b/docs/data/gas_supply.rst @@ -0,0 +1 @@ +Information on gas supply - hydrogen and methane. diff --git a/docs/data/heat_demand.rst b/docs/data/heat_demand.rst new file mode 100644 index 000000000..6e004ae28 --- /dev/null +++ b/docs/data/heat_demand.rst @@ -0,0 +1,100 @@ +Heat demands comprise space heating and dirinking hot water demands from +residential and comertial trade and service (CTS) buildings. +Process heat demands from the industry are, depending on the required temperature +level, modelled as electrcity, hydrogen or methane demand. + +The spatial distribution of annual heat demands is taken from the Pan-European +Thermal Altlas version 5.0.1 [Peta]_. +This source provides data on annual european residential and CTS heat demands +per hectar cell for the year 2015. +In order to model future demands, the demand distribution extracted by Peta is +then scaled to meet a national annual demand from external sources. +The following national demands are taken for the selected scenarios: + +.. list-table:: Heat demands per sector and scenario + :widths: 25 25 25 25 + :header-rows: 1 + + * - + - Residential sector + - CTS sector + - Sources + + * - eGon2035 + - 379 TWh + - 122 TWh + - [Energiereferenzprognose]_ + + * - eGon100RE + - 284 TWh + - 89 TWh + - [Energiereferenzprognose]_ + + +The resulting data is stored in the database table :py:class:`demand.egon_peta_heat `. +The implementation of these dataprocessing steps can be found in :py:class:`HeatDemandImport `. + +Figure :ref:`residential-heat-demand-annual` shows the distribution of residential heat demands for scenario ``eGon2035``, +categorized for different levels of annual demands. + +.. figure:: /images/residential_heat_demand.png + :name: residential-heat-demand-annual + :width: 400 + + Spatial distribution of residential heat demand in scenario ``eGon2035`` + + +Afterwards, the annual heat demands are used to create hourly heat demand profiles. +For residential heat demand profiles a pool of synthetical created bottom-up demand +profiles is used. Depending on the mean temperature per day, these profiles are +randomly assigned to each residential building. The methodology is described in +detail in [Buettner2022]_. + + +Data on residential heat demand profiles is stored in the database within the tables :py:class:`demand.egon_heat_timeseries_selected_profiles `, :py:class:`demand.egon_daily_heat_demand_per_climate_zone `, :py:class:`boundaries.egon_map_zensus_climate_zones `. To create the profiles for a selected buidling, these tables +have to be combined, e.g. like this: + +.. code-block:: none + + SELECT (b.demand/f.count * UNNEST(e.idp) * d.daily_demand_share)*1000 AS demand_profile + FROM (SELECT * FROM demand.egon_heat_timeseries_selected_profiles, + UNNEST(selected_idp_profiles) WITH ORDINALITY as selected_idp) a + JOIN demand.egon_peta_heat b + ON b.zensus_population_id = a.zensus_population_id + JOIN boundaries.egon_map_zensus_climate_zones c + ON c.zensus_population_id = a.zensus_population_id + JOIN demand.egon_daily_heat_demand_per_climate_zone d + ON (c.climate_zone = d.climate_zone AND d.day_of_year = ordinality) + JOIN demand.egon_heat_idp_pool e + ON selected_idp = e.index + JOIN (SELECT zensus_population_id, COUNT(building_id) + FROM demand.egon_heat_timeseries_selected_profiles + GROUP BY zensus_population_id + ) f + ON f.zensus_population_id = a.zensus_population_id + WHERE a.building_id = SELECTED_BUILDING_ID + AND b.scenario = 'eGon2035' + AND b.sector = 'residential'; + + +Exemplary resulting residential heat demand time series for a selected day in winter and +summer considering different aggregation levels are visualized in figures :ref:`residential-heat-demand-timeseries-winter` and :ref:`residential-heat-demand-timeseries-summer`. + +.. figure:: /images/residential_heat_demand_profile_winter.png + :name: residential-heat-demand-timeseries-winter + :width: 400 + + Temporal distribution of residential heat demand for a selected day in winter + +.. figure:: /images/residential_heat_demand_profile_summer.png + :name: residential-heat-demand-timeseries-summer + :width: 400 + + Temporal distribution of residential heat demand for a selected day in summer + +The temporal disaggregation of CTS heat demand is done using Standard Load Profiles Gas +from ``demandregio`` [demandregio]_ considering different profiles per CTS branch. + + +The heat demand time series for both sectors creation is done in the Dataset +:py:class:`HeatTimeSeries `. diff --git a/docs/data/heat_stores.rst b/docs/data/heat_stores.rst new file mode 100644 index 000000000..59a10c172 --- /dev/null +++ b/docs/data/heat_stores.rst @@ -0,0 +1,35 @@ +The heat sector can provide flexibility through stores that allow shifting energy in time. The data model includes hot water tanks as heat stores in individual buildings and pit thermal energy storage for district heating grids (further described in :ref:`district-heating`). + +Within the data model, potential locations as well as technical and economic parameters of these stores are defined. The installed store and (dis-)charging capacities are part of the grid optimization methods that can be performed by `eTraGo `_. The power-to-energy ratio is not predefined but a result of the optimization, which allows to build heat stores with various time horizons. + +Individual heat stores can be built in every building with an individual heat pump. Central heat stores can be built next to district heating grids. There are no maximum limits for the energy output as well as (dis-)charging capacities implemented yet. + +Central cost assumptions for central and decentral heat stores are listed in the table below. The parameters can differ for each scenario in order to include technology updates and learning curves. The table focuses on the scenario ``eGon2035``. + +.. list-table:: Parameters of heat stores + :widths: 16 16 16 16 16 16 + :header-rows: 1 + + * - + - Technology + - Costs for store capacity + - Costs for (dis-)charging capacity + - Round-trip efficiency + - Sources + + * - District heating + - Pit thermal energy storage + - 0.51 EUR / kWh + - 0 EUR / kW + - 70 % + - [DAE_store]_ + + * - Buildings with heat pump + - Water tank + - 1.84 EUR / kWh + - 0 EUR / kW + - 70 % + - [DAE_store]_ + +The heat stores are implemented as a part of the dataset :py:class:`HeatEtrago `, the data is written into the tables :py:class:`grid.egon_etrago_bus `, :py:class:`grid.egon_etrago_link ` and :py:class:`grid.egon_etrago_store `. + diff --git a/docs/data/heat_supply.rst b/docs/data/heat_supply.rst new file mode 100644 index 000000000..f0c633edb --- /dev/null +++ b/docs/data/heat_supply.rst @@ -0,0 +1,80 @@ +Heat demand of residential as well as commercial, trade and service (CTS) buildings can be supplied by different technologies and carriers. Within the data model creation, capacities of supply technologies are assigned to specific locations and their demands. The hourly dispatch of heat supply is not part of the data model, but a result of the grid optimization tools. + +In general, heat supply can be divided into three categories which include specific technologies: residential and CTS buildings in a district heating area, buildings supplied by individual heat pumps, and buildings supplied by conventional gas boilers. The shares of these categories are taken from external sources for each scenario. + +.. list-table:: Heat demands of different supply categories + :widths: 20 20 20 20 + :header-rows: 1 + + * - + - District heating + - Individual heat pumps + - Individual gas boilers + + * - eGon2035 + - 69 TWh + - 27.24 TWh + - 390.78 TWh + + * - eGon100RE + - 61.5 TWh + - 311.5 TWh + - 0 TWh + +The following subsections describe the heat supply methodology for each category. + +.. _district-heating: + +District heating +~~~~~~~~~~~~~~~~ + +First, district heating areas are defined for each scenario based on existing district heating areas and an overall district heating share per scenario. To reduce the model complexity, district heating areas are defined per Census cell, either all buildings within a cell are supplied by district heat or none. The first step of the extraction of district heating areas is the identification of Census cells with buildings that are currently supplied by district heating using the ``building`` dataset of Census. All Census cells where more than 30% of the buildings are currently supplied by district heat are defined as cells inside a district heating area. +The identified cells are then summarized by combining cells that have a maximum distance of 500m. + +Second, additional Census cells are assigned to district heating areas considering the heat demand density. Assuming that new district heating grids are more likely in cells with high demand, the remaining Census cells outside of a district heating grid are sorted by their demands. Until the pre-defined national district heating demand is met, cells from that list are assigned to district heating areas. This can also result in new district heating grids which cover only a few Census cells. + +To avoid unrealistic large district heating grids in areas with many cities close to each other (e.g. the Ruhr Area), district heating areas with an annual demand > 4 TWh are split by NUTS3 boundaries. + +The implementation of the district heating area demarcation is done in :py:class:`DistrictHeatingAreas `, the resulting data is stored in the tables :py:class:`demand.egon_map_zensus_district_heating_areas ` and :py:class:`demand.egon_district_heating_areas `. +The resulting district heating grids for the scenario eGon2035 are visualized in figure :ref:`district-heating-areas`, which also includes a zoom on the district heating grid in Berlin. + +.. figure:: /images/district_heating_areas.png + :name: district-heating-areas + :width: 800 + + Defined district heating grids in scenario ``eGon2035`` + +The national capacities for each supply technology are taken from the Grid Development Plan (GDP) for the scenario ``eGon2035``, in the ``eGon100RE`` scenario they are the result of the ``pypsa-eur-sec`` run. The distribution of the capacities to district heating grids is done similarly based on [FfE2017]_, which is also used in the GDP. The basic idea of this method is to use a cascade of heat supply technologies until the heat demand can be covered. + +#. Combined heat and power (CHP) plants are assigned to nearby district heating grids first. Their location and thermal capacities are from Marktstammdatenregister [MaStR]_. To identify district heating grids that need additional suppliers, the remaining annual heat demand is calculated using the thermal capacities of the CHP plants and assumed full load hours. + +#. Large district heating grids with an annual demand that is higher than 96GWh can be supplied by geothermal plants, in case of an intersection of geothermal potential areas and the district heating grid. Smaller district heating grids can be supplied by solar thermal power plants. The national capacities are distributed proportionally to the remaining heat demands. After assigning these plants, the remaining heat demands are reduced by the thermal output and assumed full load hours. + +#. Next, the national capacities for central heat pumps and resistive heaters are distributed to all district heating areas proportionally to their remaining demands. Heat pumps are modeled with a time-dependent coefficient of performance depending on the temperature data. + +#. In the last step, gas boilers are assigned to every district heating grid regardless of the remaining demand. In the optimization, this can be used as a fall-back option to not run into infeasibilities. + +The distribution of CHP plants for different carriers is shown in figure :ref:`chp-plants`. + +.. figure:: /images/combined_heat_and_power_plants.png + :name: chp-plants + :width: 400 + + Spatial distribution of CHP plants in scenario ``eGon2035`` + + +Individual heat pumps +~~~~~~~~~~~~~~~~~~~~~ + +Heat pumps supplying individual buildings are first distributed to each medium-voltage grid district, these capacities are later on further disaggregated to single buildings. Similar to central heat pumps they are modeled with a time-dependent coefficient of performance depending on the temperature data. + +The distribution of the national capacities to each medium-voltage grid district is proportional to the heat demand outside of district heating grids. + +@RLI: Distribution on building level + +Individual gas boilers +~~~~~~~~~~~~~~~~~~~~~~ + +All residential and CTS buildings that are neither supplied by a district heating grid nor an individual heat pump are supplied by gas boilers. The demand time series of these buildings are multiplied by the efficiency of gas boilers and aggregated per methane grid node. + +All heat supply categories are implemented in the dataset :py:class:`HeatSupply `. The data is stored in the tables :py:class:`demand.egon_district_heating ` and :py:class:`demand.egon_individual_heating `. diff --git a/docs/data/input_data.rst b/docs/data/input_data.rst new file mode 100644 index 000000000..b1dd53cb6 --- /dev/null +++ b/docs/data/input_data.rst @@ -0,0 +1,143 @@ +.. _data-bundle-ref: + +Data bundle +----------- + +The data bundle is published on +`zenodo `_. It contains several data +sets, which serve as a basis for egon-data: + +* Climate zones in Germany +* Data on eMobility individual trips of electric vehicles +* Spatial distribution of deep geothermal potentials in Germany +* Annual profiles in hourly resolution of electricity demand of private households +* Sample heat time series including hot water and space heating for single- and multi-familiy houses +* Hydrogen storage potentials in salt structures +* Information about industrial sites with DSM-potential in Germany +* Data extracted from the German grid development plan - power +* Parameters for the classification of gas pipelines +* Preliminary results from scenario generator pypsa-eur-sec +* German regions suitable to model dynamic line rating +* Eligible areas for wind turbines and ground-mounted PV systems +* Definitions of industrial and commercial branches +* Zensus data on households +* Geocoding of all unique combinations of ZIP code and municipality within the Marktstammdatenregister + +For further description of the data including licenses and references please refer to the Zenodo repository. + +.. _mastr-ref: + +Marktstammdatenregister +----------------------- + +The `Marktstammdatenregister `_ (MaStR) +is the register for the German electricity and gas +market holding, among others, data on electricity and gas generation plants. In eGon-data +it is used for status quo data on PV plants, wind turbines, biomass, hydro power plants, +combustion power plants, nuclear power plants, geo- and solarthermal power plants, and storage units. +The data are obtained from zenodo, where raw MaStR data, downloaded with the tool +`open-MaStR `_ using the MaStR webservice, +is provided. It contains all data from the MaStR, including possible duplicates. +Currently, two versions are used: + +* `2021-05-03 `_ +* `2022-11-17 `_ + +The download is implemented in :class:`MastrData`. + +.. _osm-ref: + +OpenStreetMap +------------- + +`OpenStreetMap `_ (OSM) is a free, editable map of the whole +world that is being built by volunteers and released with an open-content license. +In eGon-data it is, among others, used to obtain information on land use as well as +locations of buildings and amenities to spatially dissolve energy demand. +The OSM data is downloaded from the `Geofabrik `_ download +server, which holds extracts from the OpenStreetMap. Afterwards, they are imported +to the database using osm2pgsql with a custom style file. The implementation of this +can be found in :class:`OpenStreetMap`. + +In the :class:`OpenStreetMap` +dataset, the OSM data is filtered, processed and enriched with other data. This is +described in the following subsections. + +Amenity data +++++++++++++++ + +The data on amenities is used to disaggregate CTS demand data. It is filtered from the +raw OSM data using tags listed in script `osm_amenities_shops_preprocessing.sql`, e.g. +shops and restaurants. The filtered data is written to database table +`openstreetmap.osm_amenities_shops_filtered`. + +.. _building-data-ref: + +Building data +++++++++++++++ + +The data on buildings is required by several tasks in the +pipeline, such as the disaggregation of household demand profiles or PV home +systems to buildings, as well as the DIstribution Network Generat0r `ding0 +`_ (see also :ref:`ding0-grids`). + +The data processing steps are: + +* Extract buildings and filter using relevant tags, e.g. residential and + commercial, see script `osm_buildings_filter.sql` for the full list of tags. + Resulting tables: + + * All buildings: `openstreetmap.osm_buildings` + * Filtered buildings: `openstreetmap.osm_buildings_filtered` + * Residential buildings: `openstreetmap.osm_buildings_residential` + +* Create a mapping table for building's OSM IDs to the Zensus cells the + building's centroid is located in. + Resulting tables: + + * `boundaries.egon_map_zensus_buildings_filtered` (filtered) + * `boundaries.egon_map_zensus_buildings_residential` (residential only) + +* Enrich each building by number of apartments from Zensus table + `society.egon_destatis_zensus_apartment_building_population_per_ha` + by splitting up the cell's sum equally to the buildings. In some cases, a + Zensus cell does not contain buildings but there is a building nearby which + the no. of apartments is to be allocated to. To make sure apartments are + allocated to at least one building, a radius of 77m is used to catch building + geometries. +* Split filtered buildings into 3 datasets using the amenities' locations: + temporary tables are created in script `osm_buildings_temp_tables.sql`, the + final tables in `osm_buildings_amentities_results.sql`. + Resulting tables: + + * Buildings w/ amenities: `openstreetmap.osm_buildings_with_amenities` + * Buildings w/o amenities: `openstreetmap.osm_buildings_without_amenities` + * Amenities not allocated to buildings: + `openstreetmap.osm_amenities_not_in_buildings` + +As there are discrepancies between the Census data [Census]_ and OSM building data when both +datasets are used to generate electricity demand profiles of households, synthetic buildings +are added in Census cells with households but without buildings. This is done as part +of the :class:`Demand_Building_Assignment` +dataset in function :func:`generate_synthetic_buildings`. +The synthetic building data are written to table `openstreetmap.osm_buildings_synthetic`. +The same is done in case of CTS electricity demand profiles. Here, electricity demand is +disaggregated to Census cells according to heat demand information from the +Pan European Thermal Atlas [Peta]_. In case there are Census cells with electricity demand +assigned but no building or amenity data, synthetic buildings are added. +This is done as part +of the :class:`CtsDemandBuildings` +dataset in function :func:`create_synthetic_buildings`. +The synthetic building data are again written to table `openstreetmap.osm_buildings_synthetic`. + +Street data +++++++++++++++ + +The data on streets is used in the DIstribution Network Generat0r `ding0 +`_, e.g. for the routing of the grid. +It is filtered from the +raw OSM data using tags listed in script `osm_ways_preprocessing.sql`, e.g. +highway=secondary. Additionally, each way is split into its line segments and their +lengths is retained. The filtered streets data is written to database table +`openstreetmap.osm_ways_preprocessed` and the filtered streets with segments +to table `openstreetmap.osm_ways_with_segments`. diff --git a/docs/data/mobility_demand.rst b/docs/data/mobility_demand.rst new file mode 100644 index 000000000..e3b5310fe --- /dev/null +++ b/docs/data/mobility_demand.rst @@ -0,0 +1,73 @@ +.. _mobility-demand-mit-ref: + +Motorized individual travel +++++++++++++++++++++++++++++ + +The electricity demand data of motorized individual travel (MIT) for both the eGon2035 +and eGon100RE scenario is set up +in the :py:class:`MotorizedIndividualTravel` +dataset. + +The profiles are generated using a modified version of +`SimBEV v0.1.3 `_. +SimBEV generates driving profiles for battery electric vehicles (BEVs) and +plug-in hybrid electric vehicles (PHEVs) based on MID survey data [MiD2017]_ per +RegioStaR7 region type [RegioStaR7_2020]_. +These profiles include driving, parking and (user-oriented) charging times. +Different vehicle classes are taken +into account whose assumed technical data is given in table :ref:`ev-types-data`. +Moreover, charging probabilities for multiple types of charging +infrastructure are presumed based on [NOW2020]_ and [Helfenbein2021]_. +Given these assumptions, a pool of 33.000 EVs-types is pre-generated and provided through the data bundle +(see :ref:`data-bundle-ref`) as well as written +to table :py:class:`EgonEvTrip`. +The complete tech data and assumptions of the run can be found in the +metadata_simbev_run.json file, that is provided along with the trip data. + +.. csv-table:: EV types + :header: "Tecnnology", "Size", "Max. charging capacity slow in kW", "Max. charging capacity fast in kW", "Battery capacity in kWh", "Energy consumption in kWh/km" + :widths: 10, 10, 30, 30, 25, 10 + :name: ev-types-data + + "BEV", "mini", 11, 120, 60, 0.1397 + "BEV", "medium", 22, 350, 90, 0.1746 + "BEV", "luxury", 50, 350, 110, 0.2096 + "PHEV", "mini", 3.7, 40, 14, 0.1425 + "PHEV", "medium", 11, 40, 20, 0.1782 + "PHEV", "luxury", 11, 120, 30, 0.2138 + +Heavy-duty transport ++++++++++++++++++++++ + +In the context of the eGon project, it is assumed that all e-trucks will be +completely hydrogen-powered. The hydrogen demand data of all e-trucks is set up +in the :py:class:`HeavyDutyTransport` +dataset for both the eGon2035 and eGon100RE scenario. + +In both scenarios the hydrogen consumption is +assumed to be 6.68 kgH2 per 100 km with an additional supply chain leakage rate of 0.5 % +(see `here `_). + +For the eGon2035 scenario the ramp-up figures are taken from the +`network development plan (version 2021) `_ +(Scenario C 2035). According to this, 100,000 e-trucks are +expected in Germany in 2035, each covering an average of 100,000 km per year. +In total this means 10 Billion km. + +For the eGon100RE scenario it is assumed that the heavy-duty transport is +completely hydrogen-powered. The total freight traffic with 40 Billion km is +taken from the +`BMWK Langfristszenarien `_ +for heavy-duty vehicles larger 12 t allowed total weight (SNF > 12 t zGG). + +The total hydrogen demand is spatially distributed on the basis of traffic volume data from [BASt]_. +For this purpose, first a voronoi partition of Germany using the traffic measuring points is created. +Afterwards, the spatial shares of the Voronoi regions in each NUTS3 area are used to allocate +hydrogen demand to the NUTS3 regions and are then aggregated per NUTS3 region. +The refuelling is assumed to take place at a constant rate. +Finally, to +determine the hydrogen bus where the hydrogen demand is allocated to, the centroid +of each NUTS3 region is used to determine the respective hydrogen Voronoi cell (see +:py:class:`GasAreaseGon2035` and +:py:class:`GasAreaseGon100RE`) it is +located in. diff --git a/docs/images/Erzeugerpark.png b/docs/images/Erzeugerpark.png new file mode 100644 index 000000000..3d67b2af2 Binary files /dev/null and b/docs/images/Erzeugerpark.png differ diff --git a/docs/images/Logos_Projektpartner_egon_data.png b/docs/images/Logos_Projektpartner_egon_data.png new file mode 100644 index 000000000..45c35cd2a Binary files /dev/null and b/docs/images/Logos_Projektpartner_egon_data.png differ diff --git a/docs/images/Toolchain_web_desktop.svg b/docs/images/Toolchain_web_desktop.svg new file mode 100644 index 000000000..5165add10 --- /dev/null +++ b/docs/images/Toolchain_web_desktop.svg @@ -0,0 +1,2097 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/images/combined_heat_and_power_plants.png b/docs/images/combined_heat_and_power_plants.png new file mode 100644 index 000000000..3120b08a4 Binary files /dev/null and b/docs/images/combined_heat_and_power_plants.png differ diff --git a/docs/images/ding0_mv_lv_grid.png b/docs/images/ding0_mv_lv_grid.png new file mode 100644 index 000000000..0ed848db7 Binary files /dev/null and b/docs/images/ding0_mv_lv_grid.png differ diff --git a/docs/images/district_heating_areas.png b/docs/images/district_heating_areas.png new file mode 100644 index 000000000..226a3d8a4 Binary files /dev/null and b/docs/images/district_heating_areas.png differ diff --git a/docs/images/egon-modell-szenario-egon2035.png b/docs/images/egon-modell-szenario-egon2035.png new file mode 100644 index 000000000..321eca9c0 Binary files /dev/null and b/docs/images/egon-modell-szenario-egon2035.png differ diff --git a/docs/images/regions_DLR.png b/docs/images/regions_DLR.png new file mode 100644 index 000000000..b4a7aedfd Binary files /dev/null and b/docs/images/regions_DLR.png differ diff --git a/docs/images/residential_heat_demand.png b/docs/images/residential_heat_demand.png new file mode 100644 index 000000000..974c9c604 Binary files /dev/null and b/docs/images/residential_heat_demand.png differ diff --git a/docs/images/residential_heat_demand_profile_summer.png b/docs/images/residential_heat_demand_profile_summer.png new file mode 100644 index 000000000..3c41641d3 Binary files /dev/null and b/docs/images/residential_heat_demand_profile_summer.png differ diff --git a/docs/images/residential_heat_demand_profile_winter.png b/docs/images/residential_heat_demand_profile_winter.png new file mode 100644 index 000000000..e661fd289 Binary files /dev/null and b/docs/images/residential_heat_demand_profile_winter.png differ diff --git a/docs/images/table_max_capacity_DLR.png b/docs/images/table_max_capacity_DLR.png new file mode 100644 index 000000000..016b7db3f Binary files /dev/null and b/docs/images/table_max_capacity_DLR.png differ diff --git a/docs/index.rst b/docs/index.rst index 600622854..7dc38f45d 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -5,8 +5,9 @@ Contents .. toctree:: :maxdepth: 2 - getting_started + about workflow + getting_started troubleshooting data literature diff --git a/docs/literature.rst b/docs/literature.rst index d725279b4..e44a0dc0f 100644 --- a/docs/literature.rst +++ b/docs/literature.rst @@ -1,3 +1,45 @@ ********** Literature ********** + +.. [BASt] Bundesanstalt für Straßenwesen, Automatische Zählstellen 2020 (2020). URL https://www.bast.de/DE/Verkehrstechnik/Fachthemen/v2-verkehrszaehlung/Daten/2020_1/Jawe2020.cs + +.. [Brakelmann2004] H. Brakelmann, Netzverstärkungs-Trassen zur Übertragung von Windenergie: Freileitung oder Kabel? (2004). URL http://www.ets.uni-duisburg-essen.de/download/public/Freileitung_Kabel.pdf + +.. [Buettner2022] C. Büttner, J. Amme, J. Endres, A. Malla, B. Schachler, I. Cußmann, Open modeling of electricity and heat demand curves for all residential buildings in Germany, Energy Informatics 5 (1) (2022) 21. doi:10.1186/s42162-022-00201-y. URL https://doi.org/10.1186/s42162-022-00201-y + +.. [Census] S. B. (Destatis), Datensatzbeschreibung ”Haushalte im 100 Meter-Gitter” (2018). URL https://www.zensus2011.de/SharedDocs/Downloads/DE/Pressemitteilung/DemografischeGrunddaten/Datensatzbeschreibung_Haushalt_100m_Gitter.html + +.. [DAE_store] Danish Energy Agency, Technology Data – Energy storage, First published 2018 by the Danish Energy Agency and Energinet, URL https://ens.dk/en/our-services/projections-and-models/technology-data/technology-data-energy-storage + +.. [demandregio] F. Gotzens, B. Gillessen, S. Burges, W. Hennings, J. Müller-Kirchenbauer, S. Seim, P. Verwiebe, S. Tobias, F. Jetter, T. Limmer, DemandRegio - Harmonisierung und Entwicklung von Verfahren zur regionalen und zeitlichen Auflösung von Energienachfragen (2020). URL https://openaccess.ffe.de/10.34805/ffe-119-20 + +.. [Energiereferenzprognose] Prognos AG, Energiewirtschaftliches Institut an der Universität zu Köln, Gesellschaft für Wirtschaftliche Strukturforschung mbH: Entwicklung der Energiemärkte – Energiereferenzprognose (2014) + +.. [eXtremOS] A. Guminski, C. Fiedler, S. Kigle, C. Pellinger, P. Dossow, K. Ganz, F. Jetter, T. Kern, T. Limmer, A. Murmann, J. Reinhard, T. Schmid, T. Schmidt-Achert, S. von Roon, eXtremOS Summary Report (2021). doi:https://doi.org/10.34805/ffe-24-21. + +.. [FfE2017] Flexibilisierung der Kraft-Wärme-Kopplung; 2017; Forschungsstelle für Energiewirtschaft e.V. (FfE) + +.. [Helfenbein2021] K. Helfenbein, Analyse des Einflusses netzdienlicher Ladestrategien auf Verteilnetze aufgrund der zunehmenden Netzintegration von Elektrofahrzeugen, Master’s thesis, Hochschule für Technik und Wirtschaft Berlin, URL https://reiner-lemoine-institut.de/analyse-einflussesnetzdienlicher-ladestrategien-verteilnetze-zunehmender-netzintegration-elektrofahrzeugen-helfenbein-2021/ + +.. [Hotmaps] S. Pezzutto, S. Zambotti, S. Croce, P. Zambelli, G. Garegnani, C. Scaramuzzino, R. P. Pascuas, A. Zubaryeva, F. Haas, D. Exner, A. Mueller, M. Hartner, T. Fleiter, A.-L. Klingler, M. Kuehnbach, P. Manz, S. Marwitz, M. Rehfeldt, J. Steinbach, E. Popovski, Hotmaps project, d2.3 wp2 report – open data set for the eu28 (2018). URL www.hotmaps-project.eu + +.. [Huelk2017] L. Hülk, L. Wienholt, I. Cußmann, U.P. Müller, C. Matke, E. Kötter, Allocation of annual electricity consumption and power generation capacities across multiple voltage levels in a high spatial resolution, International Journal of Sustainable Energy Planning and Management Vol. 13 2017 79–92. URL https://journals.aau.dk/index.php/sepm/article/view/1833 + +.. [MiD2017] Bundesministerium für Digitales und Verkehr, Mobilität in Deutschland 2017 (2017). URL https://daten.clearingstelle-verkehr.de/279/ + +.. [Mueller2018] U. Mueller, L. Wienholt, D. Kleinhans, I. Cussmann, W.-D. Bunke, G. Pleßmann, J. Wendiggensen 2018 J. Phys.: Conf. Ser. 977 012003, DOI 10.1088/1742-6596/977/1/012003 + +.. [NEP2021] Übertragungsnetzbetreiber Deutschland (2021): *Netzentwicklungsplan Strom 2035*, Version 2021, 1. Entwurf. 2021. + +.. [NOW2020] Nationale Leitstelle Ladeinfrastruktur, Ladeinfrastruktur nach 2025/2030: Szenarien für den Markthochlauf (2020). URL https://www.now-gmbh.de/wp-content/uploads/2020/11/Studie_Ladeinfrastruktur-nach-2025-2.pdf + +.. [OSM] Geofabrik GmbH and OpenStreetMap-Contributors, OpenStreetMap Data Extracts, Stand 01.01.2022 (2022). URL https://download.geofabrik.de/europe/germany-220101.osm.pbf + +.. [Peta] Europa-Universität Flensburg, Halmstad University and Aalborg University, Pan-European Thermal Atlas - Residential heat demand (2021). URL https://s-eenergies-open-data-euf.hub.arcgis.com/maps/d7d18b63250240a49eb81db972aa573e/about + +.. [RegioStaR7_2020] Bundesministerium für Digitales und Verkehr, Regionalstatistische Raumtypologie (RegioStaR7), Gebietsstand 2020 (2020). URL https://mcloud.de/web/guest/suche/-/results/detail/536149D1-2902-4975-9F7D-253191C0AD07 + +.. [Schmidt2018] D. Schmidt, Supplementary material to the masters thesis: NUTS-3 Regionalization of Industrial Load Shifting Potential in Germany using a Time-Resolved Model (Nov. 2019). doi:10.5281/zenodo.3613767. URL https://doi.org/10.5281/zenodo.3613767 + +.. [sEEnergies] T. Fleiter, P. Manz, N. Neuwirth, F. Mildner, K. Persson, U.AND Kermeli, W. Crijns-Graus, C. Rutten, seenergies d5.1 dataset web-app.seenergies arcgis online web-apps hosted by europa-universität flensburg (2020). URL https://tinyurl.com/sEEnergies-D5-1 diff --git a/docs/reference/egon.data.datasets.emobility.rst b/docs/reference/egon.data.datasets.emobility.rst index 56aed4ca6..87fb31ee1 100644 --- a/docs/reference/egon.data.datasets.emobility.rst +++ b/docs/reference/egon.data.datasets.emobility.rst @@ -2,5 +2,14 @@ emobility ========= +.. toctree:: + :maxdepth: 1 -.. py:module:: egon.data.datasets.emobility + egon.data.datasets.emobility.heavy_duty_transport + egon.data.datasets.emobility.motorized_individual_travel + egon.data.datasets.emobility.motorized_individual_travel_charging_infrastructure + +.. automodule:: egon.data.datasets.emobility + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/reference/egon.data.datasets.rst b/docs/reference/egon.data.datasets.rst index 286ad1e13..8640aa50c 100644 --- a/docs/reference/egon.data.datasets.rst +++ b/docs/reference/egon.data.datasets.rst @@ -40,6 +40,7 @@ datasets egon.data.datasets.district_heating_areas egon.data.datasets.electricity_demand egon.data.datasets.electricity_demand_timeseries + egon.data.datasets.emobility egon.data.datasets.gas_neighbours egon.data.datasets.heat_demand egon.data.datasets.heat_demand_timeseries diff --git a/docs/workflow.rst b/docs/workflow.rst index b37aba363..45ad1ba3b 100644 --- a/docs/workflow.rst +++ b/docs/workflow.rst @@ -2,17 +2,8 @@ Workflow ******** -Project background ------------------ - -egon-data provides a transparent and reproducible open data based data processing pipeline for generating data models suitable for energy system modeling. The data is customized for the requirements of the research project eGo_n. The research project aims to develop tools for an open and cross-sectoral planning of transmission and distribution grids. For further information please visit the `eGo_n project website `_. -egon-data is a further development of the `Data processing `_ developed in the former research project `open_eGo `_. It aims for an extensions of the data models as well as for a better replicability and manageability of the data preparation and processing. -The resulting data set serves as an input for the optimization tools `eTraGo `_, `ding0 `_ and `eDisGo `_ and delivers for example data on grid topologies, demands/demand curves and generation capacities in a high spatial resolution. The outputs of egon-data are published under open source and open data licenses. - -Data ----- - -egon-data retrieves and processes data from several different external input sources which are all freely available and published under an open data license. The process handles data with different data types, such as spatial data with a high geographical resolution or load/generation time series with an hourly resolution. +Workflow management +------------------- Execution --------- diff --git a/src/egon/data/airflow/dags/pipeline.py b/src/egon/data/airflow/dags/pipeline.py index a024ed1f9..de18b85dc 100755 --- a/src/egon/data/airflow/dags/pipeline.py +++ b/src/egon/data/airflow/dags/pipeline.py @@ -4,6 +4,7 @@ import airflow from egon.data.config import set_numexpr_threads +from egon.data.metadata import Json_Metadata from egon.data.datasets import database from egon.data.datasets.calculate_dlr import Calculate_dlr from egon.data.datasets.ch4_prod import CH4Production @@ -95,7 +96,6 @@ from egon.data.datasets.zensus import ZensusMiscellaneous, ZensusPopulation from egon.data.datasets.zensus_mv_grid_districts import ZensusMvGridDistricts from egon.data.datasets.zensus_vg250 import ZensusVg250 - # Set number of threads used by numpy and pandas set_numexpr_threads() @@ -274,7 +274,7 @@ ] ) - # Get household electrical demands for cencus cells + # Get household electrical demands for census cells household_electricity_demand_annual = HouseholdElectricityDemand( dependencies=[ tasks[ @@ -285,7 +285,7 @@ ] ) - # Distribute electrical CTS demands to zensus grid + # Distribute electrical CTS demands to census grid cts_electricity_demand_annual = CtsElectricityDemand( dependencies=[ demandregio, @@ -523,7 +523,7 @@ ] ) - # DSM (demand site management) + # DSM (demand side management) components_dsm = DsmPotential( dependencies=[ cts_electricity_demand_annual, @@ -689,3 +689,12 @@ low_flex_scenario, ] ) + + # upload json metadata at the end + json_metadata = Json_Metadata( + dependencies=[ + load_areas, + cts_demand_buildings, + heat_pumps_2050 + ] + ) diff --git a/src/egon/data/datasets/DSM_cts_ind.py b/src/egon/data/datasets/DSM_cts_ind.py index 9a66cd85b..52d871766 100644 --- a/src/egon/data/datasets/DSM_cts_ind.py +++ b/src/egon/data/datasets/DSM_cts_ind.py @@ -97,11 +97,50 @@ class DsmPotential(Dataset): + """ + Calculate Demand-Side Management potentials and transfer to charactersitics of DSM components + + DSM within this work includes the shifting of loads within the sectors of + industry and CTS. Therefore, the corresponding formerly prepared demand + time sereies are used. Shiftable potentials are calculated using the + parametrization elaborated in Heitkoetter et. al (doi:https://doi.org/10.1016/j.adapen.2020.100001). + DSM is modelled as storage-equivalent operation using the methods by Kleinhans (doi:10.48550/ARXIV.1401.4121). + The potentials are transferred to characterisitcs of DSM links (minimal and + maximal shiftable power per time step) and DSM stores (minimum and maximum + capacity per time step). DSM buses are created to connect DSM components with + the electrical network. All DSM components are added to the corresponding + tables for the transmission grid level. For the distribution grids, the + respective time series are exported to the corresponding tables (for the + required higher spatial resolution). + + *Dependencies* + * :py:class:`CtsElectricityDemand ` + * :py:class:`IndustrialDemandCurves ` + * :py:class:`Osmtgmod ` + + *Resulting tables* + * :py:class:`grid.egon_etrago_bus ` is extended + * :py:class:`grid.egon_etrago_link ` is extended + * :py:class:`grid.egon_etrago_link_timeseries ` is extended + * :py:class:`grid.egon_etrago_store ` is extended + * :py:class:`grid.egon_etrago_store_timeseries ` is extended + * :py:class:`demand.egon_etrago_electricity_cts_dsm_timeseries ` is created and filled # noqa: E501 + * :py:class:`demand.egon_osm_ind_load_curves_individual_dsm_timeseries ` is created and filled # noqa: E501 + * :py:class:`demand.egon_demandregio_sites_ind_electricity_dsm_timeseries ` is created and filled # noqa: E501 + * :py:class:`demand.egon_sites_ind_load_curves_individual_dsm_timeseries ` is created and filled # noqa: E501 + + """ + + #: + name: str = "DsmPotential" + #: + version: str = "0.0.5" + def __init__(self, dependencies): super().__init__( - name="DsmPotential", - version="0.0.5", - dependencies=dependencies, + name=self.name, + version=self.version, + dependencies=self.dependencies, tasks=(dsm_cts_ind_processing,), ) diff --git a/src/egon/data/datasets/__init__.py b/src/egon/data/datasets/__init__.py index c45e0d470..3b376b35b 100644 --- a/src/egon/data/datasets/__init__.py +++ b/src/egon/data/datasets/__init__.py @@ -98,6 +98,10 @@ class Model(Base): #: a :class:`tuple` of :class:`TaskGraphs ` will be executed #: sequentially in the given order. TaskGraph = Union[Task, Set["TaskGraph"], Tuple["TaskGraph", ...]] +#: A type alias to help specifying that something can be an explicit +#: :class:`Tasks_` object or a :class:`TaskGraph`, i.e. something that +#: can be converted to :class:`Tasks_`. +Tasks = Union["Tasks_", TaskGraph] def prefix(o): @@ -107,7 +111,7 @@ def prefix(o): @dataclass -class Tasks(dict): +class Tasks_(dict): first: Set[Task] last: Set[Task] graph: TaskGraph = () @@ -115,7 +119,7 @@ class Tasks(dict): def __init__(self, graph: TaskGraph): """Connect multiple tasks into a potentially complex graph. - Parses a :class:`TaskGraph` into a :class:`Tasks` object. + Parses a :class:`TaskGraph` into a :class:`Tasks_` object. """ if isinstance(graph, Callable): graph = PythonOperator( @@ -131,14 +135,14 @@ def __init__(self, graph: TaskGraph): self.first = {} self.last = {} elif isinstance(graph, abc.Set): - results = [Tasks(subtasks) for subtasks in graph] + results = [Tasks_(subtasks) for subtasks in graph] self.first = {task for result in results for task in result.first} self.last = {task for result in results for task in result.last} self.update(reduce(lambda d1, d2: dict(d1, **d2), results, {})) self.graph = set(tasks.graph for tasks in results) elif isinstance(graph, tuple): - results = [Tasks(subtasks) for subtasks in graph] - for (left, right) in zip(results[:-1], results[1:]): + results = [Tasks_(subtasks) for subtasks in graph] + for left, right in zip(results[:-1], results[1:]): for last in left.last: for first in right.first: last.set_downstream(first) @@ -149,13 +153,17 @@ def __init__(self, graph: TaskGraph): else: raise ( TypeError( - "`egon.data.datasets.Tasks` got an argument of type:\n\n" + "`egon.data.datasets.Tasks_` got an argument of type:\n\n" f" {type(graph)}\n\n" "where only `Task`s, `Set`s and `Tuple`s are allowed." ) ) +#: A dataset can depend on other datasets or the tasks of other datasets. +Dependencies = Iterable[Union["Dataset", Task]] + + @dataclass class Dataset: #: The name of the Dataset @@ -172,10 +180,10 @@ class Dataset: #: downstream of any of the listed dependencies. In case of bare #: :class:`Task`, a direct link will be created whereas for a #: :class:`Dataset` the link will be made to all of its last tasks. - dependencies: Iterable[Union[Dataset, Task]] = () + dependencies: Dependencies = () #: The tasks of this :class:`Dataset`. A :class:`TaskGraph` will - #: automatically be converted to :class:`Tasks`. - tasks: Union[Tasks, TaskGraph] = () + #: automatically be converted to :class:`Tasks_`. + tasks: Tasks = () def check_version(self, after_execution=()): def skip_task(task, *xs, **ks): @@ -224,8 +232,8 @@ def update(self, session): def __post_init__(self): self.dependencies = list(self.dependencies) - if not isinstance(self.tasks, Tasks): - self.tasks = Tasks(self.tasks) + if not isinstance(self.tasks, Tasks_): + self.tasks = Tasks_(self.tasks) if len(self.tasks.last) > 1: # Explicitly create single final task, because we can't know # which of the multiple tasks finishes last. @@ -236,7 +244,7 @@ def __post_init__(self): # Do nothing, because updating will be added later. python_callable=lambda *xs, **ks: None, ) - self.tasks = Tasks((self.tasks.graph, update_version)) + self.tasks = Tasks_((self.tasks.graph, update_version)) # Due to the `if`-block above, there'll now always be exactly # one task in `self.tasks.last` which the next line just # selects. diff --git a/src/egon/data/datasets/calculate_dlr.py b/src/egon/data/datasets/calculate_dlr.py index 95f1fac85..3680f9abb 100644 --- a/src/egon/data/datasets/calculate_dlr.py +++ b/src/egon/data/datasets/calculate_dlr.py @@ -21,10 +21,32 @@ class Calculate_dlr(Dataset): + """Calculate DLR and assign values to each line in the db + + Parameters + ---------- + *No parameters required + + *Dependencies* + * :py:class:`DataBundle ` + * :py:class:`Osmtgmod ` + * :py:class:`WeatherData ` + * :py:class:`FixEhvSubnetworks ` + + *Resulting tables* + * :py:class:`grid.egon_etrago_line_timeseries + ` is filled + """ + + #: + name: str = "dlr" + #: + version: str = "0.0.1" + def __init__(self, dependencies): super().__init__( - name="dlr", - version="0.0.1", + name=self.name, + version=self.version, dependencies=dependencies, tasks=(dlr,), ) diff --git a/src/egon/data/datasets/ch4_prod.py b/src/egon/data/datasets/ch4_prod.py index cc7f31b6f..786d879ea 100755 --- a/src/egon/data/datasets/ch4_prod.py +++ b/src/egon/data/datasets/ch4_prod.py @@ -61,10 +61,10 @@ def load_NG_generators(scn_name): Define the fossil CH4 production units in Germany This function reads from the SciGRID_gas dataset the fossil CH4 - production units in Germany, adjuts and returns them. + production units in Germany, adjusts and returns them. Natural gas production reference: SciGRID_gas dataset (datasets/gas_data/data/IGGIELGN_Production.csv downloaded in :func:`download_SciGRID_gas_data `). - For more information on these data, refer to the + For more information on this data, refer to the `SciGRID_gas IGGIELGN documentation `_. Parameters @@ -169,11 +169,11 @@ def load_biogas_generators(scn_name): """ Define the biogas production units in Germany - This function download the Biogaspartner Einspeiseatlas into + This function downloads the Biogaspartner Einspeiseatlas into (datasets/gas_data/Biogaspartner_Einspeiseatlas_Deutschland_2021.xlsx), - reads the biogas production units in Germany data, adjuts and + reads the biogas production units in Germany data, adjusts and returns them. - For more information on these data refer, to the + For more information on this data refer to the `Einspeiseatlas website `_. Parameters @@ -282,7 +282,7 @@ def load_biogas_generators(scn_name): def import_gas_generators(scn_name="eGon2035"): """ - Insert list of gas production units into the database + Inserts list of gas production units into the database To insert the gas production units into the database, the following steps are followed: diff --git a/src/egon/data/datasets/ch4_storages.py b/src/egon/data/datasets/ch4_storages.py index ff84af686..8c9e9b60b 100755 --- a/src/egon/data/datasets/ch4_storages.py +++ b/src/egon/data/datasets/ch4_storages.py @@ -3,7 +3,7 @@ The central module containing all code dealing with importing gas stores This module contains the functions to import the existing methane stores -in Germany and to insert them into the database. They are modelled as +in Germany and inserting them into the database. They are modelled as PyPSA stores and are not extendable. """ @@ -27,9 +27,9 @@ class CH4Storages(Dataset): """ - Insert the non extendable gas stores in Germany into the database + Inserts the gas stores in Germany - Insert the non extendable gas stores into the database in Germany + Inserts the non extendable gas stores in Germany into the database for the scnenarios eGon2035 and eGon100RE using the function :py:func:`insert_ch4_storages`. @@ -47,7 +47,7 @@ class CH4Storages(Dataset): #: name: str = "CH4Storages" #: - version: str = "0.0.2" + version: str = "0.0.3" def __init__(self, dependencies): super().__init__( @@ -60,7 +60,7 @@ def __init__(self, dependencies): def import_installed_ch4_storages(scn_name): """ - Define list of CH4 stores from the SciGRID_gas data + Defines list of CH4 stores from the SciGRID_gas data This function reads from the SciGRID_gas dataset the existing CH4 cavern stores in Germany, adjusts and returns them. @@ -77,7 +77,7 @@ def import_installed_ch4_storages(scn_name): Returns ------- Gas_storages_list : - Dataframe containing the CH4 cavern stores units in Germany + Dataframe containing the CH4 cavern store units in Germany """ target_file = ( @@ -88,7 +88,7 @@ def import_installed_ch4_storages(scn_name): target_file, delimiter=";", decimal=".", - usecols=["lat", "long", "country_code", "param"], + usecols=["lat", "long", "country_code", "param", "method"], ) Gas_storages_list = Gas_storages_list[ @@ -99,13 +99,18 @@ def import_installed_ch4_storages(scn_name): max_workingGas_M_m3 = [] NUTS1 = [] end_year = [] + method_cap = [] for index, row in Gas_storages_list.iterrows(): param = ast.literal_eval(row["param"]) NUTS1.append(param["nuts_id_1"]) end_year.append(param["end_year"]) max_workingGas_M_m3.append(param["max_workingGas_M_m3"]) - Gas_storages_list = Gas_storages_list.assign(NUTS1=NUTS1) + method = ast.literal_eval(row["method"]) + method_cap.append(method["max_workingGas_M_m3"]) + + Gas_storages_list["method_cap"] = method_cap + Gas_storages_list = Gas_storages_list.assign(NUTS1=NUTS1).drop_duplicates() # Calculate e_nom conv_factor = 10830 # gross calorific value = 39 MJ/m3 (eurogas.org) @@ -114,6 +119,19 @@ def import_installed_ch4_storages(scn_name): end_year = [float("inf") if x == None else x for x in end_year] Gas_storages_list = Gas_storages_list.assign(end_year=end_year) + # Adjust the storage capacities calculated by 'Median(max_workingGas_M_m3)' + total_german_cap = 266424202 # MWh GIE https://www.gie.eu/transparency/databases/storage-database/ + ch4_estimated = Gas_storages_list[ + Gas_storages_list.method_cap == "Median(max_workingGas_M_m3)" + ] + german_cap_source = Gas_storages_list[ + Gas_storages_list.method_cap != "Median(max_workingGas_M_m3)" + ].e_nom.sum() + + Gas_storages_list.loc[ch4_estimated.index, "e_nom"] = ( + total_german_cap - german_cap_source + ) / len(ch4_estimated) + # Cut data to federal state if in testmode boundary = settings()["egon-data"]["--dataset-boundary"] if boundary != "Everything": @@ -179,6 +197,7 @@ def import_installed_ch4_storages(scn_name): "end_year", "geom", "bus_id", + "method_cap", ] ) @@ -187,7 +206,7 @@ def import_installed_ch4_storages(scn_name): def import_ch4_grid_capacity(scn_name): """ - Define the gas stores modelling the store capacity of the grid + Defines the gas stores modelling the store capacity of the grid Define dataframe containing the modelling of the grid storage capacity. The whole storage capacity of the grid (130000 MWh, @@ -195,7 +214,7 @@ def import_ch4_grid_capacity(scn_name): all the German gas nodes of the grid (without consideration of the capacities of the pipes). In eGon100RE, the storage capacity of the grid is split between H2 - and CH4 stores, with the same share than the pipes capacity (value + and CH4 stores, with the same share as the pipeline capacities (value calculated in the p-e-s run). Parameters @@ -248,17 +267,17 @@ def import_ch4_grid_capacity(scn_name): def insert_ch4_stores(scn_name): """ - Insert gas stores for specific scenario + Inserts gas stores for specific scenario Insert non extendable gas stores for specific scenario in Germany by executing the following steps: * Clean the database. - * For CH4 stores, call the functions. + * For CH4 stores, call the functions :py:func:`import_installed_ch4_storages` to get the CH4 cavern stores and :py:func:`import_ch4_grid_capacity` to get the CH4 stores modelling the storage capacity of the grid. - * Aggregate of the stores attached to the same bus. + * Aggregate the stores attached to the same bus. * Add the missing columns: store_id, scn_name, carrier, e_cyclic. * Insert the stores into the database. diff --git a/src/egon/data/datasets/chp/__init__.py b/src/egon/data/datasets/chp/__init__.py index 26c309578..5425c30d7 100644 --- a/src/egon/data/datasets/chp/__init__.py +++ b/src/egon/data/datasets/chp/__init__.py @@ -2,8 +2,10 @@ The central module containing all code dealing with combined heat and power (CHP) plants. """ - from pathlib import Path +import datetime +import json +import time from geoalchemy2 import Geometry from shapely.ops import nearest_points @@ -32,6 +34,13 @@ filter_mastr_geometry, scale_prox2now, ) +from egon.data.metadata import ( + context, + generate_resource_fields_from_sqla_model, + license_egon_data_odbl, + meta_metadata, + sources, +) Base = declarative_base() @@ -67,6 +76,106 @@ class EgonMaStRConventinalWithoutChp(Base): geometry = Column(Geometry("POINT", 4326)) +def metadata(): + """Write metadata for heat supply tables + + Returns + ------- + None. + + """ + + fields = generate_resource_fields_from_sqla_model(EgonChp) + + fields_df = pd.DataFrame(data=fields).set_index("name") + fields_df.loc["id", "description"] = "Unique identifyer" + fields_df.loc["sources", "description"] = "List of sources" + fields_df.loc[ + "source_id", "description" + ] = "Names of sources, e.g. MaStr_id" + fields_df.loc["carrier", "description"] = "Energy carrier" + fields_df.loc[ + "district_heating", "description" + ] = "Used in district heating or not" + fields_df.loc[ + "el_capacity", "description" + ] = "Installed electrical capacity" + fields_df.loc["th_capacity", "description"] = "Installed thermal capacity" + fields_df.loc[ + "electrical_bus_id", "description" + ] = "Index of corresponding electricity bus" + fields_df.loc[ + "district_heating_area_id", "description" + ] = "Index of corresponding district heating bus" + fields_df.loc[ + "ch4_bus_id", "description" + ] = "Index of corresponding methane bus" + fields_df.loc["voltage_level", "description"] = "Voltage level" + fields_df.loc["scenario", "description"] = "Name of scenario" + fields_df.loc["geom", "description"] = "Location of CHP plant" + + fields_df.loc["el_capacity", "unit"] = "MW_el" + fields_df.loc["th_capacity", "unit"] = "MW_th" + fields_df.unit.fillna("none", inplace=True) + + fields = fields_df.reset_index().to_dict(orient="records") + + meta_district = { + "name": "supply.egon_chp_plants", + "title": "eGon combined heat and power plants", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": "Combined heat and power plants", + "language": ["EN"], + "publicationDate": datetime.date.today().isoformat(), + "context": context(), + "spatial": { + "location": None, + "extent": "Germany", + "resolution": None, + }, + "sources": [ + sources()["vg250"], + sources()["egon-data"], + sources()["egon-data_bundle"], + sources()["openstreetmap"], + sources()["mastr"], + ], + "licenses": [license_egon_data_odbl()], + "contributors": [ + { + "title": "Clara Büttner", + "email": "http://github.com/ClaraBuettner", + "date": time.strftime("%Y-%m-%d"), + "object": None, + "comment": "Imported data", + }, + ], + "resources": [ + { + "profile": "tabular-data-resource", + "name": "supply.egon_chp_plants", + "path": None, + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": fields, + "primaryKey": ["index"], + "foreignKeys": [], + }, + "dialect": {"delimiter": None, "decimalSeparator": "."}, + } + ], + "metaMetadata": meta_metadata(), + } + + # Add metadata as a comment to the table + db.submit_comment( + "'" + json.dumps(meta_district) + "'", + EgonChp.__table__.schema, + EgonChp.__table__.name, + ) + + def create_tables(): """Create tables for chp data Returns @@ -529,15 +638,49 @@ def insert_chp_egon100re(): class Chp(Dataset): + """ + Extract combined heat and power plants for each scenario + + This dataset creates combined heat and power (CHP) plants for each scenario and defines their use case. + The method bases on existing CHP plants from Marktstammdatenregister. For the eGon2035 scenario, + a list of CHP plans from the grid operator is used for new largescale CHP plants. CHP < 10MW are + randomly distributed. + Depending on the distance to a district heating grid, it is decided if the CHP is used to + supply a district heating grid or used by an industrial site. + + + *Dependencies* + * :py:class:`GasAreaseGon100RE ` + * :py:class:`GasAreaseGon2035 ` + * :py:class:`DistrictHeatingAreas ` + * :py:class:`IndustrialDemandCurves ` + * :py:class:`OsmLanduse ` + * :py:func:`download_mastr_data ` + * :py:func:`define_mv_grid_districts ` + * :py:class:`ScenarioCapacities ` + + + *Resulting tables* + * :py:class:`supply.egon_chp_plants ` is created and filled + * :py:class:`supply.egon_mastr_conventional_without_chp ` is created and filled + + """ + + #: + name: str = "Chp" + #: + version: str = "0.0.7" + def __init__(self, dependencies): super().__init__( - name="Chp", - version="0.0.6", + name=self.name, + version=self.version, dependencies=dependencies, tasks=( create_tables, {insert_chp_egon2035, insert_chp_egon100re}, assign_heat_bus, extension, + metadata, ), ) diff --git a/src/egon/data/datasets/chp_etrago.py b/src/egon/data/datasets/chp_etrago.py index 822efe550..9a178ce13 100644 --- a/src/egon/data/datasets/chp_etrago.py +++ b/src/egon/data/datasets/chp_etrago.py @@ -12,16 +12,48 @@ class ChpEtrago(Dataset): + """ + Collect data related to combined heat and power plants for the eTraGo tool + + This dataset collects data for combined heat and power plants and puts it into a format that + is needed for the transmission grid optimisation within the tool eTraGo. + This data is then writting into the corresponding tables that are read by eTraGo. + + + *Dependencies* + * :py:class:`HeatEtrago ` + * :py:class:`Chp ` + + *Resulting tables* + * :py:class:`grid.egon_etrago_link ` is extended + * :py:class:`grid.egon_etrago_generator ` is extended + + """ + + #: + name: str = "ChpEtrago" + #: + version: str = "0.0.6" + def __init__(self, dependencies): super().__init__( - name="ChpEtrago", - version="0.0.6", + name=self.name, + version=self.version, dependencies=dependencies, tasks=(insert), ) def insert_egon100re(): + """Insert combined heat and power plants into eTraGo tables + for the eGon100RE scenario. + + Returns + ------- + None. + + """ + sources = config.datasets()["chp_etrago"]["sources"] targets = config.datasets()["chp_etrago"]["targets"] diff --git a/src/egon/data/datasets/district_heating_areas/__init__.py b/src/egon/data/datasets/district_heating_areas/__init__.py index 82ebde10c..68e24a17e 100644 --- a/src/egon/data/datasets/district_heating_areas/__init__.py +++ b/src/egon/data/datasets/district_heating_areas/__init__.py @@ -35,7 +35,6 @@ meta_metadata, license_ccby, sources, - generate_resource_fields_from_sqla_model, ) import json @@ -48,19 +47,52 @@ from egon.data.datasets import Dataset + # class for airflow task management (and version control) class DistrictHeatingAreas(Dataset): + """ + Create district heating grids for all scenarios + + This dataset creates district heating grids for each scenario based on a defined + district heating share, annual heat demands calcultaed within + :py:class:`HeatDemandImport ` + and information on existing heating grids from census :py:class:`ZensusMiscellaneous ` + + First the tables are created using :py:func:`create_tables`. Afterwards, the + distict heating grids for each scenario are created and inserted into the database + by applying the function :py:func:`district_heating_areas` + + + *Dependencies* + * :py:class:`HeatDemandImport ` + * :py:class:`ZensusMiscellaneous ` + * :py:class:`ScenarioParameters ` + + *Resulting tables* + * :py:class:`demand.egon_map_zensus_district_heating_areas ` + is created and filled + * :py:class:`demand.egon_district_heating_areas ` is created and filled + + """ + + #: + name: str = "district-heating-areas" + #: + version: str = "0.0.2" + def __init__(self, dependencies): super().__init__( - name="district-heating-areas", + name=self.name, # version=self.target_files + "_0.0", - version="0.0.2", # maybe rethink the naming + version=self.version, # maybe rethink the naming dependencies=dependencies, tasks=(create_tables, demarcation), ) Base = declarative_base() + + # definition of classes for saving data in the database class MapZensusDistrictHeatingAreas(Base): __tablename__ = "egon_map_zensus_district_heating_areas" diff --git a/src/egon/data/datasets/electrical_neighbours.py b/src/egon/data/datasets/electrical_neighbours.py index b1fa1ccb3..b60b198e5 100755 --- a/src/egon/data/datasets/electrical_neighbours.py +++ b/src/egon/data/datasets/electrical_neighbours.py @@ -17,10 +17,45 @@ class ElectricalNeighbours(Dataset): + """ + Add lines, loads, generation and storage for electrical neighbours + + This dataset creates data for modelling the considered foreign countries and writes + that data into the database tables that can be read by the eTraGo tool. + Neighbouring countries are modelled in a lower spatial resolution, in general one node per + country is considered. + Defined load timeseries as well as generatrion and storage capacities are connected to these nodes. + The nodes are connected by AC and DC transmission lines with the German grid and other neighbouring countries + considering the grid topology from ENTSO-E. + + + *Dependencies* + * :py:class:`Tyndp ` + * :py:class:`PypsaEurSec ` + + + *Resulting tables* + * :py:class:`grid.egon_etrago_bus ` is extended + * :py:class:`grid.egon_etrago_link ` is extended + * :py:class:`grid.egon_etrago_line ` is extended + * :py:class:`grid.egon_etrago_load ` is extended + * :py:class:`grid.egon_etrago_load_timeseries ` is extended + * :py:class:`grid.egon_etrago_storage ` is extended + * :py:class:`grid.egon_etrago_generator ` is extended + * :py:class:`grid.egon_etrago_generator_timeseries ` is extended + * :py:class:`grid.egon_etrago_transformer ` is extended + + """ + + #: + name: str = "ElectricalNeighbours" + #: + version: str = "0.0.7" + def __init__(self, dependencies): super().__init__( - name="ElectricalNeighbours", - version="0.0.7", + name=self.name, + version=self.version, dependencies=dependencies, tasks=(grid, {tyndp_generation, tyndp_demand}), ) @@ -488,7 +523,6 @@ def central_transformer(scenario, sources, targets, central_buses, new_lines): # Add one transformer per central foreign bus with v_nom != 380 for i, row in central_buses[central_buses.v_nom != 380].iterrows(): - s_nom_0 = new_lines[new_lines.bus0 == row.bus_id].s_nom.sum() s_nom_1 = new_lines[new_lines.bus1 == row.bus_id].s_nom.sum() if s_nom_0 == 0.0: @@ -674,7 +708,6 @@ def grid(): targets = config.datasets()["electrical_neighbours"]["targets"] for scenario in ["eGon2035"]: - central_buses = buses(scenario, sources, targets) foreign_lines = cross_border_lines( @@ -1078,9 +1111,12 @@ def insert_storage(capacities): ) # Add columns for additional parameters to df - store["dispatch"], store["store"], store["standing_loss"], store[ - "max_hours" - ] = (None, None, None, None) + ( + store["dispatch"], + store["store"], + store["standing_loss"], + store["max_hours"], + ) = (None, None, None, None) # Insert carrier specific parameters diff --git a/src/egon/data/datasets/electricity_demand_timeseries/cts_buildings.py b/src/egon/data/datasets/electricity_demand_timeseries/cts_buildings.py index 3ac66c1ad..748ddac94 100644 --- a/src/egon/data/datasets/electricity_demand_timeseries/cts_buildings.py +++ b/src/egon/data/datasets/electricity_demand_timeseries/cts_buildings.py @@ -1,168 +1,12 @@ """ CTS electricity and heat demand time series for scenarios in 2035 and 2050 -assigned to OSM-buildings. +assigned to OSM-buildings are generated. -Disaggregation of cts heat & electricity demand time series from MV Substation +Disaggregation of CTS heat & electricity demand time series from MV substation to census cells via annual demand and then to OSM buildings via amenity tags or randomly if no sufficient OSM-data is available in the respective census cell. If no OSM-buildings or synthetic residential buildings are available new synthetic 5x5m buildings are generated. - -The resulting data is stored in separate tables - -* `openstreetmap.osm_buildings_synthetic`: - Lists generated synthetic building with id, zensus_population_id and - building type. This table is already created within - :func:`hh_buildings.map_houseprofiles_to_buildings()` -* `openstreetmap.egon_cts_buildings`: - Table of all selected cts buildings with id, census cell id, geometry and - amenity count in building. This table is created within - :func:`cts_buildings()` -* `demand.egon_cts_electricity_demand_building_share`: - Table including the mv substation electricity profile share of all selected - cts buildings for scenario eGon2035 and eGon100RE. This table is created - within :func:`cts_electricity()` -* `demand.egon_cts_heat_demand_building_share`: - Table including the mv substation heat profile share of all selected - cts buildings for scenario eGon2035 and eGon100RE. This table is created - within :func:`cts_heat()` -* `demand.egon_building_electricity_peak_loads`: - Mapping of electricity demand time series and buildings including cell_id, - building area and peak load. This table is already created within - :func:`hh_buildings.get_building_peak_loads()` -* `boundaries.egon_map_zensus_mvgd_buildings`: - A final mapping table including all buildings used for residential and - cts, heat and electricity timeseries. Including census cells, mvgd bus_id, - building type (osm or synthetic) - -**The following datasets from the database are mainly used for creation:** - -* `openstreetmap.osm_buildings_filtered`: - Table of OSM-buildings filtered by tags to selecting residential and cts - buildings only. -* `openstreetmap.osm_amenities_shops_filtered`: - Table of OSM-amenities filtered by tags to select cts only. -* `openstreetmap.osm_amenities_not_in_buildings_filtered`: - Table of amenities which do not intersect with any building from - `openstreetmap.osm_buildings_filtered` -* `openstreetmap.osm_buildings_synthetic`: - Table of synthetic residential buildings -* `boundaries.egon_map_zensus_buildings_filtered_all`: - Mapping table of census cells and buildings filtered even if population - in census cell = 0. -* `demand.egon_demandregio_zensus_electricity`: - Table of annual electricity load demand for residential and cts at census - cell level. Residential load demand is derived from aggregated residential - building profiles. DemandRegio CTS load demand at NUTS3 is distributed to - census cells linearly to heat demand from peta5. -* `demand.egon_peta_heat`: - Table of annual heat load demand for residential and cts at census cell - level from peta5. -* `demand.egon_etrago_electricity_cts`: - Scaled cts electricity time series for every MV substation. Derived from - DemandRegio SLP for selected economic sectors at nuts3. Scaled with annual - demand from `demand.egon_demandregio_zensus_electricity` -* `demand.egon_etrago_heat_cts`: - Scaled cts heat time series for every MV substation. Derived from - DemandRegio SLP Gas for selected economic sectors at nuts3. Scaled with - annual demand from `demand.egon_peta_heat`. - -**What is the goal?** - -To disaggregate cts heat and electricity time series from MV substation level -to geo-referenced buildings, the annual demand from DemandRegio and Peta5 is -used to identify census cells with load demand. We use Openstreetmap data and -filter tags to identify buildings and count the amenities within. The number -of amenities and the annual demand serve to assign a demand share of the MV -substation profile to the building. - -**What is the challenge?** - -The OSM, DemandRegio and Peta5 dataset differ from each other. The OSM dataset -is a community based dataset which is extended throughout and does not claim to -be complete. Therefore, not all census cells which have a demand assigned by -DemandRegio or Peta5 methodology also have buildings with respective tags or -sometimes even any building at all. Furthermore, the substation load areas are -determined dynamically in a previous dataset. Merging these datasets different -scopes (census cell shapes, building shapes) and their inconsistencies need to -be addressed. For example: not yet tagged buildings or amenities in OSM, or -building shapes exceeding census cells. - - -**How are these datasets combined?** - - -The methodology for heat and electricity is the same and only differs in the -annual demand and MV/HV Substation profile. In a previous dataset -(openstreetmap), we filter all OSM buildings and amenities for tags, we relate -to the cts sector. Amenities are mapped to intersecting buildings and then -intersected with the annual demand which exists at census cell level. We obtain -census cells with demand and amenities and without amenities. If there is no -data on amenities, n synthetic ones are assigned to existing buildings. We use -the median value of amenities/census cell for n and all filtered buildings + -synthetic residential buildings. If no building data is available a synthetic -buildings is randomly generated. This also happens for amenities which couldn't -be assigned to any osm building. All census cells with an annual demand are -covered this way, and we obtain four different categories of buildings with -amenities: - -* Buildings with amenities -* Synthetic buildings with amenities -* Buildings with synthetic amenities -* Synthetics buildings with synthetic amenities - -The amenities are summed per census cell (of amenity) and building to derive -the building amenity share per census cell. Multiplied with the annual demand, -we receive the profile demand share for each cell. Some buildings exceed the -census cell shape and have amenities in different cells although mapped to one -building only. To have unique buildings the demand share is summed once more -per building id. This factor can now be used to obtain the profile for each -building. - -A schematic flow chart exist in the correspondent issue #671: -https://github.com/openego/eGon-data/issues/671#issuecomment-1260740258 - - -**What are central assumptions during the data processing?** - -* We assume OSM data to be the most reliable and complete open source dataset. -* We assume building and amenity tags to be truthful and accurate. -* Mapping census to OSM data is not trivial. Discrepancies are substituted. -* Missing OSM buildings are generated for each amenity. -* Missing amenities are generated by median value of amenities/census cell. - - -**Drawbacks and limitations of the data** - -* Shape of profiles for each building is similar within a MVGD and only scaled -with a different factor. -* MVGDs are generated dynamically. In case of buildings with amenities -exceeding MVGD borders, amenities which are assigned to a different MVGD than -the assigned building centroid, the amenities are dropped for sake of -simplicity. One building should not have a connection to two MVGDs. -* The completeness of the OSM data depends on community contribution and is -crucial to the quality of our results. -* Randomly selected buildings and generated amenities may inadequately reflect -reality, but are chosen for sake of simplicity as a measure to fill data gaps. -* Since this dataset is a cascade after generation of synthetic residential -buildings also check drawbacks and limitations in hh_buildings.py. -* Synthetic buildings may be placed within osm buildings which exceed multiple -census cells. This is currently accepted but may be solved in #953 -* Scattered high peak loads occur and might lead to single MV grid connections -in ding0. In some cases this might not be viable. Postprocessing is needed and -may be solved in #954. - - -Example Query ------ - - -Notes ------ - -This module docstring is rather a dataset documentation. Once, a decision -is made in ... the content of this module docstring needs to be moved to -docs attribute of the respective dataset class. """ from geoalchemy2 import Geometry @@ -212,6 +56,13 @@ class EgonCtsElectricityDemandBuildingShare(Base): + """ + Class definition of table demand.egon_cts_electricity_demand_building_share. + + Table including the MV substation electricity profile share of all selected + CTS buildings for scenario eGon2035 and eGon100RE. This table is created + within :func:`cts_electricity()`. + """ __tablename__ = "egon_cts_electricity_demand_building_share" __table_args__ = {"schema": "demand"} @@ -222,6 +73,13 @@ class EgonCtsElectricityDemandBuildingShare(Base): class EgonCtsHeatDemandBuildingShare(Base): + """ + Class definition of table demand.egon_cts_heat_demand_building_share. + + Table including the MV substation heat profile share of all selected + CTS buildings for scenario eGon2035 and eGon100RE. This table is created + within :func:`cts_heat()`. + """ __tablename__ = "egon_cts_heat_demand_building_share" __table_args__ = {"schema": "demand"} @@ -232,6 +90,13 @@ class EgonCtsHeatDemandBuildingShare(Base): class CtsBuildings(Base): + """ + Class definition of table openstreetmap.egon_cts_buildings. + + Table of all selected CTS buildings with id, census cell id, geometry and + amenity count in building. This table is created within + :func:`cts_buildings()`. + """ __tablename__ = "egon_cts_buildings" __table_args__ = {"schema": "openstreetmap"} @@ -244,6 +109,10 @@ class CtsBuildings(Base): class BuildingHeatPeakLoads(Base): + """ + Class definition of table demand.egon_building_heat_peak_loads. + + """ __tablename__ = "egon_building_heat_peak_loads" __table_args__ = {"schema": "demand"} @@ -254,10 +123,160 @@ class BuildingHeatPeakLoads(Base): class CtsDemandBuildings(Dataset): + """ + Generates CTS electricity and heat demand time series for scenarios in 2035 and 2050 + assigned to OSM-buildings. + + Disaggregation of CTS heat & electricity demand time series from MV Substation + to census cells via annual demand and then to OSM buildings via + amenity tags or randomly if no sufficient OSM-data is available in the + respective census cell. If no OSM-buildings or synthetic residential buildings + are available new synthetic 5x5m buildings are generated. + + + *Dependencies* + * :py:class:`OsmBuildingsStreets ` + * :py:class:`CtsElectricityDemand ` + * :py:class:`hh_buildings ` + * :py:class:`HeatTimeSeries ` (more specifically the :func:`export_etrago_cts_heat_profiles ` task) + + *Resulting tables* + * :py:class:`openstreetmap.osm_buildings_synthetic ` is extended + * :py:class:`openstreetmap.egon_cts_buildings is created + * :py:class:`demand.egon_cts_electricity_demand_building_share ` is created + * :py:class:`demand.egon_cts_heat_demand_building_share ` is created + * :py:class:`demand.egon_building_electricity_peak_loads ` is extended + * :py:class:`boundaries.egon_map_zensus_mvgd_buildings ` is extended. + + + **The following datasets from the database are mainly used for creation:** + + * `openstreetmap.osm_buildings_filtered`: + Table of OSM-buildings filtered by tags to selecting residential and cts + buildings only. + * `openstreetmap.osm_amenities_shops_filtered`: + Table of OSM-amenities filtered by tags to select cts only. + * `openstreetmap.osm_amenities_not_in_buildings_filtered`: + Table of amenities which do not intersect with any building from + `openstreetmap.osm_buildings_filtered` + * `openstreetmap.osm_buildings_synthetic`: + Table of synthetic residential buildings + * `boundaries.egon_map_zensus_buildings_filtered_all`: + Mapping table of census cells and buildings filtered even if population + in census cell = 0. + * `demand.egon_demandregio_zensus_electricity`: + Table of annual electricity load demand for residential and cts at census + cell level. Residential load demand is derived from aggregated residential + building profiles. DemandRegio CTS load demand at NUTS3 is distributed to + census cells linearly to heat demand from peta5. + * `demand.egon_peta_heat`: + Table of annual heat load demand for residential and cts at census cell + level from peta5. + * `demand.egon_etrago_electricity_cts`: + Scaled cts electricity time series for every MV substation. Derived from + DemandRegio SLP for selected economic sectors at nuts3. Scaled with annual + demand from `demand.egon_demandregio_zensus_electricity` + * `demand.egon_etrago_heat_cts`: + Scaled cts heat time series for every MV substation. Derived from + DemandRegio SLP Gas for selected economic sectors at nuts3. Scaled with + annual demand from `demand.egon_peta_heat`. + + **What is the goal?** + + To disaggregate cts heat and electricity time series from MV substation level + to geo-referenced buildings, the annual demand from DemandRegio and Peta5 is + used to identify census cells with load demand. We use Openstreetmap data and + filter tags to identify buildings and count the amenities within. The number + of amenities and the annual demand serve to assign a demand share of the MV + substation profile to the building. + + **What is the challenge?** + + The OSM, DemandRegio and Peta5 dataset differ from each other. The OSM dataset + is a community based dataset which is extended throughout and does not claim to + be complete. Therefore, not all census cells which have a demand assigned by + DemandRegio or Peta5 methodology also have buildings with respective tags or + sometimes even any building at all. Furthermore, the substation load areas are + determined dynamically in a previous dataset. Merging these datasets different + scopes (census cell shapes, building shapes) and their inconsistencies need to + be addressed. For example: not yet tagged buildings or amenities in OSM, or + building shapes exceeding census cells. + + + **How are these datasets combined?** + + + The methodology for heat and electricity is the same and only differs in the + annual demand and MV/HV Substation profile. In a previous dataset + (openstreetmap), we filter all OSM buildings and amenities for tags, we relate + to the cts sector. Amenities are mapped to intersecting buildings and then + intersected with the annual demand which exists at census cell level. We obtain + census cells with demand and amenities and without amenities. If there is no + data on amenities, n synthetic ones are assigned to existing buildings. We use + the median value of amenities/census cell for n and all filtered buildings + + synthetic residential buildings. If no building data is available a synthetic + buildings is randomly generated. This also happens for amenities which couldn't + be assigned to any osm building. All census cells with an annual demand are + covered this way, and we obtain four different categories of buildings with + amenities: + + * Buildings with amenities + * Synthetic buildings with amenities + * Buildings with synthetic amenities + * Synthetics buildings with synthetic amenities + + The amenities are summed per census cell (of amenity) and building to derive + the building amenity share per census cell. Multiplied with the annual demand, + we receive the profile demand share for each cell. Some buildings exceed the + census cell shape and have amenities in different cells although mapped to one + building only. To have unique buildings the demand share is summed once more + per building id. This factor can now be used to obtain the profile for each + building. + + A schematic flow chart exist in the correspondent issue #671: + https://github.com/openego/eGon-data/issues/671#issuecomment-1260740258 + + + **What are central assumptions during the data processing?** + + * We assume OSM data to be the most reliable and complete open source dataset. + * We assume building and amenity tags to be truthful and accurate. + * Mapping census to OSM data is not trivial. Discrepancies are substituted. + * Missing OSM buildings are generated for each amenity. + * Missing amenities are generated by median value of amenities/census cell. + + + **Drawbacks and limitations of the data** + + * Shape of profiles for each building is similar within a MVGD and only scaled + with a different factor. + * MVGDs are generated dynamically. In case of buildings with amenities + exceeding MVGD borders, amenities which are assigned to a different MVGD than + the assigned building centroid, the amenities are dropped for sake of + simplicity. One building should not have a connection to two MVGDs. + * The completeness of the OSM data depends on community contribution and is + crucial to the quality of our results. + * Randomly selected buildings and generated amenities may inadequately reflect + reality, but are chosen for sake of simplicity as a measure to fill data gaps. + * Since this dataset is a cascade after generation of synthetic residential + buildings also check drawbacks and limitations in hh_buildings.py. + * Synthetic buildings may be placed within osm buildings which exceed multiple + census cells. This is currently accepted but may be solved in #953. + * Scattered high peak loads occur and might lead to single MV grid connections + in ding0. In some cases this might not be viable. Postprocessing is needed and + may be solved in #954. + + """ + + #: + name: str = "CtsDemandBuildings" + #: + version: str = "0.0.3" + def __init__(self, dependencies): super().__init__( - name="CtsDemandBuildings", - version="0.0.3", + name=self.name, + version=self.version, dependencies=dependencies, tasks=( cts_buildings, diff --git a/src/egon/data/datasets/electricity_demand_timeseries/hh_buildings.py b/src/egon/data/datasets/electricity_demand_timeseries/hh_buildings.py index 9d5cbd4a9..89ac83c50 100755 --- a/src/egon/data/datasets/electricity_demand_timeseries/hh_buildings.py +++ b/src/egon/data/datasets/electricity_demand_timeseries/hh_buildings.py @@ -2,115 +2,8 @@ Household electricity demand time series for scenarios in 2035 and 2050 assigned to OSM-buildings. -Assignment of household electricity demand timeseries to OSM buildings and -generation of randomly placed synthetic 5x5m buildings if no sufficient OSM-data -available in the respective cencus cell. - -The resulting data is stored in separate tables - -* `openstreetmap.osm_buildings_synthetic`: - Lists generated synthetic building with id and cell_id -* `demand.egon_household_electricity_profile_of_buildings`: - Mapping of demand timeseries and buildings including cell_id, building - area and peak load - -Both tables are created within :func:`map_houseprofiles_to_buildings`. - - -**The following datasets from the database are used for creation:** - -* `demand.household_electricity_profiles_in_census_cells`: - Lists references and scaling parameters to time series data for each - household in a cell by identifiers. This table is fundamental for creating - subsequent data like demand profiles on MV grid level or for determining - the peak load at load. Only the profile reference and the cell identifiers - are used. - -* `society.egon_destatis_zensus_apartment_building_population_per_ha`: - Lists number of apartments, buildings and population for each census cell. - -* `boundaries.egon_map_zensus_buildings_residential`: - List of OSM tagged buildings which are considered to be residential. - - -**What is the goal?** - -To assign every household demand timeseries, which already exist at cell level, -to a specific OSM building. - -**What is the challenge?** - -The census and the OSM dataset differ from each other. The census uses -statistical methods and therefore lacks accuracy at high spatial resolution. -The OSM datasets is community based dataset which is extended throughout and -does not claim to be complete. By merging these datasets inconsistencies need -to be addressed. For example: not yet tagged buildings in OSM or new building -areas not considered in census 2011. - -**How are these datasets combined?** - -The assignment of household demand timeseries to buildings takes place at cell -level. Within each cell a pool of profiles exists, produced by the 'HH Demand" -module. These profiles are randomly assigned to a filtered list of OSM buildings -within this cell. Every profile is assigned to a building and every building -get a profile assigned if there is enough households by the census data. If -there are more profiles then buildings, all additional profiles are randomly -assigned. Therefore multiple profiles can be assigned to one building, making -it a multi-household building. - - -**What are central assumptions during the data processing?** - -* Mapping zensus data to OSM data is not trivial. Discrepancies are substituted. -* Missing OSM buildings are generated by census building count. -* If no census building count data is available, the number of buildings is -derived by an average rate of households/buildings applied to the number of -households. - -**Drawbacks and limitations of the data** - -* Missing OSM buildings in cells without census building count are derived by -an average rate of households/buildings applied to the number of households. -As only whole houses can exist, the substitute is ceiled to the next higher -integer. Ceiling is applied to avoid rounding to amount of 0 buildings. - -* As this datasets is a cascade after profile assignement at census cells -also check drawbacks and limitations in hh_profiles.py. - - - -Example Query ------ - - -* Get a list with number of houses, households and household types per census cell - -.. code-block:: SQL - - SELECT t1.cell_id, building_count, hh_count, hh_types - FROM( - SELECT cell_id, Count(distinct(building_id)) as building_count, - count(profile_id) as hh_count - FROM demand.egon_household_electricity_profile_of_buildings - Group By cell_id - ) as t1 - FULL OUTER JOIN( - SELECT cell_id, array_agg(array[cast(hh_10types as char), - hh_type]) as hh_types - FROM society.egon_destatis_zensus_household_per_ha_refined - GROUP BY cell_id - ) as t2 - ON t1.cell_id = t2.cell_id - - -Notes ------ - -This module docstring is rather a dataset documentation. Once, a decision -is made in ... the content of this module docstring needs to be moved to -docs attribute of the respective dataset class. """ -from functools import partial + import random from geoalchemy2 import Geometry @@ -140,6 +33,14 @@ class HouseholdElectricityProfilesOfBuildings(Base): + """ + Class definition of table demand.egon_household_electricity_profile_of_buildings. + + Mapping of demand timeseries and buildings and cell_id. This table is created within + :py:func:`hh_buildings.map_houseprofiles_to_buildings()`. + + """ + __tablename__ = "egon_household_electricity_profile_of_buildings" __table_args__ = {"schema": "demand"} @@ -150,6 +51,14 @@ class HouseholdElectricityProfilesOfBuildings(Base): class OsmBuildingsSynthetic(Base): + """ + Class definition of table demand.osm_buildings_synthetic. + + Lists generated synthetic building with id, zensus_population_id and + building type. This table is created within + :py:func:`hh_buildings.map_houseprofiles_to_buildings()`. + """ + __tablename__ = "osm_buildings_synthetic" __table_args__ = {"schema": "openstreetmap"} @@ -163,6 +72,14 @@ class OsmBuildingsSynthetic(Base): class BuildingElectricityPeakLoads(Base): + """ + Class definition of table demand.egon_building_electricity_peak_loads. + + Mapping of electricity demand time series and buildings including cell_id, + building area and peak load. This table is created within + :func:`hh_buildings.get_building_peak_loads()`. + """ + __tablename__ = "egon_building_electricity_peak_loads" __table_args__ = {"schema": "demand"} @@ -446,7 +363,6 @@ def generate_mapping_table( """ def create_pool(buildings, profiles): - if profiles > buildings: surplus = profiles - buildings surplus = rng.integers(0, buildings, surplus) @@ -618,7 +534,9 @@ def reduce_synthetic_buildings( # ) # time expensive because of regex - # mapping_profiles_to_buildings['building_id'] = mapping_profiles_to_buildings['building_id'].replace(id_mapping) + # mapping_profiles_to_buildings['building_id'] = ( + # mapping_profiles_to_buildings['building_id'].replace(id_mapping) + # ) return synthetic_buildings @@ -805,6 +723,7 @@ def map_houseprofiles_to_buildings(): OsmBuildingsSynthetic.__table__.create(bind=engine, checkfirst=True) # Write new buildings incl coord into db + n_amenities_inside_type = OsmBuildingsSynthetic.n_amenities_inside.type synthetic_buildings.to_postgis( "osm_buildings_synthetic", con=engine, @@ -815,7 +734,7 @@ def map_houseprofiles_to_buildings(): "cell_id": OsmBuildingsSynthetic.cell_id.type, "geom_building": OsmBuildingsSynthetic.geom_building.type, "geom_point": OsmBuildingsSynthetic.geom_point.type, - "n_amenities_inside": OsmBuildingsSynthetic.n_amenities_inside.type, + "n_amenities_inside": n_amenities_inside_type, "building": OsmBuildingsSynthetic.building.type, "area": OsmBuildingsSynthetic.area.type, }, @@ -836,10 +755,134 @@ def map_houseprofiles_to_buildings(): ) -setup = partial( - Dataset, - name="Demand_Building_Assignment", - version="0.0.5", - dependencies=[], - tasks=(map_houseprofiles_to_buildings, get_building_peak_loads), -) +class setup(Dataset): + """ + Household electricity demand time series for scenarios in 2035 and 2050 + assigned to OSM-buildings. + + Assignment of household electricity demand timeseries to OSM buildings + and generation of randomly placed synthetic 5x5m buildings if no + sufficient OSM-data available in the respective census cell. + + *Dependencies* + * :py:func:`houseprofiles_in_census_cells + ` + + *Resulting tables* + * :py:class:`OsmBuildingsSynthetic + ` + is created and filled + * :py:class:`HouseholdElectricityProfilesOfBuildings + ` + is created and filled + * :py:class:`BuildingElectricityPeakLoads + ` + is created and filled + + **The following datasets from the database are used for creation:** + + * `demand.household_electricity_profiles_in_census_cells`: + Lists references and scaling parameters to time series data for each + household in a cell by identifiers. This table is fundamental for + creating subsequent data like demand profiles on MV grid level or + for determining the peak load at load. Only the profile reference + and the cell identifiers are used. + + * `society.egon_destatis_zensus_apartment_building_population_per_ha`: + Lists number of apartments, buildings and population for each census + cell. + + * `boundaries.egon_map_zensus_buildings_residential`: + List of OSM tagged buildings which are considered to be residential. + + + **What is the goal?** + + To assign every household demand timeseries, which already exist at cell + level, to a specific OSM building. + + **What is the challenge?** + + The census and the OSM dataset differ from each other. The census uses + statistical methods and therefore lacks accuracy at high spatial + resolution. The OSM datasets is community based dataset which is + extended throughout and does not claim to be complete. By merging these + datasets inconsistencies need to be addressed. For example: not yet + tagged buildings in OSM or new building areas not considered in census + 2011. + + **How are these datasets combined?** + + The assignment of household demand timeseries to buildings takes place + at cell level. Within each cell a pool of profiles exists, produced by + the 'HH Demand" module. These profiles are randomly assigned to a + filtered list of OSM buildings within this cell. Every profile is + assigned to a building and every building get a profile assigned if + there is enough households by the census data. If there are more + profiles than buildings, all additional profiles are randomly assigned. + Therefore, multiple profiles can be assigned to one building, making it a + multi-household building. + + **What are central assumptions during the data processing?** + + * Mapping zensus data to OSM data is not trivial. + Discrepancies are substituted. + * Missing OSM buildings are generated by census building count. + * If no census building count data is available, the number of buildings + is derived by an average rate of households/buildings applied to the + number of households. + + **Drawbacks and limitations of the data** + + * Missing OSM buildings in cells without census building count are + derived by an average rate of households/buildings applied to the + number of households. As only whole houses can exist, the substitute + is ceiled to the next higher integer. Ceiling is applied to avoid + rounding to amount of 0 buildings. + + * As this datasets is a cascade after profile assignement at census + cells also check drawbacks and limitations in hh_profiles.py. + + Example Query + ----- + + * Get a list with number of houses, households and household types per + census cell + + .. code-block:: SQL + + SELECT t1.cell_id, building_count, hh_count, hh_types FROM ( + SELECT + cell_id, + COUNT(DISTINCT(building_id)) AS building_count, + COUNT(profile_id) AS hh_count + FROM demand.egon_household_electricity_profile_of_buildings + GROUP BY cell_id + ) AS t1 + FULL OUTER JOIN ( + SELECT + cell_id, + array_agg( + array[CAST(hh_10types AS char), hh_type] + ) AS hh_types + FROM society.egon_destatis_zensus_household_per_ha_refined + GROUP BY cell_id + ) AS t2 + ON t1.cell_id = t2.cell_id + + """ + + #: + name: str = "Demand_Building_Assignment" + #: + version: str = "0.0.5" + #: + tasks = (map_houseprofiles_to_buildings, get_building_peak_loads) + + def __init__(self, dependencies): + super().__init__( + name=self.name, + version=self.version, + dependencies=dependencies, + tasks=self.tasks, + ) diff --git a/src/egon/data/datasets/electricity_demand_timeseries/hh_profiles.py b/src/egon/data/datasets/electricity_demand_timeseries/hh_profiles.py index fc12de792..6598e7695 100644 --- a/src/egon/data/datasets/electricity_demand_timeseries/hh_profiles.py +++ b/src/egon/data/datasets/electricity_demand_timeseries/hh_profiles.py @@ -1,125 +1,12 @@ """ -Household electricity demand time series for scenarios in 2035 and 2050 at -census cell level. +Household electricity demand time series for scenarios eGon2035 and eGon100RE at +census cell level are set up. Electricity demand data for households in Germany in 1-hourly resolution for an entire year. Spatially, the data is resolved to 100 x 100 m cells and provides individual and distinct time series for each household in a cell. The cells are defined by the dataset Zensus 2011. -The resulting data is stored in two separate tables - -* `demand.household_electricity_profiles_in_census_cells`: - Lists references and scaling parameters to time series data for each - household in a cell by identifiers. This table is fundamental for creating - subsequent data like demand profiles on MV grid level or for determining - the peak load at load area level. - The table is created by:func:`houseprofiles_in_census_cells`. -* `demand.household_electricity_profiles_hvmv_substation`: - Household electricity demand profiles aggregated at MV grid district level - in MWh. Primarily used to create the eTraGo data model. - The table is created with :func:`mv_grid_district_HH_electricity_load`. - -The following datasets are used for creating the data: - -* Electricity demand time series for household categories - produced by demand profile generator (DPG) from Fraunhofer IEE - (see :func:`get_iee_hh_demand_profiles_raw`) -* Spatial information about people living in households by Zensus 2011 at - federal state level - * Type of household (family status) - * Age - * Number of people -* Spatial information about number of households per ha, categorized by type - of household (family status) with 5 categories (also from Zensus 2011) -* Demand-Regio annual household demand at NUTS3 level - -**What is the goal?** - -To use the electricity demand time series from the `demand profile generator` -to created spatially reference household demand time series for Germany at a -resolution of 100 x 100 m cells. - -**What is the challenge?** - -The electricity demand time series produced by demand profile generator offer -12 different household profile categories. -To use most of them, the spatial information about the number of households -per cell (5 categories) needs to be enriched by supplementary data to match -the household demand profile categories specifications. Hence, 10 out of 12 -different household profile categories can be distinguished by increasing -the number of categories of cell-level household data. - -**How are these datasets combined?** - -* Spatial information about people living in households by zensus (2011) at - federal state NUTS1 level :var:`df_zensus` is aggregated to be compatible - to IEE household profile specifications. - * exclude kids and reduce to adults and seniors - * group as defined in :var:`HH_TYPES` - * convert data from people living in households to number of households - by :var:`mapping_people_in_households` - * calculate fraction of fine household types (10) within subgroup of rough - household types (5) :var:`df_dist_households` -* Spatial information about number of households per ha - :var:`df_census_households_nuts3` is mapped to NUTS1 and NUTS3 level. - Data is refined with household subgroups via - :var:`df_dist_households` to :var:`df_census_households_grid_refined`. -* Enriched 100 x 100 m household dataset is used to sample and aggregate - household profiles. A table including individual profile id's for each cell - and scaling factor to match Demand-Regio annual sum projections for 2035 - and 2050 at NUTS3 level is created in the database as - `demand.household_electricity_profiles_in_census_cells`. - -**What are central assumptions during the data processing?** - -* Mapping zensus data to IEE household categories is not trivial. In - conversion from persons in household to number of - households, number of inhabitants for multi-person households is estimated - as weighted average in :var:`OO_factor` -* The distribution to refine household types at cell level are the same for - each federal state -* Refining of household types lead to float number of profiles drew at cell - level and need to be rounded to nearest int by np.rint(). -* 100 x 100 m cells are matched to NUTS via cells centroid location -* Cells with households in unpopulated areas are removed - -**Drawbacks and limitations of the data** - -* The distribution to refine household types at cell level are the same for - each federal state -* Household profiles aggregated annual demand matches Demand Regio demand at - NUTS-3 level, but it is not matching the demand regio time series profile -* Due to secrecy, some census data are highly modified under certain attributes - (quantity_q = 2). This cell data is not corrected, but excluded. -* There is deviation in the Census data from table to table. The statistical - methods are not stringent. Hence, there are cases in which data contradicts. -* Census data with attribute 'HHTYP_FAM' is missing for some cells with small - amount of households. This data is generated using the average share of - household types for cells with similar household number. For some cells the - summed amount of households per type deviates from the total number with - attribute 'INSGESAMT'. As the profiles are scaled with demand-regio data at - nuts3-level the impact at a higher aggregation level is negligible. - For sake of simplicity, the data is not corrected. -* There are cells without household data but a population. A randomly chosen - household distribution is taken from a subgroup of cells with same population - value and applied to all cells with missing household distribution and the - specific population value. - -Helper functions ----- -* To access the DB, select specific profiles at various aggregation levels -use:func:`get_hh_profiles_from_db' -* To access the DB, select specific profiles at various aggregation levels -and scale profiles use :func:`get_scaled_profiles_from_db` - - -Notes ------ - -This module docstring is rather a dataset documentation. Once, a decision -is made in ... the content of this module docstring needs to be moved to -docs attribute of the respective dataset class. """ from itertools import cycle, product from pathlib import Path @@ -147,6 +34,9 @@ class IeeHouseholdLoadProfiles(Base): + """ + Class definition of table demand.iee_household_load_profiles. + """ __tablename__ = "iee_household_load_profiles" __table_args__ = {"schema": "demand"} @@ -156,6 +46,15 @@ class IeeHouseholdLoadProfiles(Base): class HouseholdElectricityProfilesInCensusCells(Base): + """ + Class definition of table demand.egon_household_electricity_profile_in_census_cell. + + Lists references and scaling parameters of time series data for each + household in a cell by identifiers. This table is fundamental for creating + subsequent data like demand profiles on MV grid level or for determining + the peak load at load area level. + + """ __tablename__ = "egon_household_electricity_profile_in_census_cell" __table_args__ = {"schema": "demand"} @@ -169,6 +68,9 @@ class HouseholdElectricityProfilesInCensusCells(Base): class EgonDestatisZensusHouseholdPerHaRefined(Base): + """ + Class definition of table society.egon_destatis_zensus_household_per_ha_refined. + """ __tablename__ = "egon_destatis_zensus_household_per_ha_refined" __table_args__ = {"schema": "society"} @@ -184,6 +86,12 @@ class EgonDestatisZensusHouseholdPerHaRefined(Base): class EgonEtragoElectricityHouseholds(Base): + """ + Class definition of table demand.egon_etrago_electricity_households. + + The table contains household electricity demand profiles aggregated at MV grid + district level in MWh. + """ __tablename__ = "egon_etrago_electricity_households" __table_args__ = {"schema": "demand"} @@ -194,6 +102,134 @@ class EgonEtragoElectricityHouseholds(Base): class HouseholdDemands(Dataset): + """ + Household electricity demand time series for scenarios eGon2035 and eGon100RE at + census cell level are set up. + + Electricity demand data for households in Germany in 1-hourly resolution for + an entire year. Spatially, the data is resolved to 100 x 100 m cells and + provides individual and distinct time series for each household in a cell. + The cells are defined by the dataset Zensus 2011. + + *Dependencies* + * :py:class:`DemandRegio ` + * :py:class:`Vg250 ` + * :py:class:`OsmBuildingsStreets ` + * :py:func:`create_buildings_residential_zensus_mapping ` + * :py:class:`ZensusMiscellaneous ` + * :py:class:`ZensusMvGridDistricts ` + * :py:class:`ZensusVg250 ` + + *Resulting tables* + * :py:class:`demand.iee_household_load_profiles + ` is created and filled + * :py:class:`demand.egon_household_electricity_profile_in_census_cell + ` is created and filled + * :py:class:`society.egon_destatis_zensus_household_per_ha_refined + ` is created and filled + * :py:class:`demand.egon_etrago_electricity_households + ` is created and filled + + The following datasets are used for creating the data: + + * Electricity demand time series for household categories + produced by demand profile generator (DPG) from Fraunhofer IEE + (see :func:`get_iee_hh_demand_profiles_raw`) + * Spatial information about people living in households by Zensus 2011 at + federal state level + * Type of household (family status) + * Age + * Number of people + * Spatial information about number of households per ha, categorized by type + of household (family status) with 5 categories (also from Zensus 2011) + * Demand-Regio annual household demand at NUTS3 level + + *What is the goal?* + + To use the electricity demand time series from the `demand profile generator` + to created spatially reference household demand time series for Germany at a + resolution of 100 x 100 m cells. + + *What is the challenge?* + + The electricity demand time series produced by demand profile generator offer + 12 different household profile categories. + To use most of them, the spatial information about the number of households + per cell (5 categories) needs to be enriched by supplementary data to match + the household demand profile categories specifications. Hence, 10 out of 12 + different household profile categories can be distinguished by increasing + the number of categories of cell-level household data. + + *How are these datasets combined?* + + * Spatial information about people living in households by zensus (2011) at + federal state NUTS1 level `df_zensus` is aggregated to be compatible + to IEE household profile specifications. + * exclude kids and reduce to adults and seniors + * group as defined in `HH_TYPES` + * convert data from people living in households to number of households + by `mapping_people_in_households` + * calculate fraction of fine household types (10) within subgroup of rough + household types (5) `df_dist_households` + * Spatial information about number of households per ha + `df_census_households_nuts3` is mapped to NUTS1 and NUTS3 level. + Data is refined with household subgroups via + `df_dist_households` to `df_census_households_grid_refined`. + * Enriched 100 x 100 m household dataset is used to sample and aggregate + household profiles. A table including individual profile id's for each cell + and scaling factor to match Demand-Regio annual sum projections for 2035 + and 2050 at NUTS3 level is created in the database as + `demand.household_electricity_profiles_in_census_cells`. + + *What are central assumptions during the data processing?* + + * Mapping zensus data to IEE household categories is not trivial. In + conversion from persons in household to number of + households, number of inhabitants for multi-person households is estimated + as weighted average in `OO_factor` + * The distribution to refine household types at cell level are the same for + each federal state + * Refining of household types lead to float number of profiles drew at cell + level and need to be rounded to nearest int by np.rint(). + * 100 x 100 m cells are matched to NUTS via cells centroid location + * Cells with households in unpopulated areas are removed + + *Drawbacks and limitations of the data* + + * The distribution to refine household types at cell level are the same for + each federal state + * Household profiles aggregated annual demand matches Demand Regio demand at + NUTS-3 level, but it is not matching the demand regio time series profile + * Due to secrecy, some census data are highly modified under certain attributes + (quantity_q = 2). This cell data is not corrected, but excluded. + * There is deviation in the Census data from table to table. The statistical + methods are not stringent. Hence, there are cases in which data contradicts. + * Census data with attribute 'HHTYP_FAM' is missing for some cells with small + amount of households. This data is generated using the average share of + household types for cells with similar household number. For some cells the + summed amount of households per type deviates from the total number with + attribute 'INSGESAMT'. As the profiles are scaled with demand-regio data at + nuts3-level the impact at a higher aggregation level is negligible. + For sake of simplicity, the data is not corrected. + * There are cells without household data but a population. A randomly chosen + household distribution is taken from a subgroup of cells with same population + value and applied to all cells with missing household distribution and the + specific population value. + + *Helper functions* + * To access the DB, select specific profiles at various aggregation levels + use :py:func:`get_hh_profiles_from_db` + * To access the DB, select specific profiles at various aggregation levels + and scale profiles use :py:func:`get_scaled_profiles_from_db` + + + """ + + #: + name: str = "Household Demands" + #: + version: str = "0.0.10" + def __init__(self, dependencies): mv_hh_electricity_load_2035 = PythonOperator( task_id="MV-hh-electricity-load-2035", @@ -209,8 +245,8 @@ def __init__(self, dependencies): ) super().__init__( - name="Household Demands", - version="0.0.10", + name=self.name, + version=self.version, dependencies=dependencies, tasks=( houseprofiles_in_census_cells, @@ -295,8 +331,7 @@ def get_iee_hh_demand_profiles_raw(): Household electricity demand profiles generated by Fraunhofer IEE. Methodology is described in - :ref:`Erzeugung zeitlich hochaufgelöster Stromlastprofile für verschiedene - Haushaltstypen + `Erzeugung zeitlich hochaufgelöster Stromlastprofile für verschiedene Haushaltstypen `_. It is used and further described in the following theses by: @@ -456,6 +491,9 @@ def create_missing_zensus_data( df_households_typ, df_missing_data, missing_cells ): """ + Generate missing data as average share of the household types for cell groups with + the same amount of households. + There is missing data for specific attributes in the zensus dataset because of secrecy reasons. Some cells with only small amount of households are missing with attribute HHTYP_FAM. However the total amount of households @@ -622,6 +660,7 @@ def process_nuts1_census_data(df_census_households_raw): def regroup_nuts1_census_data(df_census_households_nuts1): """Regroup census data and map according to demand-profile types. + For more information look at the respective publication: https://www.researchgate.net/publication/273775902_Erzeugung_zeitlich_hochaufgeloster_Stromlastprofile_fur_verschiedene_Haushaltstypen @@ -819,7 +858,11 @@ def inhabitants_to_households(df_hh_people_distribution_abs): def impute_missing_hh_in_populated_cells(df_census_households_grid): - """There are cells without household data but a population. A randomly + """ + Fills in missing household data in populated cells based on a random selection from + a subgroup of cells with the same population value. + + There are cells without household data but a population. A randomly chosen household distribution is taken from a subgroup of cells with same population value and applied to all cells with missing household distribution and the specific population value. In the case, in which there @@ -893,7 +936,11 @@ def impute_missing_hh_in_populated_cells(df_census_households_grid): def get_census_households_grid(): - """Query census household data at 100x100m grid level from database. As + """ + Retrieves and adjusts census household data at 100x100m grid level, accounting for + missing or divergent data. + + Query census household data at 100x100m grid level from database. As there is a divergence in the census household data depending which attribute is used. There also exist cells without household but with population data. The missing data in these cases are substituted. First @@ -1079,7 +1126,11 @@ def refine_census_data_at_cell_level( df_census_households_grid, df_census_households_nuts1, ): - """The census data is processed to define the number and type of households + """ + Processes and merges census data to specify household numbers and types per census + cell according to IEE profiles. + + The census data is processed to define the number and type of households per zensus cell. Two subsets of the census data are merged to fit the IEE profiles specifications. To do this, proportionate allocation is applied at nuts1 level and within household type clusters. @@ -1460,11 +1511,14 @@ def houseprofiles_in_census_cells(): """ Allocate household electricity demand profiles for each census cell. - Creates a table that maps household electricity demand profiles to census - cells. Each row represents one cell and contains a list of profile IDs. + Creates table `emand.egon_household_electricity_profile_in_census_cell` that maps + household electricity demand profiles to census cells. Each row represents one cell + and contains a list of profile IDs. This table is fundamental + for creating subsequent data like demand profiles on MV grid level or for + determining the peak load at load area level. Use :func:`get_houseprofiles_in_census_cells` to retrieve the data from - the database as pandas + the database as pandas. """ @@ -1721,6 +1775,9 @@ def mv_grid_district_HH_electricity_load( Calculate the aggregated demand time series based on the demand profiles of each zensus cell inside each MV grid district. Profiles are read from local hdf5-file. + Creates table `demand.egon_etrago_electricity_households` with + Household electricity demand profiles aggregated at MV grid district level + in MWh. Primarily used to create the eTraGo data model. Parameters ---------- diff --git a/src/egon/data/datasets/electricity_demand_timeseries/mapping.py b/src/egon/data/datasets/electricity_demand_timeseries/mapping.py index 2f6e7ca95..e5fe664e6 100644 --- a/src/egon/data/datasets/electricity_demand_timeseries/mapping.py +++ b/src/egon/data/datasets/electricity_demand_timeseries/mapping.py @@ -8,6 +8,11 @@ class EgonMapZensusMvgdBuildings(Base): + """ + A final mapping table including all buildings used for residential and + cts, heat and electricity timeseries. Including census cells, mvgd bus_id, + building type (osm or synthetic) + """ __tablename__ = "egon_map_zensus_mvgd_buildings" __table_args__ = {"schema": "boundaries"} diff --git a/src/egon/data/datasets/emobility/__init__.py b/src/egon/data/datasets/emobility/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/src/egon/data/datasets/emobility/heavy_duty_transport/__init__.py b/src/egon/data/datasets/emobility/heavy_duty_transport/__init__.py index 5699719ed..46a1e51ca 100644 --- a/src/egon/data/datasets/emobility/heavy_duty_transport/__init__.py +++ b/src/egon/data/datasets/emobility/heavy_duty_transport/__init__.py @@ -1,55 +1,16 @@ """ -Heavy Duty Transport / Heavy Goods Vehicle (HGV) - Main module for preparation of model data (static and timeseries) for heavy duty transport. **Contents of this module** + * Creation of DB tables * Download and preprocessing of vehicle registration data from BAST * Calculation of hydrogen demand based on a Voronoi distribution of counted truck traffic among NUTS 3 regions. -* Write results to DB -* Map demand to H2 buses and write to DB - -**Configuration** - -The config of this dataset can be found in *datasets.yml* in section -*mobility_hgv*. - -**Scenarios and variations** - -Assumptions can be changed within the *datasets.yml*. - -In the context of the eGon project, it is assumed that e-trucks will be -completely hydrogen-powered and in both scenarios the hydrogen consumption is -assumed to be 6.68 kgH2 per 100 km with an additional -[supply chain leakage rate of 0.5 %]( -https://www.energy.gov/eere/fuelcells/doe-technical-targets-hydrogen-delivery). - -### Scenario NEP C 2035 +* Writing results to DB +* Mapping demand to H2 buses and writing to DB -The ramp-up figures are taken from -[Scenario C 2035 Grid Development Plan 2021-2035]( -https://www.netzentwicklungsplan.de/sites/default/files/paragraphs-files/ -NEP_2035_V2021_2_Entwurf_Teil1.pdf). According to this, 100,000 e-trucks are -expected in Germany in 2035, each covering an average of 100,000 km per year. -In total this means 10 Billion km. - -### Scenario eGon100RE - -In the case of the eGon100RE scenario it is assumed that the HGV traffic is -completely hydrogen-powered. The total freight traffic with 40 Billion km is -taken from the -[BMWk Langfristszenarien GHG-emission free scenarios (SNF > 12 t zGG)]( -https://www.langfristszenarien.de/enertile-explorer-wAssets/docs/ -LFS3_Langbericht_Verkehr_final.pdf#page=17). - -## Methodology - -Using a Voronoi interpolation, the censuses of the BASt data is distributed -according to the area fractions of the Voronoi fields within each mv grid or -any other geometries like NUTS-3. """ from pathlib import Path import csv @@ -77,6 +38,11 @@ def create_tables(): + """ + Drops existing :py:class:`demand.egon_heavy_duty_transport_voronoi ` is extended + table and creates new one. + + """ engine = db.engine() EgonHeavyDutyTransportVoronoi.__table__.drop(bind=engine, checkfirst=True) EgonHeavyDutyTransportVoronoi.__table__.create( @@ -87,6 +53,13 @@ def create_tables(): def download_hgv_data(): + """ + Downloads BAST data. + + The data is downloaded to file specified in *datasets.yml* in section + *mobility_hgv/original_data/sources/BAST/file*. + + """ sources = DATASET_CFG["original_data"]["sources"] # Create the folder, if it does not exist @@ -106,10 +79,71 @@ def download_hgv_data(): class HeavyDutyTransport(Dataset): + """ + Class for preparation of static and timeseries data for heavy duty transport. + + *Dependencies* + * :py:class:`Vg250 ` + * :py:class:`EtragoSetup ` + * :py:class:`GasAreaseGon2035 ` + + *Resulting tables* + * :py:class:`demand.egon_heavy_duty_transport_voronoi + ` + is created and filled + * :py:class:`grid.egon_etrago_load` + is extended + * :py:class:`grid.egon_etrago_load_timeseries + ` is extended + + *Configuration* + + The config of this dataset can be found in *datasets.yml* in section + *mobility_hgv*. + + *Scenarios and variations* + + Assumptions can be changed within the *datasets.yml*. + + In the context of the eGon project, it is assumed that e-trucks will be + completely hydrogen-powered and in both scenarios the hydrogen consumption is + assumed to be 6.68 kg H2 per 100 km with an additional + `supply chain leakage rate of 0.5 % + `_. + + ### Scenario NEP C 2035 + + The ramp-up figures are taken from + `Scenario C 2035 Grid Development Plan 2021-2035 + `_. According to this, 100,000 e-trucks are + expected in Germany in 2035, each covering an average of 100,000 km per year. + In total this means 10 billion km. + + ### Scenario eGon100RE + + In the case of the eGon100RE scenario it is assumed that the HGV traffic is + completely hydrogen-powered. The total freight traffic with 40 Billion km is + taken from the + `BMWK Langfristszenarien GHG-emission free scenarios (SNF > 12 t zGG) + `_. + + *Methodology* + + Using a Voronoi interpolation, the censuses of the BASt data is distributed + according to the area fractions of the Voronoi fields within each mv grid or + any other geometries like NUTS-3. + + """ + #: + name: str = "HeavyDutyTransport" + #: + version: str = "0.0.2" def __init__(self, dependencies): super().__init__( - name="HeavyDutyTransport", - version="0.0.2", + name=self.name, + version=self.version, dependencies=dependencies, tasks=( { diff --git a/src/egon/data/datasets/emobility/heavy_duty_transport/create_h2_buses.py b/src/egon/data/datasets/emobility/heavy_duty_transport/create_h2_buses.py index 7611db043..c9d0eef75 100644 --- a/src/egon/data/datasets/emobility/heavy_duty_transport/create_h2_buses.py +++ b/src/egon/data/datasets/emobility/heavy_duty_transport/create_h2_buses.py @@ -1,5 +1,5 @@ """ -Map demand to H2 buses and write to DB +Map demand to H2 buses and write to DB. """ from __future__ import annotations @@ -23,7 +23,7 @@ def insert_hgv_h2_demand(): """ - Insert list of hgv H2 demand (one per NUTS3) in database + Insert list of hgv H2 demand (one per NUTS3) in database. """ for scenario in SCENARIOS: delete_old_entries(scenario) @@ -73,10 +73,12 @@ def kg_per_year_to_mega_watt(df: pd.DataFrame | gpd.GeoDataFrame): def insert_new_entries(hgv_h2_demand_gdf: gpd.GeoDataFrame): """ Insert loads. + Parameters ---------- hgv_h2_demand_gdf : geopandas.GeoDataFrame Load data to insert. + """ new_id = db.next_etrago_id("load") hgv_h2_demand_gdf["load_id"] = range( @@ -111,10 +113,12 @@ def insert_new_entries(hgv_h2_demand_gdf: gpd.GeoDataFrame): def delete_old_entries(scenario: str): """ Delete loads and load timeseries. + Parameters ---------- scenario : str Name of the scenario. + """ # Clean tables db.execute_sql( diff --git a/src/egon/data/datasets/emobility/heavy_duty_transport/data_io.py b/src/egon/data/datasets/emobility/heavy_duty_transport/data_io.py index 2f9bec715..386b17d8c 100644 --- a/src/egon/data/datasets/emobility/heavy_duty_transport/data_io.py +++ b/src/egon/data/datasets/emobility/heavy_duty_transport/data_io.py @@ -1,5 +1,5 @@ """ -Read data from DB and downloads +Read data from DB and download. """ from pathlib import Path diff --git a/src/egon/data/datasets/emobility/heavy_duty_transport/db_classes.py b/src/egon/data/datasets/emobility/heavy_duty_transport/db_classes.py index feae149fa..90b6df823 100644 --- a/src/egon/data/datasets/emobility/heavy_duty_transport/db_classes.py +++ b/src/egon/data/datasets/emobility/heavy_duty_transport/db_classes.py @@ -1,5 +1,5 @@ """ -DB tables / SQLAlchemy ORM classes for heavy duty transport +DB tables / SQLAlchemy ORM classes for heavy duty transport. """ from geoalchemy2 import Geometry @@ -14,7 +14,9 @@ class EgonHeavyDutyTransportVoronoi(Base): - + """ + Class definition of table demand.egon_heavy_duty_transport_voronoi. + """ __tablename__ = "egon_heavy_duty_transport_voronoi" __table_args__ = {"schema": "demand"} diff --git a/src/egon/data/datasets/emobility/heavy_duty_transport/h2_demand_distribution.py b/src/egon/data/datasets/emobility/heavy_duty_transport/h2_demand_distribution.py index daa8cb08b..b2511dd74 100644 --- a/src/egon/data/datasets/emobility/heavy_duty_transport/h2_demand_distribution.py +++ b/src/egon/data/datasets/emobility/heavy_duty_transport/h2_demand_distribution.py @@ -1,6 +1,6 @@ """ -Calculation of hydrogen demand based on a Voronoi distribution of counted truck traffic -among NUTS 3 regions +Calculation of hydrogen demand based on a Voronoi partition of counted truck traffic +used to allocate it to NUTS3 regions and finally aggregate it on NUTS3 level. """ from geovoronoi import points_to_coords, voronoi_regions_from_coords from loguru import logger @@ -60,7 +60,7 @@ def run_egon_truck(): def calculate_total_hydrogen_consumption(scenario: str = "eGon2035"): - """Calculate the total hydrogen demand for trucking in Germany""" + """Calculate the total hydrogen demand for trucking in Germany.""" constants = DATASET_CFG["constants"] hgv_mileage = DATASET_CFG["hgv_mileage"] @@ -133,7 +133,7 @@ def voronoi( points: gpd.GeoDataFrame, boundary: gpd.GeoDataFrame, ): - """Building a Voronoi Field from points and a boundary""" + """Building a Voronoi Field from points and a boundary.""" logger.info("Building Voronoi Field.") sources = DATASET_CFG["original_data"]["sources"] diff --git a/src/egon/data/datasets/emobility/motorized_individual_travel/__init__.py b/src/egon/data/datasets/emobility/motorized_individual_travel/__init__.py index 5e4820b96..cc5fca4e7 100644 --- a/src/egon/data/datasets/emobility/motorized_individual_travel/__init__.py +++ b/src/egon/data/datasets/emobility/motorized_individual_travel/__init__.py @@ -1,74 +1,13 @@ """ -Motorized Individual Travel (MIT) - Main module for preparation of model data (static and timeseries) for -motorized individual travel. +motorized individual travel (MIT). **Contents of this module** -* Creation of DB tables -* Download and preprocessing of vehicle registration data from KBA and BMVI -* Calculate number of electric vehicles and allocate on different spatial - levels. See :py:mod:`egon.data.metadata` -* Extract and write pre-generated trips to DB - -**Configuration** - -The config of this dataset can be found in *datasets.yml* in section -*emobility_mit*. - -**Scenarios and variations** - -* Scenario overview -* Change scenario variation for 2050: adjust in -emobility_mit->scenario->variation->eGon100RE - -**Trip data** - -The electric vehicles' trip data for each scenario have been generated using -`simBEV `_. The methodical background -is given in its `documentation `_. - -6 different vehicle types are used: -* Battery Electric Vehicle (BEV): mini, medium, luxury -* Plug-in Hybrid Electric Vehicle (PHEV): mini, medium, luxury - -.. csv-table:: EV types - :header: "Tecnnology", "Size", "Max. charging capacity slow [kW]", - "Max. charging capacity fast [kW]", "Battery capacity [kWh]", - "Energy consumption [kWh/km]" - :widths: 10, 10, 30, 30, 25, 30 - - "BEV", "mini", 11, 120, 60, 0.1397 - "BEV", "medium", 22, 350, 90, 0.1746 - "BEV", "luxury", 50, 350, 110, 0.2096 - "PHEV", "mini", 3.7, 40, 14, 0.1425 - "PHEV", "medium", 11, 40, 20, 0.1782 - "PHEV", "luxury", 11, 120, 30, 0.2138 - -The complete tech data and assumptions of the run can be found in the metadata: -*/data_bundle_egon_data/emobility/mit_trip_data// -metadata_simbev_run.json*.efficiency_fixed - -* explain scenario parameters - -* run params (all in meta file?) - -**EV allocation** - -The EVs per registration district (Zulassungsbezirk) is taken from KBA's -vehicle registration data. The numbers per EV type (BEV and PHEV) - -* RegioStaR7 -* scenario parameters: shares - -**Further notes** - -* Sanity checks - -**Model paametrization** - -**Example queries** - + * Creation of DB tables + * Download and preprocessing of vehicle registration data from KBA and BMVI + * Calculate number of electric vehicles and allocate on different spatial + levels. + * Extract and write pre-generated trips to DB """ @@ -398,6 +337,68 @@ def write_metadata_to_db(): class MotorizedIndividualTravel(Dataset): + """ + Class to set up static and timeseries data for motorized individual travel (MIT). + + For more information see data documentation on :ref:`mobility-demand-mit-ref`. + + *Dependencies* + * :py:class:`DataBundle ` + * :py:class:`MvGridDistricts + ` + * :py:class:`ScenarioParameters + ` + * :py:class:`EtragoSetup ` + * :py:class:`ZensusMvGridDistricts + ` + * :py:class:`ZensusVg250 ` + * :py:class:`StorageEtrago ` + * :py:class:`HtsEtragoTable + ` + * :py:class:`ChpEtrago ` + * :py:class:`DsmPotential ` + * :py:class:`HeatEtrago ` + * :py:class:`Egon_etrago_gen ` + * :py:class:`OpenCycleGasTurbineEtrago + ` + * :py:class:`HydrogenStoreEtrago + ` + * :py:class:`HydrogenPowerLinkEtrago + ` + * :py:class:`HydrogenMethaneLinkEtrago + ` + * :py:class:`GasAreaseGon100RE ` + * :py:class:`CH4Production ` + * :py:class:`CH4Storages ` + + *Resulting Tables* + * :py:class:`EgonEvPool ` + is created and filled + * :py:class:`EgonEvTrip ` + is created and filled + * :py:class:`EgonEvCountRegistrationDistrict ` + is created and filled + * :py:class:`EgonEvCountMunicipality ` + is created and filled + * :py:class:`EgonEvCountMvGridDistrict ` + is created and filled + * :py:class:`EgonEvMvGridDistrict ` + is created and filled + * :py:class:`EgonEvMetadata ` + is created and filled + + *Configuration* + + The config of this dataset can be found in *datasets.yml* in section + *emobility_mit*. + + """ + + #: + name: str = "MotorizedIndividualTravel" + #: + version: str = "0.0.7" + def __init__(self, dependencies): def generate_model_data_tasks(scenario_name): """Dynamically generate tasks for model data creation. @@ -453,8 +454,8 @@ def generate_model_data_tasks(scenario_name): return tasks super().__init__( - name="MotorizedIndividualTravel", - version="0.0.7", + name=self.name, + version=self.version, dependencies=dependencies, tasks=( create_tables, diff --git a/src/egon/data/datasets/emobility/motorized_individual_travel/db_classes.py b/src/egon/data/datasets/emobility/motorized_individual_travel/db_classes.py index 5f9bff10b..889f2271c 100644 --- a/src/egon/data/datasets/emobility/motorized_individual_travel/db_classes.py +++ b/src/egon/data/datasets/emobility/motorized_individual_travel/db_classes.py @@ -40,7 +40,8 @@ class EgonEvPool(Base): - """Motorized individual travel: EV pool + """ + Class definition of table demand.egon_ev_pool. Each row is one EV, uniquely defined by either (`ev_id`) or (`rs7_id`, `type`, `simbev_id`). @@ -81,7 +82,8 @@ class EgonEvPool(Base): class EgonEvTrip(Base): - """Motorized individual travel: EVs' trips + """ + Class definition of table demand.egon_ev_trip. Each row is one event of a specific electric vehicle which is uniquely defined by `rs7_id`, `ev_id` and `event_id`. @@ -181,7 +183,12 @@ class EgonEvTrip(Base): class EgonEvCountRegistrationDistrict(Base): - """Electric vehicle counts per registration district""" + """ + Class definition of table demand.egon_ev_count_registration_district. + + Contains electric vehicle counts per registration district. + + """ __tablename__ = "egon_ev_count_registration_district" __table_args__ = {"schema": "demand"} @@ -199,7 +206,12 @@ class EgonEvCountRegistrationDistrict(Base): class EgonEvCountMunicipality(Base): - """Electric vehicle counts per municipality""" + """ + Class definition of table demand.egon_ev_count_municipality. + + Contains electric vehicle counts per municipality. + + """ __tablename__ = "egon_ev_count_municipality" __table_args__ = {"schema": "demand"} @@ -217,7 +229,12 @@ class EgonEvCountMunicipality(Base): class EgonEvCountMvGridDistrict(Base): - """Electric vehicle counts per MV grid district""" + """ + Class definition of table demand.egon_ev_count_mv_grid_district. + + Contains electric vehicle counts per MV grid district. + + """ __tablename__ = "egon_ev_count_mv_grid_district" __table_args__ = {"schema": "demand"} @@ -237,7 +254,12 @@ class EgonEvCountMvGridDistrict(Base): class EgonEvMvGridDistrict(Base): - """List of electric vehicles per MV grid district""" + """ + Class definition of table demand.egon_ev_mv_grid_district. + + Contains list of electric vehicles per MV grid district. + + """ __tablename__ = "egon_ev_mv_grid_district" __table_args__ = {"schema": "demand"} @@ -253,7 +275,12 @@ class EgonEvMvGridDistrict(Base): class EgonEvMetadata(Base): - """List of EV Pool Metadata""" + """ + Class definition of table demand.egon_ev_metadata. + + Contains EV Pool Metadata. + + """ __tablename__ = "egon_ev_metadata" __table_args__ = {"schema": "demand"} diff --git a/src/egon/data/datasets/emobility/motorized_individual_travel/ev_allocation.py b/src/egon/data/datasets/emobility/motorized_individual_travel/ev_allocation.py index a64ed1c50..61c2ecfb4 100644 --- a/src/egon/data/datasets/emobility/motorized_individual_travel/ev_allocation.py +++ b/src/egon/data/datasets/emobility/motorized_individual_travel/ev_allocation.py @@ -1,8 +1,9 @@ """ * Calculate number of electric vehicles and allocate on different spatial -levels: :func:`allocate_evs_numbers` + levels: :py:func:`allocate_evs_numbers` * Allocate specific EVs to MV grid districts: -:func:`allocate_evs_to_grid_districts` + :py:func:`allocate_evs_to_grid_districts` + """ from itertools import permutations @@ -47,9 +48,9 @@ def fix_missing_ags_municipality_regiostar(muns, rs7_data): """Check if all AGS of municipality dataset are included in RegioStaR7 dataset and vice versa. - As of Dec 2021, some municipalities are not included int he RegioStaR7 + As of Dec 2021, some municipalities are not included int the RegioStaR7 dataset. This is mostly caused by incorporations of a municipality by - another municipality. This is fixed by assining a RS7 id from another + another municipality. This is fixed by assigning a RS7 id from another municipality with similar AGS (most likely a neighboured one). Missing entries in the municipality dataset is printed but not fixed diff --git a/src/egon/data/datasets/emobility/motorized_individual_travel_charging_infrastructure/__init__.py b/src/egon/data/datasets/emobility/motorized_individual_travel_charging_infrastructure/__init__.py index a6e7c019c..ae8532b82 100644 --- a/src/egon/data/datasets/emobility/motorized_individual_travel_charging_infrastructure/__init__.py +++ b/src/egon/data/datasets/emobility/motorized_individual_travel_charging_infrastructure/__init__.py @@ -1,29 +1,9 @@ """ Motorized Individual Travel (MIT) Charging Infrastructure -Main module for preparation of static model data for cahrging infrastructure -for motorized individual travel. +Main module for preparation of static model data for charging infrastructure for +motorized individual travel. -**Contents of this module** -* Creation of DB tables -* Download and preprocessing of vehicle registration data from zenodo -* Determination of all potential charging locations for the four charging use - cases home, work, public and hpc per mv grid district -* Write results to DB - -**Configuration** - -The config of this dataset can be found in *datasets.yml* in section -*charging_infrastructure*. - -**Charging Infrastructure** - -The charging infrastructure allocation is based on [TracBEV[( -https://github.com/rl-institut/tracbev). TracBEV is a tool for the regional -allocation of charging infrastructure. In practice this allows users to use -results generated via [SimBEV](https://github.com/rl-institut/simbev) and place -the corresponding charging points on a map. These are split into the four use -cases home, work, public and hpc. """ from __future__ import annotations @@ -116,10 +96,52 @@ def get_tracbev_data() -> None: class MITChargingInfrastructure(Dataset): + """ + Preparation of static model data for charging infrastructure for + motorized individual travel. + + The following is done: + + * Creation of DB tables + * Download and preprocessing of vehicle registration data from zenodo + * Determination of all potential charging locations for the four charging use cases + home, work, public and hpc per MV grid district + * Write results to DB + + *Dependencies* + * :py:class:`MvGridDistricts ` + * :py:func:`map_houseprofiles_to_buildings ` + + *Resulting tables* + * :py:class:`grid.egon_emob_charging_infrastructure + ` + is created and filled + + *Configuration* + + The config of this dataset can be found in *datasets.yml* in section + *charging_infrastructure*. + + *Charging Infrastructure* + + The charging infrastructure allocation is based on + `TracBEV `_. TracBEV is a tool for the + regional allocation of charging infrastructure. In practice this allows users to + use results generated via `SimBEV `_ and + place the corresponding charging + points on a map. These are split into the four use cases home, work, public and hpc. + + """ + + #: + name: str = "MITChargingInfrastructure" + #: + version: str = "0.0.1" + def __init__(self, dependencies): super().__init__( - name="MITChargingInfrastructure", - version="0.0.2", + name=self.name, + version=self.version, dependencies=dependencies, tasks=( { diff --git a/src/egon/data/datasets/emobility/motorized_individual_travel_charging_infrastructure/db_classes.py b/src/egon/data/datasets/emobility/motorized_individual_travel_charging_infrastructure/db_classes.py index a6e368fe4..5699627a6 100644 --- a/src/egon/data/datasets/emobility/motorized_individual_travel_charging_infrastructure/db_classes.py +++ b/src/egon/data/datasets/emobility/motorized_individual_travel_charging_infrastructure/db_classes.py @@ -25,6 +25,9 @@ class EgonEmobChargingInfrastructure(Base): + """ + Class definition of table grid.egon_emob_charging_infrastructure. + """ __tablename__ = DATASET_CFG["targets"]["charging_infrastructure"]["table"] __table_args__ = { diff --git a/src/egon/data/datasets/era5.py b/src/egon/data/datasets/era5.py index 4a0391ab8..b32061373 100644 --- a/src/egon/data/datasets/era5.py +++ b/src/egon/data/datasets/era5.py @@ -20,10 +20,34 @@ class WeatherData(Dataset): + """ + Download weather data from ERA5 using atlite + + This dataset downloads weather data for the selected representative weather year. + This is done by applying functions from the atlite-tool.The downloaded wetaher data is stored into + files within the subfolder 'cutouts'. + + + *Dependencies* + * :py:class:`ScenarioParameters ` + * :py:class:`Vg250 ` + * :py:func:`Setup ` + + *Resulting tables* + * :py:class:`supply.egon_era5_weather_cells ` is created and filled + * :py:class:`supply.egon_era5_renewable_feedin ` is created + + """ + + #: + name: str = "Era5" + #: + version: str = "0.0.2" + def __init__(self, dependencies): super().__init__( - name="Era5", - version="0.0.2", + name=self.name, + version=self.version, dependencies=dependencies, tasks=({create_tables, download_era5}, insert_weather_cells), ) @@ -47,7 +71,6 @@ class EgonRenewableFeedIn(Base): def create_tables(): - db.execute_sql("CREATE SCHEMA IF NOT EXISTS supply;") engine = db.engine() db.execute_sql( @@ -134,25 +157,21 @@ def download_era5(): ) if not os.path.exists(directory): - os.mkdir(directory) cutout = import_cutout() if not cutout.prepared: - cutout.prepare() cutout = import_cutout("Germany") if not cutout.prepared: - cutout.prepare() cutout = import_cutout("Germany-offshore") if not cutout.prepared: - cutout.prepare() diff --git a/src/egon/data/datasets/etrago_setup.py b/src/egon/data/datasets/etrago_setup.py index 92bb70441..d3f4975cc 100755 --- a/src/egon/data/datasets/etrago_setup.py +++ b/src/egon/data/datasets/etrago_setup.py @@ -4,7 +4,6 @@ from geoalchemy2.types import Geometry from shapely.geometry import LineString -from sqlalchemy import schema # ??? from sqlalchemy import ( ARRAY, BigInteger, @@ -25,7 +24,13 @@ from egon.data import db from egon.data.datasets import Dataset -from egon.data.metadata import context, contributors, meta_metadata, sources +from egon.data.metadata import ( + context, + contributors, + license_egon_data_odbl, + meta_metadata, + sources +) Base = declarative_base() metadata = Base.metadata @@ -70,7 +75,6 @@ def get_meta( component, description="TODO", source_list=[], - license_list=[], contributor_list=[], timeseries=False, ): @@ -102,7 +106,7 @@ def get_meta( "resolution": None, }, "sources": source_list, - "licenses": license_list, + "licenses": [license_egon_data_odbl()], "contributors": contributor_list, "resources": [ { @@ -158,7 +162,6 @@ class EgonPfHvBus(Base): contributor_list[3]["comment"] = "Added CH4 sector buses" contributor_list[4]["comment"] = "Added H2 sector buses" - license_list = [data["licenses"][0] for data in source_list] __tablename__ = "egon_etrago_bus" __table_args__ = { @@ -167,7 +170,6 @@ class EgonPfHvBus(Base): "grid", "Bus", source_list=source_list, - license_list=license_list, contributor_list=contributor_list, ), } @@ -195,8 +197,6 @@ class EgonPfHvBusTimeseries(Base): contributor_list = contributors(["cb"]) contributor_list[0]["comment"] = "Added metadata" - license_list = [data["licenses"][0] for data in source_list] - __tablename__ = "egon_etrago_bus_timeseries" __table_args__ = { "schema": "grid", @@ -204,7 +204,6 @@ class EgonPfHvBusTimeseries(Base): "grid", "Bus", source_list=source_list, - license_list=license_list, contributor_list=contributor_list, timeseries=True, ), @@ -236,7 +235,6 @@ class EgonPfHvGenerator(Base): contributor_list[3]["comment"] = "Added gas feedin generators" contributor_list[4]["comment"] = "Added pv ground mounted" - license_list = [data["licenses"][0] for data in source_list] __tablename__ = "egon_etrago_generator" __table_args__ = { @@ -245,7 +243,6 @@ class EgonPfHvGenerator(Base): "grid", "Generator", source_list=source_list, - license_list=license_list, contributor_list=contributor_list, ), } @@ -298,7 +295,6 @@ class EgonPfHvGeneratorTimeseries(Base): "comment" ] = "Added p_max_pu timeseries for pv and wind" - license_list = [data["licenses"][0] for data in source_list] __tablename__ = "egon_etrago_generator_timeseries" __table_args__ = { @@ -307,7 +303,6 @@ class EgonPfHvGeneratorTimeseries(Base): "grid", "Generator", source_list=source_list, - license_list=license_list, contributor_list=contributor_list, timeseries=True, ), @@ -334,8 +329,6 @@ class EgonPfHvLine(Base): contributor_list[0]["comment"] = "Added lines from osmTGmod tables" contributor_list[1]["comment"] = "Added meta data" - license_list = [data["licenses"][0] for data in source_list] - __tablename__ = "egon_etrago_line" __table_args__ = { "schema": "grid", @@ -343,7 +336,6 @@ class EgonPfHvLine(Base): "grid", "Line", source_list=source_list, - license_list=license_list, contributor_list=contributor_list, ), } @@ -389,8 +381,6 @@ class EgonPfHvLineTimeseries(Base): contributor_list[0]["comment"] = "Added s_max_pu timeseries" contributor_list[1]["comment"] = "Added meta data" - license_list = [data["licenses"][0] for data in source_list] - __tablename__ = "egon_etrago_line_timeseries" __table_args__ = { "schema": "grid", @@ -398,7 +388,6 @@ class EgonPfHvLineTimeseries(Base): "grid", "Line", source_list=source_list, - license_list=license_list, contributor_list=contributor_list, timeseries=True, ), @@ -435,7 +424,6 @@ class EgonPfHvLink(Base): contributor_list[4]["comment"] = "Added H2 related links" contributor_list[5]["comment"] = "Added CH4 links" - license_list = [data["licenses"][0] for data in source_list] __tablename__ = "egon_etrago_link" __table_args__ = { @@ -444,7 +432,6 @@ class EgonPfHvLink(Base): "grid", "Link", source_list=source_list, - license_list=license_list, contributor_list=contributor_list, ), } @@ -494,8 +481,6 @@ class EgonPfHvLinkTimeseries(Base): contributor_list[1]["comment"] = "Added dsm link timeseries" contributor_list[2]["comment"] = "Added e mobility link timeseries" - license_list = [data["licenses"][0] for data in source_list] - __tablename__ = "egon_etrago_link_timeseries" __table_args__ = { "schema": "grid", @@ -503,7 +488,6 @@ class EgonPfHvLinkTimeseries(Base): "grid", "Link", source_list=source_list, - license_list=license_list, contributor_list=contributor_list, timeseries=True, ), @@ -539,7 +523,6 @@ class EgonPfHvLoad(Base): contributor_list[2]["comment"] = "Added gas demands" contributor_list[3]["comment"] = "Added mobility demands" - license_list = [data["licenses"][0] for data in source_list] __tablename__ = "egon_etrago_load" __table_args__ = { @@ -548,7 +531,6 @@ class EgonPfHvLoad(Base): "grid", "Load", source_list=source_list, - license_list=license_list, contributor_list=contributor_list, ), } @@ -583,7 +565,6 @@ class EgonPfHvLoadTimeseries(Base): contributor_list[2]["comment"] = "Added e mobility load timeseries" contributor_list[3]["comment"] = "Added gas load timeseries" - license_list = [data["licenses"][0] for data in source_list] __tablename__ = "egon_etrago_load_timeseries" __table_args__ = { @@ -592,7 +573,6 @@ class EgonPfHvLoadTimeseries(Base): "grid", "Load", source_list=source_list, - license_list=license_list, contributor_list=contributor_list, timeseries=True, ), @@ -613,7 +593,6 @@ class EgonPfHvCarrier(Base): contributor_list = contributors(["fw"]) contributor_list[0]["comment"] = "Added list of carriers" - license_list = [data["licenses"][0] for data in source_list] __tablename__ = "egon_etrago_carrier" __table_args__ = { @@ -622,7 +601,6 @@ class EgonPfHvCarrier(Base): "grid", "Carrier", source_list=source_list, - license_list=license_list, contributor_list=contributor_list, ), } @@ -648,7 +626,6 @@ class EgonPfHvStorage(Base): "comment" ] = "Added battery and pumped hydro storage units" - license_list = [data["licenses"][0] for data in source_list] __tablename__ = "egon_etrago_storage" __table_args__ = { @@ -657,7 +634,6 @@ class EgonPfHvStorage(Base): "grid", "Storage", source_list=source_list, - license_list=license_list, contributor_list=contributor_list, ), } @@ -699,7 +675,6 @@ class EgonPfHvStorageTimeseries(Base): contributor_list = contributors(["cb"]) contributor_list[0]["comment"] = "Added metadata" - license_list = [data["licenses"][0] for data in source_list] __tablename__ = "egon_etrago_storage_timeseries" __table_args__ = { @@ -708,7 +683,6 @@ class EgonPfHvStorageTimeseries(Base): "grid", "Storage", source_list=source_list, - license_list=license_list, contributor_list=contributor_list, timeseries=True, ), @@ -749,7 +723,6 @@ class EgonPfHvStore(Base): contributor_list[3]["comment"] = "Add heat storage" contributor_list[4]["comment"] = "Add e-mobility storage" - license_list = [data["licenses"][0] for data in source_list] __tablename__ = "egon_etrago_store" __table_args__ = { "schema": "grid", @@ -757,7 +730,6 @@ class EgonPfHvStore(Base): "grid", "Store", source_list=source_list, - license_list=license_list, contributor_list=contributor_list, ), } @@ -800,7 +772,6 @@ class EgonPfHvStoreTimeseries(Base): contributor_list[0]["comment"] = "Add DSM storage" contributor_list[1]["comment"] = "Add e-mobility storage" - license_list = [data["licenses"][0] for data in source_list] __tablename__ = "egon_etrago_store_timeseries" __table_args__ = { "schema": "grid", @@ -808,7 +779,6 @@ class EgonPfHvStoreTimeseries(Base): "grid", "Store", source_list=source_list, - license_list=license_list, contributor_list=contributor_list, timeseries=True, ), @@ -844,7 +814,6 @@ class EgonPfHvTransformer(Base): contributor_list[0]["comment"] = "Added transformes from osmTGmod tables" contributor_list[1]["comment"] = "Added meta data" - license_list = [data["licenses"][0] for data in source_list] __tablename__ = "egon_etrago_transformer" __table_args__ = { @@ -853,7 +822,6 @@ class EgonPfHvTransformer(Base): "grid", "Transformer", source_list=source_list, - license_list=license_list, contributor_list=contributor_list, ), } @@ -895,8 +863,6 @@ class EgonPfHvTransformerTimeseries(Base): contributor_list = contributors(["cb"]) contributor_list[0]["comment"] = "Added meta data" - license_list = [data["licenses"][0] for data in source_list] - __tablename__ = "egon_etrago_transformer_timeseries" __table_args__ = { "schema": "grid", @@ -904,7 +870,6 @@ class EgonPfHvTransformerTimeseries(Base): "grid", "Transformer", source_list=source_list, - license_list=license_list, contributor_list=contributor_list, timeseries=True, ), diff --git a/src/egon/data/datasets/fill_etrago_gen.py b/src/egon/data/datasets/fill_etrago_gen.py index c9d8ea68b..cae8fff4f 100644 --- a/src/egon/data/datasets/fill_etrago_gen.py +++ b/src/egon/data/datasets/fill_etrago_gen.py @@ -9,10 +9,32 @@ class Egon_etrago_gen(Dataset): + """ + Group generators based on Scenario, carrier and bus. Marginal costs are + assigned to generators without this data. Grouped generators + are sent to the egon_etrago_generator table and a timeseries is assigned + to the weather dependent ones. + + *Dependencies* + * :py:class:`PowerPlants ` + * :py:class:`WeatherData ` + + *Resulting tables* + * :py:class:`grid.egon_etrago_generator + ` is extended + * :py:class:`grid.egon_etrago_generator_timeseries + ` is filled + + """ + #: + name: str = "etrago_generators" + #: + version: str = "0.0.8" + def __init__(self, dependencies): super().__init__( - name="etrago_generators", - version="0.0.8", + name=self.name, + version=self.version, dependencies=dependencies, tasks=(fill_etrago_generators,), ) @@ -65,7 +87,6 @@ def fill_etrago_generators(): def group_power_plants(power_plants, renew_feedin, etrago_gen_orig, cfg): - # group power plants by bus and carrier agg_func = { @@ -89,7 +110,6 @@ def group_power_plants(power_plants, renew_feedin, etrago_gen_orig, cfg): def add_marginal_costs(power_plants): - scenarios = power_plants.scenario.unique() pp = pd.DataFrame() @@ -124,7 +144,6 @@ def add_marginal_costs(power_plants): def fill_etrago_gen_table(etrago_pp2, etrago_gen_orig, cfg, con): - etrago_pp = etrago_pp2[ ["carrier", "el_capacity", "bus_id", "scenario", "marginal_cost"] ] @@ -243,7 +262,6 @@ def power_timeser(weather_data): def adjust_renew_feedin_table(renew_feedin, cfg): - # Define carrier 'pv' as 'solar' carrier_pv_mask = renew_feedin["carrier"] == "pv" renew_feedin.loc[carrier_pv_mask, "carrier"] = "solar" diff --git a/src/egon/data/datasets/fix_ehv_subnetworks.py b/src/egon/data/datasets/fix_ehv_subnetworks.py index eca901938..96d780ef6 100644 --- a/src/egon/data/datasets/fix_ehv_subnetworks.py +++ b/src/egon/data/datasets/fix_ehv_subnetworks.py @@ -11,17 +11,41 @@ class FixEhvSubnetworks(Dataset): + """ + Manually fix grid topology in the extra high voltage grid to avoid subnetworks + + This dataset includes fixes for the topology of the German extra high voltage grid. + The initial grid topology from openstreetmap resp. osmTGmod includes some issues, eg. because of + incomplete data. Thsi dataset does not fix all those issues, but deals only with subnetworks + in the extra high voltage grid that would result into problems in the grid optimisation. + + + *Dependencies* + * :py:class:`Osmtgmod ` + + + *Resulting tables* + * :py:class:`grid.egon_etrago_bus ` is updated + * :py:class:`grid.egon_etrago_line ` is updated + * :py:class:`grid.egon_etrago_transformer ` is updated + + """ + + #: + name: str = "FixEhvSubnetworks" + #: + version: str = "0.0.2" + def __init__(self, dependencies): super().__init__( - name="FixEhvSubnetworks", - version="0.0.2", + name=self.name, + version=self.version, dependencies=dependencies, tasks=run, ) def select_bus_id(x, y, v_nom, scn_name, carrier): - bus_id = db.select_dataframe( f""" SELECT bus_id @@ -135,7 +159,6 @@ def drop_line(x0, y0, x1, y1, v_nom, scn_name): def add_trafo(x, y, v_nom0, v_nom1, scn_name, n=1): - bus0 = select_bus_id(x, y, v_nom0, scn_name, carrier="AC") bus1 = select_bus_id(x, y, v_nom1, scn_name, carrier="AC") @@ -169,7 +192,6 @@ def add_trafo(x, y, v_nom0, v_nom1, scn_name, n=1): def drop_trafo(x, y, v_nom0, v_nom1, scn_name): - bus0 = select_bus_id(x, y, v_nom0, scn_name, carrier="AC") bus1 = select_bus_id(x, y, v_nom1, scn_name, carrier="AC") @@ -186,7 +208,6 @@ def drop_trafo(x, y, v_nom0, v_nom1, scn_name): def fix_subnetworks(scn_name): - # Missing 220kV line to Lübeck Siems # add 220kV bus at substation Lübeck Siems add_bus(10.760835327266625, 53.90974536547805, 220, scn_name) @@ -218,7 +239,6 @@ def fix_subnetworks(scn_name): ) if settings()["egon-data"]["--dataset-boundary"] == "Everything": - # Missing line from USW Uchtelfangen to 'Kraftwerk Weiher' add_line( 7.032657738999395, # Kraftwerk Weiher @@ -368,14 +388,12 @@ def fix_subnetworks(scn_name): drop_trafo(11.121774798935334, 51.00038603925895, 220, 380, scn_name) drop_bus(11.121774798935334, 51.00038603925895, 380, scn_name) - # Umspannwerk Waldlaubersheim # delete isolated bus and trafo drop_trafo(7.815993836091339, 49.92211102637183, 110, 380, scn_name) drop_bus(7.815993836091339, 49.92211102637183, 380, scn_name) - def run(): fix_subnetworks("eGon2035") fix_subnetworks("eGon100RE") diff --git a/src/egon/data/datasets/gas_areas.py b/src/egon/data/datasets/gas_areas.py index 103090ae8..70caf7f8e 100755 --- a/src/egon/data/datasets/gas_areas.py +++ b/src/egon/data/datasets/gas_areas.py @@ -1,7 +1,9 @@ -""" -The central module containing code to create CH4 and H2 voronoi polygons +"""The central module containing code to create CH4 and H2 voronoi polygons """ +import datetime +import json + from geoalchemy2.types import Geometry from sqlalchemy import BigInteger, Column, Text from sqlalchemy.ext.declarative import declarative_base @@ -9,6 +11,13 @@ from egon.data import db from egon.data.datasets import Dataset from egon.data.datasets.generate_voronoi import get_voronoi_geodataframe +from egon.data.metadata import ( + context, + contributors, + license_egon_data_odbl, + meta_metadata, + sources, +) class GasAreaseGon2035(Dataset): @@ -77,8 +86,78 @@ class EgonPfHvGasVoronoi(Base): Class definition of table grid.egon_gas_voronoi """ + + source_list = [ + sources()["openstreetmap"], + sources()["SciGRID_gas"], + sources()["bgr_inspeeds_data_bundle"], + ] + meta = { + "name": "grid.egon_gas_voronoi", + "title": "Gas voronoi areas", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": "H2 and CH4 voronoi cells", + "language": ["en-EN"], + "publicationDate": datetime.date.today().isoformat(), + "context": context(), + "spatial": { + "location": None, + "extent": "Germany", + "resolution": None, + }, + "sources": source_list, + "licenses": [license_egon_data_odbl()], + "contributors": contributors(["fw"]), + "resources": [ + { + "profile": "tabular-data-resource", + "name": "grid.egon_gas_voronoi", + "path": None, + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": [ + { + "name": "scn_name", + "description": "Name of the scenario", + "type": "str", + "unit": None, + }, + { + "name": "bus_id", + "description": "Unique identifier", + "type": "integer", + "unit": None, + }, + { + "name": "carrier", + "description": "Carrier of the voronoi cell", + "type": "str", + "unit": None, + }, + { + "name": "geom", + "description": "Voronoi cell geometry", + "type": "Geometry(Polygon, 4326)", + "unit": None, + }, + ], + "primaryKey": ["scn_name", "bus_id"], + "foreignKeys": [], + }, + "dialect": {"delimiter": None, "decimalSeparator": "."}, + } + ], + "metaMetadata": meta_metadata(), + } + # Create json dump + meta_json = "'" + json.dumps(meta, indent=4, ensure_ascii=False) + "'" + __tablename__ = "egon_gas_voronoi" - __table_args__ = {"schema": "grid"} + __table_args__ = { + "schema": "grid", + "comment": meta_json, + } #: Name of the scenario scn_name = Column(Text, primary_key=True, nullable=False) @@ -92,7 +171,7 @@ class EgonPfHvGasVoronoi(Base): def create_gas_voronoi_table(): """ - Create voronoi gas voronoi table + Create gas voronoi table """ engine = db.engine() EgonPfHvGasVoronoi.__table__.drop(bind=engine, checkfirst=True) diff --git a/src/egon/data/datasets/gas_grid.py b/src/egon/data/datasets/gas_grid.py index 27af06b4f..6fc3c18bd 100755 --- a/src/egon/data/datasets/gas_grid.py +++ b/src/egon/data/datasets/gas_grid.py @@ -1,15 +1,15 @@ # -*- coding: utf-8 -*- """ -The module containing code aiming to insert the methane grid into the database +The module contains code used to insert the methane grid into the database -The central module containing all code dealing with the import of data -from SciGRID_gas (IGGIELGN dataset) and with the insertion fo the CH4 -buses and links into the database for the scenarios eGon2035 and eGon100RE. +The central module contains all code dealing with the import of data +from SciGRID_gas (IGGIELGN dataset) and inserting the CH4 buses and links +into the database for the scenarios eGon2035 and eGon100RE. The SciGRID_gas data downloaded with :py:func:`download_SciGRID_gas_data` -into the folder ./datasets/gas_data/data are also used by other modules. +into the folder ./datasets/gas_data/data is also used by other modules. -In this module, only the IGGIELGN_Nodes and IGGIELGN_PipeSegments cvs files +In this module, only the IGGIELGN_Nodes and IGGIELGN_PipeSegments csv files are used in the function :py:func:`insert_gas_data` that inserts the CH4 buses and links, which for the case of gas represent pipelines, into the database. @@ -75,7 +75,7 @@ def download_SciGRID_gas_data(): """ Download SciGRID_gas IGGIELGN data from Zenodo - The following data for CH4 are downloaded into the folder + The following data for CH4 is downloaded into the folder ./datasets/gas_data/data: * Buses (file IGGIELGN_Nodes.csv), * Pipelines (file IGGIELGN_PipeSegments.csv), @@ -83,7 +83,7 @@ def download_SciGRID_gas_data(): * Storages (file IGGIELGN_Storages.csv), * LNG terminals (file IGGIELGN_LNGs.csv). - For more information on these data refer, to the + For more information on this data refer, to the `SciGRID_gas IGGIELGN documentation `_. Returns @@ -125,9 +125,9 @@ def define_gas_nodes_list(): Define list of CH4 buses from SciGRID_gas IGGIELGN data The CH4 nodes are modelled as buses. Therefore the SciGRID_gas nodes - are red from the IGGIELGN_Nodes cvs file previously downloaded in the + are read from the IGGIELGN_Nodes csv file previously downloaded in the function :py:func:`download_SciGRID_gas_data`, corrected (erroneous country), - and returned as dataframe. + and returned in a dataframe. Returns ------- @@ -294,26 +294,28 @@ def insert_CH4_nodes_list(gas_nodes_list): ) -def insert_gas_buses_abroad(scn_name="eGon2035"): +def define_gas_buses_abroad(scn_name="eGon2035"): """ - Insert CH4 buses in neighbouring countries to database for eGon2035 + Define central CH4 buses in foreign countries for eGon2035 - For the scenario eGon2035, insert central CH4 buses in foreign - countries to the database. The considered foreign countries are the - direct neighbouring countries, with the addition of Russia that is + For the scenario eGon2035, define central CH4 buses in foreign + countries. The considered foreign countries are the direct + neighbouring countries, with the addition of Russia that is considered as a source of fossil CH4. Therefore, the following steps are executed: * Definition of the foreign buses with the function - :py:func:`import_central_buses_egon100 ` from + :py:func:`central_buses_egon100 ` from the module :py:mod:`electrical_neighbours ` * Removal of the superfluous buses in order to have only one bus in each neighbouring country - * Removal of the the irrelevant columns + * Removal of the irrelevant columns * Addition of the missing information: scn_name and carrier * Attribution of an id to each bus - * Cleaning of the database table grid.egon_etrago_bus of the - CH4 buses of the specific scenario (eGon2035) out of Germany - * Insertion of the neighbouring buses in the table grid.egon_etrago_bus. + + Parameters + ---------- + scn_name : str + Name of the scenario Returns ------- @@ -329,15 +331,6 @@ def insert_gas_buses_abroad(scn_name="eGon2035"): "main_gas_carrier" ] - # Connect to local database - engine = db.engine() - db.execute_sql( - f""" - DELETE FROM grid.egon_etrago_bus WHERE "carrier" = '{main_gas_carrier}' AND - scn_name = '{scn_name}' AND country != 'DE'; - """ - ) - # Select the foreign buses gdf_abroad_buses = central_buses_egon100(sources) gdf_abroad_buses = gdf_abroad_buses.drop_duplicates(subset=["country"]) @@ -396,6 +389,46 @@ def insert_gas_buses_abroad(scn_name="eGon2035"): columns={"geometry": "geom"} ).set_geometry("geom", crs=4326) + return gdf_abroad_buses + + +def insert_gas_buses_abroad(scn_name="eGon2035"): + """ + Insert CH4 buses in neighbouring countries into database for eGon2035 + + * Definition of the CH4 buses abroad with the function + :py:func:`define_gas_buses_abroad` + * Cleaning of the database table grid.egon_etrago_bus of the + foreign CH4 buses of the specific scenario (eGon2035) + * Insertion of the neighbouring buses into the table grid.egon_etrago_bus. + + Parameters + ---------- + scn_name : str + Name of the scenario + + Returns + ------- + gdf_abroad_buses : dataframe + Dataframe containing the CH4 buses in the neighbouring countries + and one in the center of Germany in test mode + + """ + main_gas_carrier = get_sector_parameters("gas", scenario=scn_name)[ + "main_gas_carrier" + ] + + # Connect to local database + engine = db.engine() + db.execute_sql( + f""" + DELETE FROM grid.egon_etrago_bus WHERE "carrier" = '{main_gas_carrier}' AND + scn_name = '{scn_name}' AND country != 'DE'; + """ + ) + + gdf_abroad_buses = define_gas_buses_abroad(scn_name) + # Insert to db print(gdf_abroad_buses) gdf_abroad_buses.to_postgis( @@ -409,28 +442,26 @@ def insert_gas_buses_abroad(scn_name="eGon2035"): return gdf_abroad_buses -def insert_gas_pipeline_list( +def define_gas_pipeline_list( gas_nodes_list, abroad_gas_nodes_list, scn_name="eGon2035" ): """ - Insert list of gas pipelines into the database + Define gas pipelines in Germany from SciGRID_gas IGGIELGN data - The gas pipelines, modelled as Pypsa links are red from the IGGIELGN_PipeSegments - csv file previously downloded in the function :py:func:`download_SciGRID_gas_data`, - adapted and inserted in the database for the eGon2035 scenario. - The manual corrections allows to: - * Delete gas pipelines disconnected of the rest of the gas grid - * Connect one pipeline (also connected to Norway) disconnected of - the rest of the gas grid - * Correct erroneous country of some pipelines + The gas pipelines, modelled as PyPSA links are read from the IGGIELGN_PipeSegments + csv file previously downloded in the function :py:func:`download_SciGRID_gas_data`. The capacities of the pipelines are determined by the correspondance - table given by the Parameters for the classification of gas pipelines + table given by the parameters for the classification of gas pipelines in `Electricity, heat, and gas sector data for modeling the German system `_ related to the pipeline diameter given in the SciGRID_gas dataset. - The database is cleaned before the insertion of the pipelines. + The manual corrections allow to: + * Delete gas pipelines disconnected of the rest of the gas grid + * Connect one pipeline (also connected to Norway) disconnected of + the rest of the gas grid + * Correct countries of some erroneous pipelines Parameters ---------- @@ -444,7 +475,8 @@ def insert_gas_pipeline_list( Returns ------- - None + gas_pipelines_list : pandas.DataFrame + Dataframe containing the gas pipelines in Germany """ abroad_gas_nodes_list = abroad_gas_nodes_list.set_index("country") @@ -453,8 +485,6 @@ def insert_gas_pipeline_list( "main_gas_carrier" ] - engine = db.engine() - # Select next id value new_id = db.next_etrago_id("link") @@ -771,8 +801,6 @@ def insert_gas_pipeline_list( "NUTS1_0", "NUTS1_1", "country_code", - "country_0", - "country_1", "diameter", "pipe_class", "classification", @@ -783,6 +811,36 @@ def insert_gas_pipeline_list( ] ) + return gas_pipelines_list + + +def insert_gas_pipeline_list(gas_pipelines_list, scn_name="eGon2035"): + """ + Insert list of gas pipelines into the database + + Receive as argument a list of gas pipelines and insert them into the + database after cleaning it. + + Parameters + ---------- + gas_pipelines_list : pandas.DataFrame + Dataframe containing the gas pipelines in Germany + scn_name : str + Name of the scenario + + Returns + ------- + None + + """ + main_gas_carrier = get_sector_parameters("gas", scenario=scn_name)[ + "main_gas_carrier" + ] + engine = db.engine() + gas_pipelines_list = gas_pipelines_list.drop( + columns=["country_0", "country_1"] + ) + # Clean db db.execute_sql( f"""DELETE FROM grid.egon_etrago_link @@ -828,7 +886,9 @@ def remove_isolated_gas_buses(): """ Delete CH4 buses which are disconnected of the CH4 grid for the eGon2035 scenario - This function deletes directly in the database and has no return. + Returns + ------- + None """ targets = config.datasets()["gas_grid"]["targets"] @@ -853,9 +913,9 @@ def remove_isolated_gas_buses(): def insert_gas_data(): """ - Overall function for importing methane data for eGon2035 + Function for importing methane data for eGon2035 - This function import the methane data (buses and pipelines) for + This function imports the methane data (buses and pipelines) for eGon2035, by executing the following steps: * Download the SciGRID_gas datasets with the function :py:func:`download_SciGRID_gas_data` * Define CH4 buses with the function :py:func:`define_gas_nodes_list` @@ -868,7 +928,9 @@ def insert_gas_data(): * Remove the isolated CH4 buses directly from the database using the function :py:func:`remove_isolated_gas_buses` - This function inserts data into the database and has no return. + Returns + ------- + None """ download_SciGRID_gas_data() @@ -878,24 +940,29 @@ def insert_gas_data(): insert_CH4_nodes_list(gas_nodes_list) abroad_gas_nodes_list = insert_gas_buses_abroad() - insert_gas_pipeline_list(gas_nodes_list, abroad_gas_nodes_list) + gas_pipeline_list = define_gas_pipeline_list( + gas_nodes_list, abroad_gas_nodes_list + ) + insert_gas_pipeline_list(gas_pipeline_list) remove_isolated_gas_buses() def insert_gas_data_eGon100RE(): """ - Overall function for importing methane data for eGon100RE + Function for importing methane data for eGon100RE - This function import the methane data (buses and pipelines) for + This function imports the methane data (buses and pipelines) for eGon100RE, by copying the CH4 buses from the eGon2035 scenario using the function :py:func:`copy_and_modify_buses ` from the module :py:mod:`etrago_helpers `. The methane pipelines are also copied and their capacities are adapted: one share of the methane grid is retroffited into an hydrogen grid, so - the methane pieplines nominal capacities are reduced from this share + the methane pipelines nominal capacities are reduced from this share (calculated in the pyspa-eur-sec run). - This function inserts data into the database and has no return. + Returns + ------- + None """ # copy buses diff --git a/src/egon/data/datasets/gas_neighbours/eGon100RE.py b/src/egon/data/datasets/gas_neighbours/eGon100RE.py index c748ae4be..16d6f8ff2 100755 --- a/src/egon/data/datasets/gas_neighbours/eGon100RE.py +++ b/src/egon/data/datasets/gas_neighbours/eGon100RE.py @@ -1,6 +1,6 @@ -"""Module containing code dealing with crossbording gas pipelines for eGon100RE +"""Module containing code dealing with cross border gas pipelines for eGon100RE -In this module the crossbordering pipelines for H2 and CH4, exclusively +In this module the cross border pipelines for H2 and CH4, exclusively between Germany and its neighbouring countries, in eGon100RE are defined and inserted in the database. @@ -38,23 +38,23 @@ def insert_gas_neigbours_eGon100RE(): - """Insert missing gas crossbordering grid capacities for eGon100RE + """Insert missing gas cross border grid capacities for eGon100RE - This function insert the crossbordering pipelines for H2 and CH4, + This function insert the cross border pipelines for H2 and CH4, exclusively between Germany and its neighbouring countries, for eGon100RE in the database by executing the following steps: - * call of the the function + * call of the function :py:func:`define_DE_crossbording_pipes_geom_eGon100RE`, that - defines the crossbordering pipelines (H2 and CH4) between + defines the cross border pipelines (H2 and CH4) between Germany and its neighbouring countries - * call of the the function + * call of the function :py:func:`read_DE_crossbordering_cap_from_pes`, that calculates - the crossbordering total exchange capactities for H2 and CH4 + the cross border total exchange capactities for H2 and CH4 between Germany and its neighbouring countries based on the pypsa-eur-sec results - * call of the the function + * call of the function :py:func:`calculate_crossbordering_gas_grid_capacities_eGon100RE`, - that attributes to each crossbordering pipeline (H2 and CH4) + that attributes to each cross border pipeline (H2 and CH4) between Germany and its neighbouring countries its capacity * insertion of the H2 and CH4 pipelines between Germany and its neighbouring countries in the database with function @@ -91,12 +91,12 @@ def insert_gas_neigbours_eGon100RE(): def define_DE_crossbording_pipes_geom_eGon100RE(scn_name="eGon100RE"): - """Define the missing crossbordering gas pipelines in eGon100RE + """Define the missing cross border gas pipelines in eGon100RE - This function defines the crossbordering pipelines (for H2 and CH4) + This function defines the cross border pipelines (for H2 and CH4) between Germany and its neighbouring countries. These pipelines are defined as links and there are copied from the corresponding - CH4 crossbering pipelines from eGon2035. + CH4 cross border pipelines from eGon2035. Parameters ---------- @@ -106,7 +106,7 @@ def define_DE_crossbording_pipes_geom_eGon100RE(scn_name="eGon100RE"): Returns ------- gas_pipelines_list_DE : pandas.DataFrame - List of the crossbordering H2 and CH4 pipelines between + List of the cross border H2 and CH4 pipelines between Germany and its neighbouring countries in eGon100RE, with geometry (geom and topo) but no capacity. @@ -192,7 +192,7 @@ def define_DE_crossbording_pipes_geom_eGon100RE(scn_name="eGon100RE"): f""" SELECT * FROM grid.egon_etrago_bus WHERE scn_name = 'eGon100RE' - AND carrier = '{carrier}' + AND carrier = '{carrier_bus_DE}' AND country != 'DE' """, epsg=4326, @@ -262,9 +262,9 @@ def define_DE_crossbording_pipes_geom_eGon100RE(scn_name="eGon100RE"): def read_DE_crossbordering_cap_from_pes(): - """Read gas pipelines crossbordering capacities from pes run + """Read gas pipelines cross border capacities from pes run - This function calculates the crossbordering total exchange + This function calculates the cross border total exchange capactities for H2 and CH4 between Germany and its neighbouring countries based on the pypsa-eur-sec results. @@ -349,9 +349,9 @@ def read_DE_crossbordering_cap_from_pes(): def calculate_crossbordering_gas_grid_capacities_eGon100RE( cap_DE, DE_pipe_capacities_list ): - """Attribute gas crossbordering grid capacities for eGon100RE + """Attribute gas cross border grid capacities for eGon100RE - This function attributes to each crossbordering pipeline (H2 and + This function attributes to each cross border pipeline (H2 and CH4) between Germany and its neighbouring countries its capacity. Parameters @@ -360,14 +360,14 @@ def calculate_crossbordering_gas_grid_capacities_eGon100RE( List of the H2 and CH4 exchange capacity for each neighbouring country of Germany. DE_pipe_capacities_list : pandas.DataFrame - List of the crossbordering for H2 and CH4 pipelines between + List of the cross border for H2 and CH4 pipelines between Germany and its neighbouring countries in eGon100RE, with geometry (geom and topo) but no capacity. Returns ------- Crossbordering_pipe_capacities_list : pandas.DataFrame - List of the crossbordering H2 and CH4 pipelines between + List of the cross border H2 and CH4 pipelines between Germany and its neighbouring countries in eGon100RE. """ diff --git a/src/egon/data/datasets/gas_neighbours/eGon2035.py b/src/egon/data/datasets/gas_neighbours/eGon2035.py index 7420a6faa..c488657b2 100755 --- a/src/egon/data/datasets/gas_neighbours/eGon2035.py +++ b/src/egon/data/datasets/gas_neighbours/eGon2035.py @@ -176,10 +176,10 @@ def calc_capacities(): For each neigbouring country, this function calculates the gas generation capacity in 2035 using the function :py:func:`calc_capacity_per_year` for 2030 and 2040 and - interpolating the results. These capacities include LNG import, as + interpolates the results. These capacities include LNG import, as well as conventional and biogas production. - Two conventional gas generators for are added for Norway and Russia - interpolating the supply potential (min) values from the TYNPD 2020 + Two conventional gas generators are added for Norway and Russia + interpolating the supply potential values from the TYNPD 2020 for 2030 and 2040. Returns @@ -334,13 +334,13 @@ def calc_capacity_per_year(df, lng, year): """Calculates gas production capacities for a specified year For a specified year and for the foreign country nodes this function - calculates the gas production capacity, considering the gas - (conventional and bio) production capacity from TYNDP data and the - LGN import capacity from Scigrid gas data. + calculates the gas production capacities, considering the gas + (conventional and bio) production capacities from TYNDP data and the + LNG import capacities from Scigrid gas data. The columns of the returned dataframe are the following: - * Value_bio_year: biogas capacity prodution (in GWh/d) - * Value_conv_year: conventional gas capacity prodution including + * Value_bio_year: biogas production capacity (in GWh/d) + * Value_conv_year: conventional gas production capacity including LNG imports (in GWh/d) * CH4_year: total gas production capacity (in GWh/d). This value is calculated using the peak production value from the TYNDP. @@ -364,7 +364,7 @@ def calc_capacity_per_year(df, lng, year): LNG terminal capacities per foreign country node (in GWh/d) year : int - Year to calculate gas production capacity for. + Year to calculate gas production capacities for Returns ------- @@ -453,14 +453,14 @@ def calc_capacity_per_year(df, lng, year): def insert_generators(gen): - """Insert gas generators for foreign countries in the database + """Insert gas generators for foreign countries into the database - Insert gas generators for foreign countries in the data base. + Insert gas generators for foreign countries into the database. The marginal cost of the methane is calculated as the sum of the - imported LNG cost, of the conventional natural gas cost and of the + imported LNG cost, the conventional natural gas cost and the biomethane cost, weighted by their share in the total import/ production capacity. - LNG is considerate to be 30% more expensive than the natural gas + LNG gas is considered to be 30% more expensive than the natural gas transported by pipelines (source: iwd, 2022). Parameters @@ -470,7 +470,7 @@ def insert_generators(gen): Returns ------- - None. + None """ sources = config.datasets()["gas_neighbours"]["sources"] @@ -539,7 +539,7 @@ def calc_global_ch4_demand(Norway_global_demand_1y): """Calculates global CH4 demands abroad for eGon2035 scenario The data comes from TYNDP 2020 according to NEP 2021 from the - scenario 'Distributed Energy', linear interpolate between 2030 + scenario 'Distributed Energy'; linear interpolates between 2030 and 2040. Returns @@ -618,17 +618,13 @@ def calc_global_ch4_demand(Norway_global_demand_1y): def import_ch4_demandTS(): """Calculate global CH4 demand in Norway and CH4 demand profile - Import from the PyPSA-eur-sec run the timeseries of residential - rural heat per neighbor country. This timeserie is used to + Import from the PyPSA-eur-sec run the time series of residential + rural heat per neighbor country. This time series is used to calculate: * the global (yearly) heat demand of Norway (that will be supplied by CH4) * the normalized CH4 hourly resolved demand profile - Parameters - ---------- - None. - Returns ------- Norway_global_demand: Float @@ -674,18 +670,18 @@ def import_ch4_demandTS(): def insert_ch4_demand(global_demand, normalized_ch4_demandTS): - """Insert CH4 demands abroad in the database for eGon2035 + """Insert CH4 demands abroad into the database for eGon2035 Parameters ---------- global_demand : pandas.DataFrame Global CH4 demand per foreign node in 1 year gas_demandTS : pandas.DataFrame - Normalized time serie of the demand per foreign country + Normalized time series of the demand per foreign country Returns ------- - None. + None """ sources = config.datasets()["gas_neighbours"]["sources"] @@ -699,7 +695,11 @@ def insert_ch4_demand(global_demand, normalized_ch4_demandTS): db.execute_sql( f""" DELETE FROM - {targets['load_timeseries']['schema']}.{targets['load_timeseries']['table']} + { + targets['load_time series']['schema'] + }.{ + targets['load_time series']['table'] + } WHERE "load_id" IN ( SELECT load_id FROM {targets['loads']['schema']}.{targets['loads']['table']} @@ -788,15 +788,23 @@ def insert_ch4_demand(global_demand, normalized_ch4_demandTS): # Insert data to DB ch4_demand_TS.to_sql( - targets["load_timeseries"]["table"], + targets["load_time series"]["table"], db.engine(), - schema=targets["load_timeseries"]["schema"], + schema=targets["load_time series"]["schema"], index=False, if_exists="append", ) def calc_ch4_storage_capacities(): + """Calculate CH4 storage capacities for neighboring countries + + Returns + ------- + ch4_storage_capacities: pandas.DataFrame + Methane gas storage capacities per country in MWh + + """ target_file = ( Path(".") / "datasets" / "gas_data" / "data" / "IGGIELGN_Storages.csv" ) @@ -885,6 +893,18 @@ def calc_ch4_storage_capacities(): def insert_storage(ch4_storage_capacities): + """Insert CH4 storage capacities into the database for eGon2035 + + Parameters + ---------- + ch4_storage_capacities : pandas.DataFrame + Methane gas storage capacities per country in MWh + + Returns + ------- + None + + """ sources = config.datasets()["gas_neighbours"]["sources"] targets = config.datasets()["gas_neighbours"]["targets"] @@ -929,11 +949,11 @@ def insert_storage(ch4_storage_capacities): def calc_global_power_to_h2_demand(): - """Calculates H2 demand abroad for eGon2035 scenario + """Calculate H2 demand abroad for eGon2035 scenario Calculates global power demand abroad linked to H2 production. The data comes from TYNDP 2020 according to NEP 2021 from the - scenario 'Distributed Energy', linear interpolate between 2030 + scenario 'Distributed Energy'; linear interpolate between 2030 and 2040. Returns @@ -1025,14 +1045,15 @@ def calc_global_power_to_h2_demand(): def insert_power_to_h2_demand(global_power_to_h2_demand): """Insert H2 demands into database for eGon2035 - Detailled description - This function inserts data in the database and has no return. - Parameters ---------- global_power_to_h2_demand : pandas.DataFrame Global hourly power-to-h2 demand per foreign node + Returns + ------- + None + """ sources = config.datasets()["gas_neighbours"]["sources"] targets = config.datasets()["gas_neighbours"]["targets"] @@ -1106,13 +1127,11 @@ def insert_power_to_h2_demand(global_power_to_h2_demand): def calculate_ch4_grid_capacities(): """Calculates CH4 grid capacities for foreign countries based on TYNDP-data - Parameters - ---------- - None. - Returns ------- Neighbouring_pipe_capacities_list : pandas.DataFrame + Table containing the CH4 grid capacity for each foreign + country """ sources = config.datasets()["gas_neighbours"]["sources"] @@ -1285,7 +1304,6 @@ def calculate_ch4_grid_capacities(): schema = sources["buses"]["schema"] table = sources["buses"]["table"] for country_code in [e for e in countries if e not in ("GB", "SE", "UK")]: - # Select cross-bording links cap_DE = db.select_dataframe( f"""SELECT link_id, bus0, bus1 @@ -1383,12 +1401,12 @@ def calculate_ch4_grid_capacities(): def tyndp_gas_generation(): - """Insert data from TYNDP 2020 accordning to NEP 2021 - Scenario 'Distributed Energy', linear interpolate between 2030 and 2040 + """Insert data from TYNDP 2020 according to NEP 2021 + Scenario 'Distributed Energy'; linear interpolate between 2030 and 2040 Returns ------- - None. + None """ capacities = calc_capacities() insert_generators(capacities) @@ -1403,13 +1421,13 @@ def tyndp_gas_demand(): Insert CH4 and H2 demands abroad for eGon2035 by executing the following steps: * CH4 - * Calculation of the global CH4 demand in Norway and of the + * Calculation of the global CH4 demand in Norway and the CH4 demand profile by executing the function :py:func:`import_ch4_demandTS` * Calculation of the global CH4 demands by executing the function :py:func:`calc_global_ch4_demand` - * Insertion the CH4 loads and their associated time series - in the database by executing the function + * Insertion of the CH4 loads and their associated time + series in the database by executing the function :py:func:`insert_ch4_demand` * H2 * Calculation of the global power demand abroad linked @@ -1417,7 +1435,10 @@ def tyndp_gas_demand(): :py:func:`calc_global_power_to_h2_demand` * Insertion of these loads in the database by executing the function :py:func:`insert_power_to_h2_demand` - This function insert data in the database and has no return. + + Returns + ------- + None """ Norway_global_demand_1y, normalized_ch4_demandTS = import_ch4_demandTS() @@ -1429,12 +1450,12 @@ def tyndp_gas_demand(): def grid(): - """Insert data from TYNDP 2020 accordning to NEP 2021 - Scenario 'Distributed Energy', linear interpolate between 2030 and 2040 + """Insert data from TYNDP 2020 according to NEP 2021 + Scenario 'Distributed Energy; linear interpolate between 2030 and 2040 Returns ------- - None. + None """ Neighbouring_pipe_capacities_list = calculate_ch4_grid_capacities() insert_gas_grid_capacities( @@ -1446,7 +1467,7 @@ def calculate_ocgt_capacities(): """Calculate gas turbine capacities abroad for eGon2035 Calculate gas turbine capacities abroad for eGon2035 based on TYNDP - 2020, scenario "Distributed Energy", interpolated between 2030 and 2040 + 2020, scenario "Distributed Energy"; interpolated between 2030 and 2040 Returns ------- @@ -1501,15 +1522,17 @@ def calculate_ocgt_capacities(): def insert_ocgt_abroad(): - """Insert gas turbine capicities abroad for eGon2035 in the database - - This function inserts data in the database and has no return. + """Insert gas turbine capacities abroad for eGon2035 in the database Parameters ---------- df_ocgt: pandas.DataFrame Gas turbine capacities per foreign node + Returns + ------- + None + """ scn_name = "eGon2035" carrier = "OCGT" diff --git a/src/egon/data/datasets/gas_neighbours/gas_abroad.py b/src/egon/data/datasets/gas_neighbours/gas_abroad.py index 860075dde..fa7c1a9a5 100755 --- a/src/egon/data/datasets/gas_neighbours/gas_abroad.py +++ b/src/egon/data/datasets/gas_neighbours/gas_abroad.py @@ -1,6 +1,6 @@ """Module containing functions to insert gas abroad -In this module, functions useful to insert the gas components (H2 and +In this module, functions used to insert the gas components (H2 and CH4) abroad for eGon2035 and eGon100RE are defined. """ @@ -11,14 +11,14 @@ def insert_gas_grid_capacities(Neighbouring_pipe_capacities_list, scn_name): - """Insert crossbordering gas pipelines in the database + """Insert crossbordering gas pipelines into the database - This function insert a list of crossbordering gas pipelines after + This function inserts a list of crossbordering gas pipelines after cleaning the database. For eGon2035, all the CH4 crossbordering pipelines are inserted - there (no H2 grid in this scenario). - For eGon100RE, only the the crossbordering pipelines with Germany - are inserted there (the other ones are inerted in PypsaEurSec), + (no H2 grid in this scenario). + For eGon100RE, only the crossbordering pipelines with Germany + are inserted (the other ones are inserted in PypsaEurSec), but in this scenario there are H2 and CH4 pipelines. Parameters @@ -63,10 +63,7 @@ def insert_gas_grid_capacities(Neighbouring_pipe_capacities_list, scn_name): """ ) - carriers = { - "CH4": {"bus_inDE": "CH4", "bus_abroad": "CH4"}, - "H2_retrofit": {"bus_inDE": "H2_grid", "bus_abroad": "H2"}, - } + carriers = {"CH4": "CH4", "H2_retrofit": "H2_grid"} if scn_name == "eGon100RE": for c in carriers: @@ -78,27 +75,27 @@ def insert_gas_grid_capacities(Neighbouring_pipe_capacities_list, scn_name): SELECT bus_id FROM {sources['buses']['schema']}.{sources['buses']['table']} WHERE country != 'DE' - AND carrier = '{carriers[c]["bus_abroad"]}' + AND carrier = '{carriers[c]}' AND scn_name = '{scn_name}') AND "bus1" IN (SELECT bus_id FROM {sources['buses']['schema']}.{sources['buses']['table']} WHERE country = 'DE' - AND carrier = '{carriers[c]["bus_inDE"]}' + AND carrier = '{carriers[c]}' AND scn_name = '{scn_name}')) OR ("bus0" IN ( SELECT bus_id FROM {sources['buses']['schema']}.{sources['buses']['table']} WHERE country = 'DE' - AND carrier = '{carriers[c]["bus_inDE"]}' + AND carrier = '{carriers[c]}' AND scn_name = '{scn_name}') - AND "bus1" IN ( + AND "bus1" IN ( SELECT bus_id FROM {sources['buses']['schema']}.{sources['buses']['table']} WHERE country != 'DE' - AND carrier = '{carriers[c]["bus_abroad"]}' + AND carrier = '{carriers[c]}' AND scn_name = '{scn_name}')) AND scn_name = '{scn_name}' - AND carrier = '{c.index}' + AND carrier = '{c}' ; """ ) diff --git a/src/egon/data/datasets/heat_demand/__init__.py b/src/egon/data/datasets/heat_demand/__init__.py index ee9537483..77267d883 100644 --- a/src/egon/data/datasets/heat_demand/__init__.py +++ b/src/egon/data/datasets/heat_demand/__init__.py @@ -10,11 +10,11 @@ This module obtains the residential and service-sector heat demand data for 2015 from Peta5.0.1, calculates future heat demands and saves them in the database with assigned census cell IDs. + """ from pathlib import Path # for database import from urllib.request import urlretrieve -import datetime # for metadata creation import json @@ -41,7 +41,6 @@ ) from egon.data.metadata import ( context, - generate_resource_fields_from_sqla_model, license_ccby, meta_metadata, sources, @@ -49,12 +48,41 @@ import egon.data.config -# class for airflow task management (and version control) + class HeatDemandImport(Dataset): + + """ + Insert the annual heat demand per census cell for each scenario + + This dataset downloads the heat demand raster data for private households + and CTS from Peta 5.0.1 (https://s-eenergies-open-data-euf.hub.arcgis.com/maps/d7d18b63250240a49eb81db972aa573e/about) + and stores it into files in the working directory. + The data from Peta 5.0.1 represents the status quo of the year 2015. + To model future heat demands, the data is scaled to meet target values + from external sources. These target values are defined for each scenario + in :py:class:`ScenarioParameters `. + + *Dependencies* + * :py:class:`ScenarioParameters ` + * :py:class:`Vg250 ` + * :py:class:`ZensusVg250 ` + + *Resulting tables* + * :py:class:`demand.egon_peta_heat ` is created and filled + + """ + + + #: + name: str = "heat-demands" + #: + version: str = "0.0.1" + def __init__(self, dependencies): super().__init__( - name="heat-demands", - version="0.0.3", + name=self.name, + # version=self.target_files + "_0.0", + version=self.version, # maybe rethink the naming dependencies=dependencies, tasks=(scenario_data_import), ) @@ -62,6 +90,7 @@ def __init__(self, dependencies): Base = declarative_base() + # class for the final dataset in the database class EgonPetaHeat(Base): __tablename__ = "egon_peta_heat" @@ -469,7 +498,6 @@ def heat_demand_to_db_table(): db.execute_sql("DELETE FROM demand.egon_peta_heat;") for source in sources: - if not "2015" in source.stem: # Create a temporary table and fill the final table using the sql script rasters = f"heat_demand_rasters_{source.stem.lower()}" diff --git a/src/egon/data/datasets/heat_demand_europe.py b/src/egon/data/datasets/heat_demand_europe.py index f5f832f33..5bd1f729c 100644 --- a/src/egon/data/datasets/heat_demand_europe.py +++ b/src/egon/data/datasets/heat_demand_europe.py @@ -22,20 +22,35 @@ class HeatDemandEurope(Dataset): + """ + Downloads annual heat demands for European countries from hotmaps - data_config = egon.data.config.datasets() + This dataset downloads annual heat demands for all European countries for the year 2050 from + hotmaps and stores the results into files. These are later used by pypsa-eur-sec. - hotmapsheatdemands_config = data_config[ - "hotmaps_current_policy_scenario_heat_demands_buildings"] - - target_file = hotmapsheatdemands_config["targets"]["path"] + + *Dependencies* + * :py:func:`Setup ` + + """ + + #: + name: str = "heat-demands-europe" + #: + version: str = ( + egon.data.config.datasets()[ + "hotmaps_current_policy_scenario_heat_demands_buildings" + ]["targets"]["path"] + + "_hotmaps.0.1" + ) def __init__(self, dependencies): super().__init__( - name="heat-demands-europe", - version=self.target_file + "_hotmaps.0.1", + name=self.name, + version=self.version, dependencies=dependencies, - tasks=(download)) + tasks=(download), + ) def download(): @@ -59,7 +74,8 @@ def download(): # heat demands hotmapsheatdemands_config = data_config[ - "hotmaps_current_policy_scenario_heat_demands_buildings"] + "hotmaps_current_policy_scenario_heat_demands_buildings" + ] target_file = hotmapsheatdemands_config["targets"]["path"] diff --git a/src/egon/data/datasets/heat_demand_timeseries/__init__.py b/src/egon/data/datasets/heat_demand_timeseries/__init__.py index 80e9d8010..5fe5b3bd2 100644 --- a/src/egon/data/datasets/heat_demand_timeseries/__init__.py +++ b/src/egon/data/datasets/heat_demand_timeseries/__init__.py @@ -1,6 +1,8 @@ -from datetime import datetime +from datetime import date, datetime from pathlib import Path +import json import os +import time from sqlalchemy import ARRAY, Column, Float, Integer, String, Text from sqlalchemy.ext.declarative import declarative_base @@ -27,7 +29,12 @@ from egon.data.datasets.heat_demand_timeseries.service_sector import ( CTS_demand_scale, ) -import egon +from egon.data.metadata import ( + context, + license_egon_data_odbl, + meta_metadata, + sources, +) Base = declarative_base() @@ -87,7 +94,11 @@ def create_timeseries_for_building(building_id, scenario): SELECT building_demand * UNNEST(idp) as demand FROM ( - SELECT demand.demand / building.count * daily_demand.daily_demand_share as building_demand, daily_demand.day_of_year + SELECT + demand.demand + / building.count + * daily_demand.daily_demand_share as building_demand, + daily_demand.day_of_year FROM (SELECT demand FROM @@ -111,8 +122,12 @@ def create_timeseries_for_building(building_id, scenario): WHERE climate_zone = ( SELECT climate_zone FROM boundaries.egon_map_zensus_climate_zones WHERE zensus_population_id = - (SELECT zensus_population_id FROM demand.egon_heat_timeseries_selected_profiles - WHERE building_id = {building_id}))) as daily_demand) as daily_demand + ( + SELECT zensus_population_id + FROM demand.egon_heat_timeseries_selected_profiles + WHERE building_id = {building_id} + ) + )) as daily_demand) as daily_demand JOIN (SELECT b.idp, ordinality as day FROM demand.egon_heat_timeseries_selected_profiles a, @@ -126,20 +141,23 @@ def create_timeseries_for_building(building_id, scenario): def create_district_heating_profile(scenario, area_id): - """Create heat demand profile for district heating grid including demands of - households and service sector. + """Create a heat demand profile for a district heating grid. + + The created heat demand profile includes the demands of households + and the service sector. Parameters ---------- scenario : str - Name of the selected scenario. + The name of the selected scenario. area_id : int - Index of the selected district heating grid + The index of the selected district heating grid. Returns ------- - df : pandas,DataFrame - Hourly heat demand timeseries in MW for the selected district heating grid + pd.DataFrame + An hourly heat demand timeseries in MW for the selected district + heating grid. """ @@ -178,7 +196,8 @@ def create_district_heating_profile(scenario, area_id): JOIN demand.egon_daily_heat_demand_per_climate_zone c ON c.climate_zone = b.climate_zone - JOIN (SELECT e.idp, ordinality as day, zensus_population_id, building_id + JOIN ( + SELECT e.idp, ordinality as day, zensus_population_id, building_id FROM demand.egon_heat_timeseries_selected_profiles d, UNNEST (d.selected_idp_profiles) WITH ORDINALITY as selected_idp JOIN demand.egon_heat_idp_pool e @@ -203,7 +222,7 @@ def create_district_heating_profile(scenario, area_id): WHERE scenario = '{scenario}' AND area_id = {area_id} )) - GROUP BY zensus_population_id) building + GROUP BY zensus_population_id) building ON building.zensus_population_id = b.zensus_population_id, UNNEST(demand_profile.idp) WITH ORDINALITY as hourly_demand @@ -216,9 +235,9 @@ def create_district_heating_profile(scenario, area_id): ) print( - f"Time to create time series for district heating grid {scenario} {area_id}:" + f"Time to create time series for district heating grid {scenario}" + f" {area_id}:\n{datetime.now() - start_time}" ) - print(datetime.now() - start_time) return df @@ -258,7 +277,12 @@ def create_district_heating_profile_python_like(scenario="eGon2035"): annual_demand = db.select_dataframe( f""" - SELECT a.zensus_population_id, demand/c.count as per_building , area_id, demand as demand_total FROM + SELECT + a.zensus_population_id, + demand / c.count as per_building, + area_id, + demand as demand_total + FROM demand.egon_peta_heat a INNER JOIN ( SELECT * FROM demand.egon_map_zensus_district_heating_areas @@ -320,14 +344,13 @@ def create_district_heating_profile_python_like(scenario="eGon2035"): WHERE scenario = '{scenario}' AND area_id = '{area}' ) b ON a.zensus_population_id = b.zensus_population_id , - + UNNEST (selected_idp_profiles) WITH ORDINALITY as selected_idp - + """ ) if not selected_profiles.empty: - df = pd.merge( selected_profiles, daily_demand_shares, @@ -364,7 +387,7 @@ def create_district_heating_profile_python_like(scenario="eGon2035"): assert ( abs(diff) < 0.03 - ), f"""Deviation of residential heat demand time + ), f"""Deviation of residential heat demand time series for district heating grid {str(area)} is {diff}""" hh = np.concatenate( @@ -438,7 +461,9 @@ def create_individual_heat_per_mv_grid(scenario="eGon2035", mv_grid_id=1564): JOIN demand.egon_daily_heat_demand_per_climate_zone c ON c.climate_zone = b.climate_zone - JOIN (SELECT e.idp, ordinality as day, zensus_population_id, building_id + JOIN ( + SELECT + e.idp, ordinality as day, zensus_population_id, building_id FROM demand.egon_heat_timeseries_selected_profiles d, UNNEST (d.selected_idp_profiles) WITH ORDINALITY as selected_idp JOIN demand.egon_heat_idp_pool e @@ -461,7 +486,7 @@ def create_individual_heat_per_mv_grid(scenario="eGon2035", mv_grid_id=1564): boundaries.egon_map_zensus_grid_districts WHERE bus_id = {mv_grid_id} )) - GROUP BY zensus_population_id) building + GROUP BY zensus_population_id) building ON building.zensus_population_id = b.zensus_population_id, UNNEST(demand_profile.idp) WITH ORDINALITY as hourly_demand @@ -480,7 +505,6 @@ def create_individual_heat_per_mv_grid(scenario="eGon2035", mv_grid_id=1564): def calulate_peak_load(df, scenario): - # peat load in W_th data = ( df.groupby("building_id") @@ -505,7 +529,6 @@ def calulate_peak_load(df, scenario): def create_individual_heating_peak_loads(scenario="eGon2035"): - engine = db.engine() EgonIndividualHeatingPeakLoads.__table__.drop(bind=engine, checkfirst=True) @@ -547,7 +570,8 @@ def create_individual_heating_peak_loads(scenario="eGon2035"): WHERE a.scenario = '{scenario}' AND a.sector = 'residential' AND a.zensus_population_id NOT IN ( - SELECT zensus_population_id FROM demand.egon_map_zensus_district_heating_areas + SELECT zensus_population_id + FROM demand.egon_map_zensus_district_heating_areas WHERE scenario = '{scenario}' ) @@ -564,7 +588,6 @@ def create_individual_heating_peak_loads(scenario="eGon2035"): start_time = datetime.now() for grid in annual_demand.bus_id.unique(): - selected_profiles = db.select_dataframe( f""" SELECT a.zensus_population_id, building_id, c.climate_zone, @@ -577,7 +600,8 @@ def create_individual_heating_peak_loads(scenario="eGon2035"): UNNEST (selected_idp_profiles) WITH ORDINALITY as selected_idp WHERE a.zensus_population_id NOT IN ( - SELECT zensus_population_id FROM demand.egon_map_zensus_district_heating_areas + SELECT zensus_population_id + FROM demand.egon_map_zensus_district_heating_areas WHERE scenario = '{scenario}' ) AND a.zensus_population_id IN ( @@ -615,7 +639,6 @@ def create_individual_heating_peak_loads(scenario="eGon2035"): def create_individual_heating_profile_python_like(scenario="eGon2035"): - start_time = datetime.now() idp_df = db.select_dataframe( @@ -627,7 +650,11 @@ def create_individual_heating_profile_python_like(scenario="eGon2035"): annual_demand = db.select_dataframe( f""" - SELECT a.zensus_population_id, demand/c.count as per_building, demand as demand_total, bus_id + SELECT + a.zensus_population_id, + demand / c.count as per_building, + demand as demand_total, + bus_id FROM demand.egon_peta_heat a @@ -649,7 +676,8 @@ def create_individual_heating_profile_python_like(scenario="eGon2035"): WHERE a.scenario = '{scenario}' AND a.sector = 'residential' AND a.zensus_population_id NOT IN ( - SELECT zensus_population_id FROM demand.egon_map_zensus_district_heating_areas + SELECT zensus_population_id + FROM demand.egon_map_zensus_district_heating_areas WHERE scenario = '{scenario}' ) @@ -658,7 +686,7 @@ def create_individual_heating_profile_python_like(scenario="eGon2035"): ) daily_demand_shares = db.select_dataframe( - f""" + """ SELECT climate_zone, day_of_year as day, daily_demand_share FROM demand.egon_daily_heat_demand_per_climate_zone """ @@ -738,10 +766,10 @@ def create_individual_heating_profile_python_like(scenario="eGon2035"): - annual_demand[annual_demand.bus_id == grid].demand_total.sum() ) / (annual_demand[annual_demand.bus_id == grid].demand_total.sum()) - assert ( - abs(diff) < 0.03 - ), f"""Deviation of residential heat demand time - series for mv grid {str(grid)} is {diff}""" + assert abs(diff) < 0.03, ( + "Deviation of residential heat demand time series for mv" + f" grid {grid} is {diff}" + ) if not (slice_df[hour].empty or cts.empty): entry = EgonEtragoTimeseriesIndividualHeating( @@ -773,7 +801,6 @@ def create_individual_heating_profile_python_like(scenario="eGon2035"): def district_heating(method="python"): - engine = db.engine() EgonTimeseriesDistrictHeating.__table__.drop(bind=engine, checkfirst=True) EgonTimeseriesDistrictHeating.__table__.create( @@ -785,7 +812,6 @@ def district_heating(method="python"): create_district_heating_profile_python_like("eGon100RE") else: - CTS_demand_dist, CTS_demand_grid, CTS_demand_zensus = CTS_demand_scale( aggregation_level="district" ) @@ -861,7 +887,6 @@ def individual_heating_per_mv_grid_100(method="python"): def individual_heating_per_mv_grid(method="python"): - if method == "python": engine = db.engine() EgonEtragoTimeseriesIndividualHeating.__table__.drop( @@ -875,7 +900,6 @@ def individual_heating_per_mv_grid(method="python"): create_individual_heating_profile_python_like("eGon100RE") else: - engine = db.engine() EgonEtragoTimeseriesIndividualHeating.__table__.drop( bind=engine, checkfirst=True @@ -897,7 +921,6 @@ def individual_heating_per_mv_grid(method="python"): ) for index, row in ids.iterrows(): - for scenario in ["eGon2035", "eGon100RE"]: series = create_individual_heat_per_mv_grid( scenario, row.bus_id @@ -938,7 +961,6 @@ def individual_heating_per_mv_grid(method="python"): def store_national_profiles(): - scenario = "eGon100RE" df = db.select_dataframe( @@ -969,7 +991,8 @@ def store_national_profiles(): JOIN demand.egon_daily_heat_demand_per_climate_zone c ON c.climate_zone = b.climate_zone - JOIN (SELECT e.idp, ordinality as day, zensus_population_id, building_id + JOIN ( + SELECT e.idp, ordinality as day, zensus_population_id, building_id FROM demand.egon_heat_timeseries_selected_profiles d, UNNEST (d.selected_idp_profiles) WITH ORDINALITY as selected_idp JOIN demand.egon_heat_idp_pool e @@ -984,7 +1007,7 @@ def store_national_profiles(): SELECT zensus_population_id FROM demand.egon_heat_timeseries_selected_profiles ) - GROUP BY zensus_population_id) building + GROUP BY zensus_population_id) building ON building.zensus_population_id = b.zensus_population_id, UNNEST(demand_profile.idp) WITH ORDINALITY as hourly_demand @@ -1062,11 +1085,132 @@ def export_etrago_cts_heat_profiles(): ) +def metadata(): + fields = [ + { + "description": "Index of corresponding district heating area", + "name": "area_id", + "type": "integer", + "unit": "none", + }, + { + "description": "Name of scenario", + "name": "scenario", + "type": "str", + "unit": "none", + }, + { + "description": "Heat demand time series", + "name": "dist_aggregated_mw", + "type": "array of floats", + "unit": "MW", + }, + ] + + meta_district = { + "name": "demand.egon_timeseries_district_heating", + "title": "eGon heat demand time series for district heating grids", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": "Heat demand time series for district heating grids", + "language": ["EN"], + "publicationDate": date.today().isoformat(), + "context": context(), + "spatial": { + "location": None, + "extent": "Germany", + "resolution": None, + }, + "sources": [ + sources()["era5"], + sources()["vg250"], + sources()["egon-data"], + sources()["egon-data_bundle"], + sources()["peta"], + ], + "licenses": [license_egon_data_odbl()], + "contributors": [ + { + "title": "Clara Büttner", + "email": "http://github.com/ClaraBuettner", + "date": time.strftime("%Y-%m-%d"), + "object": None, + "comment": "Imported data", + }, + ], + "resources": [ + { + "profile": "tabular-data-resource", + "name": "demand.egon_timeseries_district_heating", + "path": None, + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": fields, + "primaryKey": ["index"], + "foreignKeys": [], + }, + "dialect": {"delimiter": None, "decimalSeparator": "."}, + } + ], + "metaMetadata": meta_metadata(), + } + + # Add metadata as a comment to the table + db.submit_comment( + "'" + json.dumps(meta_district) + "'", + EgonTimeseriesDistrictHeating.__table__.schema, + EgonTimeseriesDistrictHeating.__table__.name, + ) + + + class HeatTimeSeries(Dataset): + """ + Chooses heat demand profiles for each residential and CTS building + + This dataset creates heat demand profiles in an hourly resoultion. + Time series for CTS buildings are created using the SLP-gas method implemented + in the demandregio disagregator with the function :py:func:`export_etrago_cts_heat_profiles` + and stored in the database. + Time series for residential buildings are created based on a variety of synthetical created + individual demand profiles that are part of :py:class:`DataBundle `. + This method is desribed within the functions and in this publication: + C. Büttner, J. Amme, J. Endres, A. Malla, B. Schachler, I. Cußmann, + Open modeling of electricity and heat demand curves for all + residential buildings in Germany, Energy Informatics 5 (1) (2022) 21. + doi:10.1186/s42162-022-00201-y. + + + *Dependencies* + * :py:class:`DataBundle ` + * :py:class:`DemandRegio ` + * :py:class:`HeatDemandImport ` + * :py:class:`DistrictHeatingAreas ` + * :py:class:`Vg250 ` + * :py:class:`ZensusMvGridDistricts ` + * :py:func:`hh_demand_buildings_setup ` + * :py:class:`WeatherData ` + + + *Resulting tables* + * :py:class:`demand.egon_timeseries_district_heating ` is created and filled + * :py:class:`demand.egon_etrago_heat_cts ` is created and filled + * :py:class:`demand.egon_heat_timeseries_selected_profiles ` is created and filled + * :py:class:`demand.egon_daily_heat_demand_per_climate_zone ` + is created and filled + * :py:class:`boundaries.egon_map_zensus_climate_zones ` is created and filled + + """ + + #: + name: str = "HeatTimeSeries" + #: + version: str = "0.0.8" + def __init__(self, dependencies): super().__init__( - name="HeatTimeSeries", - version="0.0.7", + name=self.name, + version=self.version, dependencies=dependencies, tasks=( { @@ -1077,6 +1221,7 @@ def __init__(self, dependencies): }, select, district_heating, + metadata, # store_national_profiles, ), ) diff --git a/src/egon/data/datasets/heat_etrago/__init__.py b/src/egon/data/datasets/heat_etrago/__init__.py index 9004de2d3..2ff427f5c 100644 --- a/src/egon/data/datasets/heat_etrago/__init__.py +++ b/src/egon/data/datasets/heat_etrago/__init__.py @@ -69,14 +69,14 @@ def insert_buses(carrier, scenario): SELECT ST_Centroid(geom) AS geom FROM {sources['mv_grids']['schema']}. {sources['mv_grids']['table']} - WHERE bus_id IN - (SELECT DISTINCT bus_id + WHERE bus_id IN + (SELECT DISTINCT bus_id FROM boundaries.egon_map_zensus_grid_districts a - JOIN demand.egon_peta_heat b + JOIN demand.egon_peta_heat b ON a.zensus_population_id = b.zensus_population_id WHERE b.scenario = '{scenario}' AND b.zensus_population_id NOT IN ( - SELECT zensus_population_id FROM + SELECT zensus_population_id FROM demand.egon_map_zensus_district_heating_areas WHERE scenario = '{scenario}' ) @@ -288,8 +288,8 @@ def insert_central_direct_heat(scenario="eGon2035"): {targets['heat_generators']['table']} WHERE carrier IN ('solar_thermal_collector', 'geo_thermal') AND scn_name = '{scenario}' - AND bus IN - (SELECT bus_id + AND bus IN + (SELECT bus_id FROM {targets['heat_buses']['schema']}. {targets['heat_buses']['table']} WHERE scn_name = '{scenario}' @@ -509,14 +509,14 @@ def insert_rural_gas_boilers(scenario="eGon2035"): {targets['heat_links']['table']} WHERE carrier = 'rural_gas_boiler' AND scn_name = '{scenario}' - AND bus0 IN - (SELECT bus_id + AND bus0 IN + (SELECT bus_id FROM {targets['heat_buses']['schema']}. {targets['heat_buses']['table']} WHERE scn_name = '{scenario}' AND country = 'DE') - AND bus1 IN - (SELECT bus_id + AND bus1 IN + (SELECT bus_id FROM {targets['heat_buses']['schema']}. {targets['heat_buses']['table']} WHERE scn_name = '{scenario}' @@ -615,10 +615,42 @@ def supply(): class HeatEtrago(Dataset): + """ + Collect data related to the heat sector for the eTraGo tool + + This dataset collects data from the heat sector and puts it into a format that + is needed for the transmission grid optimisation within the tool eTraGo. + It includes the creation of individual and central heat nodes, aggregates the + heat supply technologies (apart from CHP) per medium voltage grid district and + adds extendable heat stores to each bus. This data is then writing into the + corresponding tables that are read by eTraGo. + + + *Dependencies* + * :py:class:`HeatSupply ` + * :py:class:`MvGridDistricts ` + * :py:class:`EtragoSetup ` + * :py:class:`RenewableFeedin ` + * :py:class:`HeatTimeSeries ` + + *Resulting tables* + * :py:class:`grid.egon_etrago_bus ` is extended + * :py:class:`grid.egon_etrago_link ` is extended + * :py:class:`grid.egon_etrago_link_timeseries ` is extended + * :py:class:`grid.egon_etrago_store ` is extended + * :py:class:`grid.egon_etrago_generator ` is extended + + """ + + #: + name: str = "HeatEtrago" + #: + version: str = "0.0.10" + def __init__(self, dependencies): super().__init__( - name="HeatEtrago", - version="0.0.10", + name=self.name, + version=self.version, dependencies=dependencies, tasks=(buses, supply, store), ) diff --git a/src/egon/data/datasets/heat_etrago/hts_etrago.py b/src/egon/data/datasets/heat_etrago/hts_etrago.py index 7777baa41..112919457 100644 --- a/src/egon/data/datasets/heat_etrago/hts_etrago.py +++ b/src/egon/data/datasets/heat_etrago/hts_etrago.py @@ -1,3 +1,6 @@ +""" +The central module creating heat demand time series for the eTraGo tool +""" from egon.data import config, db from egon.data.db import next_etrago_id from egon.data.datasets import Dataset @@ -89,7 +92,6 @@ def hts_to_etrago(): bus_ts.loc[:, "bus_id"] = bus_ts.loc[:, "heat_bus_id"] else: - efficiency_gas_boiler = get_sector_parameters("heat", "eGon2035")[ "efficiency" ]["rural_gas_boiler"] @@ -211,10 +213,34 @@ def hts_to_etrago(): class HtsEtragoTable(Dataset): + """ + Collect heat demand time series for the eTraGo tool + + This dataset collects data for individual and district heating demands + and writes that into the tables that can be read by the eTraGo tool. + + *Dependencies* + * :py:class:`DistrictHeatingAreas ` + * :py:class:`HeatEtrago ` + * :py:class:`MvGridDistricts ` + * :py:class:`HeatPumps2035 ` + * :py:class:`HeatTimeSeries ` + + *Resulting tables* + * :py:class:`grid.egon_etrago_load ` is extended + * :py:class:`grid.egon_etrago_load_timeseries ` is extended + + """ + + #: + name: str = "HtsEtragoTable" + #: + version: str = "0.0.6" + def __init__(self, dependencies): super().__init__( - name="HtsEtragoTable", - version="0.0.6", + name=self.name, + version=self.version, dependencies=dependencies, tasks=(hts_to_etrago,), ) diff --git a/src/egon/data/datasets/heat_supply/__init__.py b/src/egon/data/datasets/heat_supply/__init__.py index f12b151f9..36b8afae3 100644 --- a/src/egon/data/datasets/heat_supply/__init__.py +++ b/src/egon/data/datasets/heat_supply/__init__.py @@ -2,22 +2,35 @@ """ -from egon.data import db, config +import datetime +import json +import time +from geoalchemy2.types import Geometry +from sqlalchemy import Column, Float, ForeignKey, Integer, String +from sqlalchemy.ext.declarative import declarative_base +import pandas as pd + +from egon.data import config, db +from egon.data.datasets import Dataset +from egon.data.datasets.district_heating_areas import EgonDistrictHeatingAreas from egon.data.datasets.heat_supply.district_heating import ( - cascade_heat_supply, backup_gas_boilers, backup_resistive_heaters, + cascade_heat_supply, ) +from egon.data.datasets.heat_supply.geothermal import potential_germany from egon.data.datasets.heat_supply.individual_heating import ( cascade_heat_supply_indiv, ) -from egon.data.datasets.heat_supply.geothermal import potential_germany -from egon.data.datasets.district_heating_areas import EgonDistrictHeatingAreas -from sqlalchemy import Column, String, Float, Integer, ForeignKey -from sqlalchemy.ext.declarative import declarative_base -from geoalchemy2.types import Geometry -from egon.data.datasets import Dataset +from egon.data.metadata import ( + context, + generate_resource_fields_from_sqla_model, + license_ccby, + license_egon_data_odbl, + meta_metadata, + sources, +) # Will later be imported from another file. Base = declarative_base() @@ -167,11 +180,210 @@ def individual_heating(): ) +def metadata(): + """Write metadata for heat supply tables + + Returns + ------- + None. + + """ + + fields = generate_resource_fields_from_sqla_model( + EgonDistrictHeatingSupply + ) + + fields_df = pd.DataFrame(data=fields).set_index("name") + fields_df.loc["index", "description"] = "Unique identifyer" + fields_df.loc[ + "district_heating_id", "description" + ] = "Index of the corresponding district heating grid" + fields_df.loc["carrier", "description"] = "Name of energy carrier" + fields_df.loc[ + "category", "description" + ] = "Size-category of district heating grid" + fields_df.loc["capacity", "description"] = "Installed heating capacity" + fields_df.loc[ + "geometry", "description" + ] = "Location of thermal power plant" + fields_df.loc["scenario", "description"] = "Name of corresponing scenario" + + fields_df.loc["capacity", "unit"] = "MW_th" + fields_df.unit.fillna("none", inplace=True) + + fields = fields_df.reset_index().to_dict(orient="records") + + meta_district = { + "name": "supply.egon_district_heating", + "title": "eGon heat supply for district heating grids", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": "Heat supply technologies for district heating grids", + "language": ["EN"], + "publicationDate": datetime.date.today().isoformat(), + "context": context(), + "spatial": { + "location": None, + "extent": "Germany", + "resolution": None, + }, + "sources": [ + sources()["era5"], + sources()["vg250"], + sources()["egon-data"], + sources()["egon-data_bundle"], + sources()["openstreetmap"], + sources()["mastr"], + sources()["peta"], + ], + "licenses": [license_egon_data_odbl()], + "contributors": [ + { + "title": "Clara Büttner", + "email": "http://github.com/ClaraBuettner", + "date": time.strftime("%Y-%m-%d"), + "object": None, + "comment": "Imported data", + }, + ], + "resources": [ + { + "profile": "tabular-data-resource", + "name": "supply.egon_district_heating", + "path": None, + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": fields, + "primaryKey": ["index"], + "foreignKeys": [], + }, + "dialect": {"delimiter": None, "decimalSeparator": "."}, + } + ], + "metaMetadata": meta_metadata(), + } + + # Add metadata as a comment to the table + db.submit_comment( + "'" + json.dumps(meta_district) + "'", + EgonDistrictHeatingSupply.__table__.schema, + EgonDistrictHeatingSupply.__table__.name, + ) + + fields = generate_resource_fields_from_sqla_model( + EgonIndividualHeatingSupply + ) + + fields_df = pd.DataFrame(data=fields).set_index("name") + fields_df.loc["index", "description"] = "Unique identifyer" + fields_df.loc[ + "mv_grid_id", "description" + ] = "Index of the corresponding mv grid district" + fields_df.loc["carrier", "description"] = "Name of energy carrier" + fields_df.loc["category", "description"] = "Size-category" + fields_df.loc["capacity", "description"] = "Installed heating capacity" + fields_df.loc[ + "geometry", "description" + ] = "Location of thermal power plant" + fields_df.loc["scenario", "description"] = "Name of corresponing scenario" + + fields_df.loc["capacity", "unit"] = "MW_th" + fields_df.unit.fillna("none", inplace=True) + + fields = fields_df.reset_index().to_dict(orient="records") + + meta_district = { + "name": "supply.egon_individual_heating", + "title": "eGon heat supply for individual supplied buildings", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": "Heat supply technologies for individual supplied buildings", + "language": ["EN"], + "publicationDate": datetime.date.today().isoformat(), + "context": context(), + "spatial": { + "location": None, + "extent": "Germany", + "resolution": None, + }, + "sources": [ + sources()["era5"], + sources()["vg250"], + sources()["egon-data"], + sources()["egon-data_bundle"], + sources()["openstreetmap"], + sources()["mastr"], + sources()["peta"], + ], + "licenses": [license_egon_data_odbl()], + "contributors": [ + { + "title": "Clara Büttner", + "email": "http://github.com/ClaraBuettner", + "date": time.strftime("%Y-%m-%d"), + "object": None, + "comment": "Imported data", + }, + ], + "resources": [ + { + "profile": "tabular-data-resource", + "name": "supply.egon_individual_heating", + "path": None, + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": fields, + "primaryKey": ["index"], + "foreignKeys": [], + }, + "dialect": {"delimiter": None, "decimalSeparator": "."}, + } + ], + "metaMetadata": meta_metadata(), + } + + # Add metadata as a comment to the table + db.submit_comment( + "'" + json.dumps(meta_district) + "'", + EgonIndividualHeatingSupply.__table__.schema, + EgonIndividualHeatingSupply.__table__.name, + ) + + class HeatSupply(Dataset): + """ + Select and store heat supply technologies for inidvidual and district heating + + This dataset distributes heat supply technologies to each district heating grid + and individual supplies buildings per medium voltage grid district. + National installed capacities are predefined from external sources within + :py:class:`ScenarioCapacities `. + The further distribution is done using a cascade that follows a specific order of supply technologies + and the heat demand. + + + *Dependencies* + * :py:class:`DataBundle ` + * :py:class:`DistrictHeatingAreas ` + * :py:class:`ZensusMvGridDistricts ` + * :py:class:`Chp ` + + + *Resulting tables* + * :py:class:`demand.egon_district_heating ` is created and filled + * :py:class:`demand.egon_individual_heating ` is created and filled + + """ + + #: + name: str = "HeatSupply" + #: + version: str = "0.0.9" + def __init__(self, dependencies): super().__init__( - name="HeatSupply", - version="0.0.8", + name=self.name, + version=self.version, dependencies=dependencies, tasks=( create_tables, @@ -180,5 +392,6 @@ def __init__(self, dependencies): individual_heating, potential_germany, }, + metadata, ), ) diff --git a/src/egon/data/datasets/hydrogen_etrago/__init__.py b/src/egon/data/datasets/hydrogen_etrago/__init__.py index a18264701..5ca7162dd 100755 --- a/src/egon/data/datasets/hydrogen_etrago/__init__.py +++ b/src/egon/data/datasets/hydrogen_etrago/__init__.py @@ -29,10 +29,10 @@ insert_power_to_h2_to_power_eGon100RE, ) from egon.data.datasets.hydrogen_etrago.storage import ( - calculate_and_map_saltcavern_storage_potential, insert_H2_overground_storage, insert_H2_saltcavern_storage, insert_H2_storage_eGon100RE, + write_saltcavern_potential, ) @@ -67,7 +67,7 @@ def __init__(self, dependencies): version=self.version, dependencies=dependencies, tasks=( - calculate_and_map_saltcavern_storage_potential, + write_saltcavern_potential, insert_hydrogen_buses, insert_hydrogen_buses_eGon100RE, ), diff --git a/src/egon/data/datasets/hydrogen_etrago/bus.py b/src/egon/data/datasets/hydrogen_etrago/bus.py index d780551b1..a2d101bf0 100755 --- a/src/egon/data/datasets/hydrogen_etrago/bus.py +++ b/src/egon/data/datasets/hydrogen_etrago/bus.py @@ -6,15 +6,19 @@ The H2 buses in the neighbouring countries (only present in eGon100RE) are defined in :py:mod:`pypsaeursec `. In both scenarios, there are two types of H2 buses in Germany: - * H2_grid buses: defined in :py:func:`insert_H2_buses_from_CH4_grid`, - these buses are located at the places than the CH4 buses. - * H2_saltcavern buses: defined in :py:func:`insert_H2_buses_from_saltcavern`, + * H2_grid buses: defined in :py:func:`insert_H2_buses_from_CH4_grid`; + these buses are located at the places of the CH4 buses. + * H2_saltcavern buses: defined in :py:func:`insert_H2_buses_from_saltcavern`; these buses are located at the intersection of AC buses and - potential for H2 saltcavern. + potential H2 saltcaverns. """ +import datetime +import json from geoalchemy2 import Geometry +from sqlalchemy import BigInteger, Column, Text +from sqlalchemy.ext.declarative import declarative_base from egon.data import config, db from egon.data.datasets.etrago_helpers import ( @@ -22,6 +26,13 @@ finalize_bus_insertion, initialise_bus_insertion, ) +from egon.data.metadata import ( + context, + contributors, + license_egon_data_odbl, + meta_metadata, + sources, +) def insert_hydrogen_buses(scenario="eGon2035"): @@ -37,6 +48,10 @@ def insert_hydrogen_buses(scenario="eGon2035"): ---------- scenario : str, optional Name of the scenario, the default is 'eGon2035'. + + Returns + ------- + None """ sources = config.datasets()["etrago_hydrogen"]["sources"] @@ -57,12 +72,95 @@ def insert_hydrogen_buses(scenario="eGon2035"): insert_H2_buses_from_CH4_grid(hydrogen_buses, carrier, target, scenario) +Base = declarative_base() + + +class EgonMapACH2(Base): + source_list = [ + sources()["openstreetmap"], + sources()["SciGRID_gas"], + sources()["bgr_inspeeds_data_bundle"], + ] + meta_ac_h2 = { + "name": "grid.egon_etrago_ac_h2", + "title": "Mapping table of AC-H2 buses", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": "Table mapping AC and H2 buses in Germany", + "language": ["en-EN"], + "publicationDate": datetime.date.today().isoformat(), + "context": context(), + "spatial": { + "location": None, + "extent": "Germany", + "resolution": None, + }, + "sources": source_list, + "licenses": [license_egon_data_odbl()], + "contributors": contributors(["fw"]), + "resources": [ + { + "profile": "tabular-data-resource", + "name": "grid.egon_etrago_ac_h2", + "path": None, + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": [ + { + "name": "scn_name", + "description": "Name of the scenario", + "type": "str", + "unit": None, + }, + { + "name": "bus_H2", + "description": "H2 bus_id", + "type": "integer", + "unit": None, + }, + { + "name": "bus_AC", + "description": "AC bus_id", + "type": "integer", + "unit": None, + }, + ], + "primaryKey": ["scn_name", "bus_H2"], + "foreignKeys": [], + }, + "dialect": {"delimiter": None, "decimalSeparator": "."}, + } + ], + "metaMetadata": meta_metadata(), + } + # Create json dump + meta_json_ac_h2 = ( + "'" + json.dumps(meta_ac_h2, indent=4, ensure_ascii=False) + "'" + ) + + __tablename__ = "egon_etrago_ac_h2" + __table_args__ = { + "schema": "grid", + "comment": meta_json_ac_h2, + } + + scn_name = Column(Text, primary_key=True, nullable=False) + bus_H2 = Column(BigInteger, primary_key=True, nullable=False) + bus_AC = Column(BigInteger, primary_key=False, nullable=False) + + +def create_AC_H2_table(): + engine = db.engine() + EgonMapACH2.__table__.drop(bind=engine, checkfirst=True) + EgonMapACH2.__table__.create(bind=engine, checkfirst=True) + + def insert_H2_buses_from_saltcavern(gdf, carrier, sources, target, scn_name): """ - Insert the H2 buses based saltcavern locations into the database. + Insert the H2 buses based on saltcavern locations into the database. These buses are located at the intersection of AC buses and - potential for H2 saltcavern. + potential H2 saltcaverns. Parameters ---------- @@ -76,6 +174,11 @@ def insert_H2_buses_from_saltcavern(gdf, carrier, sources, target, scn_name): Target schema and table information. scn_name : str Name of the scenario. + + Returns + ------- + None + """ # electrical buses related to saltcavern storage el_buses = db.select_dataframe( @@ -114,21 +217,104 @@ def insert_H2_buses_from_saltcavern(gdf, carrier, sources, target, scn_name): gdf_H2_cavern["bus_AC"] = AC_bus_ids gdf_H2_cavern["scn_name"] = hydrogen_bus_ids["scn_name"] + create_AC_H2_table() + # Insert data to db gdf_H2_cavern.to_sql( "egon_etrago_ac_h2", db.engine(), schema="grid", index=False, - if_exists="replace", + if_exists="append", ) +class EgonMapH2CH4(Base): + source_list = [ + sources()["openstreetmap"], + sources()["SciGRID_gas"], + sources()["bgr_inspeeds_data_bundle"], + ] + meta_H2_CH4 = { + "name": "grid.egon_etrago_ch4_h2", + "title": "Mapping table of CH4-H2 buses", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": "Table mapping CH4 and H2 buses in Germany", + "language": ["en-EN"], + "publicationDate": datetime.date.today().isoformat(), + "context": context(), + "spatial": { + "location": None, + "extent": "Germany", + "resolution": None, + }, + "sources": source_list, + "licenses": [license_egon_data_odbl()], + "contributors": contributors(["fw"]), + "resources": [ + { + "profile": "tabular-data-resource", + "name": "grid.egon_etrago_ch4_h2", + "path": None, + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": [ + { + "name": "scn_name", + "description": "Name of the scenario", + "type": "str", + "unit": None, + }, + { + "name": "bus_H2", + "description": "H2 bus_id", + "type": "integer", + "unit": None, + }, + { + "name": "bus_CH4", + "description": "CH4 bus_id", + "type": "integer", + "unit": None, + }, + ], + "primaryKey": ["scn_name", "bus_H2"], + "foreignKeys": [], + }, + "dialect": {"delimiter": None, "decimalSeparator": "."}, + } + ], + "metaMetadata": meta_metadata(), + } + + # Create json dump + meta_json_H2_CH4 = ( + "'" + json.dumps(meta_H2_CH4, indent=4, ensure_ascii=False) + "'" + ) + + __tablename__ = "egon_etrago_ch4_h2" + __table_args__ = { + "schema": "grid", + "comment": meta_json_H2_CH4, + } + + scn_name = Column(Text, primary_key=True, nullable=False) + bus_H2 = Column(BigInteger, primary_key=True, nullable=False) + bus_CH4 = Column(BigInteger, primary_key=False, nullable=False) + + +def create_H2_CH4_table(): + engine = db.engine() + EgonMapH2CH4.__table__.drop(bind=engine, checkfirst=True) + EgonMapH2CH4.__table__.create(bind=engine, checkfirst=True) + + def insert_H2_buses_from_CH4_grid(gdf, carrier, target, scn_name): """ Insert the H2 buses based on CH4 grid into the database. - At each CH4 location, in other words at each intersection of the CH4 + At each CH4 location, respectively at each intersection of the CH4 grid, a H2 bus is created. Parameters @@ -141,7 +327,11 @@ def insert_H2_buses_from_CH4_grid(gdf, carrier, target, scn_name): Target schema and table information. scn_name : str Name of the scenario. - + + Returns + ------- + None + """ # Connect to local database engine = db.engine() @@ -165,18 +355,28 @@ def insert_H2_buses_from_CH4_grid(gdf, carrier, target, scn_name): gdf_H2_CH4["bus_CH4"] = CH4_bus_ids["bus_id"] gdf_H2_CH4["scn_name"] = CH4_bus_ids["scn_name"] + create_H2_CH4_table() + # Insert data to db gdf_H2_CH4.to_sql( "egon_etrago_ch4_h2", engine, schema="grid", index=False, - if_exists="replace", + if_exists="append", ) def insert_hydrogen_buses_eGon100RE(): - """Copy H2 buses from the eGon2035 to the eGon100RE scenario.""" + """Copy H2 buses from the eGon2035 to the eGon100RE scenario. + + Returns + ------- + None + + """ copy_and_modify_buses( - "eGon2035", "eGon100RE", {"carrier": ["H2_grid", "H2_saltcavern"]}, + "eGon2035", + "eGon100RE", + {"carrier": ["H2_grid", "H2_saltcavern"]}, ) diff --git a/src/egon/data/datasets/hydrogen_etrago/h2_grid.py b/src/egon/data/datasets/hydrogen_etrago/h2_grid.py index 3646eac29..ff0776d26 100755 --- a/src/egon/data/datasets/hydrogen_etrago/h2_grid.py +++ b/src/egon/data/datasets/hydrogen_etrago/h2_grid.py @@ -2,13 +2,13 @@ The central module containing all code dealing with the H2 grid in eGon100RE The H2 grid, present only in eGon100RE, is composed of two parts: - * a fixed part with the same topology than the CH4 grid and with - carrier 'H2_retrofit' corresponding to the retrofiting of a share of - the CH4 grid into an hydrogen grid, + * a fixed part with the same topology as the CH4 grid and with + carrier 'H2_retrofit' corresponding to the retrofitted share of + the CH4 grid into a hydrogen grid, * an extendable part with carrier 'H2_gridextension', linking each - H2_salcavern bus to the closest H2_grid bus: this part as no - capacity (p_nom = 0) but it could be extended. -As the CH4 grid, the H2 pipelines are modelled by PyPSA links. + H2_salcavern bus to the closest H2_grid bus: this part has no + capacity (p_nom = 0) but it can be extended. +As for the CH4 grid, the H2 pipelines are modelled by PyPSA links. """ from geoalchemy2.types import Geometry @@ -30,16 +30,19 @@ def insert_h2_pipelines(): * bus0 and bus1 using the grid.egon_etrago_ch4_h2 table * carrier, scn_name * p_nom: the value attributed there corresponds to the share - of p_nom of the specific pipe that could be retrofited into - H2 pipe. This share is the same for every pipeline and is + of p_nom of the specific pipeline that could be retrofited into + H2 pipeline. This share is the same for every pipeline and is calculated in the PyPSA-eur-sec run. * Create new extendable pipelines to link the existing grid to the H2_saltcavern buses * Clean database * Attribute link_id to the links - * Insert the into the database + * Insert into the database - This function inserts data into the database and has no return. + Returns + ------- + None + """ H2_buses = db.select_geodataframe( diff --git a/src/egon/data/datasets/hydrogen_etrago/h2_to_ch4.py b/src/egon/data/datasets/hydrogen_etrago/h2_to_ch4.py index 1fb80335b..6c26caea6 100755 --- a/src/egon/data/datasets/hydrogen_etrago/h2_to_ch4.py +++ b/src/egon/data/datasets/hydrogen_etrago/h2_to_ch4.py @@ -2,8 +2,8 @@ """ Module containing the definition of the links between H2 and CH4 buses -In this module the functions used to define and insert into the database -the links between H2 and CH4 buses are to be found. +In this module the functions used to define and insert the links between +H2 and CH4 buses into the database are to be found. These links are modelling: * Methanisation (carrier name: 'H2_to_CH4'): technology to produce CH4 from H2 @@ -23,7 +23,7 @@ def insert_h2_to_ch4_to_h2(): """ - Inserts methanisation, feed in and SMR links into the database + Inserts methanisation, feedin and SMR links into the database Define the potentials for methanisation and Steam Methane Reaction (SMR) modelled as extendable links as well as the H2 feedin @@ -33,10 +33,12 @@ def insert_h2_to_ch4_to_h2(): The capacity of the H2_feedin links is considerated as constant and calculated as the sum of the capacities of the CH4 links connected - to the CH4 bus multiplied by the H2 energy share allowed to be fed. + to the CH4 bus multiplied by the H2 energy share allowed to be fed in. This share is calculated in the function :py:func:`H2_CH4_mix_energy_fractions`. - This function inserts data into the database and has no return. + Returns + ------- + None """ # Connect to local database diff --git a/src/egon/data/datasets/hydrogen_etrago/power_to_h2.py b/src/egon/data/datasets/hydrogen_etrago/power_to_h2.py index 72fdc6738..78e4a09b0 100755 --- a/src/egon/data/datasets/hydrogen_etrago/power_to_h2.py +++ b/src/egon/data/datasets/hydrogen_etrago/power_to_h2.py @@ -2,8 +2,8 @@ """ Module containing the definition of the AC grid to H2 links -In this module the functions used to define and insert into the database -the links between H2 and AC buses are to be found. +In this module the functions used to define and insert the links +between H2 and AC buses into the database are to be found. These links are modelling: * Electrolysis (carrier name: 'power_to_H2'): technology to produce H2 from AC @@ -34,13 +34,15 @@ def insert_power_to_h2_to_power(scn_name="eGon2035"): between the AC and the H2 bus is > 500m, the maximum capacity of the installation is limited to 1 MW. - This function inserts data into the database and has no return. - Parameters ---------- scn_name : str Name of the scenario + Returns + ------- + None + """ # Connect to local database @@ -229,7 +231,13 @@ def map_buses(scn_name): def insert_power_to_h2_to_power_eGon100RE(): - """Copy H2/power links from the eGon2035 to the eGon100RE scenario.""" + """Copy H2/power links from the eGon2035 to the eGon100RE scenario. + + Returns + ------- + None + + """ copy_and_modify_links( "eGon2035", "eGon100RE", ["H2_to_power", "power_to_H2"], "gas" ) diff --git a/src/egon/data/datasets/hydrogen_etrago/storage.py b/src/egon/data/datasets/hydrogen_etrago/storage.py index 55f6f7161..c85013d23 100755 --- a/src/egon/data/datasets/hydrogen_etrago/storage.py +++ b/src/egon/data/datasets/hydrogen_etrago/storage.py @@ -25,7 +25,11 @@ def insert_H2_overground_storage(scn_name="eGon2035"): Insert H2_overground stores into the database. Insert extendable H2_overground stores (steel tanks) at each H2_grid - bus. This function inserts data into the database and has no return. + bus. + + Returns + ------- + None """ # The targets of etrago_hydrogen also serve as source here ಠ_ಠ @@ -90,8 +94,11 @@ def insert_H2_saltcavern_storage(scn_name="eGon2035"): Insert H2_underground stores into the database. Insert extendable H2_underground stores (saltcavern potentials) at - every H2_saltcavern bus.This function inserts data into the database - and has no return. + every H2_saltcavern bus. + + Returns + ------- + None """ @@ -112,7 +119,7 @@ def insert_H2_saltcavern_storage(scn_name="eGon2035"): f""" SELECT * FROM {sources['H2_AC_map']['schema']}. - {sources['H2_AC_map']['table']}""", + {sources['H2_AC_map']['table']}""" ) storage_potentials["storage_potential"] = ( @@ -182,13 +189,14 @@ def calculate_and_map_saltcavern_storage_potential(): """ Calculate site specific storage potential based on InSpEE-DS report. - This function inserts data into the database and has no return. + Returns + ------- + None """ # select onshore vg250 data sources = config.datasets()["bgr"]["sources"] - targets = config.datasets()["bgr"]["targets"] vg250_data = db.select_geodataframe( f"""SELECT * FROM {sources['vg250_federal_states']['schema']}. @@ -366,7 +374,21 @@ def calculate_and_map_saltcavern_storage_potential(): epsg=25832 ).area / potential_areas.groupby("gen")["shape_star"].transform("sum") + return potential_areas + + +def write_saltcavern_potential(): + """Write saltcavern potentials into the database + + Returns + ------- + None + + """ + potential_areas = calculate_and_map_saltcavern_storage_potential() + # write information to saltcavern data + targets = config.datasets()["bgr"]["targets"] potential_areas.to_crs(epsg=4326).to_postgis( targets["storage_potential"]["table"], db.engine(), @@ -378,7 +400,13 @@ def calculate_and_map_saltcavern_storage_potential(): def insert_H2_storage_eGon100RE(): - """Copy H2 storage from the eGon2035 to the eGon100RE scenario.""" + """Copy H2 storage from the eGon2035 to the eGon100RE scenario. + + Returns + ------- + None + + """ copy_and_modify_stores( "eGon2035", "eGon100RE", ["H2_underground", "H2_overground"], "gas" ) diff --git a/src/egon/data/datasets/industrial_gas_demand.py b/src/egon/data/datasets/industrial_gas_demand.py index df25ee1a9..d57ccfcba 100755 --- a/src/egon/data/datasets/industrial_gas_demand.py +++ b/src/egon/data/datasets/industrial_gas_demand.py @@ -2,8 +2,8 @@ """ The central module containing code dealing with gas industrial demand -In this this module, the functions to import the industrial hydrogen and -methane demands from the opendata.ffe database and to insert them in +In this module, the functions to import the industrial hydrogen and +methane demands from the opendata.ffe database and to insert them into the database after modification are to be found. """ @@ -33,7 +33,7 @@ class IndustrialGasDemand(Dataset): """ Download the industrial gas demands from the opendata.ffe database - Data are downloaded in the folder ./datasets/gas_data/demand using + Data is downloaded to the folder ./datasets/gas_data/demand using the function :py:func:`download_industrial_gas_demand` and no dataset is resulting. *Dependencies* @@ -232,9 +232,9 @@ def read_and_process_demand( following steps: * Read the industrial demand time series in Germany with the - fonction :py:func:`read_industrial_demand` + function :py:func:`read_industrial_demand` * Attribute the bus_id to which each load and it associated time - serie is associated by calling the function :py:func:`assign_gas_bus_id ` + series is associated by calling the function :py:func:`assign_gas_bus_id ` from :py:mod:`egon.data.db ` * Adjust the columns: add "carrier" and remove useless ones @@ -287,13 +287,15 @@ def delete_old_entries(scn_name): """ Delete CH4 and H2 loads and load time series for the specified scenario - This function cleans the database and has no return. - Parameters ---------- scn_name : str Name of the scenario. + Returns + ------- + None + """ # Clean tables db.execute_sql( @@ -329,10 +331,10 @@ def insert_new_entries(industrial_gas_demand, scn_name): """ Insert industrial gas loads into the database - This function prepares and imports the industrial gas loads, by + This function prepares and imports the industrial gas loads by executing the following steps: - * Attribution of an id to each load in the list received as paramater + * Attribution of an id to each load in the list received as parameter * Deletion of the column containing the time series (they will be inserted in another table (grid.egon_etrago_load_timeseries) in the :py:func:`insert_industrial_gas_demand_time_series`) @@ -385,13 +387,13 @@ def insert_industrial_gas_demand_egon2035(): """Insert industrial gas demands into the database for eGon2035 Insert the industrial CH4 and H2 demands and their associated time - series into the database for the eGon2035 scenario. The data, + series into the database for the eGon2035 scenario. The data previously downloaded in :py:func:`download_industrial_gas_demand` - are adjusted by executing the following steps: + is adjusted by executing the following steps: - * Clean the database with the fonction :py:func:`delete_old_entries` + * Clean the database with the function :py:func:`delete_old_entries` * Read and prepare the CH4 and the H2 industrial demands and their - associated time series in Germany with the fonction :py:func:`read_and_process_demand` + associated time series in Germany with the function :py:func:`read_and_process_demand` * Aggregate the demands with the same properties at the same gas bus * Insert the loads into the database by executing :py:func:`insert_new_entries` * Insert the time series associated to the loads into the database @@ -438,12 +440,12 @@ def insert_industrial_gas_demand_egon100RE(): previously downloaded in :py:func:`download_industrial_gas_demand` are adapted by executing the following steps: - * Clean the database with the fonction :py:func:`delete_old_entries` + * Clean the database with the function :py:func:`delete_old_entries` * Read and prepare the CH4 and the H2 industrial demands and their - associated time series in Germany with the fonction :py:func:`read_and_process_demand` + associated time series in Germany with the function :py:func:`read_and_process_demand` * Identify and adjust the total industrial CH4 and H2 loads for Germany generated by PyPSA-Eur-Sec - * For the CH4, the time serie used is the one from H2, because + * For CH4, the time series used is the one from H2, because the industrial CH4 demand in the opendata.ffe database is 0 * In test mode, the total values are obtained by evaluating the share of H2 demand in the test region @@ -562,15 +564,15 @@ def insert_industrial_gas_demand_egon100RE(): def insert_industrial_gas_demand_time_series(egon_etrago_load_gas): """ - Insert list of industrial gas demand time series (one per NUTS3) + Insert list of industrial gas demand time series (one per NUTS3 region) - These loads are hourly and NUTS3-geographic resolved. + These loads are hourly and on NUTS3 level resolved. Parameters ---------- industrial_gas_demand : pandas.DataFrame - Dataframe containing the loads that have been inserted in - the database and whose time serie will be inserted into the + Dataframe containing the loads that have been inserted into + the database and whose time series will be inserted into the database. Returns @@ -603,11 +605,14 @@ def download_industrial_gas_demand(): """Download the industrial gas demand data from opendata.ffe database The industrial demands for hydrogen and methane are downloaded in - the folder ./datasets/gas_data/demand and the function has no - return. - These loads are hourly and NUTS3-geographic resolved. For more + the folder ./datasets/gas_data/demand + These loads are hourly and NUTS3-level resolved. For more information on these data, refer to the `Extremos project documentation `_. + Returns + ------- + None + """ correspondance_url = ( "http://opendata.ffe.de:3000/region?id_region_type=eq.38" diff --git a/src/egon/data/datasets/loadarea/__init__.py b/src/egon/data/datasets/loadarea/__init__.py index 0cda809d2..c5512f174 100644 --- a/src/egon/data/datasets/loadarea/__init__.py +++ b/src/egon/data/datasets/loadarea/__init__.py @@ -1,21 +1,6 @@ """ OSM landuse extraction and load areas creation. -**Landuse** - -* Landuse data is extracted from OpenStreetMap: residential, retail, - industrial, Agricultural -* Data is cut with German borders (VG 250), data outside is dropped -* Invalid geometries are fixed -* Results are stored in table `openstreetmap.osm_landuse` - -**Load Areas** - -TBD - -Note: industrial demand contains: -* voltage levels 4-7 -* only demand from ind. sites+osm located in LA! """ import os @@ -36,6 +21,9 @@ class OsmPolygonUrban(Base): + """ + Class definition of table openstreetmap.osm_landuse. + """ __tablename__ = "osm_landuse" __table_args__ = {"schema": "openstreetmap"} id = Column(Integer, primary_key=True) @@ -50,10 +38,30 @@ class OsmPolygonUrban(Base): class OsmLanduse(Dataset): + """ + OSM landuse extraction. + + * Landuse data is extracted from OpenStreetMap: residential, retail, + industrial, Agricultural + * Data is cut with German borders (VG 250), data outside is dropped + * Invalid geometries are fixed + * Results are stored in table `openstreetmap.osm_landuse` + + Note: industrial demand contains: + * voltage levels 4-7 + * only demand from ind. sites+osm located in LA! + + """ + + #: + name: str = "OsmLanduse" + #: + version: str = "0.0.0" + def __init__(self, dependencies): super().__init__( - name="OsmLanduse", - version="0.0.0", + name=self.name, + version=self.version, dependencies=dependencies, tasks=( create_landuse_table, @@ -70,10 +78,41 @@ def __init__(self, dependencies): class LoadArea(Dataset): + """ + Creates load area data based on OSM and census data. + + *Dependencies* + * :py:class:`OsmLanduse ` + * :py:class:`ZensusVg250 ` + * :py:class:`HouseholdElectricityDemand ` + * :py:func:`get_building_peak_loads ` + * :py:class:`CtsDemandBuildings ` + * :py:class:`IndustrialDemandCurves ` + + *Resulting tables* + * :class:`demand.egon_loadarea` is created and filled (no associated Python class) + + Create and update the `demand.egon_loadarea` table with new data, based on OSM and + census data. Among other things, area updates are carried out, smaller load areas + are removed, center calculations are performed, and census data are added. + Statistics for various OSM sectors are also calculated and inserted. + See also documentation section :ref:`load-areas-ref` for more information. + + Note: industrial demand contains: + * voltage levels 4-7 + * only demand from ind. sites+osm located in LA! + + """ + + #: + name: str = "LoadArea" + #: + version: str = "0.0.1" + def __init__(self, dependencies): super().__init__( - name="LoadArea", - version="0.0.1", + name=self.name, + version=self.version, dependencies=dependencies, tasks=( osm_landuse_melt, diff --git a/src/egon/data/datasets/mastr.py b/src/egon/data/datasets/mastr.py index 32f38b3d7..0881e9dd3 100644 --- a/src/egon/data/datasets/mastr.py +++ b/src/egon/data/datasets/mastr.py @@ -1,25 +1,8 @@ """ -Download Marktstammdatenregister (MaStR) datasets unit registry. -It incorporates two different datasets: - -Dump 2021-05-03 -* Source: https://sandbox.zenodo.org/record/808086 -* Used technologies: PV plants, wind turbines, biomass, hydro plants, - combustion, nuclear, gsgk, storage -* Data is further processed in dataset - :py:class:`egon.data.datasets.power_plants.PowerPlants` - -Dump 2022-11-17 -* Source: https://sandbox.zenodo.org/record/1132839 -* Used technologies: PV plants, wind turbines, biomass, hydro plants -* Data is further processed in module - :py:mod:`egon.data.datasets.power_plants.mastr` `PowerPlants` - -Todo: Finish docstring -TBD +Download Marktstammdatenregister (MaStR) from Zenodo. + """ -from functools import partial from pathlib import Path from urllib.request import urlretrieve import os @@ -32,7 +15,7 @@ def download_mastr_data(): - """Download MaStR data from Zenodo""" + """Download MaStR data from Zenodo.""" def download(dataset_name, download_dir): print(f"Downloading dataset {dataset_name} to {download_dir} ...") @@ -66,10 +49,44 @@ def download(dataset_name, download_dir): download(dataset_name="mastr_new", download_dir=WORKING_DIR_MASTR_NEW) -mastr_data_setup = partial( - Dataset, - name="MastrData", - version="0.0.1", - dependencies=[], - tasks=(download_mastr_data,), -) +class mastr_data_setup(Dataset): + """ + Download Marktstammdatenregister (MaStR) from Zenodo. + + *Dependencies* + * :py:func:`Setup ` + + The downloaded data incorporates two different datasets: + + Dump 2021-05-03 + * Source: https://sandbox.zenodo.org/record/808086 + * Used technologies: PV plants, wind turbines, biomass, hydro plants, + combustion, nuclear, gsgk, storage + * Data is further processed in the :py:class:`PowerPlants + ` dataset + + Dump 2022-11-17 + * Source: https://sandbox.zenodo.org/record/1132839 + * Used technologies: PV plants, wind turbines, biomass, hydro plants + * Data is further processed in module :py:mod:`mastr + ` and :py:class:`PowerPlants + ` + + See documentation section :ref:`mastr-ref` for more information. + + """ + + #: + name: str = "MastrData" + #: + version: str = "0.0.1" + #: + tasks = (download_mastr_data,) + + def __init__(self, dependencies): + super().__init__( + name=self.name, + version=self.version, + dependencies=dependencies, + tasks=self.tasks, + ) diff --git a/src/egon/data/datasets/mv_grid_districts.py b/src/egon/data/datasets/mv_grid_districts.py index bba34ee9e..d13e5fb5d 100644 --- a/src/egon/data/datasets/mv_grid_districts.py +++ b/src/egon/data/datasets/mv_grid_districts.py @@ -1,66 +1,10 @@ """ -Medium-voltage grid districts describe the area supplied by one MV grid - -Medium-voltage grid districts are defined by one polygon that represents the -supply area. Each MV grid district is connected to the HV grid via a single -substation. - -The methods used for identifying the MV grid districts are heavily inspired -by `Hülk et al. (2017) -`_ -(section 2.3), but the implementation differs in detail. -The main difference is that direct adjacency is preferred over proximity. -For polygons of municipalities -without a substation inside, it is iteratively checked for direct adjacent -other polygons that have a substation inside. Speaking visually, a MV grid -district grows around a polygon with a substation inside. - -The grid districts are identified using three data sources - -1. Polygons of municipalities (:class:`Vg250GemClean`) -2. HV-MV substations (:class:`EgonHvmvSubstation`) -3. HV-MV substation voronoi polygons (:class:`EgonHvmvSubstationVoronoi`) - -Fundamentally, it is assumed that grid districts (supply areas) often go -along borders of administrative units, in particular along the borders of -municipalities due to the concession levy. -Furthermore, it is assumed that one grid district is supplied via a single -substation and that locations of substations and grid districts are designed -for aiming least lengths of grid line and cables. - -With these assumptions, the three data sources from above are processed as -follows: - -* Find the number of substations inside each municipality -* Split municipalities with more than one substation inside - * Cut polygons of municipalities with voronoi polygons of respective - substations - * Assign resulting municipality polygon fragments to nearest substation -* Assign municipalities without a single substation to nearest substation in - the neighborhood -* Merge all municipality polygons and parts of municipality polygons to a - single polygon grouped by the assigned substation - -For finding the nearest substation, as already said, direct adjacency is -preferred over closest distance. This means, the nearest substation does not -necessarily have to be the closest substation in the sense of beeline distance. -But it is the substation definitely located in a neighboring polygon. This -prevents the algorithm to find solutions where a MV grid districts consists of -multi-polygons with some space in between. -Nevertheless, beeline distance still plays an important role, as the algorithm -acts in two steps - -1. Iteratively look for neighboring polygons until there are no further - polygons -2. Find a polygon to assign to by minimum beeline distance - -The second step is required in order to cover edge cases, such as islands. - -For understanding how this is implemented into separate functions, please -see :func:`define_mv_grid_districts`. -""" +The module containing all code to generate MV grid district polygons. + +Medium-voltage grid districts describe the area supplied by one MV grid and +are defined by one polygon that represents the supply area. -from functools import partial +""" from geoalchemy2.types import Geometry from sqlalchemy import ( @@ -87,6 +31,10 @@ class Vg250GemClean(Base): + """ + Class definition of table boundaries.vg250_gem_clean. + """ + __tablename__ = "vg250_gem_clean" __table_args__ = {"schema": "boundaries"} @@ -106,6 +54,10 @@ class Vg250GemClean(Base): class HvmvSubstPerMunicipality(Base): + """ + Class definition of temporary table grid.hvmv_subst_per_municipality. + """ + __tablename__ = "hvmv_subst_per_municipality" __table_args__ = {"schema": "grid"} @@ -136,6 +88,10 @@ class VoronoiMunicipalityCutsBase(object): class VoronoiMunicipalityCuts(VoronoiMunicipalityCutsBase, Base): + """ + Class definition of temporary table grid.voronoi_municipality_cuts. + """ + __tablename__ = "voronoi_municipality_cuts" __table_args__ = {"schema": "grid"} @@ -147,6 +103,11 @@ class VoronoiMunicipalityCuts(VoronoiMunicipalityCutsBase, Base): class VoronoiMunicipalityCutsAssigned(VoronoiMunicipalityCutsBase, Base): + """ + Class definition of temporary table + grid.voronoi_municipality_cuts_assigned. + """ + __tablename__ = "voronoi_municipality_cuts_assigned" __table_args__ = {"schema": "grid"} @@ -159,6 +120,10 @@ class VoronoiMunicipalityCutsAssigned(VoronoiMunicipalityCutsBase, Base): class MvGridDistrictsDissolved(Base): + """ + Class definition of temporary table grid.egon_mv_grid_district_dissolved. + """ + __tablename__ = "egon_mv_grid_district_dissolved" __table_args__ = {"schema": "grid"} @@ -173,6 +138,10 @@ class MvGridDistrictsDissolved(Base): class MvGridDistricts(Base): + """ + Class definition of table grid.egon_mv_grid_district. + """ + __tablename__ = "egon_mv_grid_district" __table_args__ = {"schema": "grid"} @@ -183,14 +152,17 @@ class MvGridDistricts(Base): def substations_in_municipalities(): """ - Create a table that counts number of HV-MV substations in each MV grid + Create a table that counts number of HV-MV substations in each MV grid. - Counting is performed in two steps + Counting is performed in two steps: 1. HV-MV substations are spatially joined on municipalities, grouped by - municipality and number of substations counted + municipality and number of substations counted. 2. Because (1) works only for number of substations >0, all municipalities - not containing a substation, are added + not containing a substation, are added. + + Data is written to temporary table grid.hvmv_subst_per_municipality. + """ engine = db.engine() HvmvSubstPerMunicipality.__table__.drop(bind=engine, checkfirst=True) @@ -251,7 +223,7 @@ def substations_in_municipalities(): def split_multi_substation_municipalities(): """ - Split municipalities that have more than one substation + Split municipalities that have more than one substation. Municipalities that contain more than one HV-MV substation in their polygon are cut by HV-MV voronoi polygons. Resulting fragments are then @@ -259,13 +231,15 @@ def split_multi_substation_municipalities(): In detail, the following steps are performed: - * Step 1: cut municipalities with voronoi polygons - * Step 2: Determine number of substations inside cut polygons - * Step 3: separate cut polygons with exactly one substation inside + * Step 1: Cut municipalities with voronoi polygons. + * Step 2: Determine number of substations inside cut polygons. + * Step 3: Separate cut polygons with exactly one substation inside. * Step 4: Assign polygon without a substation to next neighboring - polygon with a substation - * Step 5: Assign remaining polygons that are non-touching + polygon with a substation. + * Step 5: Assign remaining polygons that are non-touching. + Data is written to temporary tables grid.voronoi_municipality_cuts and + grid.voronoi_municipality_cuts_assigned. """ engine = db.engine() @@ -378,7 +352,7 @@ def split_multi_substation_municipalities(): # without any space in between (aka. polygons touch each other) # Initialize with very large number - remaining_polygons = [10 ** 10] + remaining_polygons = [10**10] while True: # This loop runs until all polygon that inital haven't had a @@ -464,12 +438,13 @@ def assign_substation_municipality_fragments( * "within": Only polygons within a radius of 100 km of polygons without substation are considered for assignment session: SQLAlchemy session - SQLAlchemy session obejct + SQLAlchemy session object - See Also + Notes -------- - The function :func:`nearest_polygon_with_substation` is very similar, but - different in detail. + The function :py:func:`nearest_polygon_with_substation` is very similar, + but different in detail. + """ # Determine nearest neighboring polygon that has a substation columns_from_cut1_subst = ["bus_id", "subst_count", "geom_sub"] @@ -547,18 +522,23 @@ def assign_substation_municipality_fragments( def merge_polygons_to_grid_district(): """ - Merge municipality polygon (parts) to MV grid districts + Merge municipality polygon (parts) to MV grid districts. Polygons of municipalities and cut parts of such polygons are merged to a single grid district per one HV-MV substation. Prior determined assignment of cut polygons parts is used as well as proximity of entire municipality polygons to polygons with a substation inside. - * Step 1: Merge municipality parts that are assigned to the same substation - * Step 2: Insert municipality polygons with exactly one substation + * Step 1: Merge municipality parts that are assigned to the same + substation. + * Step 2: Insert municipality polygons with exactly one substation. * Step 3: Assign municipality polygons without a substation and insert - to table - * Step 4: Merge MV grid district parts + to table. + * Step 4: Merge MV grid district parts. + + Data is written to table grid.egon_mv_grid_district and + to temporary table grid.egon_mv_grid_district_dissolved. + """ engine = db.engine() @@ -686,7 +666,7 @@ def nearest_polygon_with_substation( with_substation, without_substation, strategy, session ): """ - Assign next neighboring polygon + Assign next neighboring polygon. For municipalities without a substation inside their polygon the next MV grid district (part) polygon is found and assigned. @@ -708,13 +688,13 @@ def nearest_polygon_with_substation( * "within": Only polygons within a radius of 100 km of polygons without substation are considered for assignment session: SQLAlchemy session - SQLAlchemy session obejct + SQLAlchemy session object Returns ------- list IDs of polygons that were already assigned to a polygon with a - substation + substation. """ if strategy == "touches": neighboring_criterion = func.ST_Touches( @@ -787,17 +767,17 @@ def nearest_polygon_with_substation( def define_mv_grid_districts(): """ - Define spatial extent of MV grid districts + Define spatial extent of MV grid districts. The process of identifying the boundary of medium-voltage grid districts - is organized in three steps + is organized in three steps: 1. :func:`substations_in_municipalities`: The number of substations - located inside each municipality is calculated + located inside each municipality is calculated. 2. :func:`split_multi_substation_municipalities`: The municipalities with - >1 substation inside are split by Voronoi polygons around substations + >1 substation inside are split by Voronoi polygons around substations. 3. :func:`merge_polygons_to_grid_district`: All polygons are merged such - that one polygon has exactly one single substation inside + that one polygon has exactly one single substation inside. Finally, intermediate tables used for storing data temporarily are deleted. """ @@ -814,10 +794,34 @@ def define_mv_grid_districts(): MvGridDistrictsDissolved.__table__.drop(bind=engine, checkfirst=True) -mv_grid_districts_setup = partial( - Dataset, - name="MvGridDistricts", - version="0.0.2", - dependencies=[], - tasks=(define_mv_grid_districts), -) +class mv_grid_districts_setup(Dataset): + """ + Sets up medium-voltage grid districts that describe the area supplied by one + MV grid. + + See documentation section :ref:`mv-grid-districts` for more information. + + *Dependencies* + * :py:class:`SubstationVoronoi + ` + + *Resulting tables* + * :py:class:`grid.egon_mv_grid_district ` + is created and filled + * :py:class:`boundaries.vg250_gem_clean ` + is created and filled + + """ + + #: + name: str = "MvGridDistricts" + #: + version: str = "0.0.2" + + def __init__(self, dependencies): + super().__init__( + name=self.name, + version=self.version, + dependencies=dependencies, + tasks=define_mv_grid_districts, + ) diff --git a/src/egon/data/datasets/osm/__init__.py b/src/egon/data/datasets/osm/__init__.py index b0cfaa11b..635e00e5e 100644 --- a/src/egon/data/datasets/osm/__init__.py +++ b/src/egon/data/datasets/osm/__init__.py @@ -296,10 +296,34 @@ def modify_tables(): class OpenStreetMap(Dataset): + """ + Downloads OpenStreetMap data from Geofabrik and writes it to database. + + *Dependencies* + * :py:func:`Setup ` + + *Resulting Tables* + * openstreetmap.osm_line is created and filled (table has no associated python class) + * openstreetmap.osm_nodes is created and filled (table has no associated python class) + * openstreetmap.osm_point is created and filled (table has no associated python class) + * openstreetmap.osm_polygon is created and filled (table has no associated python class) + * openstreetmap.osm_rels is created and filled (table has no associated python class) + * openstreetmap.osm_roads is created and filled (table has no associated python class) + * openstreetmap.osm_ways is created and filled (table has no associated python class) + + See documentation section :ref:`osm-ref` for more information. + + """ + + #: + name: str = "OpenStreetMap" + #: + version: str = "0.0.4" + def __init__(self, dependencies): super().__init__( - name="OpenStreetMap", - version="0.0.4", + name=self.name, + version=self.version, dependencies=dependencies, tasks=(download, to_postgres, modify_tables, add_metadata), ) diff --git a/src/egon/data/datasets/osm_buildings_streets/__init__.py b/src/egon/data/datasets/osm_buildings_streets/__init__.py index cfa3aacf6..e0cb8a56f 100644 --- a/src/egon/data/datasets/osm_buildings_streets/__init__.py +++ b/src/egon/data/datasets/osm_buildings_streets/__init__.py @@ -1,60 +1,6 @@ """ -Filtered and preprocessed buildings, streets and amenities from OpenStreetMap -(OSM) - -This dataset on buildings and amenities is required by several tasks in the -pipeline, such as the distribution of household demand profiles or PV home -systems to buildings. This data is enriched by population and apartments from -Zensus 2011. Those derived datasets and the data on streets will be used in the -DIstribution Network Generat0r :ref:`ding0 -`_ e.g. to cluster loads and create low -voltage grids. - -**Details and Steps** - -* Extract buildings and filter using relevant tags, e.g. residential and - commercial, see script `osm_buildings_filter.sql` for the full list of tags. - Resulting tables: - * All buildings: `openstreetmap.osm_buildings` - * Filtered buildings: `openstreetmap.osm_buildings_filtered` - * Residential buildings: `openstreetmap.osm_buildings_residential` -* Extract amenities and filter using relevant tags, e.g. shops and restaurants, - see script `osm_amenities_shops_preprocessing.sql` for the full list of tags. - Resulting table: `openstreetmap.osm_amenities_shops_filtered` -* Create a mapping table for building's osm IDs to the Zensus cells the - building's centroid is located in. - Resulting tables: - * `boundaries.egon_map_zensus_buildings_filtered` (filtered) - * `boundaries.egon_map_zensus_buildings_residential` (residential only) -* Enrich each building by number of apartments from Zensus table - `society.egon_destatis_zensus_apartment_building_population_per_ha` - by splitting up the cell's sum equally to the buildings. In some cases, a - Zensus cell does not contain buildings but there's a building nearby which - the no. of apartments is to be allocated to. To make sure apartments are - allocated to at least one building, a radius of 77m is used to catch building - geometries. -* Split filtered buildings into 3 datasets using the amenities' locations: - temporary tables are created in script `osm_buildings_temp_tables.sql` the - final tables in `osm_buildings_amentities_results.sql`. - Resulting tables: - * Buildings w/ amenities: `openstreetmap.osm_buildings_with_amenities` - * Buildings w/o amenities: `openstreetmap.osm_buildings_without_amenities` - * Amenities not allocated to buildings: - `openstreetmap.osm_amenities_not_in_buildings` -* Extract streets (OSM ways) and filter using relevant tags, e.g. - highway=secondary, see script `osm_ways_preprocessing.sql` for the full list - of tags. Additionally, each way is split into its line segments and their - lengths is retained. - Resulting tables: - * Filtered streets: `openstreetmap.osm_ways_preprocessed` - * Filtered streets w/ segments: `openstreetmap.osm_ways_with_segments` - -Notes ------ - -This module docstring is rather a dataset documentation. Once, a decision -is made in ... the content of this module docstring needs to be moved to -docs attribute of the respective dataset class. +Filtering and preprocessing of buildings, streets and amenities from OpenStreetMap + """ import os @@ -151,10 +97,86 @@ def add_metadata(): class OsmBuildingsStreets(Dataset): + """ + Filter and preprocess buildings, streets and amenities from OpenStreetMap (OSM). + + This dataset on buildings and amenities is required by several tasks in the + pipeline, such as the distribution of household demand profiles or PV home + systems to buildings. This data is enriched by population and apartments from + Zensus 2011. Those derived datasets and the data on streets will be used in the + DIstribution Network Generat0r + `ding0 `_ e.g. to cluster loads and create low + voltage grids. + + *Dependencies* + * :py:class:`OpenStreetMap ` + * :py:class:`ZensusMiscellaneous ` + + *Resulting Tables* + * openstreetmap.osm_buildings is created and filled (table has no associated python class) + * openstreetmap.osm_buildings_filtered is created and filled (table has no associated python class) + * openstreetmap.osm_buildings_residential is created and filled (table has no associated python class) + * openstreetmap.osm_amenities_shops_filtered is created and filled (table has no associated python class) + * openstreetmap.osm_buildings_with_amenities is created and filled (table has no associated python class) + * openstreetmap.osm_buildings_without_amenities is created and filled (table has no associated python class) + * openstreetmap.osm_amenities_not_in_buildings is created and filled (table has no associated python class) + * openstreetmap.osm_ways_preprocessed is created and filled (table has no associated python class) + * openstreetmap.osm_ways_with_segments is created and filled (table has no associated python class) + * boundaries.egon_map_zensus_buildings_filtered is created and filled (table has no associated python class) + * boundaries.egon_map_zensus_buildings_residential is created and filled (table has no associated python class) + * openstreetmap.osm_buildings is created and filled (table has no associated python class) + + **Details and Steps** + + * Extract buildings and filter using relevant tags, e.g. residential and + commercial, see script `osm_buildings_filter.sql` for the full list of tags. + Resulting tables: + * All buildings: `openstreetmap.osm_buildings` + * Filtered buildings: `openstreetmap.osm_buildings_filtered` + * Residential buildings: `openstreetmap.osm_buildings_residential` + * Extract amenities and filter using relevant tags, e.g. shops and restaurants, + see script `osm_amenities_shops_preprocessing.sql` for the full list of tags. + Resulting table: `openstreetmap.osm_amenities_shops_filtered` + * Create a mapping table for building's osm IDs to the Zensus cells the + building's centroid is located in. + Resulting tables: + * `boundaries.egon_map_zensus_buildings_filtered` (filtered) + * `boundaries.egon_map_zensus_buildings_residential` (residential only) + * Enrich each building by number of apartments from Zensus table + `society.egon_destatis_zensus_apartment_building_population_per_ha` + by splitting up the cell's sum equally to the buildings. In some cases, a + Zensus cell does not contain buildings but there's a building nearby which + the no. of apartments is to be allocated to. To make sure apartments are + allocated to at least one building, a radius of 77m is used to catch building + geometries. + * Split filtered buildings into 3 datasets using the amenities' locations: + temporary tables are created in script `osm_buildings_temp_tables.sql` the + final tables in `osm_buildings_amentities_results.sql`. + Resulting tables: + + * Buildings w/ amenities: `openstreetmap.osm_buildings_with_amenities` + * Buildings w/o amenities: `openstreetmap.osm_buildings_without_amenities` + * Amenities not allocated to buildings: + `openstreetmap.osm_amenities_not_in_buildings` + * Extract streets (OSM ways) and filter using relevant tags, e.g. + highway=secondary, see script `osm_ways_preprocessing.sql` for the full list + of tags. Additionally, each way is split into its line segments and their + lengths is retained. + Resulting tables: + * Filtered streets: `openstreetmap.osm_ways_preprocessed` + * Filtered streets w/ segments: `openstreetmap.osm_ways_with_segments` + + """ + + #: + name: str = "OsmBuildingsStreets" + #: + version: str = "0.0.6" + def __init__(self, dependencies): super().__init__( - name="OsmBuildingsStreets", - version="0.0.6", + name=self.name, + version=self.version, dependencies=dependencies, tasks=( preprocessing, diff --git a/src/egon/data/datasets/power_etrago/match_ocgt.py b/src/egon/data/datasets/power_etrago/match_ocgt.py index f5cf29c53..bef670bbc 100755 --- a/src/egon/data/datasets/power_etrago/match_ocgt.py +++ b/src/egon/data/datasets/power_etrago/match_ocgt.py @@ -19,6 +19,11 @@ def insert_open_cycle_gas_turbines(scn_name="eGon2035"): ---------- scn_name : str Name of the scenario. + + Returns + ---------- + None + """ # Connect to local database diff --git a/src/egon/data/datasets/power_plants/__init__.py b/src/egon/data/datasets/power_plants/__init__.py index 71213e66f..8a293a678 100755 --- a/src/egon/data/datasets/power_plants/__init__.py +++ b/src/egon/data/datasets/power_plants/__init__.py @@ -33,6 +33,7 @@ ) import egon.data.config import egon.data.datasets.power_plants.assign_weather_data as assign_weather_data # noqa: E501 +import egon.data.datasets.power_plants.metadata as pp_metadata import egon.data.datasets.power_plants.pv_ground_mounted as pv_ground_mounted import egon.data.datasets.power_plants.wind_farms as wind_onshore import egon.data.datasets.power_plants.wind_offshore as wind_offshore @@ -56,10 +57,50 @@ class EgonPowerPlants(Base): class PowerPlants(Dataset): + """ + This module creates all electrical generators for different scenarios. It + also calculates the weather area for each weather dependent generator. + + *Dependencies* + * :py:class:`Chp ` + * :py:class:`CtsElectricityDemand + ` + * :py:class:`HouseholdElectricityDemand + ` + * :py:class:`mastr_data ` + * :py:func:`define_mv_grid_districts + ` + * :py:class:`RePotentialAreas + ` + * :py:class:`ZensusVg250 + ` + * :py:class:`ScenarioCapacities + ` + * :py:class:`ScenarioParameters + ` + * :py:func:`Setup ` + * :py:class:`substation_extraction + ` + * :py:class:`Vg250MvGridDistricts + ` + * :py:class:`ZensusMvGridDistricts + ` + + *Resulting tables* + * :py:class:`supply.egon_power_plants + ` is filled + + """ + + #: + name: str = "PowerPlants" + #: + version: str = "0.0.18" + def __init__(self, dependencies): super().__init__( - name="PowerPlants", - version="0.0.18", + name=self.name, + version=self.version, dependencies=dependencies, tasks=( create_tables, @@ -77,6 +118,7 @@ def __init__(self, dependencies): }, wind_offshore.insert, assign_weather_data.weatherId_and_busId, + pp_metadata.metadata, ), ) @@ -347,7 +389,6 @@ def insert_hydro_plants(scenario): } for carrier in map_carrier.keys(): - # import target values target = select_target(carrier, scenario) @@ -500,9 +541,7 @@ def assign_voltage_level(mastr_loc, cfg, mastr_working_dir): def assign_voltage_level_by_capacity(mastr_loc): - for i, row in mastr_loc[mastr_loc.voltage_level.isnull()].iterrows(): - if row.Nettonennleistung > 120: level = 1 elif row.Nettonennleistung > 20: @@ -610,7 +649,6 @@ def insert_hydro_biomass(): def allocate_conventional_non_chp_power_plants(): - carrier = ["oil", "gas"] cfg = egon.data.config.datasets()["power_plants"] @@ -625,14 +663,12 @@ def allocate_conventional_non_chp_power_plants(): ) for carrier in carrier: - nep = select_nep_power_plants(carrier) if nep.empty: print(f"DataFrame from NEP for carrier {carrier} is empty!") else: - mastr = select_no_chp_combustion_mastr(carrier) # Assign voltage level to MaStR @@ -771,7 +807,6 @@ def allocate_conventional_non_chp_power_plants(): def allocate_other_power_plants(): - # Get configuration cfg = egon.data.config.datasets()["power_plants"] boundary = egon.data.config.settings()["egon-data"]["--dataset-boundary"] diff --git a/src/egon/data/datasets/power_plants/metadata.py b/src/egon/data/datasets/power_plants/metadata.py new file mode 100644 index 000000000..91fe353e6 --- /dev/null +++ b/src/egon/data/datasets/power_plants/metadata.py @@ -0,0 +1,266 @@ +"""Metadata for power plants table +""" +import datetime +import json + +from egon.data import db +from egon.data.metadata import ( + license_agpl, + license_ccby, + licenses_datenlizenz_deutschland, +) + + +def metadata(): + """Add metdata to power plants table + + Returns + ------- + None. + + """ + meta = { + "name": "supply.egon_power_plants", + "title": "supply.egon_power_plants", + "id": "", + "description": "Database of powerplants ", + "language": "en-GB", + "keywords": [], + "publicationDate": datetime.date.today().isoformat(), + "context": { + "homepage": "https://ego-n.org/tools_data/", + "documentation": "https://egon-data.readthedocs.io/en/latest/", + "sourceCode": "https://github.com/openego/eGon-data", + "contact": "https://ego-n.org/partners/", + "grantNo": "03EI1002", + "fundingAgency": "Bundesministerium für Wirtschaft und Energie", + "fundingAgencyLogo": "https://www.innovation-beratung-foerderung.de/INNO/Redaktion/DE/Bilder/Titelbilder/titel_foerderlogo_bmwi.jpg?__blob=normal&v=3", + "publisherLogo": "https://ego-n.org/images/eGon_logo_noborder_transbg.svg", + }, + "spatial": {"location": "", "extent": "Germany", "resolution": ""}, + "sources": [ + { + "title": '"open-MaStR power unit registry"', + "description": "Raw data download Marktstammdatenregister (MaStR) data using the webservice. All data from the Marktstammdatenregister is included. There are duplicates included. For further information read in the documentationg of the original data source: https://www.marktstammdatenregister.de/MaStRHilfe/subpages/statistik.html", + "path": "https://sandbox.zenodo.org/record/808086", + "licenses": [ + { + "name": licenses_datenlizenz_deutschland( + '"© 2021 Bundesnetzagentur für Elektrizität, Gas, Telekommunikation, Post und Eisenbahnen"' + ), + "title": "", + "path": "", + "instruction": "", + "attribution": "eGon developers", + } + ], + }, + { + "title": '"ERA5 global reanalysis"', + "description": "ERA5 is the fifth generation ECMWF reanalysis for the global climate " + "and weather for the past 4 to 7 decades. Currently data is available from 1950, " + "split into Climate Data Store entries for 1950-1978 (preliminary back extension) and f" + "rom 1979 onwards (final release plus timely updates, this page). ERA5 replaces the ERA-Interim reanalysis. " + "See the online ERA5 documentation " + "(https://confluence.ecmwf.int/display/CKB/ERA5%3A+data+documentation#ERA5:datadocumentation-Dataupdatefrequency) " + "for more information.", + "path": "https://confluence.ecmwf.int/display/CKB/ERA5%3A+data+documentation#ERA5:datadocumentation-Dataupdatefrequency", + "licenses": [ + { + "name": "Licence to use Copernicus Products", + "title": "Licence to use Copernicus Products", + "path": "https://cds.climate.copernicus.eu/api/v2/terms/static/licence-to-use-copernicus-products.pdf", + "instruction": "This Licence is free of charge, worldwide, non-exclusive, royalty free and perpetual. " + "Access to Copernicus Products is given for any purpose in so far as it is lawful, whereas use " + "may include, but is not limited to: reproduction; distribution; communication to the public; " + "adaptation, modification and combination with other data and information; or any " + "combination of the foregoing", + "attribution": "Copernicus Climate Change Service (C3S) Climate Data Store", + }, + ], + }, + { + "title": '"Verwaltungsgebiete 1:250 000 (Ebenen)"', + "description": "Der Datenbestand umfasst sämtliche Verwaltungseinheiten der " + "hierarchischen Verwaltungsebenen vom Staat bis zu den Gemeinden " + "mit ihren Grenzen, statistischen Schlüsselzahlen, Namen der " + "Verwaltungseinheit sowie die spezifische Bezeichnung der " + "Verwaltungsebene des jeweiligen Landes.", + "path": "https://daten.gdz.bkg.bund.de/produkte/vg/vg250_ebenen_0101/2020/vg250_01-01.geo84.shape.ebenen.zip", + "licenses": [ + licenses_datenlizenz_deutschland( + "© Bundesamt für Kartographie und Geodäsie " + "2020 (Daten verändert)" + ) + ], + }, + { + "title": '"eGon-data"', + "description": "Workflow to download, process and generate data sets" + "suitable for the further research conducted in the project eGon (https://ego-n.org/)", + "path": "https://github.com/openego/eGon-data", + "licenses": [ + license_agpl( + "© Jonathan Amme, Clara Büttner, Ilka Cußmann, Julian Endres, Carlos Epia, Stephan Günther, Ulf Müller, Amélia Nadal, Guido Pleßmann, Francesco Witte" + ) + ], + }, + { + "title": '"Netzentwicklungsplan Strom 2035, Version 2021, erster Entwurf"', + "description": "Die vier deutschen Übertragungsnetzbetreiber zeigen mit " + "diesem ersten Entwurf des Netzentwicklungsplans 2035, Version 2021, den " + "benötigten Netzausbau für die nächsten Jahre auf. Der NEP-Bericht beschreibt " + "keine konkreten Trassenverläufe von Übertragungsleitungen, sondern er " + "dokumentiert den notwendigen Übertragungsbedarf zwischen Netzknoten. " + "Das heißt, es werden Anfangs- und Endpunkte von zukünftigen Leitungsverbindungen " + "definiert sowie konkrete Empfehlungen für den Aus- und Neubau der Übertragungsnetze " + "an Land und auf See in Deutschland gemäß den Detailanforderungen im § 12 EnWG gegeben.", + "path": "https://zenodo.org/record/5743452#.YbCoz7so8go", + "licenses": [license_ccby("© Übertragungsnetzbetreiber")], + }, + { + "title": '"Data bundle for egon-data"', + "description": "Zenodo repository to provide several different input data sets for eGon-data", + "path": "https://sandbox.zenodo.org/record/1167119", + "licenses": [license_ccby("© eGon development team")], + }, + ], + "licenses": [ + { + "name": "CC-BY-4.0", + "title": "Creative Commons Attribution 4.0 International", + "path": "https://creativecommons.org/licenses/by/4.0/legalcode", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute.", + "attribution": "eGon developers", + } + ], + "contributors": [ + { + "title": "Ilka Cußmann", + "email": "", + "object": "", + "comment": "Added hydro and biomass plants", + }, + { + "title": "Clara Büttner", + "email": "", + "object": "", + "comment": "Added table structure", + }, + { + "title": "Katharina Esterl", + "email": "", + "object": "", + "comment": "Add pv ground mounted", + }, + { + "title": "Carlos Epia", + "email": "", + "object": "", + "comment": "Added wind power plants", + }, + ], + "resources": [ + { + "profile": "tabular-data-resource", + "name": "supply.egon_power_plants", + "path": "", + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": [ + { + "name": "id", + "description": "Unique identifier for power plants", + "type": "Integer ", + "unit": "", + }, + { + "name": "sources", + "description": "sources of power plants", + "type": "Dictionary", + "unit": "", + }, + { + "name": "source_id", + "description": "Unique Identifier of sources ", + "type": "Dictionary", + "unit": "", + }, + { + "name": "carrier", + "description": "Energy carrier such as biomass, wind_onshore", + "type": "String", + "unit": "", + }, + { + "name": "el_capacity", + "description": "Installed electrical capacity in MW", + "type": "", + "unit": "MW", + }, + { + "name": "bus_id", + "description": "bus ID where the power plant is connected", + "type": "Integer", + "unit": "", + }, + { + "name": "voltage_level", + "description": "voltage level of power plant", + "type": "Integer", + "unit": "", + }, + { + "name": "weather_cell_id", + "description": "Unique identifier of the corresponding weather cell", + "type": "Integer", + "unit": "", + }, + { + "name": "scenario", + "description": "eGon scenario, for example eGon235 or eGon100RE", + "type": "String", + "unit": "", + }, + { + "name": "geometry", + "description": "geometry of the power plant", + "type": "String", + "unit": "", + }, + { + "name": "", + "description": "", + "type": "", + "unit": "", + }, + ], + "primaryKey": "", + }, + "dialect": {"delimiter": "", "decimalSeparator": ""}, + } + ], + "review": {"path": "", "badge": ""}, + "metaMetadata": { + "metadataVersion": "OEP-1.4.1", + "metadataLicense": { + "name": "CC0-1.0", + "title": "Creative Commons Zero v1.0 Universal", + "path": "https://creativecommons.org/publicdomain/zero/1.0/", + }, + }, + "_comment": { + "metadata": "Metadata documentation and explanation (https://github.com/OpenEnergyPlatform/oemetadata/blob/master/metadata/v141/metadata_key_description.md)", + "dates": "Dates and time must follow the ISO8601 including time zone (YYYY-MM-DD or YYYY-MM-DDThh:mm:ss±hh)", + "units": "Use a space between numbers and units (100 m)", + "languages": "Languages must follow the IETF (BCP47) format (en-GB, en-US, de-DE)", + "licenses": "License name must follow the SPDX License List (https://spdx.org/licenses/)", + "review": "Following the OEP Data Review (https://github.com/OpenEnergyPlatform/data-preprocessing/wiki)", + "none": "If not applicable use (none)", + }, + } + + db.submit_comment( + "'" + json.dumps(meta) + "'", "supply", "egon_power_plants" + ) diff --git a/src/egon/data/datasets/power_plants/pv_rooftop.py b/src/egon/data/datasets/power_plants/pv_rooftop.py index 11dfd0ad1..dc74a6184 100644 --- a/src/egon/data/datasets/power_plants/pv_rooftop.py +++ b/src/egon/data/datasets/power_plants/pv_rooftop.py @@ -1,4 +1,4 @@ -"""The module containing all code dealing with pv rooftop distribution. +"""The module containing all code dealing with pv rooftop distribution to MV grid level. """ from pathlib import Path diff --git a/src/egon/data/datasets/power_plants/pv_rooftop_buildings.py b/src/egon/data/datasets/power_plants/pv_rooftop_buildings.py index 872a85754..5843d8007 100644 --- a/src/egon/data/datasets/power_plants/pv_rooftop_buildings.py +++ b/src/egon/data/datasets/power_plants/pv_rooftop_buildings.py @@ -2,41 +2,8 @@ Distribute MaStR PV rooftop capacities to OSM and synthetic buildings. Generate new PV rooftop generators for scenarios eGon2035 and eGon100RE. -Data cleaning and inference: -* Drop duplicates and entries with missing critical data. -* Determine most plausible capacity from multiple values given in MaStR data. -* Drop generators which don't have any plausible capacity data - (23.5MW > P > 0.1). -* Randomly and weighted add a start-up date if it is missing. -* Extract zip and municipality from 'site' given in MaStR data. -* Geocode unique zip and municipality combinations with Nominatim (1 sec - delay). Drop generators for which geocoding failed or which are located - outside the municipalities of Germany. -* Add some visual sanity checks for cleaned data. - -Allocation of MaStR data: -* Allocate each generator to an existing building from OSM. -* Determine the quantile each generator and building is in depending on the - capacity of the generator and the area of the polygon of the building. -* Randomly distribute generators within each municipality preferably within - the same building area quantile as the generators are capacity wise. -* If not enough buildings exists within a municipality and quantile additional - buildings from other quantiles are chosen randomly. - -Desegregation of pv rooftop scenarios: -* The scenario data per federal state is linearly distributed to the mv grid - districts according to the pv rooftop potential per mv grid district. -* The rooftop potential is estimated from the building area given from the OSM - buildings. -* Grid districts, which are located in several federal states, are allocated - PV capacity according to their respective roof potential in the individual - federal states. -* The desegregation of PV plants within a grid districts respects existing - plants from MaStR, which did not reach their end of life. -* New PV plants are randomly and weighted generated using a breakdown of MaStR - data as generator basis. -* Plant metadata (e.g. plant orientation) is also added random and weighted - from MaStR data as basis. +See documentation section :ref:`pv-rooftop-ref` for more information. + """ from __future__ import annotations @@ -381,6 +348,10 @@ def load_mastr_data(): class OsmBuildingsFiltered(Base): + """ + Class definition of table openstreetmap.osm_buildings_filtered. + + """ __tablename__ = "osm_buildings_filtered" __table_args__ = {"schema": "openstreetmap"} @@ -991,6 +962,10 @@ def scenario_data( class Vg250Lan(Base): + """ + Class definition of table boundaries.vg250_lan. + + """ __tablename__ = "vg250_lan" __table_args__ = {"schema": "boundaries"} @@ -2100,6 +2075,10 @@ def allocate_scenarios( class EgonPowerPlantPvRoofBuilding(Base): + """ + Class definition of table supply.egon_power_plants_pv_roof_building. + + """ __tablename__ = "egon_power_plants_pv_roof_building" __table_args__ = {"schema": "supply"} diff --git a/src/egon/data/datasets/pypsaeursec/__init__.py b/src/egon/data/datasets/pypsaeursec/__init__.py index 0a43de772..7d2a43897 100755 --- a/src/egon/data/datasets/pypsaeursec/__init__.py +++ b/src/egon/data/datasets/pypsaeursec/__init__.py @@ -609,6 +609,7 @@ def neighbor_reduction(): { "gas": "CH4", "gas_for_industry": "CH4_for_industry", + "H2": "H2_grid", }, inplace=True, ) @@ -785,6 +786,17 @@ def links_to_etrago(neighbor_links, scn="eGon100RE", extendable=True): inplace=True, ) + for c in [ + "H2_to_CH4", + "H2_to_power", + "power_to_H2", + "CH4_to_H2", + ]: + neighbor_links.loc[ + (neighbor_links.carrier == c), + "lifetime", + ] = get_sector_parameters("gas", "eGon100RE")["lifetime"][c] + neighbor_links.to_postgis( "egon_etrago_link", engine, @@ -918,6 +930,12 @@ def links_to_etrago(neighbor_links, scn="eGon100RE", extendable=True): for i in ["name", "p_set", "q_set", "e_nom_opt", "lifetime"]: neighbor_stores = neighbor_stores.drop(i, axis=1) + for c in ["H2_underground", "H2_overground"]: + neighbor_stores.loc[ + (neighbor_stores.carrier == c), + "lifetime", + ] = get_sector_parameters("gas", "eGon100RE")["lifetime"][c] + neighbor_stores.to_sql( "egon_etrago_store", engine, @@ -1127,7 +1145,7 @@ class PypsaEurSec(Dataset): def __init__(self, dependencies): super().__init__( name="PypsaEurSec", - version="0.0.9", + version="0.0.10", dependencies=dependencies, tasks=tasks, ) diff --git a/src/egon/data/datasets/re_potential_areas/__init__.py b/src/egon/data/datasets/re_potential_areas/__init__.py index e1738d413..5a66332f7 100644 --- a/src/egon/data/datasets/re_potential_areas/__init__.py +++ b/src/egon/data/datasets/re_potential_areas/__init__.py @@ -2,7 +2,6 @@ potential areas for wind onshore and ground-mounted PV. """ -from functools import partial from pathlib import Path from geoalchemy2 import Geometry @@ -18,6 +17,10 @@ class EgonRePotentialAreaPvAgriculture(Base): + """ + Class definition of table supply.egon_re_potential_area_pv_agriculture. + + """ __tablename__ = "egon_re_potential_area_pv_agriculture" __table_args__ = {"schema": "supply"} id = Column(Integer, primary_key=True) @@ -25,6 +28,10 @@ class EgonRePotentialAreaPvAgriculture(Base): class EgonRePotentialAreaPvRoadRailway(Base): + """ + Class definition of table supply.egon_re_potential_area_pv_road_railway. + + """ __tablename__ = "egon_re_potential_area_pv_road_railway" __table_args__ = {"schema": "supply"} id = Column(Integer, primary_key=True) @@ -32,6 +39,10 @@ class EgonRePotentialAreaPvRoadRailway(Base): class EgonRePotentialAreaWind(Base): + """ + Class definition of table supply.egon_re_potential_area_wind. + + """ __tablename__ = "egon_re_potential_area_wind" __table_args__ = {"schema": "supply"} id = Column(Integer, primary_key=True) @@ -109,11 +120,32 @@ def insert_data(): ) -# create re_potential_areas dataset partial object -re_potential_area_setup = partial( - Dataset, - name="RePotentialAreas", - version="0.0.1", - dependencies=[], - tasks=(create_tables, insert_data), -) +class re_potential_area_setup(Dataset): + """ + Downloads potential areas for PV and wind power plants from data bundle and + writes them to the database. + + *Dependencies* + * :py:func:`Setup ` + * :py:class:`DataBundle ` + + *Resulting Tables* + * :py:class:`EgonRePotentialAreaPvAgriculture ` + * :py:class:`EgonRePotentialAreaPvRoadRailway ` + * :py:class:`EgonRePotentialAreaWind ` + + """ + #: + name: str = "RePotentialAreas" + #: + version: str = "0.0.1" + #: + tasks = (create_tables, insert_data) + + def __init__(self, dependencies): + super().__init__( + name=self.name, + version=self.version, + dependencies=dependencies, + tasks=self.tasks, + ) diff --git a/src/egon/data/datasets/renewable_feedin.py b/src/egon/data/datasets/renewable_feedin.py index 1ded14b72..acb9f0163 100644 --- a/src/egon/data/datasets/renewable_feedin.py +++ b/src/egon/data/datasets/renewable_feedin.py @@ -17,7 +17,6 @@ from egon.data.datasets.scenario_parameters import get_sector_parameters from egon.data.metadata import ( context, - generate_resource_fields_from_sqla_model, license_ccby, meta_metadata, sources, @@ -27,10 +26,35 @@ class RenewableFeedin(Dataset): + """ + Calculate possible feedin time series for renewable energy generators + + This dataset calculates possible feedin timeseries for fluctuation renewable generators + and coefficient of performance time series for heat pumps. Relevant input is the + downloaded weather data. Parameters for the time series calcultaion are also defined by + representative types of pv plants and wind turbines that are selected within this dataset. + The resulting profiles are stored in the database. + + + *Dependencies* + * :py:class:`WeatherData ` + * :py:class:`Vg250 ` + * :py:class:`ZensusVg250 ` + + *Resulting tables* + * :py:class:`supply.egon_era5_renewable_feedin ` is filled + + """ + + #: + name: str = "RenewableFeedin" + #: + version: str = "0.0.7" + def __init__(self, dependencies): super().__init__( - name="RenewableFeedin", - version="0.0.8", + name=self.name, + version=self.version, dependencies=dependencies, tasks={ wind, @@ -145,7 +169,6 @@ def federal_states_per_weather_cell(): while (buffer < 30000) & ( len(weather_cells[weather_cells["federal_state"].isnull()]) > 0 ): - cells = weather_cells[weather_cells["federal_state"].isnull()] cells.loc[:, "geom_point"] = cells.geom_point.buffer(buffer) diff --git a/src/egon/data/datasets/sanity_checks.py b/src/egon/data/datasets/sanity_checks.py old mode 100644 new mode 100755 index bb8d68e63..aae4702fa --- a/src/egon/data/datasets/sanity_checks.py +++ b/src/egon/data/datasets/sanity_checks.py @@ -7,6 +7,7 @@ """ from math import isclose from pathlib import Path +import ast from sqlalchemy import Numeric from sqlalchemy.sql import and_, cast, func, or_ @@ -41,6 +42,22 @@ EgonPfHvStore, EgonPfHvStoreTimeseries, ) +from egon.data.datasets.gas_grid import ( + define_gas_buses_abroad, + define_gas_nodes_list, + define_gas_pipeline_list, +) +from egon.data.datasets.gas_neighbours.eGon2035 import ( + calc_capacities, + calc_ch4_storage_capacities, + calc_global_ch4_demand, + calc_global_power_to_h2_demand, + calculate_ch4_grid_capacities, + import_ch4_demandTS, +) +from egon.data.datasets.hydrogen_etrago.storage import ( + calculate_and_map_saltcavern_storage_potential, +) from egon.data.datasets.power_plants.pv_rooftop_buildings import ( PV_CAP_PER_SQ_M, ROOF_FACTOR, @@ -48,6 +65,7 @@ load_building_data, scenario_data, ) +from egon.data.datasets.pypsaeursec import read_network from egon.data.datasets.scenario_parameters import get_sector_parameters from egon.data.datasets.storages.home_batteries import get_cbat_pbat_ratio import egon.data @@ -58,10 +76,15 @@ class SanityChecks(Dataset): + #: + name: str = "SanityChecks" + #: + version: str = "0.0.8" + def __init__(self, dependencies): super().__init__( - name="SanityChecks", - version="0.0.8", + name=self.name, + version=self.version, dependencies=dependencies, tasks={ etrago_eGon2035_electricity, @@ -73,6 +96,8 @@ def __init__(self, dependencies): sanitycheck_emobility_mit, sanitycheck_pv_rooftop_buildings, sanitycheck_home_batteries, + etrago_eGon2035_gas_DE, + etrago_eGon2035_gas_abroad, sanitycheck_dsm, }, ) @@ -1407,6 +1432,911 @@ def sanitycheck_home_batteries(): assert (home_batteries_df.round(6) == df.round(6)).all().all() +def sanity_check_gas_buses(scn): + """Execute sanity checks for the gas buses in Germany + + Returns print statements as sanity checks for the CH4, H2_grid and + H2_saltcavern buses. + * For all of them, it is checked if they are not isolated. + * For the grid buses, the deviation is calculated between the + number of gas grid buses in the database and the original + Scigrid_gas number of gas buses in Germany. + + Parameters + ---------- + scn_name : str + Name of the scenario + + """ + logger.info("BUSES") + + # Are gas buses isolated? + corresponding_carriers = { + "eGon2035": { + "CH4": "CH4", + "H2_grid": "H2_feedin", + "H2_saltcavern": "power_to_H2", + }, + # "eGon100RE": { + # "CH4": "CH4", + # "H2_grid": "H2_retrofit", + # "H2_saltcavern": "H2_extension", + # } + } + for key in corresponding_carriers[scn]: + isolated_gas_buses = db.select_dataframe( + f""" + SELECT bus_id, carrier, country + FROM grid.egon_etrago_bus + WHERE scn_name = '{scn}' + AND carrier = '{key}' + AND country = 'DE' + AND bus_id NOT IN + (SELECT bus0 + FROM grid.egon_etrago_link + WHERE scn_name = '{scn}' + AND carrier = '{corresponding_carriers[scn][key]}') + AND bus_id NOT IN + (SELECT bus1 + FROM grid.egon_etrago_link + WHERE scn_name = '{scn}' + AND carrier = '{corresponding_carriers[scn][key]}') + ; + """, + warning=False, + ) + if not isolated_gas_buses.empty: + logger.info(f"Isolated {key} buses:") + logger.info(isolated_gas_buses) + + # Deviation of the gas grid buses number + target_file = ( + Path(".") / "datasets" / "gas_data" / "data" / "IGGIELGN_Nodes.csv" + ) + + Grid_buses_list = pd.read_csv( + target_file, + delimiter=";", + decimal=".", + usecols=["country_code"], + ) + + Grid_buses_list = Grid_buses_list[ + Grid_buses_list["country_code"].str.match("DE") + ] + input_grid_buses = len(Grid_buses_list.index) + + for carrier in ["CH4", "H2_grid"]: + output_grid_buses_df = db.select_dataframe( + f""" + SELECT bus_id + FROM grid.egon_etrago_bus + WHERE scn_name = '{scn}' + AND country = 'DE' + AND carrier = '{carrier}'; + """, + warning=False, + ) + output_grid_buses = len(output_grid_buses_df.index) + + e_grid_buses = ( + round( + (output_grid_buses - input_grid_buses) / input_grid_buses, + 2, + ) + * 100 + ) + logger.info(f"Deviation {carrier} buses: {e_grid_buses} %") + + +def sanity_check_CH4_stores(scn): + """Execute sanity checks for the CH4 stores in Germany + + Returns print statements as sanity checks for the CH4 stores + capacity in Germany. The deviation is calculated between: + * the sum of the capacities of the stores with carrier 'CH4' + in the database (for one scenario) and + * the sum of: + * the capacity the gas grid allocated to CH4 (total capacity + in eGon2035 and capacity reduced the share of the grid + allocated to H2 in eGon100RE) + * the total capacity of the CH4 stores in Germany (source: GIE) + + Parameters + ---------- + scn_name : str + Name of the scenario + + """ + output_CH4_stores = db.select_dataframe( + f"""SELECT SUM(e_nom::numeric) as e_nom_germany + FROM grid.egon_etrago_store + WHERE scn_name = '{scn}' + AND carrier = 'CH4' + AND bus IN + (SELECT bus_id + FROM grid.egon_etrago_bus + WHERE scn_name = '{scn}' + AND country = 'DE' + AND carrier = 'CH4'); + """, + warning=False, + )["e_nom_germany"].values[0] + + if scn == "eGon2035": + grid_cap = 130000 + elif scn == "eGon100RE": + grid_cap = 13000 * ( + 1 + - get_sector_parameters("gas", "eGon100RE")[ + "retrofitted_CH4pipeline-to-H2pipeline_share" + ] + ) + + stores_cap_D = 266424202 # MWh GIE https://www.gie.eu/transparency/databases/storage-database/ + + input_CH4_stores = stores_cap_D + grid_cap + + e_CH4_stores = ( + round( + (output_CH4_stores - input_CH4_stores) / input_CH4_stores, + 2, + ) + * 100 + ) + logger.info(f"Deviation CH4 stores: {e_CH4_stores} %") + + +def sanity_check_H2_saltcavern_stores(scn): + """Execute sanity checks for the H2 saltcavern stores in Germany + + Returns print as sanity checks for the H2 saltcavern potential + storage capacity in Germany. The deviation is calculated between: + * the sum of the of the H2 saltcavern potential storage capacity + (e_nom_max) in the database and + * the sum of the H2 saltcavern potential storage capacity + assumed to be the ratio of the areas of 500 m radius around + substations in each german federal state and the estimated + total hydrogen storage potential of the corresponding federal + state (data from InSpEE-DS report). + This test works also in test mode. + + Parameters + ---------- + scn_name : str + Name of the scenario + + """ + output_H2_stores = db.select_dataframe( + f"""SELECT SUM(e_nom_max::numeric) as e_nom_max_germany + FROM grid.egon_etrago_store + WHERE scn_name = '{scn}' + AND carrier = 'H2_underground' + AND bus IN + (SELECT bus_id + FROM grid.egon_etrago_bus + WHERE scn_name = '{scn}' + AND country = 'DE' + AND carrier = 'H2_saltcavern'); + """, + warning=False, + )["e_nom_max_germany"].values[0] + + storage_potentials = calculate_and_map_saltcavern_storage_potential() + storage_potentials["storage_potential"] = ( + storage_potentials["area_fraction"] * storage_potentials["potential"] + ) + input_H2_stores = sum(storage_potentials["storage_potential"].to_list()) + + e_H2_stores = ( + round( + (output_H2_stores - input_H2_stores) / input_H2_stores, + 2, + ) + * 100 + ) + logger.info(f"Deviation H2 saltcavern stores: {e_H2_stores} %") + + +def sanity_check_gas_one_port(scn): + """Check connections of gas one-port components + + Verify that gas one-port component (loads, generators, stores) are + all connected to a bus (of the right carrier) present in the data + base. Return print statements if this is not the case. + These sanity checks are not specific to Germany, they also include + the neighbouring countries. + + Parameters + ---------- + scn_name : str + Name of the scenario + + """ + if scn == "eGon2035": + # Loads + ## CH4_for_industry Germany + isolated_one_port_c = db.select_dataframe( + f""" + SELECT load_id, bus, carrier, scn_name + FROM grid.egon_etrago_load + WHERE scn_name = '{scn}' + AND carrier = 'CH4_for_industry' + AND bus NOT IN + (SELECT bus_id + FROM grid.egon_etrago_bus + WHERE scn_name = '{scn}' + AND country = 'DE' + AND carrier = 'CH4') + ; + """, + warning=False, + ) + if not isolated_one_port_c.empty: + logger.info("Isolated loads:") + logger.info(isolated_one_port_c) + + ## CH4_for_industry abroad + isolated_one_port_c = db.select_dataframe( + f""" + SELECT load_id, bus, carrier, scn_name + FROM grid.egon_etrago_load + WHERE scn_name = '{scn}' + AND carrier = 'CH4' + AND bus NOT IN + (SELECT bus_id + FROM grid.egon_etrago_bus + WHERE scn_name = '{scn}' + AND country != 'DE' + AND carrier = 'CH4') + ; + """, + warning=False, + ) + if not isolated_one_port_c.empty: + logger.info("Isolated loads:") + logger.info(isolated_one_port_c) + + ## H2_for_industry + isolated_one_port_c = db.select_dataframe( + f""" + SELECT load_id, bus, carrier, scn_name + FROM grid.egon_etrago_load + WHERE scn_name = '{scn}' + AND carrier = 'H2_for_industry' + AND (bus NOT IN + (SELECT bus_id + FROM grid.egon_etrago_bus + WHERE scn_name = '{scn}' + AND country = 'DE' + AND carrier = 'H2_grid') + AND bus NOT IN + (SELECT bus_id + FROM grid.egon_etrago_bus + WHERE scn_name = '{scn}' + AND country != 'DE' + AND carrier = 'AC')) + ; + """, + warning=False, + ) + if not isolated_one_port_c.empty: + logger.info("Isolated loads:") + logger.info(isolated_one_port_c) + + # Genrators + isolated_one_port_c = db.select_dataframe( + f""" + SELECT generator_id, bus, carrier, scn_name + FROM grid.egon_etrago_generator + WHERE scn_name = '{scn}' + AND carrier = 'CH4' + AND bus NOT IN + (SELECT bus_id + FROM grid.egon_etrago_bus + WHERE scn_name = '{scn}' + AND carrier = 'CH4'); + ; + """, + warning=False, + ) + if not isolated_one_port_c.empty: + logger.info("Isolated generators:") + logger.info(isolated_one_port_c) + + # Stores + ## CH4 and H2_underground + corresponding_carriers = { + "CH4": "CH4", + "H2_saltcavern": "H2_underground", + } + for key in corresponding_carriers: + isolated_one_port_c = db.select_dataframe( + f""" + SELECT store_id, bus, carrier, scn_name + FROM grid.egon_etrago_store + WHERE scn_name = '{scn}' + AND carrier = '{corresponding_carriers[key]}' + AND bus NOT IN + (SELECT bus_id + FROM grid.egon_etrago_bus + WHERE scn_name = '{scn}' + AND carrier = '{key}') + ; + """, + warning=False, + ) + if not isolated_one_port_c.empty: + logger.info("Isolated stores:") + logger.info(isolated_one_port_c) + + ## H2_overground + isolated_one_port_c = db.select_dataframe( + f""" + SELECT store_id, bus, carrier, scn_name + FROM grid.egon_etrago_store + WHERE scn_name = '{scn}' + AND carrier = 'H2_overground' + AND bus NOT IN + (SELECT bus_id + FROM grid.egon_etrago_bus + WHERE scn_name = '{scn}' + AND country = 'DE' + AND carrier = 'H2_saltcavern') + AND bus NOT IN + (SELECT bus_id + FROM grid.egon_etrago_bus + WHERE scn_name = '{scn}' + AND country = 'DE' + AND carrier = 'H2_grid') + ; + """, + warning=False, + ) + if not isolated_one_port_c.empty: + logger.info("Isolated stores:") + logger.info(isolated_one_port_c) + + # elif scn == "eGon2035": + + +def sanity_check_CH4_grid(scn): + """Execute sanity checks for the gas grid capacity in Germany + + Returns print statements as sanity checks for the CH4 links + (pipelines) in Germany. The deviation is calculated between + the sum of the power (p_nom) of all the CH4 pipelines in Germany + for one scenario in the database and the sum of the powers of the + imported pipelines. + In eGon100RE, the sum is reduced by the share of the grid that is + allocated to hydrogen (share calculated by PyPSA-eur-sec). + This test works also in test mode. + + Parameters + ---------- + scn_name : str + Name of the scenario + + Returns + ------- + scn_name : float + Sum of the power (p_nom) of all the pipelines in Germany + + """ + grid_carrier = "CH4" + output_gas_grid = db.select_dataframe( + f"""SELECT SUM(p_nom::numeric) as p_nom_germany + FROM grid.egon_etrago_link + WHERE scn_name = '{scn}' + AND carrier = '{grid_carrier}' + AND bus0 IN + (SELECT bus_id + FROM grid.egon_etrago_bus + WHERE scn_name = '{scn}' + AND country = 'DE' + AND carrier = '{grid_carrier}') + AND bus1 IN + (SELECT bus_id + FROM grid.egon_etrago_bus + WHERE scn_name = '{scn}' + AND country = 'DE' + AND carrier = '{grid_carrier}') + ; + """, + warning=False, + )["p_nom_germany"].values[0] + + gas_nodes_list = define_gas_nodes_list() + abroad_gas_nodes_list = define_gas_buses_abroad() + gas_grid = define_gas_pipeline_list(gas_nodes_list, abroad_gas_nodes_list) + gas_grid_germany = gas_grid[ + (gas_grid["country_0"] == "DE") & (gas_grid["country_1"] == "DE") + ] + p_nom_total = sum(gas_grid_germany["p_nom"].to_list()) + + if scn == "eGon2035": + input_gas_grid = p_nom_total + if scn == "eGon100RE": + input_gas_grid = p_nom_total * ( + 1 + - get_sector_parameters("gas", "eGon100RE")[ + "retrofitted_CH4pipeline-to-H2pipeline_share" + ] + ) + + e_gas_grid = ( + round( + (output_gas_grid - input_gas_grid) / input_gas_grid, + 2, + ) + * 100 + ) + logger.info(f"Deviation of the capacity of the CH4 grid: {e_gas_grid} %") + + return p_nom_total + + +def sanity_check_gas_links(scn): + """Check connections of gas links + + Verify that gas links are all connected to buses present in the data + base. Return print statements if this is not the case. + This sanity check is not specific to Germany, it also includes + the neighbouring countries. + + Parameters + ---------- + scn_name : str + Name of the scenario + + """ + carriers = [ + "CH4", + "H2_feedin", + "H2_to_CH4", + "CH4_to_H2", + "H2_to_power", + "power_to_H2", + "OCGT", + "central_gas_boiler", + "central_gas_CHP", + "central_gas_CHP_heat", + "industrial_gas_CHP", + ] + for c in carriers: + link_with_missing_bus = db.select_dataframe( + f""" + SELECT link_id, bus0, bus1, carrier, scn_name + FROM grid.egon_etrago_link + WHERE scn_name = '{scn}' + AND carrier = '{c}' + AND (bus0 NOT IN + (SELECT bus_id + FROM grid.egon_etrago_bus + WHERE scn_name = '{scn}') + OR bus1 NOT IN + (SELECT bus_id + FROM grid.egon_etrago_bus + WHERE scn_name = '{scn}')) + ; + """, + warning=False, + ) + if not link_with_missing_bus.empty: + logger.info("Links with missing bus:") + logger.info(link_with_missing_bus) + + +def etrago_eGon2035_gas_DE(): + """Execute basic sanity checks for the gas sector in eGon2035 + + Returns print statements as sanity checks for the gas sector in + the eGon2035 scenario for the following components in Germany: + * Buses: with the function :py:func:`sanity_check_gas_buses` + * Loads: for the carriers 'CH4_for_industry' and 'H2_for_industry' + the deviation is calculated between the sum of the loads in the + database and the sum the loads in the sources document + (opendata.ffe database) + * Generators: the deviation is calculated between the sums of the + nominal powers of the gas generators in the database and of + the ones in the sources document (Biogaspartner Einspeiseatlas + Deutschland from the dena and Productions from the SciGRID_gas + data) + * Stores: deviations for stores with following carriers are + calculated: + * 'CH4': with the function :py:func:`sanity_check_CH4_stores` + * 'H2_underground': with the function :py:func:`sanity_check_H2_saltcavern_stores` + * One-port components (loads, generators, stores): verification + that they are all connected to a bus present in the data base + with the function :py:func:`sanity_check_gas_one_port` + * Links: verification: + * that the gas links are all connected to buses present in + the data base with the function :py:func:`sanity_check_gas_links` + * of the capacity of the gas grid with the function + :py:func:`sanity_check_CH4_grid` + + """ + scn = "eGon2035" + + if TESTMODE_OFF: + logger.info(f"Gas sanity checks for scenario {scn}") + + # Buses + sanity_check_gas_buses(scn) + + # Loads + logger.info("LOADS") + + path = Path(".") / "datasets" / "gas_data" / "demand" + corr_file = path / "region_corr.json" + df_corr = pd.read_json(corr_file) + df_corr = df_corr.loc[:, ["id_region", "name_short"]] + df_corr.set_index("id_region", inplace=True) + + for carrier in ["CH4_for_industry", "H2_for_industry"]: + + output_gas_demand = db.select_dataframe( + f"""SELECT (SUM( + (SELECT SUM(p) + FROM UNNEST(b.p_set) p))/1000000)::numeric as load_twh + FROM grid.egon_etrago_load a + JOIN grid.egon_etrago_load_timeseries b + ON (a.load_id = b.load_id) + JOIN grid.egon_etrago_bus c + ON (a.bus=c.bus_id) + AND b.scn_name = '{scn}' + AND a.scn_name = '{scn}' + AND c.scn_name = '{scn}' + AND c.country = 'DE' + AND a.carrier = '{carrier}'; + """, + warning=False, + )["load_twh"].values[0] + + input_gas_demand = pd.read_json( + path / (carrier + "_eGon2035.json") + ) + input_gas_demand = input_gas_demand.loc[:, ["id_region", "value"]] + input_gas_demand.set_index("id_region", inplace=True) + input_gas_demand = pd.concat( + [input_gas_demand, df_corr], axis=1, join="inner" + ) + input_gas_demand["NUTS0"] = (input_gas_demand["name_short"].str)[ + 0:2 + ] + input_gas_demand = input_gas_demand[ + input_gas_demand["NUTS0"].str.match("DE") + ] + input_gas_demand = sum(input_gas_demand.value.to_list()) / 1000000 + + e_demand = ( + round( + (output_gas_demand - input_gas_demand) / input_gas_demand, + 2, + ) + * 100 + ) + logger.info(f"Deviation {carrier}: {e_demand} %") + + # Generators + logger.info("GENERATORS") + carrier_generator = "CH4" + + output_gas_generation = db.select_dataframe( + f"""SELECT SUM(p_nom::numeric) as p_nom_germany + FROM grid.egon_etrago_generator + WHERE scn_name = '{scn}' + AND carrier = '{carrier_generator}' + AND bus IN + (SELECT bus_id + FROM grid.egon_etrago_bus + WHERE scn_name = '{scn}' + AND country = 'DE' + AND carrier = '{carrier_generator}'); + """, + warning=False, + )["p_nom_germany"].values[0] + + target_file = ( + Path(".") + / "datasets" + / "gas_data" + / "data" + / "IGGIELGN_Productions.csv" + ) + + NG_generators_list = pd.read_csv( + target_file, + delimiter=";", + decimal=".", + usecols=["country_code", "param"], + ) + + NG_generators_list = NG_generators_list[ + NG_generators_list["country_code"].str.match("DE") + ] + + p_NG = 0 + for index, row in NG_generators_list.iterrows(): + param = ast.literal_eval(row["param"]) + p_NG = p_NG + param["max_supply_M_m3_per_d"] + conversion_factor = 437.5 # MCM/day to MWh/h + p_NG = p_NG * conversion_factor + + basename = "Biogaspartner_Einspeiseatlas_Deutschland_2021.xlsx" + target_file = Path(".") / "datasets" / "gas_data" / basename + + conversion_factor_b = 0.01083 # m^3/h to MWh/h + p_biogas = ( + pd.read_excel( + target_file, + usecols=["Einspeisung Biomethan [(N*m^3)/h)]"], + )["Einspeisung Biomethan [(N*m^3)/h)]"].sum() + * conversion_factor_b + ) + + input_gas_generation = p_NG + p_biogas + e_generation = ( + round( + (output_gas_generation - input_gas_generation) + / input_gas_generation, + 2, + ) + * 100 + ) + logger.info( + f"Deviation {carrier_generator} generation: {e_generation} %" + ) + + # Stores + logger.info("STORES") + sanity_check_CH4_stores(scn) + sanity_check_H2_saltcavern_stores(scn) + + # One-port components + sanity_check_gas_one_port(scn) + + # Links + logger.info("LINKS") + sanity_check_CH4_grid(scn) + sanity_check_gas_links(scn) + + else: + print("Testmode is on, skipping sanity check.") + + +def etrago_eGon2035_gas_abroad(): + """Execute basic sanity checks for the gas sector in eGon2035 abroad + + Returns print statements as sanity checks for the gas sector in + the eGon2035 scenario for the following components in Germany: + * Buses + * Loads: for the carriers 'CH4' and 'H2_for_industry' + the deviation is calculated between the sum of the loads in the + database and the sum in the sources document (TYNDP) + * Generators: the deviation is calculated between the sums of the + nominal powers of the methane generators abroad in the database + and of the ones in the sources document (TYNDP) + * Stores: the deviation for methane stores abroad is calculated + between the sum of the capacities in the data base and the one + of the source document (SciGRID_gas data) + * Links: verification of the capacity of the crossbordering gas + grid pipelines. + + """ + scn = "eGon2035" + + if TESTMODE_OFF: + logger.info(f"Gas sanity checks abroad for scenario {scn}") + + # Buses + logger.info("BUSES") + + # Are gas buses isolated? + corresponding_carriers = { + "eGon2035": { + "CH4": "CH4", + }, + # "eGon100RE": { + # "CH4": "CH4", + # "H2_grid": "H2_retrofit", + # } + } + for key in corresponding_carriers[scn]: + isolated_gas_buses_abroad = db.select_dataframe( + f""" + SELECT bus_id, carrier, country + FROM grid.egon_etrago_bus + WHERE scn_name = '{scn}' + AND carrier = '{key}' + AND country != 'DE' + AND bus_id NOT IN + (SELECT bus0 + FROM grid.egon_etrago_link + WHERE scn_name = '{scn}' + AND carrier = '{corresponding_carriers[scn][key]}') + AND bus_id NOT IN + (SELECT bus1 + FROM grid.egon_etrago_link + WHERE scn_name = '{scn}' + AND carrier = '{corresponding_carriers[scn][key]}') + ; + """, + warning=False, + ) + if not isolated_gas_buses_abroad.empty: + logger.info(f"Isolated {key} buses abroad:") + logger.info(isolated_gas_buses_abroad) + + # Loads + logger.info("LOADS") + + ( + Norway_global_demand_1y, + normalized_ch4_demandTS, + ) = import_ch4_demandTS() + input_CH4_demand_abroad = calc_global_ch4_demand( + Norway_global_demand_1y + ) + input_CH4_demand = input_CH4_demand_abroad["GlobD_2035"].sum() + + ## CH4 + output_CH4_demand = db.select_dataframe( + f"""SELECT (SUM( + (SELECT SUM(p) + FROM UNNEST(b.p_set) p)))::numeric as load_mwh + FROM grid.egon_etrago_load a + JOIN grid.egon_etrago_load_timeseries b + ON (a.load_id = b.load_id) + JOIN grid.egon_etrago_bus c + ON (a.bus=c.bus_id) + AND b.scn_name = '{scn}' + AND a.scn_name = '{scn}' + AND c.scn_name = '{scn}' + AND c.country != 'DE' + AND a.carrier = 'CH4'; + """, + warning=False, + )["load_mwh"].values[0] + + e_demand_CH4 = ( + round( + (output_CH4_demand - input_CH4_demand) / input_CH4_demand, + 2, + ) + * 100 + ) + logger.info(f"Deviation CH4 load: {e_demand_CH4} %") + + ## H2_for_industry + input_power_to_h2_demand_abroad = calc_global_power_to_h2_demand() + input_H2_demand = input_power_to_h2_demand_abroad["GlobD_2035"].sum() + + output_H2_demand = db.select_dataframe( + f"""SELECT SUM(p_set::numeric) as p_set_abroad + FROM grid.egon_etrago_load + WHERE scn_name = '{scn}' + AND carrier = 'H2_for_industry' + AND bus IN + (SELECT bus_id + FROM grid.egon_etrago_bus + WHERE scn_name = '{scn}' + AND country != 'DE' + AND carrier = 'AC'); + """, + warning=False, + )["p_set_abroad"].values[0] + + e_demand_H2 = ( + round( + (output_H2_demand - input_H2_demand) / input_H2_demand, + 2, + ) + * 100 + ) + logger.info(f"Deviation H2_for_industry load: {e_demand_H2} %") + + # Generators + logger.info("GENERATORS ") + CH4_gen = calc_capacities() + input_CH4_gen = CH4_gen["cap_2035"].sum() + + output_CH4_gen = db.select_dataframe( + f"""SELECT SUM(p_nom::numeric) as p_nom_abroad + FROM grid.egon_etrago_generator + WHERE scn_name = '{scn}' + AND carrier = 'CH4' + AND bus IN + (SELECT bus_id + FROM grid.egon_etrago_bus + WHERE scn_name = '{scn}' + AND country != 'DE' + AND carrier = 'CH4'); + """, + warning=False, + )["p_nom_abroad"].values[0] + + e_gen = ( + round( + (output_CH4_gen - input_CH4_gen) / input_CH4_gen, + 2, + ) + * 100 + ) + logger.info(f"Deviation CH4 generators: {e_gen} %") + + # Stores + logger.info("STORES") + ch4_input_capacities = calc_ch4_storage_capacities() + input_CH4_stores = ch4_input_capacities["e_nom"].sum() + + output_CH4_stores = db.select_dataframe( + f"""SELECT SUM(e_nom::numeric) as e_nom_abroad + FROM grid.egon_etrago_store + WHERE scn_name = '{scn}' + AND carrier = 'CH4' + AND bus IN + (SELECT bus_id + FROM grid.egon_etrago_bus + WHERE scn_name = '{scn}' + AND country != 'DE' + AND carrier = 'CH4'); + """, + warning=False, + )["e_nom_abroad"].values[0] + + e_stores = ( + round( + (output_CH4_stores - input_CH4_stores) / input_CH4_stores, + 2, + ) + * 100 + ) + logger.info(f"Deviation CH4 stores: {e_stores} %") + + # Links + logger.info("LINKS") + ch4_grid_input_capacities = calculate_ch4_grid_capacities() + input_CH4_grid = ch4_grid_input_capacities["p_nom"].sum() + + grid_carrier = "CH4" + output_gas_grid = db.select_dataframe( + f"""SELECT SUM(p_nom::numeric) as p_nom + FROM grid.egon_etrago_link + WHERE scn_name = '{scn}' + AND carrier = '{grid_carrier}' + AND (bus0 IN + (SELECT bus_id + FROM grid.egon_etrago_bus + WHERE scn_name = '{scn}' + AND country != 'DE' + AND carrier = '{grid_carrier}') + OR bus1 IN + (SELECT bus_id + FROM grid.egon_etrago_bus + WHERE scn_name = '{scn}' + AND country != 'DE' + AND carrier = '{grid_carrier}')) + ; + """, + warning=False, + )["p_nom"].values[0] + + e_gas_grid = ( + round( + (output_gas_grid - input_CH4_grid) / input_CH4_grid, + 2, + ) + * 100 + ) + logger.info( + f"Deviation of the capacity of the crossbordering CH4 grid: {e_gas_grid} %" + ) + + else: + print("Testmode is on, skipping sanity check.") + + def sanitycheck_dsm(): def df_from_series(s: pd.Series): return pd.DataFrame.from_dict(dict(zip(s.index, s.values))) diff --git a/src/egon/data/datasets/scenario_capacities.py b/src/egon/data/datasets/scenario_capacities.py index 008609b06..835e54358 100755 --- a/src/egon/data/datasets/scenario_capacities.py +++ b/src/egon/data/datasets/scenario_capacities.py @@ -67,20 +67,44 @@ class NEP2021ConvPowerPlants(Base): class ScenarioCapacities(Dataset): + """ + Create and fill table with installed generation capacities in Germany + + This dataset creates and fills a table with the installed generation capacities in + Germany in a lower spatial resolution (either per federal state or on national level). + This data is coming from external sources (e.g. German grid developement plan for scenario eGon2035). + The table is in downstream datasets used to define target values for the installed capacities. + + + *Dependencies* + * :py:func:`Setup ` + * :py:class:`PypsaEurSec ` + * :py:class:`Vg250 ` + * :py:class:`DataBundle ` + * :py:class:`ZensusPopulation ` + + + *Resulting tables* + * :py:class:`supply.egon_scenario_capacities ` is created and filled + * :py:class:`supply.egon_nep_2021_conventional_powerplants ` is created and filled + + """ + + #: + name: str = "ScenarioCapacities" + #: + version: str = "0.0.13" + def __init__(self, dependencies): super().__init__( - name="ScenarioCapacities", - version="0.0.13", + name=self.name, + version=self.version, dependencies=dependencies, - tasks=( - create_table, - insert_data_nep, - eGon100_capacities, - add_metadata, - ), + tasks=(create_table, insert_data_nep, eGon100_capacities, add_metadata), ) + def create_table(): """Create input tables for scenario setup @@ -99,7 +123,6 @@ def create_table(): def nuts_mapping(): - nuts_mapping = { "BW": "DE1", "NW": "DEA", @@ -190,7 +213,6 @@ def insert_capacities_per_federal_state_nep(): # df_windoff_fs for state in index_list: - df.at["Wind offshore", state] = ( df_windoff_fs.at[state, "C 2035"] / 1000 ) @@ -235,7 +257,6 @@ def insert_capacities_per_federal_state_nep(): ] for bl in map_nuts.index: - data = pd.DataFrame(df[bl]) # if distribution to federal states is not provided, @@ -767,6 +788,7 @@ def eGon100_capacities(): ) + def add_metadata(): """Add metdata to supply.egon_scenario_capacities @@ -803,9 +825,7 @@ def add_metadata(): "name": "supply.egon_scenario_capacities", "title": "eGon scenario capacities", "id": "WILL_BE_SET_AT_PUBLICATION", - "description": ( - "Installed capacities of scenarios used in the eGon project" - ), + "description": "Installed capacities of scenarios used in the eGon project", "language": ["de-DE"], "publicationDate": datetime.date.today().isoformat(), "context": context(), @@ -822,11 +842,10 @@ def add_metadata(): ], "licenses": [ license_ccby( - "© Übertragungsnetzbetreiber; © Bundesamt für Kartographie und" - " Geodäsie 2020 (Daten verändert); © Statistische Ämter des " - "Bundes und der Länder 2014; © Jonathan Amme, Clara Büttner, " - "Ilka Cußmann, Julian Endres, Carlos Epia, Stephan Günther, " - "Ulf Müller, Amélia Nadal, Guido Pleßmann, Francesco Witte", + "© Übertragungsnetzbetreiber; " + "© Bundesamt für Kartographie und Geodäsie 2020 (Daten verändert); " + "© Statistische Ämter des Bundes und der Länder 2014; " + "© Jonathan Amme, Clara Büttner, Ilka Cußmann, Julian Endres, Carlos Epia, Stephan Günther, Ulf Müller, Amélia Nadal, Guido Pleßmann, Francesco Witte", ) ], "contributors": [ diff --git a/src/egon/data/datasets/scenario_parameters/__init__.py b/src/egon/data/datasets/scenario_parameters/__init__.py index 3c7ba45b9..73ef9196e 100755 --- a/src/egon/data/datasets/scenario_parameters/__init__.py +++ b/src/egon/data/datasets/scenario_parameters/__init__.py @@ -164,33 +164,37 @@ def get_sector_parameters(sector, scenario=None): else: print(f"Scenario name {scenario} is not valid.") else: - values = pd.DataFrame( - db.select_dataframe( - f""" - SELECT {sector}_parameters as val - FROM scenario.egon_scenario_parameters - WHERE name='eGon2035'""" - ).val[0], - index=["eGon2035"], - ).append( + values = ( pd.DataFrame( db.select_dataframe( f""" + SELECT {sector}_parameters as val + FROM scenario.egon_scenario_parameters + WHERE name='eGon2035'""" + ).val[0], + index=["eGon2035"], + ) + .append( + pd.DataFrame( + db.select_dataframe( + f""" SELECT {sector}_parameters as val FROM scenario.egon_scenario_parameters WHERE name='eGon100RE'""" - ).val[0], - index=["eGon100RE"], + ).val[0], + index=["eGon100RE"], + ) ) - ).append( - pd.DataFrame( - db.select_dataframe( - f""" + .append( + pd.DataFrame( + db.select_dataframe( + f""" SELECT {sector}_parameters as val FROM scenario.egon_scenario_parameters WHERE name='eGon2021'""" - ).val[0], - index=["eGon2021"], + ).val[0], + index=["eGon2021"], + ) ) ) @@ -220,10 +224,34 @@ def download_pypsa_technology_data(): class ScenarioParameters(Dataset): + """ + Create and fill table with central parameters for each scenario + + This dataset creates and fills a table in the database that includes central parameters + for each scenarios. These parameters are mostly from extrernal sources, they are defined + and referenced within this dataset. + The table is acced by various datasets to access the parameters for all sectors. + + + *Dependencies* + * :py:func:`Setup ` + + + *Resulting tables* + * :py:class:`scenario.egon_scenario_parameters ` is created and filled + + + """ + + #: + name: str = "ScenarioParameters" + #: + version: str = "0.0.12" + def __init__(self, dependencies): super().__init__( - name="ScenarioParameters", - version="0.0.12", + name=self.name, + version=self.version, dependencies=dependencies, tasks=( create_table, diff --git a/src/egon/data/datasets/tyndp.py b/src/egon/data/datasets/tyndp.py index 10d50591c..eb7c6d979 100644 --- a/src/egon/data/datasets/tyndp.py +++ b/src/egon/data/datasets/tyndp.py @@ -1,4 +1,4 @@ -"""The central module containing all code dealing with tyndp data +"""The central module containing all code dealing with downloading tyndp data """ import os @@ -6,18 +6,40 @@ from egon.data.datasets import Dataset from urllib.request import urlretrieve + class Tyndp(Dataset): + """ + Downloads data for foreign countries from Ten-Year-Network-Developement Plan + + This dataset downloads installed generation capacities and load time series for + foreign countries from the website of the Ten-Year-Network-Developement Plan 2020 from ENTSO-E. + That data is stored into files and later on written into the database + (see :py:class:`ElectricalNeighbours `). + + + *Dependencies* + * :py:class:`Setup ` + + *Resulting tables* + + """ + + #: + name: str = "Tyndp" + #: + version: str = "0.0.1" def __init__(self, dependencies): super().__init__( - name="Tyndp", - version="0.0.1", + name=self.name, + version=self.version, dependencies=dependencies, tasks=(download), ) + def download(): - """ Download input data from TYNDP 2020 + """Download input data from TYNDP 2020 Returns ------- None. @@ -25,11 +47,10 @@ def download(): sources = config.datasets()["tyndp"]["sources"] targets = config.datasets()["tyndp"]["targets"] - if not os.path.exists('tyndp'): - os.mkdir('tyndp') - - for dataset in ['capacities', 'demand_2030', 'demand_2040']: + if not os.path.exists("tyndp"): + os.mkdir("tyndp") + for dataset in ["capacities", "demand_2030", "demand_2040"]: target_file = targets[dataset] urlretrieve(sources[dataset], f"tyndp/{target_file}") diff --git a/src/egon/data/datasets/vg250/__init__.py b/src/egon/data/datasets/vg250/__init__.py index ce4f94c97..92741a67d 100644 --- a/src/egon/data/datasets/vg250/__init__.py +++ b/src/egon/data/datasets/vg250/__init__.py @@ -32,12 +32,19 @@ def download_files(): - """Download VG250 (Verwaltungsgebiete) shape files.""" + """ + Download VG250 (Verwaltungsgebiete) shape files. + + Data is downloaded from source specified in *datasets.yml* in section + *vg250/original_data/source/url* and saved to file specified in + *vg250/original_data/target/file*. + + """ data_config = egon.data.config.datasets() vg250_config = data_config["vg250"]["original_data"] download_directory = Path(".") / "vg250" - # Create the folder, if it does not exists already + # Create the folder, if it does not exist already if not os.path.exists(download_directory): os.mkdir(download_directory) @@ -48,8 +55,16 @@ def download_files(): def to_postgres(): + """ + Writes original VG250 data to database. + + Creates schema boundaries if it does not yet exist. + Newly creates all tables specified as keys in *datasets.yml* in section + *vg250/processed/file_table_map*. - # Get information from data configuraiton file + """ + + # Get information from data configuration file data_config = egon.data.config.datasets() vg250_orig = data_config["vg250"]["original_data"] vg250_processed = data_config["vg250"]["processed"] @@ -244,13 +259,34 @@ def add_metadata(): def nuts_mview(): + """ + Creates MView boundaries.vg250_lan_nuts_id. + """ db.execute_sql_script( os.path.join(os.path.dirname(__file__), "vg250_lan_nuts_id_mview.sql") ) def cleaning_and_preperation(): + """ + Creates tables and MViews with cleaned and corrected geometry data. + + The following table is created: + * boundaries.vg250_gem_clean where municipalities (Gemeinden) that are fragmented + are cleaned from ringholes + + The following MViews are created: + * boundaries.vg250_gem_hole + * boundaries.vg250_gem_valid + * boundaries.vg250_krs_area + * boundaries.vg250_lan_union + * boundaries.vg250_sta_bbox + * boundaries.vg250_sta_invalid_geometry + * boundaries.vg250_sta_tiny_buffer + * boundaries.vg250_sta_union + + """ db.execute_sql_script( os.path.join(os.path.dirname(__file__), "cleaning_and_preparation.sql") @@ -258,6 +294,10 @@ def cleaning_and_preperation(): def vg250_metadata_resources_fields(): + """ + Returns metadata string for VG250 tables. + + """ return [ { @@ -432,15 +472,54 @@ def vg250_metadata_resources_fields(): class Vg250(Dataset): - + """ + Obtains and processes VG250 data and writes it to database. + + Original data is downloaded using :py:func:`download_files` function and written + to database using :py:func:`to_postgres` function. + + *Dependencies* + No dependencies + + *Resulting tables* + * :py:func:`boundaries.vg250_gem ` is created and filled + * :py:func:`boundaries.vg250_krs ` is created and filled + * :py:func:`boundaries.vg250_lan ` is created and filled + * :py:func:`boundaries.vg250_rbz ` is created and filled + * :py:func:`boundaries.vg250_sta ` is created and filled + * :py:func:`boundaries.vg250_vwg ` is created and filled + * :py:func:`boundaries.vg250_lan_nuts_id ` is created and filled + * :py:func:`boundaries.vg250_gem_hole ` is created and + filled + * :py:func:`boundaries.vg250_gem_valid ` is created and + filled + * :py:func:`boundaries.vg250_krs_area ` is created and + filled + * :py:func:`boundaries.vg250_lan_union ` is created and + filled + * :py:func:`boundaries.vg250_sta_bbox ` is created and + filled + * :py:func:`boundaries.vg250_sta_invalid_geometry ` is + created and filled + * :py:func:`boundaries.vg250_sta_tiny_buffer ` is + created and filled + * :py:func:`boundaries.vg250_sta_union ` is + created and filled + + """ filename = egon.data.config.datasets()["vg250"]["original_data"]["source"][ "url" ] + #: + name: str = "VG250" + #: + version: str = filename + "-0.0.4" + def __init__(self, dependencies): super().__init__( - name="VG250", - version=self.filename + "-0.0.4", + name=self.name, + version=self.version, dependencies=dependencies, tasks=( download_files, diff --git a/src/egon/data/datasets/vg250_mv_grid_districts.py b/src/egon/data/datasets/vg250_mv_grid_districts.py index d6f556b37..6d4c8d42d 100644 --- a/src/egon/data/datasets/vg250_mv_grid_districts.py +++ b/src/egon/data/datasets/vg250_mv_grid_districts.py @@ -1,4 +1,4 @@ -"""The module containing all code dealing with pv rooftop distribution. +"""The module containing all code to map MV grid districts to federal states. """ from sqlalchemy import Column, Integer, String from sqlalchemy.ext.declarative import declarative_base @@ -12,16 +12,36 @@ class Vg250MvGridDistricts(Dataset): + """ + Maps MV grid districts to federal states and writes it to database. + + *Dependencies* + * :py:class:`Vg250 ` + * :py:class:`MvGridDistricts ` + + *Resulting tables* + * :py:class:`boundaries.egon_map_mvgriddistrict_vg250 ` + is created and filled + + """ + #: + name: str = "Vg250MvGridDistricts" + #: + version: str = "0.0.1" + def __init__(self, dependencies): super().__init__( - name="Vg250MvGridDistricts", - version="0.0.1", + name=self.name, + version=self.version, dependencies=dependencies, tasks=(mapping), ) class MapMvgriddistrictsVg250(Base): + """ + Class definition of table boundaries.egon_map_mvgriddistrict_vg250. + """ __tablename__ = "egon_map_mvgriddistrict_vg250" __table_args__ = {"schema": "boundaries"} bus_id = Column(Integer, primary_key=True) @@ -29,10 +49,9 @@ class MapMvgriddistrictsVg250(Base): def create_tables(): - """Create tables for mapping grid districts to federal state - Returns - ------- - None. + """ + Create table for mapping grid districts to federal states. + """ db.execute_sql("CREATE SCHEMA IF NOT EXISTS boundaries;") @@ -42,11 +61,10 @@ def create_tables(): def mapping(): - """Map mv grid distrcits to federal states + """ + Map MV grid districts to federal states and write to database. - Returns - ------- - None. + Newly creates and fills table boundaries.egon_map_mvgriddistrict_vg250. """ # Create table diff --git a/src/egon/data/datasets/zensus_mv_grid_districts.py b/src/egon/data/datasets/zensus_mv_grid_districts.py index 7c8d98c79..aac4736ce 100644 --- a/src/egon/data/datasets/zensus_mv_grid_districts.py +++ b/src/egon/data/datasets/zensus_mv_grid_districts.py @@ -14,10 +14,26 @@ class ZensusMvGridDistricts(Dataset): + """ + Maps zensus cells to MV grid districts and writes it to database. + + *Dependencies* + * :py:class:`ZensusPopulation ` + * :py:class:`MvGridDistricts ` + + *Resulting tables* + * :py:class:`boundaries.egon_map_zensus_grid_districts ` + is created and filled + + """ + #: + name: str = "ZensusMvGridDistricts" + #: + version: str = "0.0.1" def __init__(self, dependencies): super().__init__( - name="ZensusMvGridDistricts", - version="0.0.1", + name=self.name, + version=self.version, dependencies=dependencies, tasks=(mapping), ) @@ -28,6 +44,9 @@ def __init__(self, dependencies): class MapZensusGridDistricts(Base): + """ + Class definition of table boundaries.egon_map_zensus_grid_districts. + """ __tablename__ = "egon_map_zensus_grid_districts" __table_args__ = {"schema": "boundaries"} @@ -41,7 +60,12 @@ class MapZensusGridDistricts(Base): def mapping(): - """Perform mapping between mv grid districts and zensus grid""" + """ + Map zensus cells and MV grid districts and write to database. + + Newly creates and fills table boundaries.egon_map_zensus_grid_districts. + + """ MapZensusGridDistricts.__table__.drop(bind=db.engine(), checkfirst=True) MapZensusGridDistricts.__table__.create(bind=db.engine(), checkfirst=True) diff --git a/src/egon/data/metadata.py b/src/egon/data/metadata.py deleted file mode 100755 index f5a503044..000000000 --- a/src/egon/data/metadata.py +++ /dev/null @@ -1,719 +0,0 @@ -from geoalchemy2 import Geometry -from sqlalchemy import MetaData, Table -from sqlalchemy.dialects.postgresql.base import ischema_names - -from egon.data.db import engine - -EGON_ATTRIBUTION: str = "© eGon development team" - - -def oep_metadata_version(): - """ - Get OEP metadata version string. - - Returns - ------- - str - OEP metadata version string - - """ - return "oep-v1.4" - - -def context(): - """ - Project context information for metadata - - Returns - ------- - dict - OEP metadata conform data license information - """ - - return { - "homepage": "https://ego-n.org/", - "documentation": "https://egon-data.readthedocs.io/en/latest/", - "sourceCode": "https://github.com/openego/eGon-data", - "contact": "https://ego-n.org/partners/", - "grantNo": "03EI1002", - "fundingAgency": "Bundesministerium für Wirtschaft und Energie", - "fundingAgencyLogo": "https://www.innovation-beratung-" - "foerderung.de/INNO/Redaktion/DE/Bilder/" - "Titelbilder/titel_foerderlogo_bmwi.jpg?" - "__blob=normal&v=3", - "publisherLogo": "https://ego-n.org/images/eGon_logo_" - "noborder_transbg.svg", - } - - -def meta_metadata(): - """ - Meta data on metadata - - Returns - ------- - dict - OEP metadata conform metadata on metadata - """ - - return { - "metadataVersion": "OEP-1.4.1", - "metadataLicense": { - "name": "CC0-1.0", - "title": "Creative Commons Zero v1.0 Universal", - "path": ("https://creativecommons.org/publicdomain/zero/1.0/"), - }, - } - - -def licenses_datenlizenz_deutschland(attribution=EGON_ATTRIBUTION): - """ - License information for Datenlizenz Deutschland - - Parameters - ---------- - attribution : str - Attribution for the dataset incl. © symbol, e.g. '© GeoBasis-DE / BKG' - - Returns - ------- - dict - OEP metadata conform data license information - """ - - return { - "name": "dl-by-de/2.0", - "title": "Datenlizenz Deutschland – Namensnennung – Version 2.0", - "path": "www.govdata.de/dl-de/by-2-0", - "instruction": ( - "Jede Nutzung ist unter den Bedingungen dieser „Datenlizenz " - "Deutschland - Namensnennung - Version 2.0 zulässig.\nDie " - "bereitgestellten Daten und Metadaten dürfen für die " - "kommerzielle und nicht kommerzielle Nutzung insbesondere:" - "(1) vervielfältigt, ausgedruckt, präsentiert, verändert, " - "bearbeitet sowie an Dritte übermittelt werden;\n " - "(2) mit eigenen Daten und Daten Anderer zusammengeführt und " - "zu selbständigen neuen Datensätzen verbunden werden;\n " - "(3) in interne und externe Geschäftsprozesse, Produkte und " - "Anwendungen in öffentlichen und nicht öffentlichen " - "elektronischen Netzwerken eingebunden werden.\n" - "Bei der Nutzung ist sicherzustellen, dass folgende Angaben " - "als Quellenvermerk enthalten sind:\n" - "(1) Bezeichnung des Bereitstellers nach dessen Maßgabe,\n" - "(2) der Vermerk Datenlizenz Deutschland – Namensnennung – " - "Version 2.0 oder dl-de/by-2-0 mit Verweis auf den Lizenztext " - "unter www.govdata.de/dl-de/by-2-0 sowie\n" - "(3) einen Verweis auf den Datensatz (URI)." - "Dies gilt nur soweit die datenhaltende Stelle die Angaben" - "(1) bis (3) zum Quellenvermerk bereitstellt.\n" - "Veränderungen, Bearbeitungen, neue Gestaltungen oder " - "sonstige Abwandlungen sind im Quellenvermerk mit dem Hinweis " - "zu versehen, dass die Daten geändert wurden." - ), - "attribution": attribution, - } - - -def license_odbl(attribution=EGON_ATTRIBUTION): - """ - License information for Open Data Commons Open Database License (ODbL-1.0) - - Parameters - ---------- - attribution : str - Attribution for the dataset incl. © symbol, e.g. - '© OpenStreetMap contributors' - - Returns - ------- - dict - OEP metadata conform data license information - """ - return { - "name": "ODbL-1.0", - "title": "Open Data Commons Open Database License 1.0", - "path": "https://opendatacommons.org/licenses/odbl/1.0/index.html", - "instruction": "You are free: To Share, To Create, To Adapt; " - "As long as you: Attribute, Share-Alike, Keep open!", - "attribution": attribution, - } - - -def license_ccby(attribution=EGON_ATTRIBUTION): - """ - License information for Creative Commons Attribution 4.0 International - (CC-BY-4.0) - - Parameters - ---------- - attribution : str - Attribution for the dataset incl. © symbol, e.g. '© GeoBasis-DE / BKG' - - Returns - ------- - dict - OEP metadata conform data license information - """ - return { - "name": "CC-BY-4.0", - "title": "Creative Commons Attribution 4.0 International", - "path": "https://creativecommons.org/licenses/by/4.0/legalcode", - "instruction": "You are free: To Share, To Create, To Adapt; " - "As long as you: Attribute.", - "attribution": attribution, - } - - -def license_geonutzv(attribution=EGON_ATTRIBUTION): - """ - License information for GeoNutzV - - Parameters - ---------- - attribution : str - Attribution for the dataset incl. © symbol, e.g. '© GeoBasis-DE / BKG' - - Returns - ------- - dict - OEP metadata conform data license information - """ - return { - "name": "geonutzv-de-2013-03-19", - "title": "Verordnung zur Festlegung der Nutzungsbestimmungen für die " - "Bereitstellung von Geodaten des Bundes", - "path": "https://www.gesetze-im-internet.de/geonutzv/", - "instruction": "Geodaten und Geodatendienste, einschließlich " - "zugehöriger Metadaten, werden für alle derzeit " - "bekannten sowie für alle zukünftig bekannten Zwecke " - "kommerzieller und nicht kommerzieller Nutzung " - "geldleistungsfrei zur Verfügung gestellt, soweit " - "durch besondere Rechtsvorschrift nichts anderes " - "bestimmt ist oder vertragliche oder gesetzliche " - "Rechte Dritter dem nicht entgegenstehen.", - "attribution": attribution, - } - - -def license_agpl(attribution=EGON_ATTRIBUTION): - """ - License information for GNU Affero General Public License v3.0 - - Parameters - ---------- - attribution : str - Attribution for the dataset incl. © symbol, e.g. '© GeoBasis-DE / BKG' - - Returns - ------- - dict - OEP metadata conform data license information - """ - return { - "name": "AGPL-3.0 License", - "title": "GNU Affero General Public License v3.0", - "path": "https://www.gnu.org/licenses/agpl-3.0.de.html", - "instruction": "Permissions of this strongest copyleft license are" - "conditioned on making available complete source code of licensed " - "works and modifications, which include larger works using a licensed" - "work, under the same license. Copyright and license notices must be" - "preserved. Contributors provide an express grant of patent rights." - "When a modified version is used to provide a service over a network," - "the complete source code of the modified version must be made " - "available.", - "attribution": attribution, - } - - -def license_dedl(attribution=EGON_ATTRIBUTION): - """ - License information for Data licence Germany – attribution – version 2.0 - - Parameters - ---------- - attribution : str - Attribution for the dataset incl. © symbol, e.g. '© GeoBasis-DE / BKG' - - Returns - ------- - dict - OEP metadata conform data license information - """ - return { - "name": "DL-DE-BY-2.0", - "title": "Data licence Germany – attribution – version 2.0", - "path": "https://www.govdata.de/dl-de/by-2-0", - "instruction": ( - "Any use will be permitted provided it fulfils the requirements of" - ' this "Data licence Germany – attribution – Version 2.0". The ' - "data and meta-data provided may, for commercial and " - "non-commercial use, in particular be copied, printed, presented, " - "altered, processed and transmitted to third parties; be merged " - "with own data and with the data of others and be combined to form" - " new and independent datasets; be integrated in internal and " - "external business processes, products and applications in public " - "and non-public electronic networks. The user must ensure that the" - " source note contains the following information: the name of the " - 'provider, the annotation "Data licence Germany – attribution – ' - 'Version 2.0" or "dl-de/by-2-0" referring to the licence text ' - "available at www.govdata.de/dl-de/by-2-0, and a reference to the " - "dataset (URI). This applies only if the entity keeping the data " - "provides the pieces of information 1-3 for the source note. " - "Changes, editing, new designs or other amendments must be marked " - "as such in the source note." - ), - "attribution": attribution, - } - - -def generate_resource_fields_from_sqla_model(model): - """Generate a template for the resource fields for metadata from a SQL - Alchemy model. - - For details on the fields see field 14.6.1 of `Open Energy Metadata - `_ standard. - The fields `name` and `type` are automatically filled, the `description` - and `unit` must be filled manually. - - Examples - -------- - >>> from egon.data.metadata import generate_resource_fields_from_sqla_model - >>> from egon.data.datasets.zensus_vg250 import Vg250Sta - >>> resources = generate_resource_fields_from_sqla_model(Vg250Sta) - - Parameters - ---------- - model : sqlalchemy.ext.declarative.declarative_base() - SQLA model - - Returns - ------- - list of dict - Resource fields - """ - - return [ - { - "name": col.name, - "description": "", - "type": str(col.type).lower(), - "unit": "none", - } - for col in model.__table__.columns - ] - - -def generate_resource_fields_from_db_table(schema, table, geom_columns=None): - """Generate a template for the resource fields for metadata from a - database table. - - For details on the fields see field 14.6.1 of `Open Energy Metadata - `_ standard. - The fields `name` and `type` are automatically filled, the `description` - and `unit` must be filled manually. - - Examples - -------- - >>> from egon.data.metadata import generate_resource_fields_from_db_table - >>> resources = generate_resource_fields_from_db_table( - ... 'openstreetmap', 'osm_point', ['geom', 'geom_centroid'] - ... ) # doctest: +SKIP - - Parameters - ---------- - schema : str - The target table's database schema - table : str - Database table on which to put the given comment - geom_columns : list of str - Names of all geometry columns in the table. This is required to return - Geometry data type for those columns as SQL Alchemy does not recognize - them correctly. Defaults to ['geom']. - - Returns - ------- - list of dict - Resource fields - """ - - # handle geometry columns - if geom_columns is None: - geom_columns = ["geom"] - for col in geom_columns: - ischema_names[col] = Geometry - - table = Table( - table, MetaData(), schema=schema, autoload=True, autoload_with=engine() - ) - - return [ - { - "name": col.name, - "description": "", - "type": str(col.type).lower(), - "unit": "none", - } - for col in table.c - ] - - -def sources(): - return { - "bgr_inspee": { - "title": "Salt structures in Northern Germany", - "description": 'The application "Information System Salt Structures" provides information about the ' - "areal distribution of salt structures (stocks and pillows) in Northern Germany. With general structural " - "describing information, such as depth, secondary thickness, types of use or state of exploration, queries " - "can be conducted. Contours of the salt structures can be displayed at horizontal cross-sections at four " - "different depths up to a maximum depth of 2000 m below NN. A data sheet with information and further " - "reading is provided for every single salt structure. Taking into account the fact that this work was " - "undertaken at a scale for providing an overview and not for investigation of single structures, the scale " - "of display is limited to a minimum of 1:300.000. This web application is the product of a BMWi-funded " - 'research project "InSpEE" running from the year 2012 to 2015. The acronym stands for "Information system ' - "salt structures: planning basis, selection criteria and estimation of the potential for the construction " - 'of salt caverns for the storage of renewable energies (hydrogen and compressed air)".', - "path": "https://produktcenter.bgr.de/terraCatalog/DetailResult.do?fileIdentifier=338136ea-261a-4569-a2bf-92999d09bad2", - "licenses": [license_geonutzv("© BGR, Hannover, 2015")], - }, - "bgr_inspeeds": { - "title": "Flat layered salts in Germany", - "description": "Which salt formations are suitable for storing hydrogen or compressed air? " - "In the InSpEE-DS research project, scientists developed requirements and criteria for the assessment " - "of suitable sites even if their exploration is still at an early stage and there is little knowledge of " - "the salinaries structures. Scientists at DEEP.KBB GmbH in Hanover, worked together with their project " - "partners at the Federal Institute for Geosciences and Natural Resources and the Leibniz University " - "Hanover, Institute for Geotechnics Hanover, to develop the planning basis for the site selection and for " - "the construction of storage caverns in flat layered salt and multiple or double saliniferous formations. " - "Such caverns could store renewable energy in the form of hydrogen or compressed air. While the previous " - "project InSpEE was limited to salt formations of great thickness in Northern Germany, salt horizons of " - "different ages have now been examined all over Germany. To estimate the potential, depth contour maps of " - "the top and the base as well as thickness maps of the respective stratigraphic units and reference " - "profiles were developed. Information on compressed air and hydrogen storage potential were given for the " - "identified areas and for the individual federal states. The web service " - '"Information system for flat layered salt" gives access to this data. The scale of display is limited ' - "to a minimum of 1:300.000. This geographic information is product of a BMWi-funded research project " - '"InSpEE-DS" running from the year 2015 to 2019. The acronym stands for "Information system salt: ' - "planning basis, selection criteria and estimation of the potential for the construction of salt caverns " - 'for the storage of renewable energies (hydrogen and compressed air) - double saline and flat salt layers".', - "path": "https://produktcenter.bgr.de/terraCatalog/DetailResult.do?fileIdentifier=630430b8-4025-4d6f-9a62-025b53bc8b3d", - "licenses": [license_geonutzv("© BGR, Hannover, 2021")], - }, - "bgr_inspeeds_data_bundle": { - "title": "Informationssystem Salz: Planungsgrundlagen, Auswahlkriterien und Potenzialabschätzung für die " - "Errichtung von Salzkavernen zur Speicherung von Erneuerbaren Energien (Wasserstoff und Druckluft) – " - "Doppelsalinare und flach lagernde Salzschichten. Teilprojekt Bewertungskriterien und Potenzialabschätzung", - "description": "Shapefiles corresponding to the data provided in figure 7-1 " - "(Donadei, S., et al., 2020, p. 7-5). The energy storage potential data are provided per federal state " - " in table 7-1 (Donadei, S., et al., 2020, p. 7-4). Note: Please include all bgr data sources when using " - "the data.", - "path": "https://dx.doi.org/10.5281/zenodo.4896526", - "licenses": [license_geonutzv("© BGR, Hannover, 2021")], - }, - "bgr_inspeeds_report": { - "title": "Informationssystem Salz: Planungsgrundlagen, Auswahlkriterien und Potenzialabschätzung für die " - "Errichtung von Salzkavernen zur Speicherung von Erneuerbaren Energien (Wasserstoff und Druckluft) – " - "Doppelsalinare und flach lagernde Salzschichten. Teilprojekt Bewertungskriterien und Potenzialabschätzung", - "description": "The report includes availability of saltstructures for energy storage and energy " - "storage potential accumulated per federal state in Germany.", - "path": "https://www.bgr.bund.de/DE/Themen/Nutzung_tieferer_Untergrund_CO2Speicherung/Downloads/InSpeeDS_TP_Bewertungskriterien.pdf?__blob=publicationFile&v=3", - "licenses": [license_geonutzv("© BGR, Hannover, 2021")], - }, - "demandregio": { - "title": "DemandRegio", - "description": "Harmonisierung und Entwicklung von Verfahren zur regionalen und " - "zeitlichen Auflösung von Energienachfragen", - "path": "https://doi.org/10.34805/ffe-119-20", - "licenses": [license_ccby("© FZJ, TUB, FfE")], - }, - "egon-data": { - "titel": "eGon-data", - "description": "Workflow to download, process and generate data sets" - "suitable for the further research conducted in the project eGon (https://ego-n.org/)", - "path": "https://github.com/openego/eGon-data", - "licenses": [license_agpl(EGON_ATTRIBUTION)], - }, - "Einspeiseatlas": { - "title": "Einspeiseatlas", - "description": "Im Einspeiseatlas finden sie sich die Informationen " - "zu realisierten und geplanten Biomethanaufbereitungsanlagen - mit " - "und ohne Einspeisung ins Gasnetz - in Deutschland und weltweit.", - "path": "https://www.biogaspartner.de/einspeiseatlas/", - "licenses": [ - license_ccby("Deutsche Energie-Agentur (dena, 2021)") - ], - }, - "era5": { - "title": "ERA5 global reanalysis", - "description": "ERA5 is the fifth generation ECMWF reanalysis for the global climate " - "and weather for the past 4 to 7 decades. Currently data is available from 1950, " - "split into Climate Data Store entries for 1950-1978 (preliminary back extension) and f" - "rom 1979 onwards (final release plus timely updates, this page). ERA5 replaces the ERA-Interim reanalysis. " - "See the online ERA5 documentation " - "(https://confluence.ecmwf.int/display/CKB/ERA5%3A+data+documentation#ERA5:datadocumentation-Dataupdatefrequency) " - "for more information.", - "path": "https://confluence.ecmwf.int/display/CKB/ERA5%3A+data+documentation#ERA5:datadocumentation-Dataupdatefrequency", - "licenses": [ - { - "name": "Licence to use Copernicus Products", - "title": "Licence to use Copernicus Products", - "path": "https://cds.climate.copernicus.eu/api/v2/terms/static/licence-to-use-copernicus-products.pdf", - "instruction": "This Licence is free of charge, worldwide, non-exclusive, royalty free and perpetual. " - "Access to Copernicus Products is given for any purpose in so far as it is lawful, whereas use " - "may include, but is not limited to: reproduction; distribution; communication to the public; " - "adaptation, modification and combination with other data and information; or any " - "combination of the foregoing", - "attribution": "Copernicus Climate Change Service (C3S) Climate Data Store", - }, - ], - }, - "dsm-heitkoetter": { - "title": "Assessment of the regionalised demand response potential " - "in Germany using an open source tool and dataset", - "description": "With the expansion of renewable energies in Germany, " - "imminent grid congestion events occur more often. One approach for " - "avoiding curtailment of renewable energies is to cover excess feed-in " - "by demand response. As curtailment is often a local phenomenon, in " - "this work we determine the regional demand response potential for " - "the 401 German administrative districts with a temporal resolution " - "of 15 min, including technical, socio-technical and economic " - "restrictions.", - "path": "https://doi.org/10.1016/j.adapen.2020.100001", - "licenses": [ - license_ccby( - "© 2020 German Aerospace Center (DLR), " - "Institute of Networked Energy Systems." - ) - ], - }, - "hotmaps_industrial_sites": { - "titel": "industrial_sites_Industrial_Database", - "description": "Georeferenced industrial sites of energy-intensive industry sectors in EU28", - "path": "https://gitlab.com/hotmaps/industrial_sites/industrial_sites_Industrial_Database", - "licenses": [ - license_ccby("© 2016-2018: Pia Manz, Tobias Fleiter") - ], - }, - "hotmaps_scen_buildings": { - "titel": "scen_current_building_demand", - "description": "Energy demand scenarios in buidlings until the year 2050 - current policy scenario", - "path": "https://gitlab.com/hotmaps/scen_current_building_demand", - "licenses": [ - license_ccby( - "© 2016-2018: Michael Hartner, Lukas Kranzl, Sebastian Forthuber, Sara Fritz, Andreas Müller" - ) - ], - }, - "mastr": { - "title": "open-MaStR power unit registry", - "description": "Raw data download Marktstammdatenregister (MaStR) data " - "using the webservice. All data from the Marktstammdatenregister is included." - "There are duplicates included. For further information read in the documentation" - "of the original data source: https://www.marktstammdatenregister.de/MaStRHilfe/subpages/statistik.html", - "path": "https://sandbox.zenodo.org/record/808086", - "licenses": [ - licenses_datenlizenz_deutschland( - "© 2021 Bundesnetzagentur für Elektrizität, Gas, Telekommunikation, Post und Eisenbahnen" - ) - ], - }, - "nep2021": { - "title": "Netzentwicklungsplan Strom 2035, Version 2021, erster Entwurf", - "description": "Die vier deutschen Übertragungsnetzbetreiber zeigen mit " - "diesem ersten Entwurf des Netzentwicklungsplans 2035, Version 2021, den " - "benötigten Netzausbau für die nächsten Jahre auf. Der NEP-Bericht beschreibt " - "keine konkreten Trassenverläufe von Übertragungsleitungen, sondern er " - "dokumentiert den notwendigen Übertragungsbedarf zwischen Netzknoten. " - "Das heißt, es werden Anfangs- und Endpunkte von zukünftigen Leitungsverbindungen " - "definiert sowie konkrete Empfehlungen für den Aus- und Neubau der Übertragungsnetze " - "an Land und auf See in Deutschland gemäß den Detailanforderungen im § 12 EnWG gegeben.", - "path": "https://zenodo.org/record/5743452#.YbCoz7so8go", - "licenses": [license_ccby("© Übertragungsnetzbetreiber")], - }, - "openffe_gas": { - "title": "Load Curves of the Industry Sector – eXtremOS solidEU Scenario (Europe NUTS-3)", - "description": "Load Curves of the Industry Sector for the eXtremOS solidEU Scenario Scenario at NUTS-3-Level. " - "More information at https://extremos.ffe.de/.", - "path": "http://opendata.ffe.de/dataset/load-curves-of-the-industry-sector-extremos-solideu-scenario-europe-nuts-3/", - "licenses": [license_ccby("© FfE, eXtremOS Project")], - }, - "openstreetmap": { - "title": "OpenStreetMap Data Extracts (Geofabrik)", - "description": "Full data extract of OpenStreetMap data for defined" - "spatial extent at ''referenceDate''", - "path": "https://download.geofabrik.de/europe/germany-210101.osm.pbf", - "licenses": [license_odbl("© OpenStreetMap contributors")], - }, - "peta": { - "title": "Pan-European Thermal Atlas, Peta version 5.0.1", - "description": "Modelled Heat Demand distribution (in GJ per hectare grid cell) for residential and service " - "heat demands for space heating and hot water for the year 2015 using HRE4 data and the combined " - "top-down bottom-up approach of HRE4. " - "National sector-specific heat demand data, derived by the FORECAST model in HRE4 for residential " - "(delivered energy, for space heating and hot water) and service-sector (delivered energy, for space heating, hot " - "water and process heat) buildings for the year 2015, were distributed using modelled, spatial " - "statistics based floor areas in 100x100m grids and a population grid. " - "For further information please see the documentation available on the Heat Roadmap Europe website, " - "in particular D2.3 report: Methodologies and assumptions used in the mapping.", - "path": "https://s-eenergies-open-data-euf.hub.arcgis.com/search", - "licenses": [ - license_ccby( - "© Europa-Universität Flensburg, Halmstad University and Aalborg University" - ) - ], - }, - "pipeline_classification": { - "title": "Technical pipeline characteristics for high pressure pipelines", - "description": "Parameters for the classification of gas pipelines, " - "the whole documentation could is available at: " - "https://www.econstor.eu/bitstream/10419/173388/1/1011162628.pdf", - "path": "https://zenodo.org/record/5743452", - "licenses": [license_ccby("© DIW Berlin, 2017")], - }, - "schmidt": { - "title": "Supplementary material to the masters thesis: " - "NUTS-3 Regionalization of Industrial Load Shifting Potential in Germany using a Time-Resolved Model", - "description": "Supplementary material to the named masters thesis, containing data on industrial processes" - "for the estimation of NUTS-3 load shifting potential of suitable electrically powered industrial processes" - "(cement milling, mechanical pulping, paper production, air separation).", - "path": "https://zenodo.org/record/3613767", - "licenses": [license_ccby("© 2019 Danielle Schmidt")], - }, - "SciGRID_gas": { - "title": "SciGRID_gas IGGIELGN", - "description": "The SciGRID_gas dataset represents the European " - "gas transport network (pressure levels of 20 bars and higher) " - "including the geo-referenced transport pipelines, compressor " - "stations, LNG terminals, storage, production sites, gas power " - "plants, border points, and demand time series. ", - "path": "https://dx.doi.org/10.5281/zenodo.4896526", - "licenses": [ - license_ccby( - "Jan Diettrich; Adam Pluta; Wided Medjroubi (DLR-VE)" - ), - ], - }, - "seenergies": { - "title": "D5 1 Industry Dataset With Demand Data", - "description": "Georeferenced EU28 industrial sites with quantified annual excess heat volumes and demand data" - "within main sectors: Chemical industry, Iron and steel, Non-ferrous metals, Non-metallic minerals, Paper and printing, and Refineries.", - "path": "https://s-eenergies-open-data-euf.hub.arcgis.com/datasets/5e36c0af918040ed936b4e4c101f611d_0/about", - "licenses": [license_ccby("© Europa-Universität Flensburg")], - }, - "technology-data": { - "titel": "Energy System Technology Data v0.3.0", - "description": "This script compiles assumptions on energy system " - "technologies (such as costs, efficiencies, lifetimes, etc.) for " - "chosen years (e.g. [2020, 2030, 2050]) from a variety of sources " - "into CSV files to be read by energy system modelling software. " - "The merged outputs have standardized cost years, technology names, " - "units and source information.", - "path": "https://github.com/PyPSA/technology-data/tree/v0.3.0", - "licenses": [ - license_agpl( - "© Marta Victoria (Aarhus University), Kun Zhu (Aarhus University), Elisabeth Zeyen (TUB), Tom Brown (TUB)" - ) - ], - }, - "tyndp": { - "title": "Ten-Year Network Development Plan (TYNDP) 2020 Scenarios", - "description": "ENTSOs’ TYNDP 2020 Scenario Report describes possible European energy futures up to 2050. " - "Scenarios are not forecasts; they set out a range of possible futures used by the ENTSOs to test future " - "electricity and gas infrastructure needs and projects. The scenarios are ambitious as they deliver " - "a low carbon energy system for Europe by 2050. The ENTSOs have developed credible scenarios that are " - "guided by technically sound pathways, while reflecting country by country specifics, so that a pan-European " - "low carbon future is achieved.", - "path": "https://tyndp.entsoe.eu/maps-data", - "licenses": [license_ccby("© ENTSO-E and ENTSOG")], - }, - "vg250": { - "title": "Verwaltungsgebiete 1:250 000 (Ebenen)", - "description": "Der Datenbestand umfasst sämtliche Verwaltungseinheiten der " - "hierarchischen Verwaltungsebenen vom Staat bis zu den Gemeinden " - "mit ihren Grenzen, statistischen Schlüsselzahlen, Namen der " - "Verwaltungseinheit sowie die spezifische Bezeichnung der " - "Verwaltungsebene des jeweiligen Landes.", - "path": "https://daten.gdz.bkg.bund.de/produkte/vg/vg250_ebenen_0101/2020/vg250_01-01.geo84.shape.ebenen.zip", - "licenses": [ - licenses_datenlizenz_deutschland( - "© Bundesamt für Kartographie und Geodäsie " - "2020 (Daten verändert)" - ) - ], - }, - "zensus": { - "title": "Statistisches Bundesamt (Destatis) - Ergebnisse des Zensus 2011 zum Download", - "description": "Als Download bieten wir Ihnen auf dieser Seite zusätzlich zur " - "Zensusdatenbank CSV- und teilweise Excel-Tabellen mit umfassenden Personen-, Haushalts- " - "und Familien- sowie Gebäude- und Wohnungs­merkmalen. Die Ergebnisse liegen auf Bundes-, " - "Länder-, Kreis- und Gemeinde­ebene vor. Außerdem sind einzelne Ergebnisse für Gitterzellen verfügbar.", - "path": "https://www.zensus2011.de/SharedDocs/Aktuelles/Ergebnisse/DemografischeGrunddaten.html;jsessionid=E0A2B4F894B258A3B22D20448F2E4A91.2_cid380?nn=3065474", - "licenses": [ - licenses_datenlizenz_deutschland( - "© Statistische Ämter des Bundes und der Länder 2014" - ) - ], - }, - } - - -def contributors(authorlist): - contributors_dict = { - "am": { - "title": "Aadit Malla", - "email": "https://github.com/aadit879", - }, - "an": { - "title": "Amélia Nadal", - "email": "https://github.com/AmeliaNadal", - }, - "cb": { - "title": "Clara Büttner", - "email": "https://github.com/ClaraBuettner", - }, - "ce": { - "title": "Carlos Epia", - "email": "https://github.com/CarlosEpia", - }, - "fw": { - "title": "Francesco Witte", - "email": "https://github.com/fwitte", - }, - "gp": { - "title": "Guido Pleßmann", - "email": "https://github.com/gplssm", - }, - "ic": { - "title": "Ilka Cußmann", - "email": "https://github.com/IlkaCu", - }, - "ja": { - "title": "Jonathan Amme", - "email": "https://github.com/nesnoj", - }, - "je": { - "title": "Jane Doe", - "email": "https://github.com/JaneDoe", - }, - "ke": { - "title": "Katharina Esterl", - "email": "https://github.com/KathiEsterl", - }, - "kh": { - "title": "Kilian Helfenbein", - "email": "https://github.com/khelfen", - }, - "sg": { - "title": "Stephan Günther", - "email": "https://github.com/gnn", - }, - "um": { - "title": "Ulf Müller", - "email": "https://github.com/ulfmueller", - }, - } - return [ - {key: value for key, value in contributors_dict[author].items()} - for author in authorlist - ] diff --git a/src/egon/data/metadata/__init__.py b/src/egon/data/metadata/__init__.py new file mode 100755 index 000000000..dff660905 --- /dev/null +++ b/src/egon/data/metadata/__init__.py @@ -0,0 +1,964 @@ +from geoalchemy2 import Geometry +from omi.dialects import get_dialect +from sqlalchemy import MetaData, Table +from sqlalchemy.dialects.postgresql.base import ischema_names +import importlib_resources + +from egon.data import db, logger +from egon.data.datasets import Dataset +from egon.data.db import engine + + +def context(): + """ + Project context information for metadata + + Returns + ------- + dict + OEP metadata conform data license information + """ + + return { + "homepage": "https://ego-n.org/", + "documentation": "https://egon-data.readthedocs.io/en/latest/", + "sourceCode": "https://github.com/openego/eGon-data", + "contact": "https://ego-n.org/partners/", + "grantNo": "03EI1002", + "fundingAgency": "Bundesministerium für Wirtschaft und Energie", + "fundingAgencyLogo": "https://www.innovation-beratung-" + "foerderung.de/INNO/Redaktion/DE/Bilder/" + "Titelbilder/titel_foerderlogo_bmwi.jpg?" + "__blob=normal&v=3", + "publisherLogo": "https://ego-n.org/images/eGon_logo_" + "noborder_transbg.svg", + } + + +def meta_metadata(): + """ + Meta data on metadata + + Returns + ------- + dict + OEP metadata conform metadata on metadata + """ + + return { + "metadataVersion": "OEP-1.4.1", + "metadataLicense": { + "name": "CC0-1.0", + "title": "Creative Commons Zero v1.0 Universal", + "path": ("https://creativecommons.org/publicdomain/zero/1.0/"), + }, + } + + +def licenses_datenlizenz_deutschland(attribution): + """ + License information for Datenlizenz Deutschland + + Parameters + ---------- + attribution : str + Attribution for the dataset incl. © symbol, e.g. '© GeoBasis-DE / BKG' + + Returns + ------- + dict + OEP metadata conform data license information + """ + + return { + "name": "dl-by-de/2.0", + "title": "Datenlizenz Deutschland – Namensnennung – Version 2.0", + "path": "www.govdata.de/dl-de/by-2-0", + "instruction": ( + "Jede Nutzung ist unter den Bedingungen dieser „Datenlizenz " + "Deutschland - Namensnennung - Version 2.0 zulässig.\nDie " + "bereitgestellten Daten und Metadaten dürfen für die " + "kommerzielle und nicht kommerzielle Nutzung insbesondere:" + "(1) vervielfältigt, ausgedruckt, präsentiert, verändert, " + "bearbeitet sowie an Dritte übermittelt werden;\n " + "(2) mit eigenen Daten und Daten Anderer zusammengeführt und " + "zu selbständigen neuen Datensätzen verbunden werden;\n " + "(3) in interne und externe Geschäftsprozesse, Produkte und " + "Anwendungen in öffentlichen und nicht öffentlichen " + "elektronischen Netzwerken eingebunden werden.\n" + "Bei der Nutzung ist sicherzustellen, dass folgende Angaben " + "als Quellenvermerk enthalten sind:\n" + "(1) Bezeichnung des Bereitstellers nach dessen Maßgabe,\n" + "(2) der Vermerk Datenlizenz Deutschland – Namensnennung – " + "Version 2.0 oder dl-de/by-2-0 mit Verweis auf den Lizenztext " + "unter www.govdata.de/dl-de/by-2-0 sowie\n" + "(3) einen Verweis auf den Datensatz (URI)." + "Dies gilt nur soweit die datenhaltende Stelle die Angaben" + "(1) bis (3) zum Quellenvermerk bereitstellt.\n" + "Veränderungen, Bearbeitungen, neue Gestaltungen oder " + "sonstige Abwandlungen sind im Quellenvermerk mit dem Hinweis " + "zu versehen, dass die Daten geändert wurden." + ), + "attribution": attribution, + } + + +def license_odbl(attribution): + """ + License information for Open Data Commons Open Database License (ODbL-1.0) + + Parameters + ---------- + attribution : str + Attribution for the dataset incl. © symbol, e.g. + '© OpenStreetMap contributors' + + Returns + ------- + dict + OEP metadata conform data license information + """ + return { + "name": "ODbL-1.0", + "title": "Open Data Commons Open Database License 1.0", + "path": "https://opendatacommons.org/licenses/odbl/1.0/index.html", + "instruction": "You are free: To Share, To Create, To Adapt; " + "As long as you: Attribute, Share-Alike, Keep open!", + "attribution": attribution, + } + + +def license_ccby(attribution): + """ + License information for Creative Commons Attribution 4.0 International + (CC-BY-4.0) + + Parameters + ---------- + attribution : str + Attribution for the dataset incl. © symbol, e.g. '© GeoBasis-DE / BKG' + + Returns + ------- + dict + OEP metadata conform data license information + """ + return { + "name": "CC-BY-4.0", + "title": "Creative Commons Attribution 4.0 International", + "path": "https://creativecommons.org/licenses/by/4.0/legalcode", + "instruction": "You are free: To Share, To Create, To Adapt; " + "As long as you: Attribute.", + "attribution": attribution, + } + + +def license_geonutzv(attribution): + """ + License information for GeoNutzV + + Parameters + ---------- + attribution : str + Attribution for the dataset incl. © symbol, e.g. '© GeoBasis-DE / BKG' + + Returns + ------- + dict + OEP metadata conform data license information + """ + return { + "name": "geonutzv-de-2013-03-19", + "title": "Verordnung zur Festlegung der Nutzungsbestimmungen für die " + "Bereitstellung von Geodaten des Bundes", + "path": "https://www.gesetze-im-internet.de/geonutzv/", + "instruction": "Geodaten und Geodatendienste, einschließlich " + "zugehöriger Metadaten, werden für alle derzeit " + "bekannten sowie für alle zukünftig bekannten Zwecke " + "kommerzieller und nicht kommerzieller Nutzung " + "geldleistungsfrei zur Verfügung gestellt, soweit " + "durch besondere Rechtsvorschrift nichts anderes " + "bestimmt ist oder vertragliche oder gesetzliche " + "Rechte Dritter dem nicht entgegenstehen.", + "attribution": attribution, + } + + +def license_agpl(attribution): + """ + License information for GNU Affero General Public License v3.0 + + Parameters + ---------- + attribution : str + Attribution for the dataset incl. © symbol, e.g. '© GeoBasis-DE / BKG' + + Returns + ------- + dict + OEP metadata conform data license information + """ + return { + "name": "AGPL-3.0 License", + "title": "GNU Affero General Public License v3.0", + "path": "https://www.gnu.org/licenses/agpl-3.0.de.html", + "instruction": "Permissions of this strongest copyleft license are" + "conditioned on making available complete source code of licensed " + "works and modifications, which include larger works using a licensed" + "work, under the same license. Copyright and license notices must be" + "preserved. Contributors provide an express grant of patent rights." + "When a modified version is used to provide a service over a network," + "the complete source code of the modified version must be made " + "available.", + "attribution": attribution, + } + + +def license_dedl(attribution): + """ + License information for Data licence Germany – attribution – version 2.0 + + Parameters + ---------- + attribution : str + Attribution for the dataset incl. © symbol, e.g. '© GeoBasis-DE / BKG' + + Returns + ------- + dict + OEP metadata conform data license information + """ + return { + "name": "DL-DE-BY-2.0", + "title": "Data licence Germany – attribution – version 2.0", + "path": "https://www.govdata.de/dl-de/by-2-0", + "instruction": ( + "Any use will be permitted provided it fulfils the requirements of" + " this 'Data licence Germany – attribution – Version 2.0'. The " + "data and meta-data provided may, for commercial and " + "non-commercial use, in particular be copied, printed, presented, " + "altered, processed and transmitted to third parties; be merged " + "with own data and with the data of others and be combined to form" + " new and independent datasets; be integrated in internal and " + "external business processes, products and applications in public " + "and non-public electronic networks. The user must ensure that the" + " source note contains the following information: the name of the " + "provider, the annotation 'Data licence Germany – attribution – " + "Version 2.0' or 'dl-de/by-2-0' referring to the licence text " + "available at www.govdata.de/dl-de/by-2-0, and a reference to the " + "dataset (URI). This applies only if the entity keeping the data " + "provides the pieces of information 1-3 for the source note. " + "Changes, editing, new designs or other amendments must be marked " + "as such in the source note." + ), + "attribution": attribution, + } + + +def license_egon_data_odbl(): + """ + ODbL license with eGon data attribution + + Returns + ------- + dict + OEP metadata conform data license information for eGon tables + """ + return license_odbl("© eGon development team") + + +def generate_resource_fields_from_sqla_model(model): + """Generate a template for the resource fields for metadata from a SQL + Alchemy model. + + For details on the fields see field 14.6.1 of `Open Energy Metadata + `_ standard. + The fields `name` and `type` are automatically filled, the `description` + and `unit` must be filled manually. + + Examples + -------- + >>> from egon.data.metadata import generate_resource_fields_from_sqla_model + >>> from egon.data.datasets.zensus_vg250 import Vg250Sta + >>> resources = generate_resource_fields_from_sqla_model(Vg250Sta) + + Parameters + ---------- + model : sqlalchemy.ext.declarative.declarative_base() + SQLA model + + Returns + ------- + list of dict + Resource fields + """ + + return [ + { + "name": col.name, + "description": "", + "type": str(col.type).lower(), + "unit": "none", + } + for col in model.__table__.columns + ] + + +def generate_resource_fields_from_db_table(schema, table, geom_columns=None): + """Generate a template for the resource fields for metadata from a + database table. + + For details on the fields see field 14.6.1 of `Open Energy Metadata + `_ standard. + The fields `name` and `type` are automatically filled, the `description` + and `unit` must be filled manually. + + Examples + -------- + >>> from egon.data.metadata import generate_resource_fields_from_db_table + >>> resources = generate_resource_fields_from_db_table( + ... 'openstreetmap', 'osm_point', ['geom', 'geom_centroid'] + ... ) # doctest: +SKIP + + Parameters + ---------- + schema : str + The target table's database schema + table : str + Database table on which to put the given comment + geom_columns : list of str + Names of all geometry columns in the table. This is required to return + Geometry data type for those columns as SQL Alchemy does not recognize + them correctly. Defaults to ['geom']. + + Returns + ------- + list of dict + Resource fields + """ + + # handle geometry columns + if geom_columns is None: + geom_columns = ["geom"] + for col in geom_columns: + ischema_names[col] = Geometry + + table = Table( + table, MetaData(), schema=schema, autoload=True, autoload_with=engine() + ) + + return [ + { + "name": col.name, + "description": "", + "type": str(col.type).lower(), + "unit": "none", + } + for col in table.c + ] + + +def sources(): + shared_licenses = [license_geonutzv("© BGR, Hannover, 2021")] + shared_path = "https://dx.doi.org/10.5281/zenodo.4896526" + shared_title = ( + "Informationssystem Salz: Planungsgrundlagen, Auswahlkriterien" + " und Potenzialabschätzung für die Errichtung von Salzkavernen" + " zur Speicherung von Erneuerbaren Energien" + " (Wasserstoff und Druckluft)" + " – Doppelsalinare und flach lagernde Salzschichten." + " Teilprojekt Bewertungskriterien und Potenzialabschätzung" + ) + return { + "bgr_inspee": { + "title": "Salt structures in Northern Germany", + "description": ( + 'The application "Information System Salt Structures"' + " provides information about the areal distribution of" + " salt structures (stocks and pillows) in Northern" + " Germany. With general structural describing" + " information, such as depth, secondary thickness," + " types of use or state of exploration, queries can be" + " conducted. Contours of the salt structures can be" + " displayed at horizontal cross-sections at four" + " different depths up to a maximum depth of 2000 m" + " below NN. A data sheet with information and further" + " reading is provided for every single salt structure." + " Taking into account the fact that this work was" + " undertaken at a scale for providing an overview and" + " not for investigation of single structures, the scale" + " of display is limited to a minimum of 1:300.000." + " This web application is the product of a BMWi-funded" + ' research project "InSpEE" running from the year 2012' + ' to 2015. The acronym stands for "Information system' + " salt structures: planning basis, selection criteria" + " and estimation of the potential for the construction" + " of salt caverns for the storage of renewable energies" + ' (hydrogen and compressed air)".' + ), + "path": ( + "https://produktcenter.bgr.de/terraCatalog/DetailResult.do" + "?fileIdentifier=338136ea-261a-4569-a2bf-92999d09bad2" + ), + "licenses": [license_geonutzv("© BGR, Hannover, 2015")], + }, + "bgr_inspeeds": { + "title": "Flat layered salts in Germany", + "description": ( + "Which salt formations are suitable for storing" + " hydrogen or compressed air?" + " In the InSpEE-DS research project, scientists" + " developed requirements and criteria for the" + " assessment of suitable sites even if their" + " exploration is still at an early stage and there is" + " little knowledge of the salinaries' structures." + " Scientists at DEEP.KBB GmbH in Hanover, worked" + " together with their project partners at the Federal" + " Institute for Geosciences and Natural Resources and" + " the Leibniz University Hanover, Institute for" + " Geotechnics Hanover, to develop the planning basis" + " for the site selection and for the construction of" + " storage caverns in flat layered salt and multiple or" + " double saliniferous formations." + " Such caverns could store renewable energy in the form" + " of hydrogen or compressed air." + " While the previous project InSpEE was limited to salt" + " formations of great thickness in Northern Germany," + " salt horizons of different ages have now been" + " examined all over Germany. To estimate the potential," + " depth contour maps of the top and the base as well as" + " thickness maps of the respective stratigraphic units" + " and reference profiles were developed. Information on" + " compressed air and hydrogen storage potential were" + " given for the identified areas and for the individual" + " federal states. The web service" + ' "Information system for flat layered salt"' + " gives access to this data. The scale of display is" + " limited to a minimum of 1:300.000. This geographic" + " information is product of a BMWi-funded research" + ' project "InSpEE-DS" running from the year 2015 to' + " 2019. The acronym stands for" + ' "Information system salt: planning basis, selection' + " criteria and estimation of the potential for the" + " construction of salt caverns for the storage of" + " renewable energies (hydrogen and compressed air)" + ' - double saline and flat salt layers".' + ), + "path": ( + "https://produktcenter.bgr.de/terraCatalog/DetailResult.do" + "?fileIdentifier=630430b8-4025-4d6f-9a62-025b53bc8b3d" + ), + "licenses": shared_licenses, + }, + "bgr_inspeeds_data_bundle": { + "title": shared_title, + "description": ( + "Shapefiles corresponding to the data provided in" + " figure 7-1 (Donadei, S., et al., 2020, p. 7-5)." + " The energy storage potential data are provided per" + " federal state in table 7-1" + " (Donadei, S., et al., 2020, p. 7-4)." + " Note: Please include all bgr data sources when using" + " the data." + ), + "path": shared_path, + "licenses": shared_licenses, + }, + "bgr_inspeeds_report": { + "title": shared_title, + "description": ( + "The report includes availability of saltstructures for" + " energy storage and energy storage potential" + " accumulated per federal state in Germany." + ), + "path": ( + "https://www.bgr.bund.de/DE/Themen" + "/Nutzung_tieferer_Untergrund_CO2Speicherung/Downloads" + "/InSpeeDS_TP_Bewertungskriterien.pdf" + "?__blob=publicationFile&v=3" + ), + "licenses": shared_licenses, + }, + "demandregio": { + "title": "DemandRegio", + "description": ( + "Harmonisierung und Entwicklung von Verfahren zur" + " regionalen und zeitlichen Auflösung von" + " Energienachfragen" + ), + "path": "https://doi.org/10.34805/ffe-119-20", + "licenses": [license_ccby("© FZJ, TUB, FfE")], + }, + "egon-data": { + "title": "eGon-data", + "description": ( + "Workflow to download, process and generate data sets" + " suitable for the further research conducted in the" + " project eGon (https://ego-n.org/)" + ), + "path": "https://github.com/openego/eGon-data", + "licenses": [license_agpl("© eGon development team")], + }, + "egon-data_bundle": { + "title": "Data bundle for egon-data", + "description": ( + "Zenodo repository to provide several different input" + " data sets for eGon-data" + ), + "path": "https://sandbox.zenodo.org/record/1167119", + "licenses": [license_ccby("© eGon development team")], + }, + "Einspeiseatlas": { + "title": "Einspeiseatlas", + "description": ( + "Im Einspeiseatlas finden sie sich die Informationen zu" + " realisierten und geplanten Biomethanaufbereitungsanlagen" + " - mit und ohne Einspeisung ins Gasnetz -" + " in Deutschland und weltweit." + ), + "path": "https://www.biogaspartner.de/einspeiseatlas/", + "licenses": [ + license_ccby("Deutsche Energie-Agentur (dena, 2021)") + ], + }, + "era5": { + "title": "ERA5 global reanalysis", + "description": ( + "ERA5 is the fifth generation ECMWF reanalysis for the" + " global climate and weather for the past 4 to 7" + " decades. Currently data is available from 1950, split" + " into Climate Data Store entries for 1950-1978" + " (preliminary back extension) and from 1979 onwards" + " (final release plus timely updates, this page)." + " ERA5 replaces the ERA-Interim reanalysis." + " See the online ERA5 documentation (" + "https://confluence.ecmwf.int/display/CKB" + "/ERA5%3A+data+documentation" + "#ERA5:datadocumentation-Dataupdatefrequency)" + " for more information." + ), + "path": ( + "https://confluence.ecmwf.int/display/CKB" + "/ERA5%3A+data+documentation" + "#ERA5:datadocumentation-Dataupdatefrequency" + ), + "licenses": [ + { + "name": "Licence to use Copernicus Products", + "title": "Licence to use Copernicus Products", + "path": ( + "https://cds.climate.copernicus.eu/api/v2/terms" + "/static/licence-to-use-copernicus-products.pdf" + ), + "instruction": ( + "This Licence is free of charge, worldwide," + " non-exclusive, royalty free and perpetual." + " Access to Copernicus Products is given for" + " any purpose in so far as it is lawful," + " whereas use may include, but is not limited" + " to: reproduction; distribution; communication" + " to the public; adaptation, modification and" + " combination with other data and information;" + " or any combination of the foregoing" + ), + "attribution": ( + "Copernicus Climate Change Service (C3S)" + " Climate Data Store" + ), + }, + ], + }, + "dsm-heitkoetter": { + "title": ( + "Assessment of the regionalised demand response" + " potential in Germany using an open source tool and" + " dataset" + ), + "description": ( + "With the expansion of renewable energies in Germany," + " imminent grid congestion events occur more often. One" + " approach for avoiding curtailment of renewable" + " energies is to cover excess feed-in by demand" + " response." + " As curtailment is often a local phenomenon, in this" + " work we determine the regional demand response" + " potential for the 401 German administrative districts" + " with a temporal resolution of 15 min, including" + " technical, socio-technical and economic restrictions." + ), + "path": "https://doi.org/10.1016/j.adapen.2020.100001", + "licenses": [ + license_ccby( + "© 2020 German Aerospace Center (DLR)," + " Institute of Networked Energy Systems." + ) + ], + }, + "hotmaps_industrial_sites": { + "titel": "industrial_sites_Industrial_Database", + "description": ( + "Georeferenced industrial sites of energy-intensive" + " industry sectors in EU28" + ), + "path": ( + "https://gitlab.com/hotmaps/industrial_sites" + "/industrial_sites_Industrial_Database" + ), + "licenses": [ + license_ccby("© 2016-2018: Pia Manz, Tobias Fleiter") + ], + }, + "hotmaps_scen_buildings": { + "titel": "scen_current_building_demand", + "description": ( + "Energy demand scenarios in buidlings until the year 2050" + " - current policy scenario" + ), + "path": "https://gitlab.com/hotmaps/scen_current_building_demand", + "licenses": [ + license_ccby( + "© 2016-2018: Michael Hartner" + ", Lukas Kranzl" + ", Sebastian Forthuber" + ", Sara Fritz" + ", Andreas Müller" + ) + ], + }, + "mastr": { + "title": "open-MaStR power unit registry", + "description": ( + "Raw data download Marktstammdatenregister (MaStR) data" + " using the webservice. All data from the" + " Marktstammdatenregister is included. There are" + " duplicates included. For further information read in" + " the documentation of the original data source:" + " https://www.marktstammdatenregister.de/MaStRHilfe" + "/subpages/statistik.html" + ), + "path": "https://sandbox.zenodo.org/record/808086", + "licenses": [ + licenses_datenlizenz_deutschland( + "© 2021 Bundesnetzagentur für Elektrizität, Gas," + " Telekommunikation, Post und Eisenbahnen" + ) + ], + }, + "nep2021": { + "title": ( + "Netzentwicklungsplan Strom 2035, Version 2021, erster" + " Entwurf" + ), + "description": ( + "Die vier deutschen Übertragungsnetzbetreiber zeigen" + " mit diesem ersten Entwurf des Netzentwicklungsplans" + " 2035, Version 2021, den benötigten Netzausbau für die" + " nächsten Jahre auf. Der NEP-Bericht beschreibt keine" + " konkreten Trassenverläufe von Übertragungsleitungen," + " sondern er dokumentiert den notwendigen" + " Übertragungsbedarf zwischen Netzknoten." + " Das heißt, es werden Anfangs- und Endpunkte von" + " zukünftigen Leitungsverbindungen definiert sowie" + " konkrete Empfehlungen für den Aus- und Neubau der" + " Übertragungsnetze an Land und auf See in Deutschland" + " gemäß den Detailanforderungen im § 12 EnWG gegeben." + ), + "path": "https://zenodo.org/record/5743452#.YbCoz7so8go", + "licenses": [license_ccby("© Übertragungsnetzbetreiber")], + }, + "openffe_gas": { + "title": ( + "Load Curves of the Industry Sector" + " – eXtremOS solidEU Scenario (Europe NUTS-3)" + ), + "description": ( + "Load Curves of the Industry Sector for the eXtremOS" + " solidEU Scenario Scenario at NUTS-3-Level." + " More information at https://extremos.ffe.de/." + ), + "path": ( + "http://opendata.ffe.de/dataset" + "/load-curves-of-the-industry-sector-extremos-solideu" + "-scenario-europe-nuts-3/" + ), + "licenses": [license_ccby("© FfE, eXtremOS Project")], + }, + "openstreetmap": { + "title": "OpenStreetMap Data Extracts (Geofabrik)", + "description": ( + "Full data extract of OpenStreetMap data for defined" + " spatial extent at ''referenceDate''" + ), + "path": ( + "https://download.geofabrik.de/europe/germany-210101.osm.pbf" + ), + "licenses": [license_odbl("© OpenStreetMap contributors")], + }, + "peta": { + "title": "Pan-European Thermal Atlas, Peta version 5.0.1", + "description": ( + "Modelled Heat Demand distribution (in GJ per hectare" + " grid cell) for residential and service heat demands" + " for space heating and hot water for the year 2015" + " using HRE4 data and the combined top-down bottom-up" + " approach of HRE4. National sector-specific heat" + " demand data, derived by the FORECAST model in HRE4" + " for residential (delivered energy, for space heating" + " and hot water) and service-sector (delivered energy," + " for space heating, hot water and process heat)" + " buildings for the year 2015, were distributed using" + " modelled, spatial statistics based floor areas in" + " 100x100m grids and a population grid. For further" + " information please see the documentation available on" + " the Heat Roadmap Europe website, in particular D2.3" + " report: Methodologies and assumptions used in the" + " mapping." + ), + "path": "https://s-eenergies-open-data-euf.hub.arcgis.com/search", + "licenses": [ + license_ccby( + "© Europa-Universität Flensburg" + ", Halmstad University and Aalborg University" + ) + ], + }, + "pipeline_classification": { + "title": ( + "Technical pipeline characteristics for high pressure" + " pipelines" + ), + "description": ( + "Parameters for the classification of gas pipelines," + " the whole documentation could is available at:" + " https://www.econstor.eu/bitstream/10419/173388/1" + "/1011162628.pdf" + ), + "path": "https://zenodo.org/record/5743452", + "licenses": [license_ccby("© DIW Berlin, 2017")], + }, + "schmidt": { + "title": ( + "Supplementary material to the masters thesis:" + " NUTS-3 Regionalization of Industrial Load Shifting" + " Potential in Germany using a Time-Resolved Model" + ), + "description": ( + "Supplementary material to the named masters thesis," + " containing data on industrial processes for the" + " estimation of NUTS-3 load shifting potential of" + " suitable electrically powered industrial processes" + " (cement milling, mechanical pulping, paper" + " production, air separation)." + ), + "path": "https://zenodo.org/record/3613767", + "licenses": [license_ccby("© 2019 Danielle Schmidt")], + }, + "SciGRID_gas": { + "title": "SciGRID_gas IGGIELGN", + "description": ( + "The SciGRID_gas dataset represents the European gas" + " transport network (pressure levels of 20 bars and" + " higher) including the geo-referenced transport" + " pipelines, compressor stations, LNG terminals," + " storage, production sites, gas power plants, border" + " points, and demand time series." + ), + "path": shared_path, + "licenses": [ + license_ccby( + "Jan Diettrich; Adam Pluta; Wided Medjroubi (DLR-VE)" + ), + ], + }, + "seenergies": { + "title": "D5 1 Industry Dataset With Demand Data", + "description": ( + "Georeferenced EU28 industrial sites with quantified" + " annual excess heat volumes and demand data within" + " main sectors: Chemical industry, Iron and steel," + " Non-ferrous metals, Non-metallic minerals, Paper and" + " printing, and Refineries." + ), + "path": ( + "https://s-eenergies-open-data-euf.hub.arcgis.com" + "/datasets/5e36c0af918040ed936b4e4c101f611d_0/about" + ), + "licenses": [license_ccby("© Europa-Universität Flensburg")], + }, + "technology-data": { + "titel": "Energy System Technology Data v0.3.0", + "description": ( + "This script compiles assumptions on energy system" + " technologies (such as costs, efficiencies, lifetimes," + " etc.) for chosen years (e.g. [2020, 2030, 2050]) from" + " a variety of sources into CSV files to be read by" + " energy system modelling software. The merged outputs" + " have standardized cost years, technology names, units" + " and source information." + ), + "path": "https://github.com/PyPSA/technology-data/tree/v0.3.0", + "licenses": [ + license_agpl( + "© Marta Victoria (Aarhus University)" + ", Kun Zhu (Aarhus University)" + ", Elisabeth Zeyen (TUB)" + ", Tom Brown (TUB)" + ) + ], + }, + "tyndp": { + "description": ( + "ENTSOs’ TYNDP 2020 Scenario Report describes possible" + " European energy futures up to 2050. Scenarios are not" + " forecasts; they set out a range of possible futures" + " used by the ENTSOs to test future electricity and gas" + " infrastructure needs and projects. The scenarios are" + " ambitious as they deliver a low carbon energy system" + " for Europe by 2050. The ENTSOs have developed" + " credible scenarios that are guided by technically" + " sound pathways, while reflecting country by country" + " specifics, so that a pan-European low carbon future" + " is achieved." + ), + "path": "https://tyndp.entsoe.eu/maps-data", + "licenses": [license_ccby("© ENTSO-E and ENTSOG")], + }, + "vg250": { + "title": "Verwaltungsgebiete 1:250 000 (Ebenen)", + "description": ( + "Der Datenbestand umfasst sämtliche Verwaltungseinheiten" + " der hierarchischen Verwaltungsebenen vom Staat bis zu" + " den Gemeinden mit ihren Grenzen, statistischen" + " Schlüsselzahlen, Namen der Verwaltungseinheit sowie" + " die spezifische Bezeichnung der Verwaltungsebene des" + " jeweiligen Landes." + ), + "path": ( + "https://daten.gdz.bkg.bund.de/produkte/vg" + "/vg250_ebenen_0101/2020" + "/vg250_01-01.geo84.shape.ebenen.zip" + ), + "licenses": [ + licenses_datenlizenz_deutschland( + "© Bundesamt für Kartographie und Geodäsie" + " 2020 (Daten verändert)" + ) + ], + }, + "zensus": { + "title": ( + "Statistisches Bundesamt (Destatis)" + " - Ergebnisse des Zensus 2011 zum Download" + ), + "description": ( + "Als Download bieten wir Ihnen auf dieser Seite" + " zusätzlich zur Zensusdatenbank CSV- und teilweise" + " Excel-Tabellen mit umfassenden Personen-, Haushalts-" + " und Familien- sowie Gebäude- und Wohnungs­merkmaln." + " Die Ergebnisse liegen auf Bundes-, Länder-, Kreis-" + " und Gemeinde­ebene vor. Außerdem sind einzele" + " Ergebnisse für Gitterzellen verfügbar." + ), + "path": ( + "https://www.zensus2011.de/DE/Home/Aktuelles" + "/DemografischeGrunddaten.html" + ), + "licenses": [ + licenses_datenlizenz_deutschland( + "© Statistische Ämter des Bundes und der Länder 2014" + ) + ], + }, + } + + +def contributors(authorlist): + contributors_dict = { + "am": { + "title": "Aadit Malla", + "email": "https://github.com/aadit879", + }, + "an": { + "title": "Amélia Nadal", + "email": "https://github.com/AmeliaNadal", + }, + "cb": { + "title": "Clara Büttner", + "email": "https://github.com/ClaraBuettner", + }, + "ce": { + "title": "Carlos Epia", + "email": "https://github.com/CarlosEpia", + }, + "fw": { + "title": "Francesco Witte", + "email": "https://github.com/fwitte", + }, + "gp": { + "title": "Guido Pleßmann", + "email": "https://github.com/gplssm", + }, + "ic": { + "title": "Ilka Cußmann", + "email": "https://github.com/IlkaCu", + }, + "ja": { + "title": "Jonathan Amme", + "email": "https://github.com/nesnoj", + }, + "je": { + "title": "Jane Doe", + "email": "https://github.com/JaneDoe", + }, + "ke": { + "title": "Katharina Esterl", + "email": "https://github.com/KathiEsterl", + }, + "kh": { + "title": "Kilian Helfenbein", + "email": "https://github.com/khelfen", + }, + "sg": { + "title": "Stephan Günther", + "email": "https://github.com/gnn", + }, + "um": { + "title": "Ulf Müller", + "email": "https://github.com/ulfmueller", + }, + } + return [ + {key: value for key, value in contributors_dict[author].items()} + for author in authorlist + ] + + +def upload_json_metadata(): + """Upload json metadata into db from zenodo""" + dialect = get_dialect("oep-v1.4")() + + for path in importlib_resources.files(__name__).glob("*.json"): + split = path.name.split(".") + if len(split) != 3: + continue + schema = split[0] + table = split[1] + + with open(path, "r") as infile: + obj = dialect.parse(infile.read()) + + metadata = f"'{dialect.compile_and_render(obj)}'" + db.submit_comment(metadata, schema, table) + logger.info(f"Metadata comment for {schema}.{table} stored.") + + +class Json_Metadata(Dataset): + def __init__(self, dependencies): + super().__init__( + name="JsonMetadata", + version="0.0.0", + dependencies=dependencies, + tasks={upload_json_metadata}, + ) diff --git a/src/egon/data/metadata/boundaries.egon_map_zensus_climate_zones.json b/src/egon/data/metadata/boundaries.egon_map_zensus_climate_zones.json new file mode 100644 index 000000000..f3f5a56e6 --- /dev/null +++ b/src/egon/data/metadata/boundaries.egon_map_zensus_climate_zones.json @@ -0,0 +1,151 @@ +{ + "name": "boundaries.egon_map_zensus_climate_zones", + "title": "eGon map census cells to climate zones", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": "spatial join of census cells and climate zones", + "language": [ + "en-EN", + "de-DE" + ], + "keywords": [], + "publicationDate": "2022-09-14", + "context": { + "homepage": "https://ego-n.org/", + "documentation": "https://egon-data.readthedocs.io/en/latest/", + "sourceCode": "https://github.com/openego/eGon-data", + "contact": "https://ego-n.org/partners/", + "grantNo": "03EI1002", + "fundingAgency": "Bundesministerium f\u00fcr Wirtschaft und Klimaschutz", + "fundingAgencyLogo": "https://www.innovation-beratung-foerderung.de/INNO/Redaktion/DE/Bilder/Titelbilder/titel_foerderlogo_bmwi.jpg?__blob=normal&v=3", + "publisherLogo": "https://ego-n.org/images/eGon_logo_noborder_transbg.svg" + }, + "spatial": { + "location": "", + "extent": "Germany", + "resolution": "100 m x 100 m" + }, + "temporal": { + "referenceDate": "2022-09-14", + "timeseries": {} + }, + "sources": [ + { + "title": "eGon-data", + "description": "Workflow to download, process and generate data sets suitable for the further research conducted in the project eGon (https://ego-n.org/)", + "path": "https://github.com/openego/eGon-data", + "licenses": [ + { + "name": "AGPL-3.0 License", + "title": "GNU Affero General Public License v3.0", + "path": "https://www.gnu.org/licenses/agpl-3.0.de.html", + "instruction": "Permissions of this strongest copyleft license are conditioned on making available complete source code of licensed works and modifications, which include larger works using a licensed work, under the same license. Copyright and license notices must be preserved. Contributors provide an express grant of patent rights. When a modified version is used to provide a service over a network, the complete source code of the modified version must be made available.", + "attribution": "\u00a9 eGon-data developers" + } + ] + }, + { + "title": "Data bundle for egon-data: A transparent and reproducible data processing pipeline for energy system modeling", + "description": "egon-data provides a transparent and reproducible open data based data processing pipeline for generating data models suitable for energy system modeling. The data is customized for the requirements of the research project eGon. The research project aims to develop tools for an open and cross-sectoral planning of transmission and distribution grids. For further information please visit the eGon project website or its Github repository. egon-data retrieves and processes data from several different external input sources. As not all data dependencies can be downloaded automatically from external sources we provide a data bundle to be downloaded by egon-data.", + "path": "https://doi.org/10.5281/zenodo.6630616", + "licenses": [ + { + "name": "CC-BY-4.0", + "title": "Creative Commons Attribution 4.0 International", + "path": "https://creativecommons.org/licenses/by/4.0/legalcode", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute.", + "attribution": "\u00a9 eGon-data developers, \u00a9 Leibniz Institute for Applied Geophysics, \u00a9 Fraunhofer IEE, \u00a9 BGR Hannover, \u00a9 Danielle Schmidt, \u00a9 \u00dcbertragungsnetzbetreiber, \u00a9 DIW Berlin, \u00a9 Reiner Lemoine Institut, \u00a9 Statistisches Bundesamt, Wiesbaden 2008, \u00a9 Statistische \u00c4mter des Bundes und der L\u00e4nder 2021" + } + ] + }, + { + "title": "Statistisches Bundesamt (Destatis) - Ergebnisse des Zensus 2011 zum Download", + "description": "Als Download bieten wir Ihnen auf dieser Seite zus\u00e4tzlich zur Zensusdatenbank CSV- und eilweise Excel-Tabellen mit umfassenden Personen-, Haushalts- und Familien- sowie Geb\u00e4ude- und Wohnungsmerkmalen. Die Ergebnisse liegen auf Bundes-, L\u00e4nder-, Kreis- und Gemeindeebene vor. Au\u00dferdem sind einzelne Ergebnisse f\u00fcr Gitterzellen verf\u00fcgbar.", + "path": "https://www.zensus2011.de/DE/Home/Aktuelles/DemografischeGrunddaten.html", + "licenses": [ + { + "name": "dl-by-de/2.0", + "title": "Datenlizenz Deutschland \u2013 Namensnennung \u2013 Version 2.0", + "path": "www.govdata.de/dl-de/by-2-0", + "instruction": "Jede Nutzung ist unter den Bedingungen dieser \u201eDatenlizenz Deutschland - Namensnennung - Version 2.0 zul\u00e4ssig. Die bereitgestellten Daten und Metadaten d\u00fcrfen f\u00fcr die kommerzielle und nicht kommerzielle Nutzung insbesondere: (1) vervielf\u00e4ltigt, ausgedruckt, pr\u00e4sentiert, ver\u00e4ndert, bearbeitet sowie an Dritte \u00fcbermittelt werden; (2) mit eigenen Daten und Daten Anderer zusammengef\u00fchrt und zu selbst\u00e4ndigen neuen Datens\u00e4tzen verbunden werden; (3) in interne und externe Gesch\u00e4ftsprozesse, Produkte und Anwendungen in \u00f6ffentlichen und nicht \u00f6ffentlichen elektronischen Netzwerken eingebunden werden. Bei der Nutzung ist sicherzustellen, dass folgende Angaben als Quellenvermerk enthalten sind: (1) Bezeichnung des Bereitstellers nach dessen Ma\u00dfgabe, der Vermerk Datenlizenz Deutschland \u2013 Namensnennung \u2013 Version 2.0 oder dl-de/by-2-0 mit Verweis auf den Lizenztext unter www.govdata.de/dl-de/by-2-0 sowie (3) einen Verweis auf den Datensatz (URI). Dies gilt nur soweit die datenhaltende Stelle die Angaben (1) bis (3) zum Quellenvermerk bereitstellt. Ver\u00e4nderungen, Bearbeitungen, neue Gestaltungen oder sonstige Abwandlungen sind im Quellenvermerk mit dem Hinweis zu versehen, dass die Daten ge\u00e4ndert wurden.", + "attribution": "\u00a9 Statistische \u00c4mter des Bundes und der L\u00e4nder 2014" + } + ] + } + ], + "licenses": [ + { + "name": "CC-BY-4.0", + "title": "Creative Commons Attribution 4.0 International", + "path": "https://creativecommons.org/licenses/by/4.0/legalcode", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute.", + "attribution": "\u00a9 eGon-data developers" + } + ], + "contributors": [ + { + "title": "ClaraBuettner", + "email": "https://github.com/ClaraBuettner", + "date": "2022-09-12", + "object": "metadata", + "comment": "Create metadata" + }, + { + "title": "nesnoj", + "email": "https://github.com/nesnoj", + "date": "2022-09-14", + "object": "metadata", + "comment": "Update metadata" + } + ], + "resources": [ + { + "profile": "tabular-data-resource", + "name": "boundaries.egon_map_zensus_climate_zones", + "path": "", + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": [ + { + "name": "climate_zone", + "description": "Name of the corresponding climate zone", + "type": "str", + "unit": "none" + }, + { + "name": "zensus_population_id", + "description": "Index of corresponding census cell (reference to: society.destatis_zensus_population_per_ha.id)", + "type": "integer", + "unit": "none" + } + ], + "primaryKey": "zensus_population_id" + }, + "dialect": { + "delimiter": "", + "decimalSeparator": "" + } + } + ], + "review": { + "path": "", + "badge": "" + }, + "metaMetadata": { + "metadataVersion": "OEP-1.4.1", + "metadataLicense": { + "name": "CC0-1.0", + "title": "Creative Commons Zero v1.0 Universal", + "path": "https://creativecommons.org/publicdomain/zero/1.0/" + } + }, + "_comment": { + "metadata": "Metadata documentation and explanation (https://github.com/OpenEnergyPlatform/oemetadata/blob/master/metadata/v141/metadata_key_description.md)", + "dates": "Dates and time must follow the ISO8601 including time zone (YYYY-MM-DD or YYYY-MM-DDThh:mm:ss\u00b1hh)", + "units": "Use a space between numbers and units (100 m)", + "languages": "Languages must follow the IETF (BCP47) format (en-GB, en-US, de-DE)", + "licenses": "License name must follow the SPDX License List (https://spdx.org/licenses/)", + "review": "Following the OEP Data Review (https://github.com/OpenEnergyPlatform/data-preprocessing/wiki)", + "none": "If not applicable use (none)" + } +} diff --git a/src/egon/data/metadata/boundaries.egon_map_zensus_mvgd_buildings.json b/src/egon/data/metadata/boundaries.egon_map_zensus_mvgd_buildings.json new file mode 100644 index 000000000..e09cf47aa --- /dev/null +++ b/src/egon/data/metadata/boundaries.egon_map_zensus_mvgd_buildings.json @@ -0,0 +1,171 @@ +{ + "name": "boundaries.egon_map_zensus_mvgd_buildings", + "title": "eGon buildings mapping table", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": "Mapping table for all used buildings via project internal building_id. Additional info about if it is a synthetic or a openstreetmap building, where it is located (census cell_id, mvgd_id), in which sector (residential, cts) the building is used, if there are heat or electricity profiles.", + "language": "undefined", + "keywords": [], + "publicationDate": "2023-03-13", + "context": { + "homepage": "https://ego-n.org/", + "documentation": "https://egon-data.readthedocs.io/en/latest/", + "sourceCode": "https://github.com/openego/eGon-data", + "contact": "https://ego-n.org/partners/", + "grantNo": "03EI1002", + "fundingAgency": "Bundesministerium f\u00fcr Wirtschaft und Klimaschutz", + "fundingAgencyLogo": "https://commons.wikimedia.org/wiki/File:BMWi_Logo_2021.svg#/media/File:BMWi_Logo_2021.svg", + "publisherLogo": "https://ego-n.org/images/eGon_logo_noborder_transbg.svg" + }, + "spatial": { + "location": "", + "extent": "Germany", + "resolution": "Buildings" + }, + "temporal": { + "referenceDate": "2023-03-13", + "timeseries": {} + }, + "sources": [ + { + "title": "Data bundle for egon-data: A transparent and reproducible data processing pipeline for energy system modeling", + "description": "egon-data provides a transparent and reproducible open data based data processing pipeline for generating data models suitable for energy system modeling. The data is customized for the requirements of the research project eGon. The research project aims to develop tools for an open and cross-sectoral planning of transmission and distribution grids. For further information please visit the eGon project website or its Github repository. egon-data retrieves and processes data from several different external input sources. As not all data dependencies can be downloaded automatically from external sources we provide a data bundle to be downloaded by egon-data.", + "path": "https://doi.org/10.5281/zenodo.6630616", + "licenses": [ + { + "name": "CC-BY-4.0", + "title": "Creative Commons Attribution 4.0 International", + "path": "https://creativecommons.org/licenses/by/4.0/legalcode", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute.", + "attribution": "\u00a9 eGon-data developers, \u00a9 Leibniz Institute for Applied Geophysics, \u00a9 Fraunhofer IEE, \u00a9 BGR Hannover, \u00a9 Danielle Schmidt, \u00a9 \u00dcbertragungsnetzbetreiber, \u00a9 DIW Berlin, \u00a9 Reiner Lemoine Institut, \u00a9 Statistisches Bundesamt, Wiesbaden 2008, \u00a9 Statistische \u00c4mter des Bundes und der L\u00e4nder 2021" + } + ] + }, + { + "title": "eGon-data", + "description": "Workflow to download, process and generate data sets suitable for the further research conducted in the project eGon (https://ego-n.org/)", + "path": "https://github.com/openego/eGon-data", + "licenses": [ + { + "name": "AGPL-3.0 License", + "title": "GNU Affero General Public License v3.0", + "path": "https://www.gnu.org/licenses/agpl-3.0.de.html", + "instruction": "Permissions of this strongest copyleft license are conditioned on making available complete source code of licensed works and modifications, which include larger works using a licensed work, under the same license. Copyright and license notices must be preserved. Contributors provide an express grant of patent rights. When a modified version is used to provide a service over a network, the complete source code of the modified version must be made available.", + "attribution": "\u00a9 eGon-data developers" + } + ] + }, + { + "title": "OpenStreetMap Data Extracts (Geofabrik)", + "description": "Full data extract of OpenStreetMap data for defined spatial extent at 2022-01-01", + "path": "https://download.geofabrik.de/europe/germany-210101.osm.pbf", + "licenses": [ + { + "name": "ODbL-1.0", + "title": "Open Data Commons Open Database License 1.0", + "path": "https://opendatacommons.org/licenses/odbl/1.0/", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute, Share-Alike, Keep open!", + "attribution": "\u00a9 OpenStreetMap contributors" + } + ] + } + ], + "licenses": [ + { + "name": "ODbL-1.0", + "title": "Open Data Commons Open Database License 1.0", + "path": "https://opendatacommons.org/licenses/odbl/1.0/", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute, Share-Alike, Keep open!", + "attribution": "\u00a9 Reiner Lemoine Institut" + } + ], + "contributors": [ + { + "title": "nailend", + "email": "https://github.com/nailend", + "date": "2023-03-13", + "object": "metadata", + "comment": "create metadata" + } + ], + "resources": [ + { + "profile": "tabular-data-resource", + "name": "boundaries.egon_map_zensus_mvgd_buildings", + "path": "", + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": [ + { + "name": "building_id", + "description": "Building id (reference to: openstreetmap.osm_buildings_filtered.id, openstreetmap.osm_buildings_synthetic.id)", + "type": "integer", + "unit": "none" + }, + { + "name": "sector", + "description": "Sector (cts, residential)", + "type": "characters varying", + "unit": "none" + }, + { + "name": "zensus_population_id", + "description": "Zensus population id (reference to: society.destatis_zensus_population_per_ha.id)", + "type": "integer", + "unit": "none" + }, + { + "name": "bus_id", + "description": "MVGD bus id (reference to: grid.egon_hvmv_substation.bus_id, grid.egon_mv_grid_district.bus_id)", + "type": "integer", + "unit": "none" + }, + { + "name": "electricity", + "description": "True if respectives electricity demand profile exists", + "type": "bool", + "unit": "None" + }, + { + "name": "heat", + "description": "True if respectives heat demand profile exists", + "type": "bool", + "unit": "None" + }, + { + "name": "osm", + "description": "is OSM building if true else syntheticaly created", + "type": "bool", + "unit": "None" + } + ], + "primaryKey": "building_id, sector" + }, + "dialect": { + "delimiter": "", + "decimalSeparator": "." + } + } + ], + "review": { + "path": "", + "badge": "" + }, + "metaMetadata": { + "metadataVersion": "OEP-1.4.1", + "metadataLicense": { + "name": "CC0-1.0", + "title": "Creative Commons Zero v1.0 Universal", + "path": "https://creativecommons.org/publicdomain/zero/1.0/" + } + }, + "_comment": { + "metadata": "Metadata documentation and explanation (https://github.com/OpenEnergyPlatform/oemetadata/blob/master/metadata/v141/metadata_key_description.md)", + "dates": "Dates and time must follow the ISO8601 including time zone (YYYY-MM-DD or YYYY-MM-DDThh:mm:ss\u00b1hh)", + "units": "Use a space between numbers and units (100 m)", + "languages": "Languages must follow the IETF (BCP47) format (en-GB, en-US, de-DE)", + "licenses": "License name must follow the SPDX License List (https://spdx.org/licenses/)", + "review": "Following the OEP Data Review (https://github.com/OpenEnergyPlatform/data-preprocessing/wiki)", + "none": "If not applicable use (none)" + } +} diff --git a/src/egon/data/metadata/boundaries.egon_map_zensus_weather_cell.json b/src/egon/data/metadata/boundaries.egon_map_zensus_weather_cell.json new file mode 100644 index 000000000..3026115aa --- /dev/null +++ b/src/egon/data/metadata/boundaries.egon_map_zensus_weather_cell.json @@ -0,0 +1,140 @@ +{ + "name": "boundaries.egon_map_zensus_climate_zones", + "title": "Map census cells to era5 climate zones", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": "Mapping table of census cells to era5 climate zones", + "language": "en-US", + "keywords": [ + "" + ], + "publicationDate": "2022-08-06", + "context": { + "homepage": "https://ego-n.org/", + "documentation": "https://egon-data.readthedocs.io/en/latest/", + "sourceCode": "https://github.com/openego/eGon-data", + "contact": "https://ego-n.org/partners/", + "grantNo": "03EI1002", + "fundingAgency": "Bundesministerium f\\u00fcr Wirtschaft und Energie", + "fundingAgencyLogo": "https://www.innovation-beratung-foerderung.de/INNO/Redaktion/DE/Bilder/Titelbilder/titel_foerderlogo_bmwi.jpg?__blob=normal&v=3", + "publisherLogo": "https://ego-n.org/images/eGon_logo_noborder_transbg.svg" + }, + "spatial": { + "location": "", + "extent": "Germany", + "resolution": "1 ha" + }, + "temporal": {}, + "sources": [ + { + "title": "Statistisches Bundesamt (Destatis) - Ergebnisse des Zensus 2011 zum Download", + "description": "Als Download bieten wir Ihnen auf dieser Seite zus\\u00e4tzlich zur Zensusdatenbank CSV- und teilweise Excel-Tabellen mit umfassenden Personen-, Haushalts- und Familien- sowie Geb\\u00e4ude- und Wohnungsmerkmalen. Die Ergebnisse liegen auf Bundes-, L\\u00e4nder-, Kreis- und Gemeindeebene vor. Au\\u00dferdem sind einzelne Ergebnisse f\\u00fcr Gitterzellen verf\\u00fcgbar.", + "path": "https://www.zensus2011.de/DE/Home/Aktuelles/DemografischeGrunddaten.html", + "licenses": [ + { + "name": "dl-by-de/2.0", + "title": "Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0", + "path": "www.govdata.de/dl-de/by-2-0", + "instruction": "Jede Nutzung ist unter den Bedingungen dieser \\u201eDatenlizenz Deutschland - Namensnennung - Version 2.0 zul\\u00e4ssig.\\nDie bereitgestellten Daten und Metadaten d\\u00fcrfen f\\u00fcr die kommerzielle und nicht kommerzielle Nutzung insbesondere:(1) vervielf\\u00e4ltigt, ausgedruckt, pr\\u00e4sentiert, ver\\u00e4ndert, bearbeitet sowie an Dritte \\u00fcbermittelt werden;\\n (2) mit eigenen Daten und Daten Anderer zusammengef\\u00fchrt und zu selbst\\u00e4ndigen neuen Datens\\u00e4tzen verbunden werden;\\n (3) in interne und externe Gesch\\u00e4ftsprozesse, Produkte und Anwendungen in \\u00f6ffentlichen und nicht \\u00f6ffentlichen elektronischen Netzwerken eingebunden werden.\\nBei der Nutzung ist sicherzustellen, dass folgende Angaben als Quellenvermerk enthalten sind:\\n(1) Bezeichnung des Bereitstellers nach dessen Ma\\u00dfgabe,\\n(2) der Vermerk Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0 oder dl-de/by-2-0 mit Verweis auf den Lizenztext unter www.govdata.de/dl-de/by-2-0 sowie\\n(3) einen Verweis auf den Datensatz (URI).Dies gilt nur soweit die datenhaltende Stelle die Angaben(1) bis (3) zum Quellenvermerk bereitstellt.\\nVer\\u00e4nderungen, Bearbeitungen, neue Gestaltungen oder sonstige Abwandlungen sind im Quellenvermerk mit dem Hinweis zu versehen, dass die Daten ge\\u00e4ndert wurden.", + "attribution": "\\u00a9 Statistische \\u00c4mter des Bundes und der L\\u00e4nder 2014" + } + ] + }, + { + "title": "Dokumentation - Zensus 2011 - Methoden und Verfahren", + "description": "Diese Publikation beschreibt ausf\\u00fchrlich die Methoden und Verfahren des registergest\\u00fctzten Zensus 2011; von der Datengewinnung und -aufbereitung bis hin zur Ergebniserstellung und Geheimhaltung. Der vorliegende Band wurde von den Statistischen \\u00c4mtern des Bundes und der L\\u00e4nder im Juni 2015 ver\\u00f6ffentlicht.", + "path": "https://www.destatis.de/DE/Publikationen/Thematisch/Bevoelkerung/Zensus/ZensusBuLaMethodenVerfahren5121105119004.pdf?__blob=publicationFile", + "licenses": [ + { + "name": "dl-by-de/2.0", + "title": "Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0", + "path": "www.govdata.de/dl-de/by-2-0", + "instruction": "Jede Nutzung ist unter den Bedingungen dieser \\u201eDatenlizenz Deutschland - Namensnennung - Version 2.0 zul\\u00e4ssig.\\nDie bereitgestellten Daten und Metadaten d\\u00fcrfen f\\u00fcr die kommerzielle und nicht kommerzielle Nutzung insbesondere:(1) vervielf\\u00e4ltigt, ausgedruckt, pr\\u00e4sentiert, ver\\u00e4ndert, bearbeitet sowie an Dritte \\u00fcbermittelt werden;\\n (2) mit eigenen Daten und Daten Anderer zusammengef\\u00fchrt und zu selbst\\u00e4ndigen neuen Datens\\u00e4tzen verbunden werden;\\n (3) in interne und externe Gesch\\u00e4ftsprozesse, Produkte und Anwendungen in \\u00f6ffentlichen und nicht \\u00f6ffentlichen elektronischen Netzwerken eingebunden werden.\\nBei der Nutzung ist sicherzustellen, dass folgende Angaben als Quellenvermerk enthalten sind:\\n(1) Bezeichnung des Bereitstellers nach dessen Ma\\u00dfgabe,\\n(2) der Vermerk Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0 oder dl-de/by-2-0 mit Verweis auf den Lizenztext unter www.govdata.de/dl-de/by-2-0 sowie\\n(3) einen Verweis auf den Datensatz (URI).Dies gilt nur soweit die datenhaltende Stelle die Angaben(1) bis (3) zum Quellenvermerk bereitstellt.\\nVer\\u00e4nderungen, Bearbeitungen, neue Gestaltungen oder sonstige Abwandlungen sind im Quellenvermerk mit dem Hinweis zu versehen, dass die Daten ge\\u00e4ndert wurden.", + "attribution": "\\u00a9 Statistisches Bundesamt, Wiesbaden 2015 (im Auftrag der Herausgebergemeinschaft)" + } + ] + }, + { + "title": "ERA5 global reanalysis", + "description": "ERA5 is the fifth generation ECMWF reanalysis for the global climate and weather for the past 4 to 7 decades. Currently data is available from 1950, split into Climate Data Store entries for 1950-1978 (preliminary back extension) and from 1979 onwards (final release plus timely updates, this page). ERA5 replaces the ERA-Interim reanalysis. See the online ERA5 documentation (https://confluence.ecmwf.int/display/CKB/ERA5%3A+data+documentation#ERA5:datadocumentation-Dataupdatefrequency) for more information.", + "path": "https://confluence.ecmwf.int/display/CKB/ERA5%3A+data+documentation#ERA5:datadocumentation-Dataupdatefrequency", + "licenses": [ + { + "name": "Licence to use Copernicus Products", + "title": "Licence to use Copernicus Products", + "path": "https://cds.climate.copernicus.eu/api/v2/terms/static/licence-to-use-copernicus-products.pdf", + "instruction": "This Licence is free of charge, worldwide, non-exclusive, royalty free and perpetual. Access to Copernicus Products is given for any purpose in so far as it is lawful, whereas use may include, but is not limited to: reproduction; distribution; communication to the public; adaptation, modification and combination with other data and information; or any combination of the foregoing", + "attribution": "\u00a9 Copernicus Climate Change Service (C3S) Climate Data Store" + } + ] + } + ], + "licenses": [ + { + "name": "Creative Commons Attribution 4.0 International", + "title": "CC BY 4.0", + "path": "https://creativecommons.org/licenses/by/4.0/", + "instruction": "You are free: To Share, To Adapt; As long as you: Attribute!", + "attribution": "\\u00a9 eGon development team" + } + ], + "contributors": [ + { + "title": "nailend", + "email": "https://github.com/nailend", + "date": "2023-03-13", + "object": "metadata", + "comment": "Create metadata" + } + ], + "resources": [ + { + "profile": "tabular-data-resource", + "name": "boundaries.egon_map_zensus_weather_cell", + "path": "", + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": [ + { + "name": "zensus_population_id", + "description": "Zensus population id (reference to: society.destatis_zensus_population_per_ha.id)", + "type": "integer", + "unit": "none" + }, + { + "name": "w_id", + "description": "Weather cell id (reference to: supply.egon_era5_weather_cells)", + "type": "Integer", + "unit": "none" + } + ], + "primaryKey": "zensus_population_id" + }, + "dialect": { + "delimiter": "", + "decimalSeparator": "." + } + } + ], + "review": { + "path": "", + "badge": "" + }, + "metaMetadata": { + "metadataVersion": "OEP-1.4.1", + "metadataLicense": { + "name": "CC0-1.0", + "title": "Creative Commons Zero v1.0 Universal", + "path": "https://creativecommons.org/publicdomain/zero/1.0/" + } + }, + "_comment": { + "metadata": "Metadata documentation and explanation (https://github.com/OpenEnergyPlatform/oemetadata/blob/master/metadata/v141/metadata_key_description.md)", + "dates": "Dates and time must follow the ISO8601 including time zone (YYYY-MM-DD or YYYY-MM-DDThh:mm:ss\u00b1hh)", + "units": "Use a space between numbers and units (100 m)", + "languages": "Languages must follow the IETF (BCP47) format (en-GB, en-US, de-DE)", + "licenses": "License name must follow the SPDX License List (https://spdx.org/licenses/)", + "review": "Following the OEP Data Review (https://github.com/OpenEnergyPlatform/data-preprocessing/wiki)", + "none": "If not applicable use (none)" + } +} diff --git a/src/egon/data/metadata/demand.egon_building_electricity_peak_loads.json b/src/egon/data/metadata/demand.egon_building_electricity_peak_loads.json new file mode 100644 index 000000000..19d7b55cc --- /dev/null +++ b/src/egon/data/metadata/demand.egon_building_electricity_peak_loads.json @@ -0,0 +1,201 @@ +{ + "name": "demand.egon_building_electricity_peak_loads", + "title": "Electricity peak loads for egon buildings", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": "Electricity peak load table of residential and cts buildings with voltage level for two scenarios ", + "language": "undefined", + "keywords": [], + "publicationDate": "2023-03-13", + "context": { + "homepage": "https://ego-n.org/", + "documentation": "https://egon-data.readthedocs.io/en/latest/", + "sourceCode": "https://github.com/openego/eGon-data", + "contact": "https://ego-n.org/partners/", + "grantNo": "03EI1002", + "fundingAgency": "Bundesministerium f\u00fcr Wirtschaft und Klimaschutz", + "fundingAgencyLogo": "https://commons.wikimedia.org/wiki/File:BMWi_Logo_2021.svg#/media/File:BMWi_Logo_2021.svg", + "publisherLogo": "https://ego-n.org/images/eGon_logo_noborder_transbg.svg" + }, + "spatial": { + "location": "", + "extent": "Germany", + "resolution": "Buildings" + }, + "temporal": { + "referenceDate": "2023-03-13", + "timeseries": {} + }, + "sources": [ + { + "title": "Data bundle for egon-data: A transparent and reproducible data processing pipeline for energy system modeling", + "description": "egon-data provides a transparent and reproducible open data based data processing pipeline for generating data models suitable for energy system modeling. The data is customized for the requirements of the research project eGon. The research project aims to develop tools for an open and cross-sectoral planning of transmission and distribution grids. For further information please visit the eGon project website or its Github repository. egon-data retrieves and processes data from several different external input sources. As not all data dependencies can be downloaded automatically from external sources we provide a data bundle to be downloaded by egon-data.", + "path": "https://doi.org/10.5281/zenodo.6630616", + "licenses": [ + { + "name": "CC-BY-4.0", + "title": "Creative Commons Attribution 4.0 International", + "path": "https://creativecommons.org/licenses/by/4.0/legalcode", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute.", + "attribution": "\u00a9 eGon-data developers, \u00a9 Leibniz Institute for Applied Geophysics, \u00a9 Fraunhofer IEE, \u00a9 BGR Hannover, \u00a9 Danielle Schmidt, \u00a9 \u00dcbertragungsnetzbetreiber, \u00a9 DIW Berlin, \u00a9 Reiner Lemoine Institut, \u00a9 Statistisches Bundesamt, Wiesbaden 2008, \u00a9 Statistische \u00c4mter des Bundes und der L\u00e4nder 2021" + } + ] + }, + { + "title": "eGon-data", + "description": "Workflow to download, process and generate data sets suitable for the further research conducted in the project eGon (https://ego-n.org/)", + "path": "https://github.com/openego/eGon-data", + "licenses": [ + { + "name": "AGPL-3.0 License", + "title": "GNU Affero General Public License v3.0", + "path": "https://www.gnu.org/licenses/agpl-3.0.de.html", + "instruction": "Permissions of this strongest copyleft license are conditioned on making available complete source code of licensed works and modifications, which include larger works using a licensed work, under the same license. Copyright and license notices must be preserved. Contributors provide an express grant of patent rights. When a modified version is used to provide a service over a network, the complete source code of the modified version must be made available.", + "attribution": "\u00a9 eGon-data developers" + } + ] + }, + { + "title": "Statistisches Bundesamt (Destatis) - Ergebnisse des Zensus 2011 zum Download", + "description": "Als Download bieten wir Ihnen auf dieser Seite zus\\u00e4tzlich zur Zensusdatenbank CSV- und teilweise Excel-Tabellen mit umfassenden Personen-, Haushalts- und Familien- sowie Geb\\u00e4ude- und Wohnungsmerkmalen. Die Ergebnisse liegen auf Bundes-, L\\u00e4nder-, Kreis- und Gemeindeebene vor. Au\\u00dferdem sind einzelne Ergebnisse f\\u00fcr Gitterzellen verf\\u00fcgbar.", + "path": "https://www.zensus2011.de/DE/Home/Aktuelles/DemografischeGrunddaten.html", + "licenses": [ + { + "name": "dl-by-de/2.0", + "title": "Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0", + "path": "www.govdata.de/dl-de/by-2-0", + "instruction": "Jede Nutzung ist unter den Bedingungen dieser \\u201eDatenlizenz Deutschland - Namensnennung - Version 2.0 zul\\u00e4ssig.\\nDie bereitgestellten Daten und Metadaten d\\u00fcrfen f\\u00fcr die kommerzielle und nicht kommerzielle Nutzung insbesondere:(1) vervielf\\u00e4ltigt, ausgedruckt, pr\\u00e4sentiert, ver\\u00e4ndert, bearbeitet sowie an Dritte \\u00fcbermittelt werden;\\n (2) mit eigenen Daten und Daten Anderer zusammengef\\u00fchrt und zu selbst\\u00e4ndigen neuen Datens\\u00e4tzen verbunden werden;\\n (3) in interne und externe Gesch\\u00e4ftsprozesse, Produkte und Anwendungen in \\u00f6ffentlichen und nicht \\u00f6ffentlichen elektronischen Netzwerken eingebunden werden.\\nBei der Nutzung ist sicherzustellen, dass folgende Angaben als Quellenvermerk enthalten sind:\\n(1) Bezeichnung des Bereitstellers nach dessen Ma\\u00dfgabe,\\n(2) der Vermerk Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0 oder dl-de/by-2-0 mit Verweis auf den Lizenztext unter www.govdata.de/dl-de/by-2-0 sowie\\n(3) einen Verweis auf den Datensatz (URI).Dies gilt nur soweit die datenhaltende Stelle die Angaben(1) bis (3) zum Quellenvermerk bereitstellt.\\nVer\\u00e4nderungen, Bearbeitungen, neue Gestaltungen oder sonstige Abwandlungen sind im Quellenvermerk mit dem Hinweis zu versehen, dass die Daten ge\\u00e4ndert wurden.", + "attribution": "\\u00a9 Statistische \\u00c4mter des Bundes und der L\\u00e4nder 2014" + } + ] + }, + { + "title": "Dokumentation - Zensus 2011 - Methoden und Verfahren", + "description": "Diese Publikation beschreibt ausf\\u00fchrlich die Methoden und Verfahren des registergest\\u00fctzten Zensus 2011; von der Datengewinnung und -aufbereitung bis hin zur Ergebniserstellung und Geheimhaltung. Der vorliegende Band wurde von den Statistischen \\u00c4mtern des Bundes und der L\\u00e4nder im Juni 2015 ver\\u00f6ffentlicht.", + "path": "https://www.destatis.de/DE/Publikationen/Thematisch/Bevoelkerung/Zensus/ZensusBuLaMethodenVerfahren5121105119004.pdf?__blob=publicationFile", + "licenses": [ + { + "name": "dl-by-de/2.0", + "title": "Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0", + "path": "www.govdata.de/dl-de/by-2-0", + "instruction": "Jede Nutzung ist unter den Bedingungen dieser \\u201eDatenlizenz Deutschland - Namensnennung - Version 2.0 zul\\u00e4ssig.\\nDie bereitgestellten Daten und Metadaten d\\u00fcrfen f\\u00fcr die kommerzielle und nicht kommerzielle Nutzung insbesondere:(1) vervielf\\u00e4ltigt, ausgedruckt, pr\\u00e4sentiert, ver\\u00e4ndert, bearbeitet sowie an Dritte \\u00fcbermittelt werden;\\n (2) mit eigenen Daten und Daten Anderer zusammengef\\u00fchrt und zu selbst\\u00e4ndigen neuen Datens\\u00e4tzen verbunden werden;\\n (3) in interne und externe Gesch\\u00e4ftsprozesse, Produkte und Anwendungen in \\u00f6ffentlichen und nicht \\u00f6ffentlichen elektronischen Netzwerken eingebunden werden.\\nBei der Nutzung ist sicherzustellen, dass folgende Angaben als Quellenvermerk enthalten sind:\\n(1) Bezeichnung des Bereitstellers nach dessen Ma\\u00dfgabe,\\n(2) der Vermerk Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0 oder dl-de/by-2-0 mit Verweis auf den Lizenztext unter www.govdata.de/dl-de/by-2-0 sowie\\n(3) einen Verweis auf den Datensatz (URI).Dies gilt nur soweit die datenhaltende Stelle die Angaben(1) bis (3) zum Quellenvermerk bereitstellt.\\nVer\\u00e4nderungen, Bearbeitungen, neue Gestaltungen oder sonstige Abwandlungen sind im Quellenvermerk mit dem Hinweis zu versehen, dass die Daten ge\\u00e4ndert wurden.", + "attribution": "\\u00a9 Statistisches Bundesamt, Wiesbaden 2015 (im Auftrag der Herausgebergemeinschaft)" + } + ] + }, + { + "title": "OpenStreetMap Data Extracts (Geofabrik)", + "description": "Full data extract of OpenStreetMap data for defined spatial extent at 2022-01-01", + "path": "https://download.geofabrik.de/europe/germany-210101.osm.pbf", + "licenses": [ + { + "name": "ODbL-1.0", + "title": "Open Data Commons Open Database License 1.0", + "path": "https://opendatacommons.org/licenses/odbl/1.0/", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute, Share-Alike, Keep open!", + "attribution": "\u00a9 OpenStreetMap contributors" + } + ] + }, + { + "title": "DemandRegio", + "description": "Harmonisierung und Entwicklung von Verfahren zur regionalen und zeitlichen Aufl\u00f6sung von Energienachfragen", + "path": "https://doi.org/10.34805/ffe-119-20", + "license": [ + { + "name": "Creative Commons Attribution 4.0 International", + "title": "CC BY 4.0", + "path": "https://creativecommons.org/licenses/by/4.0/", + "instruction": "You are free: To Share, To Adapt; As long as you: Attribute!", + "attribution": "\u00a9 FZJ, TUB, FfE" + } + ] + } + ], + "licenses": [ + { + "name": "ODbL-1.0", + "title": "Open Data Commons Open Database License 1.0", + "path": "https://opendatacommons.org/licenses/odbl/1.0/", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute, Share-Alike, Keep open!", + "attribution": "\u00a9 Reiner Lemoine Institut" + } + ], + "contributors": [ + { + "title": "nailend", + "email": "https://github.com/nailend", + "date": "2023-03-13", + "object": "metadata", + "comment": "create metadata" + } + ], + "resources": [ + { + "profile": "tabular-data-resource", + "name": "demand.egon_building_electricity_peak_loads", + "path": "", + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": [ + { + "name": "building_id", + "description": "Building id (reference to: openstreetmap.osm_buildings_filtered.id, openstreetmap.osm_buildings_synthetic.id, boundaries.egon_map_zensus_mvgd_building)", + "type": "integer", + "unit": "none" + }, + { + "name": "scenario", + "description": "Scenario (eGon100RE, eGon2035)", + "type": "character varying", + "unit": "none" + }, + { + "name": "sector", + "description": "Sector (cts, residential)", + "type": "characters varying", + "unit": "none" + }, + { + "name": "peak_load_in_w", + "description": "Peak load value", + "type": "real", + "unit": "W" + }, + { + "name": "voltage_level", + "description": "Voltage level of building connection after MitnetzStrom Classification", + "type": "int", + "unit": "None" + } + ], + "primaryKey": "building_id, scenario, sector" + }, + "dialect": { + "delimiter": "", + "decimalSeparator": "." + } + } + ], + "review": { + "path": "", + "badge": "" + }, + "metaMetadata": { + "metadataVersion": "OEP-1.4.1", + "metadataLicense": { + "name": "CC0-1.0", + "title": "Creative Commons Zero v1.0 Universal", + "path": "https://creativecommons.org/publicdomain/zero/1.0/" + } + }, + "_comment": { + "metadata": "Metadata documentation and explanation (https://github.com/OpenEnergyPlatform/oemetadata/blob/master/metadata/v141/metadata_key_description.md)", + "dates": "Dates and time must follow the ISO8601 including time zone (YYYY-MM-DD or YYYY-MM-DDThh:mm:ss\u00b1hh)", + "units": "Use a space between numbers and units (100 m)", + "languages": "Languages must follow the IETF (BCP47) format (en-GB, en-US, de-DE)", + "licenses": "License name must follow the SPDX License List (https://spdx.org/licenses/)", + "review": "Following the OEP Data Review (https://github.com/OpenEnergyPlatform/data-preprocessing/wiki)", + "none": "If not applicable use (none)" + } +} diff --git a/src/egon/data/metadata/demand.egon_building_heat_peak_loads.json b/src/egon/data/metadata/demand.egon_building_heat_peak_loads.json new file mode 100644 index 000000000..17f7dfee0 --- /dev/null +++ b/src/egon/data/metadata/demand.egon_building_heat_peak_loads.json @@ -0,0 +1,209 @@ +{ + "name": "demand.egon_building_heat_peak_loads", + "title": "Heat peak loads for egon buildings", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": "Heat peak load table of residential and cts buildings for two scenarios ", + "language": "undefined", + "keywords": [], + "publicationDate": "2023-03-13", + "context": { + "homepage": "https://ego-n.org/", + "documentation": "https://egon-data.readthedocs.io/en/latest/", + "sourceCode": "https://github.com/openego/eGon-data", + "contact": "https://ego-n.org/partners/", + "grantNo": "03EI1002", + "fundingAgency": "Bundesministerium f\u00fcr Wirtschaft und Klimaschutz", + "fundingAgencyLogo": "https://commons.wikimedia.org/wiki/File:BMWi_Logo_2021.svg#/media/File:BMWi_Logo_2021.svg", + "publisherLogo": "https://ego-n.org/images/eGon_logo_noborder_transbg.svg" + }, + "spatial": { + "location": "", + "extent": "Germany", + "resolution": "Buildings" + }, + "temporal": { + "referenceDate": "2023-03-13", + "timeseries": {} + }, + "sources": [ + { + "title": "Data bundle for egon-data: A transparent and reproducible data processing pipeline for energy system modeling", + "description": "egon-data provides a transparent and reproducible open data based data processing pipeline for generating data models suitable for energy system modeling. The data is customized for the requirements of the research project eGon. The research project aims to develop tools for an open and cross-sectoral planning of transmission and distribution grids. For further information please visit the eGon project website or its Github repository. egon-data retrieves and processes data from several different external input sources. As not all data dependencies can be downloaded automatically from external sources we provide a data bundle to be downloaded by egon-data.", + "path": "https://doi.org/10.5281/zenodo.6630616", + "licenses": [ + { + "name": "CC-BY-4.0", + "title": "Creative Commons Attribution 4.0 International", + "path": "https://creativecommons.org/licenses/by/4.0/legalcode", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute.", + "attribution": "\u00a9 eGon-data developers, \u00a9 Leibniz Institute for Applied Geophysics, \u00a9 Fraunhofer IEE, \u00a9 BGR Hannover, \u00a9 Danielle Schmidt, \u00a9 \u00dcbertragungsnetzbetreiber, \u00a9 DIW Berlin, \u00a9 Reiner Lemoine Institut, \u00a9 Statistisches Bundesamt, Wiesbaden 2008, \u00a9 Statistische \u00c4mter des Bundes und der L\u00e4nder 2021" + } + ] + }, + { + "title": "eGon-data", + "description": "Workflow to download, process and generate data sets suitable for the further research conducted in the project eGon (https://ego-n.org/)", + "path": "https://github.com/openego/eGon-data", + "licenses": [ + { + "name": "AGPL-3.0 License", + "title": "GNU Affero General Public License v3.0", + "path": "https://www.gnu.org/licenses/agpl-3.0.de.html", + "instruction": "Permissions of this strongest copyleft license are conditioned on making available complete source code of licensed works and modifications, which include larger works using a licensed work, under the same license. Copyright and license notices must be preserved. Contributors provide an express grant of patent rights. When a modified version is used to provide a service over a network, the complete source code of the modified version must be made available.", + "attribution": "\u00a9 eGon-data developers" + } + ] + }, + { + "title": "OpenStreetMap Data Extracts (Geofabrik)", + "description": "Full data extract of OpenStreetMap data for defined spatial extent at 2022-01-01", + "path": "https://download.geofabrik.de/europe/germany-210101.osm.pbf", + "licenses": [ + { + "name": "ODbL-1.0", + "title": "Open Data Commons Open Database License 1.0", + "path": "https://opendatacommons.org/licenses/odbl/1.0/", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute, Share-Alike, Keep open!", + "attribution": "\u00a9 OpenStreetMap contributors" + } + ] + }, + { + "title": "Verwaltungsgebiete 1:250 000 (Ebenen)", + "description": "Der Datenbestand umfasst s\u00e4mtliche Verwaltungseinheiten der hierarchischen Verwaltungsebenen vom Staat bis zu den Gemeinden mit ihren Grenzen, statistischen Schl\u00fcsselzahlen, Namen der Verwaltungseinheit sowie die spezifische Bezeichnung der Verwaltungsebene des jeweiligen Landes.", + "path": "https://daten.gdz.bkg.bund.de/produkte/vg/vg250_ebenen_0101/2020/vg250_01-01.geo84.shape.ebenen.zip", + "licenses": [ + { + "name": "dl-by-de/2.0", + "title": "Datenlizenz Deutschland \u2013 Namensnennung \u2013 Version 2.0", + "path": "www.govdata.de/dl-de/by-2-0", + "instruction": "Jede Nutzung ist unter den Bedingungen dieser \u201eDatenlizenz Deutschland - Namensnennung - Version 2.0 zul\u00e4ssig.Die bereitgestellten Daten und Metadaten d\u00fcrfen f\u00fcr die kommerzielle und nicht kommerzielle Nutzung insbesondere:(1) vervielf\u00e4ltigt, ausgedruckt, pr\u00e4sentiert, ver\u00e4ndert, bearbeitet sowie an Dritte \u00fcbermittelt werden; (2) mit eigenen Daten und Daten Anderer zusammengef\u00fchrt und zu selbst\u00e4ndigen neuen Datens\u00e4tzen verbunden werden; (3) in interne und externe Gesch\u00e4ftsprozesse, Produkte und Anwendungen in \u00f6ffentlichen und nicht \u00f6ffentlichen elektronischen Netzwerken eingebunden werden.Bei der Nutzung ist sicherzustellen, dass folgende Angaben als Quellenvermerk enthalten sind:(1) Bezeichnung des Bereitstellers nach dessen Ma\u00dfgabe,(2) der Vermerk Datenlizenz Deutschland \u2013 Namensnennung \u2013 Version 2.0 oder dl-de/by-2-0 mit Verweis auf den Lizenztext unter www.govdata.de/dl-de/by-2-0 sowie(3) einen Verweis auf den Datensatz (URI).Dies gilt nur soweit die datenhaltende Stelle die Angaben(1) bis (3) zum Quellenvermerk bereitstellt.Ver\u00e4nderungen, Bearbeitungen, neue Gestaltungen oder sonstige Abwandlungen sind im Quellenvermerk mit dem Hinweis zu versehen, dass die Daten ge\u00e4ndert wurden.", + "attribution": "\u00a9 Bundesamt f\u00fcr Kartographie und Geod\u00e4sie 2020 (Daten ver\u00e4ndert)" + } + ] + }, + { + "title": "ERA5 global reanalysis", + "description": "ERA5 is the fifth generation ECMWF reanalysis for the global climate and weather for the past 4 to 7 decades. Currently data is available from 1950, split into Climate Data Store entries for 1950-1978 (preliminary back extension) and from 1979 onwards (final release plus timely updates, this page). ERA5 replaces the ERA-Interim reanalysis. See the online ERA5 documentation (https://confluence.ecmwf.int/display/CKB/ERA5%3A+data+documentation#ERA5:datadocumentation-Dataupdatefrequency) for more information.", + "path": "https://confluence.ecmwf.int/display/CKB/ERA5%3A+data+documentation#ERA5:datadocumentation-Dataupdatefrequency", + "licenses": [ + { + "name": "Licence to use Copernicus Products", + "title": "Licence to use Copernicus Products", + "path": "https://cds.climate.copernicus.eu/api/v2/terms/static/licence-to-use-copernicus-products.pdf", + "instruction": "This Licence is free of charge, worldwide, non-exclusive, royalty free and perpetual. Access to Copernicus Products is given for any purpose in so far as it is lawful, whereas use may include, but is not limited to: reproduction; distribution; communication to the public; adaptation, modification and combination with other data and information; or any combination of the foregoing", + "attribution": "\u00a9 Copernicus Climate Change Service (C3S) Climate Data Store" + } + ] + }, + { + "title": "Peta5 0 1 HD res", + "description": "modelled Heat Demand distribution (in GJ per hectare grid cell) for residential heat demands for space heating and hot water for the year 2015 using HRE4 data and the combined top-down bottom-up approach of HRE4", + "path": "https://s-eenergies-open-data-euf.hub.arcgis.com/search", + "licenses": [ + { + "name": "Creative Commons Attribution 4.0 International", + "title": "CC BY 4.0", + "path": "https://creativecommons.org/licenses/by/4.0/", + "instruction": "You are free: To Share, To Adapt; As long as you: Attribute!", + "attribution": "\\u00a9 Flensburg, Halmstad and Aalborg universities" + } + ] + }, + { + "title": "Peta5 0 1 HD ser", + "description": "modelled Heat Demand distribution (in GJ per hectare grid cell) for service-sector heat demands for space heating, hot water and process heat for the year 2015 using HRE4 data and the combined top-down bottom-up approach of HRE4", + "path": "https://s-eenergies-open-data-euf.hub.arcgis.com/search", + "licenses": [ + { + "name": "Creative Commons Attribution 4.0 International", + "title": "CC BY 4.0", + "path": "https://creativecommons.org/licenses/by/4.0/", + "instruction": "You are free: To Share, To Adapt; As long as you: Attribute!", + "attribution": "\\u00a9 Flensburg, Halmstad and Aalborg universities" + } + ] + } + ], + "licenses": [ + { + "name": "ODbL-1.0", + "title": "Open Data Commons Open Database License 1.0", + "path": "https://opendatacommons.org/licenses/odbl/1.0/", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute, Share-Alike, Keep open!", + "attribution": "\u00a9 Reiner Lemoine Institut" + } + ], + "contributors": [ + { + "title": "nailend", + "email": "https://github.com/nailend", + "date": "2023-03-13", + "object": "metadata", + "comment": "create metadata" + } + ], + "resources": [ + { + "profile": "tabular-data-resource", + "name": "demand.egon_building_heat_peak_loads", + "path": "", + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": [ + { + "name": "building_id", + "description": "Building id (reference to: openstreetmap.osm_buildings_filtered.id, openstreetmap.osm_buildings_synthetic.id, boundaries.egon_map_zensus_mvgd_building)", + "type": "integer", + "unit": "none" + }, + { + "name": "scenario", + "description": "Scenario (eGon100RE, eGon2035)", + "type": "character varying", + "unit": "none" + }, + { + "name": "sector", + "description": "Sector (cts, residential)", + "type": "characters varying", + "unit": "none" + }, + { + "name": "peak_load_in_w", + "description": "Peak load value", + "type": "real", + "unit": "W" + } + ], + "primaryKey": "building_id, scenario, sector" + }, + "dialect": { + "delimiter": "", + "decimalSeparator": "." + } + } + ], + "review": { + "path": "", + "badge": "" + }, + "metaMetadata": { + "metadataVersion": "OEP-1.4.1", + "metadataLicense": { + "name": "CC0-1.0", + "title": "Creative Commons Zero v1.0 Universal", + "path": "https://creativecommons.org/publicdomain/zero/1.0/" + } + }, + "_comment": { + "metadata": "Metadata documentation and explanation (https://github.com/OpenEnergyPlatform/oemetadata/blob/master/metadata/v141/metadata_key_description.md)", + "dates": "Dates and time must follow the ISO8601 including time zone (YYYY-MM-DD or YYYY-MM-DDThh:mm:ss\u00b1hh)", + "units": "Use a space between numbers and units (100 m)", + "languages": "Languages must follow the IETF (BCP47) format (en-GB, en-US, de-DE)", + "licenses": "License name must follow the SPDX License List (https://spdx.org/licenses/)", + "review": "Following the OEP Data Review (https://github.com/OpenEnergyPlatform/data-preprocessing/wiki)", + "none": "If not applicable use (none)" + } +} diff --git a/src/egon/data/metadata/demand.egon_cts_electricity_demand_building_share.json b/src/egon/data/metadata/demand.egon_cts_electricity_demand_building_share.json new file mode 100644 index 000000000..c7a60b34c --- /dev/null +++ b/src/egon/data/metadata/demand.egon_cts_electricity_demand_building_share.json @@ -0,0 +1,195 @@ +{ + "name": "demand.egon_cts_electricity_demand_building_share", + "title": "Electricity demand profile share of egon cts buildings", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": "Electricity demand profile share of egon cts buildings for two scenarios with bus_id", + "language": "undefined", + "keywords": [], + "publicationDate": "2023-03-13", + "context": { + "homepage": "https://ego-n.org/", + "documentation": "https://egon-data.readthedocs.io/en/latest/", + "sourceCode": "https://github.com/openego/eGon-data", + "contact": "https://ego-n.org/partners/", + "grantNo": "03EI1002", + "fundingAgency": "Bundesministerium f\u00fcr Wirtschaft und Klimaschutz", + "fundingAgencyLogo": "https://commons.wikimedia.org/wiki/File:BMWi_Logo_2021.svg#/media/File:BMWi_Logo_2021.svg", + "publisherLogo": "https://ego-n.org/images/eGon_logo_noborder_transbg.svg" + }, + "spatial": { + "location": "", + "extent": "Germany", + "resolution": "Buildings" + }, + "temporal": { + "referenceDate": "2023-03-13", + "timeseries": {} + }, + "sources": [ + { + "title": "Data bundle for egon-data: A transparent and reproducible data processing pipeline for energy system modeling", + "description": "egon-data provides a transparent and reproducible open data based data processing pipeline for generating data models suitable for energy system modeling. The data is customized for the requirements of the research project eGon. The research project aims to develop tools for an open and cross-sectoral planning of transmission and distribution grids. For further information please visit the eGon project website or its Github repository. egon-data retrieves and processes data from several different external input sources. As not all data dependencies can be downloaded automatically from external sources we provide a data bundle to be downloaded by egon-data.", + "path": "https://doi.org/10.5281/zenodo.6630616", + "licenses": [ + { + "name": "CC-BY-4.0", + "title": "Creative Commons Attribution 4.0 International", + "path": "https://creativecommons.org/licenses/by/4.0/legalcode", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute.", + "attribution": "\u00a9 eGon-data developers, \u00a9 Leibniz Institute for Applied Geophysics, \u00a9 Fraunhofer IEE, \u00a9 BGR Hannover, \u00a9 Danielle Schmidt, \u00a9 \u00dcbertragungsnetzbetreiber, \u00a9 DIW Berlin, \u00a9 Reiner Lemoine Institut, \u00a9 Statistisches Bundesamt, Wiesbaden 2008, \u00a9 Statistische \u00c4mter des Bundes und der L\u00e4nder 2021" + } + ] + }, + { + "title": "eGon-data", + "description": "Workflow to download, process and generate data sets suitable for the further research conducted in the project eGon (https://ego-n.org/)", + "path": "https://github.com/openego/eGon-data", + "licenses": [ + { + "name": "AGPL-3.0 License", + "title": "GNU Affero General Public License v3.0", + "path": "https://www.gnu.org/licenses/agpl-3.0.de.html", + "instruction": "Permissions of this strongest copyleft license are conditioned on making available complete source code of licensed works and modifications, which include larger works using a licensed work, under the same license. Copyright and license notices must be preserved. Contributors provide an express grant of patent rights. When a modified version is used to provide a service over a network, the complete source code of the modified version must be made available.", + "attribution": "\u00a9 eGon-data developers" + } + ] + }, + { + "title": "Statistisches Bundesamt (Destatis) - Ergebnisse des Zensus 2011 zum Download", + "description": "Als Download bieten wir Ihnen auf dieser Seite zus\\u00e4tzlich zur Zensusdatenbank CSV- und teilweise Excel-Tabellen mit umfassenden Personen-, Haushalts- und Familien- sowie Geb\\u00e4ude- und Wohnungsmerkmalen. Die Ergebnisse liegen auf Bundes-, L\\u00e4nder-, Kreis- und Gemeindeebene vor. Au\\u00dferdem sind einzelne Ergebnisse f\\u00fcr Gitterzellen verf\\u00fcgbar.", + "path": "https://www.zensus2011.de/DE/Home/Aktuelles/DemografischeGrunddaten.html", + "licenses": [ + { + "name": "dl-by-de/2.0", + "title": "Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0", + "path": "www.govdata.de/dl-de/by-2-0", + "instruction": "Jede Nutzung ist unter den Bedingungen dieser \\u201eDatenlizenz Deutschland - Namensnennung - Version 2.0 zul\\u00e4ssig.\\nDie bereitgestellten Daten und Metadaten d\\u00fcrfen f\\u00fcr die kommerzielle und nicht kommerzielle Nutzung insbesondere:(1) vervielf\\u00e4ltigt, ausgedruckt, pr\\u00e4sentiert, ver\\u00e4ndert, bearbeitet sowie an Dritte \\u00fcbermittelt werden;\\n (2) mit eigenen Daten und Daten Anderer zusammengef\\u00fchrt und zu selbst\\u00e4ndigen neuen Datens\\u00e4tzen verbunden werden;\\n (3) in interne und externe Gesch\\u00e4ftsprozesse, Produkte und Anwendungen in \\u00f6ffentlichen und nicht \\u00f6ffentlichen elektronischen Netzwerken eingebunden werden.\\nBei der Nutzung ist sicherzustellen, dass folgende Angaben als Quellenvermerk enthalten sind:\\n(1) Bezeichnung des Bereitstellers nach dessen Ma\\u00dfgabe,\\n(2) der Vermerk Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0 oder dl-de/by-2-0 mit Verweis auf den Lizenztext unter www.govdata.de/dl-de/by-2-0 sowie\\n(3) einen Verweis auf den Datensatz (URI).Dies gilt nur soweit die datenhaltende Stelle die Angaben(1) bis (3) zum Quellenvermerk bereitstellt.\\nVer\\u00e4nderungen, Bearbeitungen, neue Gestaltungen oder sonstige Abwandlungen sind im Quellenvermerk mit dem Hinweis zu versehen, dass die Daten ge\\u00e4ndert wurden.", + "attribution": "\\u00a9 Statistische \\u00c4mter des Bundes und der L\\u00e4nder 2014" + } + ] + }, + { + "title": "Dokumentation - Zensus 2011 - Methoden und Verfahren", + "description": "Diese Publikation beschreibt ausf\\u00fchrlich die Methoden und Verfahren des registergest\\u00fctzten Zensus 2011; von der Datengewinnung und -aufbereitung bis hin zur Ergebniserstellung und Geheimhaltung. Der vorliegende Band wurde von den Statistischen \\u00c4mtern des Bundes und der L\\u00e4nder im Juni 2015 ver\\u00f6ffentlicht.", + "path": "https://www.destatis.de/DE/Publikationen/Thematisch/Bevoelkerung/Zensus/ZensusBuLaMethodenVerfahren5121105119004.pdf?__blob=publicationFile", + "licenses": [ + { + "name": "dl-by-de/2.0", + "title": "Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0", + "path": "www.govdata.de/dl-de/by-2-0", + "instruction": "Jede Nutzung ist unter den Bedingungen dieser \\u201eDatenlizenz Deutschland - Namensnennung - Version 2.0 zul\\u00e4ssig.\\nDie bereitgestellten Daten und Metadaten d\\u00fcrfen f\\u00fcr die kommerzielle und nicht kommerzielle Nutzung insbesondere:(1) vervielf\\u00e4ltigt, ausgedruckt, pr\\u00e4sentiert, ver\\u00e4ndert, bearbeitet sowie an Dritte \\u00fcbermittelt werden;\\n (2) mit eigenen Daten und Daten Anderer zusammengef\\u00fchrt und zu selbst\\u00e4ndigen neuen Datens\\u00e4tzen verbunden werden;\\n (3) in interne und externe Gesch\\u00e4ftsprozesse, Produkte und Anwendungen in \\u00f6ffentlichen und nicht \\u00f6ffentlichen elektronischen Netzwerken eingebunden werden.\\nBei der Nutzung ist sicherzustellen, dass folgende Angaben als Quellenvermerk enthalten sind:\\n(1) Bezeichnung des Bereitstellers nach dessen Ma\\u00dfgabe,\\n(2) der Vermerk Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0 oder dl-de/by-2-0 mit Verweis auf den Lizenztext unter www.govdata.de/dl-de/by-2-0 sowie\\n(3) einen Verweis auf den Datensatz (URI).Dies gilt nur soweit die datenhaltende Stelle die Angaben(1) bis (3) zum Quellenvermerk bereitstellt.\\nVer\\u00e4nderungen, Bearbeitungen, neue Gestaltungen oder sonstige Abwandlungen sind im Quellenvermerk mit dem Hinweis zu versehen, dass die Daten ge\\u00e4ndert wurden.", + "attribution": "\\u00a9 Statistisches Bundesamt, Wiesbaden 2015 (im Auftrag der Herausgebergemeinschaft)" + } + ] + }, + { + "title": "OpenStreetMap Data Extracts (Geofabrik)", + "description": "Full data extract of OpenStreetMap data for defined spatial extent at 2022-01-01", + "path": "https://download.geofabrik.de/europe/germany-210101.osm.pbf", + "licenses": [ + { + "name": "ODbL-1.0", + "title": "Open Data Commons Open Database License 1.0", + "path": "https://opendatacommons.org/licenses/odbl/1.0/", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute, Share-Alike, Keep open!", + "attribution": "\u00a9 OpenStreetMap contributors" + } + ] + }, + { + "title": "DemandRegio", + "description": "Harmonisierung und Entwicklung von Verfahren zur regionalen und zeitlichen Aufl\u00f6sung von Energienachfragen", + "path": "https://doi.org/10.34805/ffe-119-20", + "license": [ + { + "name": "Creative Commons Attribution 4.0 International", + "title": "CC BY 4.0", + "path": "https://creativecommons.org/licenses/by/4.0/", + "instruction": "You are free: To Share, To Adapt; As long as you: Attribute!", + "attribution": "\u00a9 FZJ, TUB, FfE" + } + ] + } + ], + "licenses": [ + { + "name": "ODbL-1.0", + "title": "Open Data Commons Open Database License 1.0", + "path": "https://opendatacommons.org/licenses/odbl/1.0/", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute, Share-Alike, Keep open!", + "attribution": "\u00a9 Reiner Lemoine Institut" + } + ], + "contributors": [ + { + "title": "nailend", + "email": "https://github.com/nailend", + "date": "2023-03-13", + "object": "metadata", + "comment": "create metadata" + } + ], + "resources": [ + { + "profile": "tabular-data-resource", + "name": "demand.egon_cts_electricity_demand_building_share", + "path": "", + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": [ + { + "name": "building_id", + "description": "Building id (reference to: openstreetmap.osm_buildings_filtered.id, openstreetmap.osm_buildings_synthetic.id, boundaries.egon_map_zensus_mvgd_building)", + "type": "integer", + "unit": "none" + }, + { + "name": "scenario", + "description": "Scenario (eGon100RE, eGon2035) (reference: demand.egon_etrago_electricity_cts.scn_name)", + "type": "character varying", + "unit": "none" + }, + { + "name": "bus_id", + "description": "MVGD bus_id (reference to: demand.egon_etrago_electricity_cts.bus_id, grid.egon_hvmv_substation.bus_id, grid.egon_mv_grid_district.bus_id)", + "type": "integer", + "unit": "none" + }, + { + "name": "profile_share", + "description": "Profile share of MVGD elctricity profile", + "type": "real", + "unit": "None" + } + ], + "primaryKey": "building_id, scenario" + }, + "dialect": { + "delimiter": "", + "decimalSeparator": "." + } + } + ], + "review": { + "path": "", + "badge": "" + }, + "metaMetadata": { + "metadataVersion": "OEP-1.4.1", + "metadataLicense": { + "name": "CC0-1.0", + "title": "Creative Commons Zero v1.0 Universal", + "path": "https://creativecommons.org/publicdomain/zero/1.0/" + } + }, + "_comment": { + "metadata": "Metadata documentation and explanation (https://github.com/OpenEnergyPlatform/oemetadata/blob/master/metadata/v141/metadata_key_description.md)", + "dates": "Dates and time must follow the ISO8601 including time zone (YYYY-MM-DD or YYYY-MM-DDThh:mm:ss\u00b1hh)", + "units": "Use a space between numbers and units (100 m)", + "languages": "Languages must follow the IETF (BCP47) format (en-GB, en-US, de-DE)", + "licenses": "License name must follow the SPDX License List (https://spdx.org/licenses/)", + "review": "Following the OEP Data Review (https://github.com/OpenEnergyPlatform/data-preprocessing/wiki)", + "none": "If not applicable use (none)" + } +} diff --git a/src/egon/data/metadata/demand.egon_cts_heat_demand_building_share.json b/src/egon/data/metadata/demand.egon_cts_heat_demand_building_share.json new file mode 100644 index 000000000..0f9228270 --- /dev/null +++ b/src/egon/data/metadata/demand.egon_cts_heat_demand_building_share.json @@ -0,0 +1,181 @@ +{ + "name": "demand.egon_cts_heat_demand_building_share", + "title": "Heat demand profile share of egon cts buildings", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": "Heat demand profile share of egon cts buildings for two scenarios with bus_id", + "language": "undefined", + "keywords": [], + "publicationDate": "2023-03-13", + "context": { + "homepage": "https://ego-n.org/", + "documentation": "https://egon-data.readthedocs.io/en/latest/", + "sourceCode": "https://github.com/openego/eGon-data", + "contact": "https://ego-n.org/partners/", + "grantNo": "03EI1002", + "fundingAgency": "Bundesministerium f\u00fcr Wirtschaft und Klimaschutz", + "fundingAgencyLogo": "https://commons.wikimedia.org/wiki/File:BMWi_Logo_2021.svg#/media/File:BMWi_Logo_2021.svg", + "publisherLogo": "https://ego-n.org/images/eGon_logo_noborder_transbg.svg" + }, + "spatial": { + "location": "", + "extent": "Germany", + "resolution": "Buildings" + }, + "temporal": { + "referenceDate": "2023-03-13", + "timeseries": {} + }, + "sources": [ + { + "title": "Data bundle for egon-data: A transparent and reproducible data processing pipeline for energy system modeling", + "description": "egon-data provides a transparent and reproducible open data based data processing pipeline for generating data models suitable for energy system modeling. The data is customized for the requirements of the research project eGon. The research project aims to develop tools for an open and cross-sectoral planning of transmission and distribution grids. For further information please visit the eGon project website or its Github repository. egon-data retrieves and processes data from several different external input sources. As not all data dependencies can be downloaded automatically from external sources we provide a data bundle to be downloaded by egon-data.", + "path": "https://doi.org/10.5281/zenodo.6630616", + "licenses": [ + { + "name": "CC-BY-4.0", + "title": "Creative Commons Attribution 4.0 International", + "path": "https://creativecommons.org/licenses/by/4.0/legalcode", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute.", + "attribution": "\u00a9 eGon-data developers, \u00a9 Leibniz Institute for Applied Geophysics, \u00a9 Fraunhofer IEE, \u00a9 BGR Hannover, \u00a9 Danielle Schmidt, \u00a9 \u00dcbertragungsnetzbetreiber, \u00a9 DIW Berlin, \u00a9 Reiner Lemoine Institut, \u00a9 Statistisches Bundesamt, Wiesbaden 2008, \u00a9 Statistische \u00c4mter des Bundes und der L\u00e4nder 2021" + } + ] + }, + { + "title": "eGon-data", + "description": "Workflow to download, process and generate data sets suitable for the further research conducted in the project eGon (https://ego-n.org/)", + "path": "https://github.com/openego/eGon-data", + "licenses": [ + { + "name": "AGPL-3.0 License", + "title": "GNU Affero General Public License v3.0", + "path": "https://www.gnu.org/licenses/agpl-3.0.de.html", + "instruction": "Permissions of this strongest copyleft license are conditioned on making available complete source code of licensed works and modifications, which include larger works using a licensed work, under the same license. Copyright and license notices must be preserved. Contributors provide an express grant of patent rights. When a modified version is used to provide a service over a network, the complete source code of the modified version must be made available.", + "attribution": "\u00a9 eGon-data developers" + } + ] + }, + { + "title": "OpenStreetMap Data Extracts (Geofabrik)", + "description": "Full data extract of OpenStreetMap data for defined spatial extent at 2022-01-01", + "path": "https://download.geofabrik.de/europe/germany-210101.osm.pbf", + "licenses": [ + { + "name": "ODbL-1.0", + "title": "Open Data Commons Open Database License 1.0", + "path": "https://opendatacommons.org/licenses/odbl/1.0/", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute, Share-Alike, Keep open!", + "attribution": "\u00a9 OpenStreetMap contributors" + } + ] + }, + { + "title": "Peta5 0 1 HD ser", + "description": "modelled Heat Demand distribution (in GJ per hectare grid cell) for service-sector heat demands for space heating, hot water and process heat for the year 2015 using HRE4 data and the combined top-down bottom-up approach of HRE4", + "path": "https://s-eenergies-open-data-euf.hub.arcgis.com/search", + "licenses": [ + { + "name": "Creative Commons Attribution 4.0 International", + "title": "CC BY 4.0", + "path": "https://creativecommons.org/licenses/by/4.0/", + "instruction": "You are free: To Share, To Adapt; As long as you: Attribute!", + "attribution": "\\u00a9 Flensburg, Halmstad and Aalborg universities" + } + ] + }, + { + "title": "Peta5 0 1 HD res", + "description": "modelled Heat Demand distribution (in GJ per hectare grid cell) for residential heat demands for space heating and hot water for the year 2015 using HRE4 data and the combined top-down bottom-up approach of HRE4", + "path": "https://s-eenergies-open-data-euf.hub.arcgis.com/search", + "licenses": [ + { + "name": "Creative Commons Attribution 4.0 International", + "title": "CC BY 4.0", + "path": "https://creativecommons.org/licenses/by/4.0/", + "instruction": "You are free: To Share, To Adapt; As long as you: Attribute!", + "attribution": "\\u00a9 Flensburg, Halmstad and Aalborg universities" + } + ] + } + ], + "licenses": [ + { + "name": "ODbL-1.0", + "title": "Open Data Commons Open Database License 1.0", + "path": "https://opendatacommons.org/licenses/odbl/1.0/", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute, Share-Alike, Keep open!", + "attribution": "\u00a9 Reiner Lemoine Institut" + } + ], + "contributors": [ + { + "title": "nailend", + "email": "https://github.com/nailend", + "date": "2023-03-13", + "object": "metadata", + "comment": "create metadata" + } + ], + "resources": [ + { + "profile": "tabular-data-resource", + "name": "demand.egon_cts_heat_demand_building_share", + "path": "", + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": [ + { + "name": "building_id", + "description": "Building id (reference to: openstreetmap.osm_buildings_filtered.id, openstreetmap.osm_buildings_synthetic.id, boundaries.egon_map_zensus_mvgd_building)", + "type": "integer", + "unit": "none" + }, + { + "name": "scenario", + "description": "Scenario (eGon100RE, eGon2035)", + "type": "character varying", + "unit": "none" + }, + { + "name": "bus_id", + "description": "MVGD bus_id (reference to: demand.egon_etrago_heat_cts.bus_id, grid.egon_hvmv_substation.bus_id, grid.egon_mv_grid_district.bus_id)", + "type": "integer", + "unit": "none" + }, + { + "name": "profile_share", + "description": "Profile share of MVGD heat profile ", + "type": "real", + "unit": "None" + } + ], + "primaryKey": "building_id, scenario" + }, + "dialect": { + "delimiter": "", + "decimalSeparator": "." + } + } + ], + "review": { + "path": "", + "badge": "" + }, + "metaMetadata": { + "metadataVersion": "OEP-1.4.1", + "metadataLicense": { + "name": "CC0-1.0", + "title": "Creative Commons Zero v1.0 Universal", + "path": "https://creativecommons.org/publicdomain/zero/1.0/" + } + }, + "_comment": { + "metadata": "Metadata documentation and explanation (https://github.com/OpenEnergyPlatform/oemetadata/blob/master/metadata/v141/metadata_key_description.md)", + "dates": "Dates and time must follow the ISO8601 including time zone (YYYY-MM-DD or YYYY-MM-DDThh:mm:ss\u00b1hh)", + "units": "Use a space between numbers and units (100 m)", + "languages": "Languages must follow the IETF (BCP47) format (en-GB, en-US, de-DE)", + "licenses": "License name must follow the SPDX License List (https://spdx.org/licenses/)", + "review": "Following the OEP Data Review (https://github.com/OpenEnergyPlatform/data-preprocessing/wiki)", + "none": "If not applicable use (none)" + } +} diff --git a/src/egon/data/metadata/demand.egon_daily_heat_demand_per_climate_zone.json b/src/egon/data/metadata/demand.egon_daily_heat_demand_per_climate_zone.json new file mode 100644 index 000000000..06d950eda --- /dev/null +++ b/src/egon/data/metadata/demand.egon_daily_heat_demand_per_climate_zone.json @@ -0,0 +1,163 @@ +{ + "name": "demand.egon_daily_heat_demand_per_climate_zone", + "title": "eGon daily heat demand share per climate zone", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": "Share of daily heat demands per climate zone", + "language": [ + "en-EN", + "de-DE" + ], + "keywords": [], + "publicationDate": "2022-09-14", + "context": { + "homepage": "https://ego-n.org/", + "documentation": "https://egon-data.readthedocs.io/en/latest/", + "sourceCode": "https://github.com/openego/eGon-data", + "contact": "https://ego-n.org/partners/", + "grantNo": "03EI1002", + "fundingAgency": "Bundesministerium f\u00fcr Wirtschaft und Klimaschutz", + "fundingAgencyLogo": "https://www.innovation-beratung-foerderung.de/INNO/Redaktion/DE/Bilder/Titelbilder/titel_foerderlogo_bmwi.jpg?__blob=normal&v=3", + "publisherLogo": "https://ego-n.org/images/eGon_logo_noborder_transbg.svg" + }, + "spatial": { + "location": "", + "extent": "Germany", + "resolution": "climate zones" + }, + "temporal": { + "referenceDate": "2022-09-14", + "timeseries": {} + }, + "sources": [ + { + "title": "eGon-data", + "description": "Workflow to download, process and generate data sets suitable for the further research conducted in the project eGon (https://ego-n.org/)", + "path": "https://github.com/openego/eGon-data", + "licenses": [ + { + "name": "AGPL-3.0 License", + "title": "GNU Affero General Public License v3.0", + "path": "https://www.gnu.org/licenses/agpl-3.0.de.html", + "instruction": "Permissions of this strongest copyleft license are conditioned on making available complete source code of licensed works and modifications, which include larger works using a licensed work, under the same license. Copyright and license notices must be preserved. Contributors provide an express grant of patent rights. When a modified version is used to provide a service over a network, the complete source code of the modified version must be made available.", + "attribution": "\u00a9 eGon-data developers" + } + ] + }, + { + "title": "Data bundle for egon-data: A transparent and reproducible data processing pipeline for energy system modeling", + "description": "egon-data provides a transparent and reproducible open data based data processing pipeline for generating data models suitable for energy system modeling. The data is customized for the requirements of the research project eGon. The research project aims to develop tools for an open and cross-sectoral planning of transmission and distribution grids. For further information please visit the eGon project website or its Github repository. egon-data retrieves and processes data from several different external input sources. As not all data dependencies can be downloaded automatically from external sources we provide a data bundle to be downloaded by egon-data.", + "path": "https://doi.org/10.5281/zenodo.6630616", + "licenses": [ + { + "name": "CC-BY-4.0", + "title": "Creative Commons Attribution 4.0 International", + "path": "https://creativecommons.org/licenses/by/4.0/legalcode", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute.", + "attribution": "\u00a9 eGon-data developers, \u00a9 Leibniz Institute for Applied Geophysics, \u00a9 Fraunhofer IEE, \u00a9 BGR Hannover, \u00a9 Danielle Schmidt, \u00a9 \u00dcbertragungsnetzbetreiber, \u00a9 DIW Berlin, \u00a9 Reiner Lemoine Institut, \u00a9 Statistisches Bundesamt, Wiesbaden 2008, \u00a9 Statistische \u00c4mter des Bundes und der L\u00e4nder 2021" + } + ] + }, + { + "title": "ERA5 global reanalysis", + "description": "ERA5 is the fifth generation ECMWF reanalysis for the global climate and weather for the past 4 to 7 decades. Currently data is available from 1950, split into Climate Data Store entries for 1950-1978 (preliminary back extension) and from 1979 onwards (final release plus timely updates, this page). ERA5 replaces the ERA-Interim reanalysis. See the online ERA5 documentation (https://confluence.ecmwf.int/display/CKB/ERA5%3A+data+documentation#ERA5:datadocumentation-Dataupdatefrequency) for more information.", + "path": "https://confluence.ecmwf.int/display/CKB/ERA5%3A+data+documentation#ERA5:datadocumentation-Dataupdatefrequency", + "licenses": [ + { + "name": "Licence to use Copernicus Products", + "title": "Licence to use Copernicus Products", + "path": "https://cds.climate.copernicus.eu/api/v2/terms/static/licence-to-use-copernicus-products.pdf", + "instruction": "This Licence is free of charge, worldwide, non-exclusive, royalty free and perpetual. Access to Copernicus Products is given for any purpose in so far as it is lawful, whereas use may include, but is not limited to: reproduction; distribution; communication to the public; adaptation, modification and combination with other data and information; or any combination of the foregoing", + "attribution": "\u00a9 Copernicus Climate Change Service (C3S) Climate Data Store" + } + ] + } + ], + "licenses": [ + { + "name": "CC-BY-4.0", + "title": "Creative Commons Attribution 4.0 International", + "path": "https://creativecommons.org/licenses/by/4.0/legalcode", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute.", + "attribution": "\u00a9 eGon-data developers" + } + ], + "contributors": [ + { + "title": "ClaraBuettner", + "email": "https://github.com/ClaraBuettner", + "date": "2022-09-12", + "object": "metadata", + "comment": "Create metadata" + }, + { + "title": "nesnoj", + "email": "https://github.com/nesnoj", + "date": "2022-09-14", + "object": "metadata", + "comment": "Update metadata" + } + ], + "resources": [ + { + "profile": "tabular-data-resource", + "name": "demand.egon_daily_heat_demand_per_climate_zone", + "path": "", + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": [ + { + "name": "climate_zone", + "description": "Name of the corresponding climate zone", + "type": "str", + "unit": "none" + }, + { + "name": "daily_demand_share", + "description": "Share of annual heat demand for each day", + "type": "double precision", + "unit": "per unit" + }, + { + "name": "day_of_year", + "description": "Number of day in the year", + "type": "integer", + "unit": "none" + }, + { + "name": "temperature_class", + "description": "Temperature class (considering the mean outside temeperature during the day)", + "type": "integer", + "unit": "none" + } + ], + "primaryKey": "climate_zone" + }, + "dialect": { + "delimiter": "", + "decimalSeparator": "" + } + } + ], + "review": { + "path": "", + "badge": "" + }, + "metaMetadata": { + "metadataVersion": "OEP-1.4.1", + "metadataLicense": { + "name": "CC0-1.0", + "title": "Creative Commons Zero v1.0 Universal", + "path": "https://creativecommons.org/publicdomain/zero/1.0/" + } + }, + "_comment": { + "metadata": "Metadata documentation and explanation (https://github.com/OpenEnergyPlatform/oemetadata/blob/master/metadata/v141/metadata_key_description.md)", + "dates": "Dates and time must follow the ISO8601 including time zone (YYYY-MM-DD or YYYY-MM-DDThh:mm:ss\u00b1hh)", + "units": "Use a space between numbers and units (100 m)", + "languages": "Languages must follow the IETF (BCP47) format (en-GB, en-US, de-DE)", + "licenses": "License name must follow the SPDX License List (https://spdx.org/licenses/)", + "review": "Following the OEP Data Review (https://github.com/OpenEnergyPlatform/data-preprocessing/wiki)", + "none": "If not applicable use (none)" + } +} diff --git a/src/egon/data/metadata/demand.egon_etrago_electricity_cts.json b/src/egon/data/metadata/demand.egon_etrago_electricity_cts.json new file mode 100644 index 000000000..5db7cd176 --- /dev/null +++ b/src/egon/data/metadata/demand.egon_etrago_electricity_cts.json @@ -0,0 +1,197 @@ +{ + "name": "demand.egon_etrago_electricity_cts", + "title": "MV cts electricity demand profiles", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": "Aggregated electricity demand profile of cts sector at mvgd bus level for two scenarios. Values are given in MWh.", + "language": "undefined", + "keywords": [ + "" + ], + "publicationDate": "2022-09-14", + "context": { + "homepage": "https://ego-n.org/", + "documentation": "https://egon-data.readthedocs.io/en/latest/", + "sourceCode": "https://github.com/openego/eGon-data", + "contact": "https://ego-n.org/partners/", + "grantNo": "03EI1002", + "fundingAgency": "Bundesministerium f\u00fcr Wirtschaft und Klimaschutz", + "fundingAgencyLogo": "https://commons.wikimedia.org/wiki/File:BMWi_Logo_2021.svg#/media/File:BMWi_Logo_2021.svg", + "publisherLogo": "https://ego-n.org/images/eGon_logo_noborder_transbg.svg" + }, + "spatial": { + "location": "", + "extent": "Germany", + "resolution": "MV Grid" + }, + "temporal": { + "referenceDate": "2023-03-13", + "timeseries": { + "start": "2016-01-01", + "end": "2016-12-31", + "resolution": "1 h", + "alignment": "left", + "aggregationType": "sum" + } + }, + "sources": [ + { + "title": "Data bundle for egon-data: A transparent and reproducible data processing pipeline for energy system modeling", + "description": "egon-data provides a transparent and reproducible open data based data processing pipeline for generating data models suitable for energy system modeling. The data is customized for the requirements of the research project eGon. The research project aims to develop tools for an open and cross-sectoral planning of transmission and distribution grids. For further information please visit the eGon project website or its Github repository. egon-data retrieves and processes data from several different external input sources. As not all data dependencies can be downloaded automatically from external sources we provide a data bundle to be downloaded by egon-data.", + "path": "https://doi.org/10.5281/zenodo.6630616", + "licenses": [ + { + "name": "CC-BY-4.0", + "title": "Creative Commons Attribution 4.0 International", + "path": "https://creativecommons.org/licenses/by/4.0/legalcode", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute.", + "attribution": "\u00a9 eGon-data developers, \u00a9 Leibniz Institute for Applied Geophysics, \u00a9 Fraunhofer IEE, \u00a9 BGR Hannover, \u00a9 Danielle Schmidt, \u00a9 \u00dcbertragungsnetzbetreiber, \u00a9 DIW Berlin, \u00a9 Reiner Lemoine Institut, \u00a9 Statistisches Bundesamt, Wiesbaden 2008, \u00a9 Statistische \u00c4mter des Bundes und der L\u00e4nder 2021" + } + ] + }, + { + "title": "eGon-data", + "description": "Workflow to download, process and generate data sets suitable for the further research conducted in the project eGon (https://ego-n.org/)", + "path": "https://github.com/openego/eGon-data", + "licenses": [ + { + "name": "AGPL-3.0 License", + "title": "GNU Affero General Public License v3.0", + "path": "https://www.gnu.org/licenses/agpl-3.0.de.html", + "instruction": "Permissions of this strongest copyleft license are conditioned on making available complete source code of licensed works and modifications, which include larger works using a licensed work, under the same license. Copyright and license notices must be preserved. Contributors provide an express grant of patent rights. When a modified version is used to provide a service over a network, the complete source code of the modified version must be made available.", + "attribution": "\u00a9 eGon-data developers" + } + ] + }, + { + "title": "OpenStreetMap Data Extracts (Geofabrik)", + "description": "Full data extract of OpenStreetMap data for defined spatial extent at 2022-01-01", + "path": "https://download.geofabrik.de/europe/germany-210101.osm.pbf", + "licenses": [ + { + "name": "ODbL-1.0", + "title": "Open Data Commons Open Database License 1.0", + "path": "https://opendatacommons.org/licenses/odbl/1.0/", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute, Share-Alike, Keep open!", + "attribution": "\u00a9 OpenStreetMap contributors" + } + ] + }, + { + "title": "Statistisches Bundesamt (Destatis) - Ergebnisse des Zensus 2011 zum Download", + "description": "Als Download bieten wir Ihnen auf dieser Seite zus\\u00e4tzlich zur Zensusdatenbank CSV- und teilweise Excel-Tabellen mit umfassenden Personen-, Haushalts- und Familien- sowie Geb\\u00e4ude- und Wohnungsmerkmalen. Die Ergebnisse liegen auf Bundes-, L\\u00e4nder-, Kreis- und Gemeindeebene vor. Au\\u00dferdem sind einzelne Ergebnisse f\\u00fcr Gitterzellen verf\\u00fcgbar.", + "path": "https://www.zensus2011.de/DE/Home/Aktuelles/DemografischeGrunddaten.html", + "licenses": [ + { + "name": "dl-by-de/2.0", + "title": "Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0", + "path": "www.govdata.de/dl-de/by-2-0", + "instruction": "Jede Nutzung ist unter den Bedingungen dieser \\u201eDatenlizenz Deutschland - Namensnennung - Version 2.0 zul\\u00e4ssig.\\nDie bereitgestellten Daten und Metadaten d\\u00fcrfen f\\u00fcr die kommerzielle und nicht kommerzielle Nutzung insbesondere:(1) vervielf\\u00e4ltigt, ausgedruckt, pr\\u00e4sentiert, ver\\u00e4ndert, bearbeitet sowie an Dritte \\u00fcbermittelt werden;\\n (2) mit eigenen Daten und Daten Anderer zusammengef\\u00fchrt und zu selbst\\u00e4ndigen neuen Datens\\u00e4tzen verbunden werden;\\n (3) in interne und externe Gesch\\u00e4ftsprozesse, Produkte und Anwendungen in \\u00f6ffentlichen und nicht \\u00f6ffentlichen elektronischen Netzwerken eingebunden werden.\\nBei der Nutzung ist sicherzustellen, dass folgende Angaben als Quellenvermerk enthalten sind:\\n(1) Bezeichnung des Bereitstellers nach dessen Ma\\u00dfgabe,\\n(2) der Vermerk Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0 oder dl-de/by-2-0 mit Verweis auf den Lizenztext unter www.govdata.de/dl-de/by-2-0 sowie\\n(3) einen Verweis auf den Datensatz (URI).Dies gilt nur soweit die datenhaltende Stelle die Angaben(1) bis (3) zum Quellenvermerk bereitstellt.\\nVer\\u00e4nderungen, Bearbeitungen, neue Gestaltungen oder sonstige Abwandlungen sind im Quellenvermerk mit dem Hinweis zu versehen, dass die Daten ge\\u00e4ndert wurden.", + "attribution": "\\u00a9 Statistische \\u00c4mter des Bundes und der L\\u00e4nder 2014" + } + ] + }, + { + "title": "Dokumentation - Zensus 2011 - Methoden und Verfahren", + "description": "Diese Publikation beschreibt ausf\\u00fchrlich die Methoden und Verfahren des registergest\\u00fctzten Zensus 2011; von der Datengewinnung und -aufbereitung bis hin zur Ergebniserstellung und Geheimhaltung. Der vorliegende Band wurde von den Statistischen \\u00c4mtern des Bundes und der L\\u00e4nder im Juni 2015 ver\\u00f6ffentlicht.", + "path": "https://www.destatis.de/DE/Publikationen/Thematisch/Bevoelkerung/Zensus/ZensusBuLaMethodenVerfahren5121105119004.pdf?__blob=publicationFile", + "licenses": [ + { + "name": "dl-by-de/2.0", + "title": "Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0", + "path": "www.govdata.de/dl-de/by-2-0", + "instruction": "Jede Nutzung ist unter den Bedingungen dieser \\u201eDatenlizenz Deutschland - Namensnennung - Version 2.0 zul\\u00e4ssig.\\nDie bereitgestellten Daten und Metadaten d\\u00fcrfen f\\u00fcr die kommerzielle und nicht kommerzielle Nutzung insbesondere:(1) vervielf\\u00e4ltigt, ausgedruckt, pr\\u00e4sentiert, ver\\u00e4ndert, bearbeitet sowie an Dritte \\u00fcbermittelt werden;\\n (2) mit eigenen Daten und Daten Anderer zusammengef\\u00fchrt und zu selbst\\u00e4ndigen neuen Datens\\u00e4tzen verbunden werden;\\n (3) in interne und externe Gesch\\u00e4ftsprozesse, Produkte und Anwendungen in \\u00f6ffentlichen und nicht \\u00f6ffentlichen elektronischen Netzwerken eingebunden werden.\\nBei der Nutzung ist sicherzustellen, dass folgende Angaben als Quellenvermerk enthalten sind:\\n(1) Bezeichnung des Bereitstellers nach dessen Ma\\u00dfgabe,\\n(2) der Vermerk Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0 oder dl-de/by-2-0 mit Verweis auf den Lizenztext unter www.govdata.de/dl-de/by-2-0 sowie\\n(3) einen Verweis auf den Datensatz (URI).Dies gilt nur soweit die datenhaltende Stelle die Angaben(1) bis (3) zum Quellenvermerk bereitstellt.\\nVer\\u00e4nderungen, Bearbeitungen, neue Gestaltungen oder sonstige Abwandlungen sind im Quellenvermerk mit dem Hinweis zu versehen, dass die Daten ge\\u00e4ndert wurden.", + "attribution": "\\u00a9 Statistisches Bundesamt, Wiesbaden 2015 (im Auftrag der Herausgebergemeinschaft)" + } + ] + }, + { + "title": "DemandRegio", + "description": "Harmonisierung und Entwicklung von Verfahren zur regionalen und zeitlichen Aufl\u00f6sung von Energienachfragen", + "path": "https://doi.org/10.34805/ffe-119-20", + "license": [ + { + "name": "Creative Commons Attribution 4.0 International", + "title": "CC BY 4.0", + "path": "https://creativecommons.org/licenses/by/4.0/", + "instruction": "You are free: To Share, To Adapt; As long as you: Attribute!", + "attribution": "\u00a9 FZJ, TUB, FfE" + } + ] + } + ], + "licenses": [ + { + "name": "ODbL-1.0", + "title": "Open Data Commons Open Database License 1.0", + "path": "https://opendatacommons.org/licenses/odbl/1.0/", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute, Share-Alike, Keep open!", + "attribution": "\u00a9 Reiner Lemoine Institut" + } + ], + "contributors": [ + { + "title": "nailend", + "email": "https://github.com/nailend", + "date": "2023-03-13", + "object": "metadata", + "comment": "Create metadata" + } + ], + "resources": [ + { + "profile": "tabular-data-resource", + "name": "demand.egon_etrago_electricity_cts", + "path": "", + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": [ + { + "name": "bus_id", + "description": "MV bus_id (reference: demand.egon_cts_electricity_demand_building_share.bus_id, grid.egon_hvmv_substation.bus_id, grid.egon_mv_grid_district.bus_id))", + "type": "integer", + "unit": "none" + }, + { + "name": "scn_name", + "description": "Scenario name (reference: demand.egon_cts_electricity_demand_building_share.scenario)", + "type": "character varying", + "unit": "none" + }, + { + "name": "p_set", + "description": "Cts electricity demand load timeseries for one year (8760 hours) in MWh", + "type": "array of real", + "unit": "MWh" + } + ], + "primaryKey": "id, scn_name" + }, + "dialect": { + "delimiter": "", + "decimalSeparator": "." + } + } + ], + "review": { + "path": "", + "badge": "" + }, + "metaMetadata": { + "metadataVersion": "OEP-1.4.1", + "metadataLicense": { + "name": "CC0-1.0", + "title": "Creative Commons Zero v1.0 Universal", + "path": "https://creativecommons.org/publicdomain/zero/1.0/" + } + }, + "_comment": { + "metadata": "Metadata documentation and explanation (https://github.com/OpenEnergyPlatform/oemetadata/blob/master/metadata/v141/metadata_key_description.md)", + "dates": "Dates and time must follow the ISO8601 including time zone (YYYY-MM-DD or YYYY-MM-DDThh:mm:ss\u00b1hh)", + "units": "Use a space between numbers and units (100 m)", + "languages": "Languages must follow the IETF (BCP47) format (en-GB, en-US, de-DE)", + "licenses": "License name must follow the SPDX License List (https://spdx.org/licenses/)", + "review": "Following the OEP Data Review (https://github.com/OpenEnergyPlatform/data-preprocessing/wiki)", + "none": "If not applicable use (none)" + } +} diff --git a/src/egon/data/metadata/demand.egon_etrago_electricity_households.json b/src/egon/data/metadata/demand.egon_etrago_electricity_households.json new file mode 100644 index 000000000..b46c4b4ef --- /dev/null +++ b/src/egon/data/metadata/demand.egon_etrago_electricity_households.json @@ -0,0 +1,197 @@ +{ + "name": "demand.egon_etrago_electricity_households", + "title": "MV residential electricity demand profiles", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": "Aggregated electricity demand profile of residential sector at mvgd bus level for two scenarios. Values are given in MWh.", + "language": "en-US", + "keywords": [ + "" + ], + "publicationDate": "2022-09-14", + "context": { + "homepage": "https://ego-n.org/", + "documentation": "https://egon-data.readthedocs.io/en/latest/", + "sourceCode": "https://github.com/openego/eGon-data", + "contact": "https://ego-n.org/partners/", + "grantNo": "03EI1002", + "fundingAgency": "Bundesministerium f\u00fcr Wirtschaft und Klimaschutz", + "fundingAgencyLogo": "https://commons.wikimedia.org/wiki/File:BMWi_Logo_2021.svg#/media/File:BMWi_Logo_2021.svg", + "publisherLogo": "https://ego-n.org/images/eGon_logo_noborder_transbg.svg" + }, + "spatial": { + "location": "", + "extent": "Germany", + "resolution": "MV Grid" + }, + "temporal": { + "referenceDate": "2023-03-13", + "timeseries": { + "start": "2016-01-01", + "end": "2016-12-31", + "resolution": "1 h", + "alignment": "left", + "aggregationType": "sum" + } + }, + "sources": [ + { + "title": "Data bundle for egon-data: A transparent and reproducible data processing pipeline for energy system modeling", + "description": "egon-data provides a transparent and reproducible open data based data processing pipeline for generating data models suitable for energy system modeling. The data is customized for the requirements of the research project eGon. The research project aims to develop tools for an open and cross-sectoral planning of transmission and distribution grids. For further information please visit the eGon project website or its Github repository. egon-data retrieves and processes data from several different external input sources. As not all data dependencies can be downloaded automatically from external sources we provide a data bundle to be downloaded by egon-data.", + "path": "https://doi.org/10.5281/zenodo.6630616", + "licenses": [ + { + "name": "CC-BY-4.0", + "title": "Creative Commons Attribution 4.0 International", + "path": "https://creativecommons.org/licenses/by/4.0/legalcode", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute.", + "attribution": "\u00a9 eGon-data developers, \u00a9 Leibniz Institute for Applied Geophysics, \u00a9 Fraunhofer IEE, \u00a9 BGR Hannover, \u00a9 Danielle Schmidt, \u00a9 \u00dcbertragungsnetzbetreiber, \u00a9 DIW Berlin, \u00a9 Reiner Lemoine Institut, \u00a9 Statistisches Bundesamt, Wiesbaden 2008, \u00a9 Statistische \u00c4mter des Bundes und der L\u00e4nder 2021" + } + ] + }, + { + "title": "eGon-data", + "description": "Workflow to download, process and generate data sets suitable for the further research conducted in the project eGon (https://ego-n.org/)", + "path": "https://github.com/openego/eGon-data", + "licenses": [ + { + "name": "AGPL-3.0 License", + "title": "GNU Affero General Public License v3.0", + "path": "https://www.gnu.org/licenses/agpl-3.0.de.html", + "instruction": "Permissions of this strongest copyleft license are conditioned on making available complete source code of licensed works and modifications, which include larger works using a licensed work, under the same license. Copyright and license notices must be preserved. Contributors provide an express grant of patent rights. When a modified version is used to provide a service over a network, the complete source code of the modified version must be made available.", + "attribution": "\u00a9 eGon-data developers" + } + ] + }, + { + "title": "OpenStreetMap Data Extracts (Geofabrik)", + "description": "Full data extract of OpenStreetMap data for defined spatial extent at 2022-01-01", + "path": "https://download.geofabrik.de/europe/germany-210101.osm.pbf", + "licenses": [ + { + "name": "ODbL-1.0", + "title": "Open Data Commons Open Database License 1.0", + "path": "https://opendatacommons.org/licenses/odbl/1.0/", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute, Share-Alike, Keep open!", + "attribution": "\u00a9 OpenStreetMap contributors" + } + ] + }, + { + "title": "Statistisches Bundesamt (Destatis) - Ergebnisse des Zensus 2011 zum Download", + "description": "Als Download bieten wir Ihnen auf dieser Seite zus\\u00e4tzlich zur Zensusdatenbank CSV- und teilweise Excel-Tabellen mit umfassenden Personen-, Haushalts- und Familien- sowie Geb\\u00e4ude- und Wohnungsmerkmalen. Die Ergebnisse liegen auf Bundes-, L\\u00e4nder-, Kreis- und Gemeindeebene vor. Au\\u00dferdem sind einzelne Ergebnisse f\\u00fcr Gitterzellen verf\\u00fcgbar.", + "path": "https://www.zensus2011.de/DE/Home/Aktuelles/DemografischeGrunddaten.html", + "licenses": [ + { + "name": "dl-by-de/2.0", + "title": "Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0", + "path": "www.govdata.de/dl-de/by-2-0", + "instruction": "Jede Nutzung ist unter den Bedingungen dieser \\u201eDatenlizenz Deutschland - Namensnennung - Version 2.0 zul\\u00e4ssig.\\nDie bereitgestellten Daten und Metadaten d\\u00fcrfen f\\u00fcr die kommerzielle und nicht kommerzielle Nutzung insbesondere:(1) vervielf\\u00e4ltigt, ausgedruckt, pr\\u00e4sentiert, ver\\u00e4ndert, bearbeitet sowie an Dritte \\u00fcbermittelt werden;\\n (2) mit eigenen Daten und Daten Anderer zusammengef\\u00fchrt und zu selbst\\u00e4ndigen neuen Datens\\u00e4tzen verbunden werden;\\n (3) in interne und externe Gesch\\u00e4ftsprozesse, Produkte und Anwendungen in \\u00f6ffentlichen und nicht \\u00f6ffentlichen elektronischen Netzwerken eingebunden werden.\\nBei der Nutzung ist sicherzustellen, dass folgende Angaben als Quellenvermerk enthalten sind:\\n(1) Bezeichnung des Bereitstellers nach dessen Ma\\u00dfgabe,\\n(2) der Vermerk Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0 oder dl-de/by-2-0 mit Verweis auf den Lizenztext unter www.govdata.de/dl-de/by-2-0 sowie\\n(3) einen Verweis auf den Datensatz (URI).Dies gilt nur soweit die datenhaltende Stelle die Angaben(1) bis (3) zum Quellenvermerk bereitstellt.\\nVer\\u00e4nderungen, Bearbeitungen, neue Gestaltungen oder sonstige Abwandlungen sind im Quellenvermerk mit dem Hinweis zu versehen, dass die Daten ge\\u00e4ndert wurden.", + "attribution": "\\u00a9 Statistische \\u00c4mter des Bundes und der L\\u00e4nder 2014" + } + ] + }, + { + "title": "Dokumentation - Zensus 2011 - Methoden und Verfahren", + "description": "Diese Publikation beschreibt ausf\\u00fchrlich die Methoden und Verfahren des registergest\\u00fctzten Zensus 2011; von der Datengewinnung und -aufbereitung bis hin zur Ergebniserstellung und Geheimhaltung. Der vorliegende Band wurde von den Statistischen \\u00c4mtern des Bundes und der L\\u00e4nder im Juni 2015 ver\\u00f6ffentlicht.", + "path": "https://www.destatis.de/DE/Publikationen/Thematisch/Bevoelkerung/Zensus/ZensusBuLaMethodenVerfahren5121105119004.pdf?__blob=publicationFile", + "licenses": [ + { + "name": "dl-by-de/2.0", + "title": "Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0", + "path": "www.govdata.de/dl-de/by-2-0", + "instruction": "Jede Nutzung ist unter den Bedingungen dieser \\u201eDatenlizenz Deutschland - Namensnennung - Version 2.0 zul\\u00e4ssig.\\nDie bereitgestellten Daten und Metadaten d\\u00fcrfen f\\u00fcr die kommerzielle und nicht kommerzielle Nutzung insbesondere:(1) vervielf\\u00e4ltigt, ausgedruckt, pr\\u00e4sentiert, ver\\u00e4ndert, bearbeitet sowie an Dritte \\u00fcbermittelt werden;\\n (2) mit eigenen Daten und Daten Anderer zusammengef\\u00fchrt und zu selbst\\u00e4ndigen neuen Datens\\u00e4tzen verbunden werden;\\n (3) in interne und externe Gesch\\u00e4ftsprozesse, Produkte und Anwendungen in \\u00f6ffentlichen und nicht \\u00f6ffentlichen elektronischen Netzwerken eingebunden werden.\\nBei der Nutzung ist sicherzustellen, dass folgende Angaben als Quellenvermerk enthalten sind:\\n(1) Bezeichnung des Bereitstellers nach dessen Ma\\u00dfgabe,\\n(2) der Vermerk Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0 oder dl-de/by-2-0 mit Verweis auf den Lizenztext unter www.govdata.de/dl-de/by-2-0 sowie\\n(3) einen Verweis auf den Datensatz (URI).Dies gilt nur soweit die datenhaltende Stelle die Angaben(1) bis (3) zum Quellenvermerk bereitstellt.\\nVer\\u00e4nderungen, Bearbeitungen, neue Gestaltungen oder sonstige Abwandlungen sind im Quellenvermerk mit dem Hinweis zu versehen, dass die Daten ge\\u00e4ndert wurden.", + "attribution": "\\u00a9 Statistisches Bundesamt, Wiesbaden 2015 (im Auftrag der Herausgebergemeinschaft)" + } + ] + }, + { + "title": "DemandRegio", + "description": "Harmonisierung und Entwicklung von Verfahren zur regionalen und zeitlichen Aufl\u00f6sung von Energienachfragen", + "path": "https://doi.org/10.34805/ffe-119-20", + "license": [ + { + "name": "Creative Commons Attribution 4.0 International", + "title": "CC BY 4.0", + "path": "https://creativecommons.org/licenses/by/4.0/", + "instruction": "You are free: To Share, To Adapt; As long as you: Attribute!", + "attribution": "\u00a9 FZJ, TUB, FfE" + } + ] + } + ], + "licenses": [ + { + "name": "ODbL-1.0", + "title": "Open Data Commons Open Database License 1.0", + "path": "https://opendatacommons.org/licenses/odbl/1.0/", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute, Share-Alike, Keep open!", + "attribution": "\u00a9 Reiner Lemoine Institut" + } + ], + "contributors": [ + { + "title": "nailend", + "email": "https://github.com/nailend", + "date": "2023-03-13", + "object": "metadata", + "comment": "Create metadata" + } + ], + "resources": [ + { + "profile": "tabular-data-resource", + "name": "demand.egon_etrago_electricity_households", + "path": "", + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": [ + { + "name": "bus_id", + "description": "MV bus_id (reference: demand.egon_cts_electricity_demand_building_share.bus_id, grid.egon_hvmv_substation.bus_id, grid.egon_mv_grid_district.bus_id))", + "type": "integer", + "unit": "none" + }, + { + "name": "scn_name", + "description": "Scenario name", + "type": "character varying", + "unit": "none" + }, + { + "name": "p_set", + "description": "Residential electricity demand load timeseries for one year (8760 hours) in MWh", + "type": "array of double precission", + "unit": "MWh" + } + ], + "primaryKey": "id, scn_name" + }, + "dialect": { + "delimiter": "", + "decimalSeparator": "." + } + } + ], + "review": { + "path": "", + "badge": "" + }, + "metaMetadata": { + "metadataVersion": "OEP-1.4.1", + "metadataLicense": { + "name": "CC0-1.0", + "title": "Creative Commons Zero v1.0 Universal", + "path": "https://creativecommons.org/publicdomain/zero/1.0/" + } + }, + "_comment": { + "metadata": "Metadata documentation and explanation (https://github.com/OpenEnergyPlatform/oemetadata/blob/master/metadata/v141/metadata_key_description.md)", + "dates": "Dates and time must follow the ISO8601 including time zone (YYYY-MM-DD or YYYY-MM-DDThh:mm:ss\u00b1hh)", + "units": "Use a space between numbers and units (100 m)", + "languages": "Languages must follow the IETF (BCP47) format (en-GB, en-US, de-DE)", + "licenses": "License name must follow the SPDX License List (https://spdx.org/licenses/)", + "review": "Following the OEP Data Review (https://github.com/OpenEnergyPlatform/data-preprocessing/wiki)", + "none": "If not applicable use (none)" + } +} diff --git a/src/egon/data/metadata/demand.egon_etrago_heat_cts.json b/src/egon/data/metadata/demand.egon_etrago_heat_cts.json new file mode 100644 index 000000000..6fff2bd10 --- /dev/null +++ b/src/egon/data/metadata/demand.egon_etrago_heat_cts.json @@ -0,0 +1,211 @@ +{ + "name": "demand.egon_etrago_heat_cts", + "title": "MV cts heat demand profiles", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": "Aggregated heat demand profile of cts sector at mvgd bus level for two scenarios. Values are given in MWh.", + "language": "undefined", + "keywords": [ + "" + ], + "publicationDate": "2023-03-13", + "context": { + "homepage": "https://ego-n.org/", + "documentation": "https://egon-data.readthedocs.io/en/latest/", + "sourceCode": "https://github.com/openego/eGon-data", + "contact": "https://ego-n.org/partners/", + "grantNo": "03EI1002", + "fundingAgency": "Bundesministerium f\u00fcr Wirtschaft und Klimaschutz", + "fundingAgencyLogo": "https://commons.wikimedia.org/wiki/File:BMWi_Logo_2021.svg#/media/File:BMWi_Logo_2021.svg", + "publisherLogo": "https://ego-n.org/images/eGon_logo_noborder_transbg.svg" + }, + "spatial": { + "location": "", + "extent": "Germany", + "resolution": "MV Grid" + }, + "temporal": { + "referenceDate": "2023-03-13", + "timeseries": { + "start": "2016-01-01", + "end": "2016-12-31", + "resolution": "1 h", + "alignment": "left", + "aggregationType": "sum" + } + }, + "sources": [ + { + "title": "Data bundle for egon-data: A transparent and reproducible data processing pipeline for energy system modeling", + "description": "egon-data provides a transparent and reproducible open data based data processing pipeline for generating data models suitable for energy system modeling. The data is customized for the requirements of the research project eGon. The research project aims to develop tools for an open and cross-sectoral planning of transmission and distribution grids. For further information please visit the eGon project website or its Github repository. egon-data retrieves and processes data from several different external input sources. As not all data dependencies can be downloaded automatically from external sources we provide a data bundle to be downloaded by egon-data.", + "path": "https://doi.org/10.5281/zenodo.6630616", + "licenses": [ + { + "name": "CC-BY-4.0", + "title": "Creative Commons Attribution 4.0 International", + "path": "https://creativecommons.org/licenses/by/4.0/legalcode", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute.", + "attribution": "\u00a9 eGon-data developers, \u00a9 Leibniz Institute for Applied Geophysics, \u00a9 Fraunhofer IEE, \u00a9 BGR Hannover, \u00a9 Danielle Schmidt, \u00a9 \u00dcbertragungsnetzbetreiber, \u00a9 DIW Berlin, \u00a9 Reiner Lemoine Institut, \u00a9 Statistisches Bundesamt, Wiesbaden 2008, \u00a9 Statistische \u00c4mter des Bundes und der L\u00e4nder 2021" + } + ] + }, + { + "title": "eGon-data", + "description": "Workflow to download, process and generate data sets suitable for the further research conducted in the project eGon (https://ego-n.org/)", + "path": "https://github.com/openego/eGon-data", + "licenses": [ + { + "name": "AGPL-3.0 License", + "title": "GNU Affero General Public License v3.0", + "path": "https://www.gnu.org/licenses/agpl-3.0.de.html", + "instruction": "Permissions of this strongest copyleft license are conditioned on making available complete source code of licensed works and modifications, which include larger works using a licensed work, under the same license. Copyright and license notices must be preserved. Contributors provide an express grant of patent rights. When a modified version is used to provide a service over a network, the complete source code of the modified version must be made available.", + "attribution": "\u00a9 eGon-data developers" + } + ] + }, + { + "title": "Verwaltungsgebiete 1:250 000 (Ebenen)", + "description": "Der Datenbestand umfasst s\u00e4mtliche Verwaltungseinheiten der hierarchischen Verwaltungsebenen vom Staat bis zu den Gemeinden mit ihren Grenzen, statistischen Schl\u00fcsselzahlen, Namen der Verwaltungseinheit sowie die spezifische Bezeichnung der Verwaltungsebene des jeweiligen Landes.", + "path": "https://daten.gdz.bkg.bund.de/produkte/vg/vg250_ebenen_0101/2020/vg250_01-01.geo84.shape.ebenen.zip", + "licenses": [ + { + "name": "dl-by-de/2.0", + "title": "Datenlizenz Deutschland \u2013 Namensnennung \u2013 Version 2.0", + "path": "www.govdata.de/dl-de/by-2-0", + "instruction": "Jede Nutzung ist unter den Bedingungen dieser \u201eDatenlizenz Deutschland - Namensnennung - Version 2.0 zul\u00e4ssig.Die bereitgestellten Daten und Metadaten d\u00fcrfen f\u00fcr die kommerzielle und nicht kommerzielle Nutzung insbesondere:(1) vervielf\u00e4ltigt, ausgedruckt, pr\u00e4sentiert, ver\u00e4ndert, bearbeitet sowie an Dritte \u00fcbermittelt werden; (2) mit eigenen Daten und Daten Anderer zusammengef\u00fchrt und zu selbst\u00e4ndigen neuen Datens\u00e4tzen verbunden werden; (3) in interne und externe Gesch\u00e4ftsprozesse, Produkte und Anwendungen in \u00f6ffentlichen und nicht \u00f6ffentlichen elektronischen Netzwerken eingebunden werden.Bei der Nutzung ist sicherzustellen, dass folgende Angaben als Quellenvermerk enthalten sind:(1) Bezeichnung des Bereitstellers nach dessen Ma\u00dfgabe,(2) der Vermerk Datenlizenz Deutschland \u2013 Namensnennung \u2013 Version 2.0 oder dl-de/by-2-0 mit Verweis auf den Lizenztext unter www.govdata.de/dl-de/by-2-0 sowie(3) einen Verweis auf den Datensatz (URI).Dies gilt nur soweit die datenhaltende Stelle die Angaben(1) bis (3) zum Quellenvermerk bereitstellt.Ver\u00e4nderungen, Bearbeitungen, neue Gestaltungen oder sonstige Abwandlungen sind im Quellenvermerk mit dem Hinweis zu versehen, dass die Daten ge\u00e4ndert wurden.", + "attribution": "\u00a9 Bundesamt f\u00fcr Kartographie und Geod\u00e4sie 2020 (Daten ver\u00e4ndert)" + } + ] + }, + { + "title": "Statistisches Bundesamt (Destatis) - Ergebnisse des Zensus 2011 zum Download", + "description": "Als Download bieten wir Ihnen auf dieser Seite zus\\u00e4tzlich zur Zensusdatenbank CSV- und teilweise Excel-Tabellen mit umfassenden Personen-, Haushalts- und Familien- sowie Geb\\u00e4ude- und Wohnungsmerkmalen. Die Ergebnisse liegen auf Bundes-, L\\u00e4nder-, Kreis- und Gemeindeebene vor. Au\\u00dferdem sind einzelne Ergebnisse f\\u00fcr Gitterzellen verf\\u00fcgbar.", + "path": "https://www.zensus2011.de/DE/Home/Aktuelles/DemografischeGrunddaten.html", + "licenses": [ + { + "name": "dl-by-de/2.0", + "title": "Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0", + "path": "www.govdata.de/dl-de/by-2-0", + "instruction": "Jede Nutzung ist unter den Bedingungen dieser \\u201eDatenlizenz Deutschland - Namensnennung - Version 2.0 zul\\u00e4ssig.\\nDie bereitgestellten Daten und Metadaten d\\u00fcrfen f\\u00fcr die kommerzielle und nicht kommerzielle Nutzung insbesondere:(1) vervielf\\u00e4ltigt, ausgedruckt, pr\\u00e4sentiert, ver\\u00e4ndert, bearbeitet sowie an Dritte \\u00fcbermittelt werden;\\n (2) mit eigenen Daten und Daten Anderer zusammengef\\u00fchrt und zu selbst\\u00e4ndigen neuen Datens\\u00e4tzen verbunden werden;\\n (3) in interne und externe Gesch\\u00e4ftsprozesse, Produkte und Anwendungen in \\u00f6ffentlichen und nicht \\u00f6ffentlichen elektronischen Netzwerken eingebunden werden.\\nBei der Nutzung ist sicherzustellen, dass folgende Angaben als Quellenvermerk enthalten sind:\\n(1) Bezeichnung des Bereitstellers nach dessen Ma\\u00dfgabe,\\n(2) der Vermerk Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0 oder dl-de/by-2-0 mit Verweis auf den Lizenztext unter www.govdata.de/dl-de/by-2-0 sowie\\n(3) einen Verweis auf den Datensatz (URI).Dies gilt nur soweit die datenhaltende Stelle die Angaben(1) bis (3) zum Quellenvermerk bereitstellt.\\nVer\\u00e4nderungen, Bearbeitungen, neue Gestaltungen oder sonstige Abwandlungen sind im Quellenvermerk mit dem Hinweis zu versehen, dass die Daten ge\\u00e4ndert wurden.", + "attribution": "\\u00a9 Statistische \\u00c4mter des Bundes und der L\\u00e4nder 2014" + } + ] + }, + { + "title": "Dokumentation - Zensus 2011 - Methoden und Verfahren", + "description": "Diese Publikation beschreibt ausf\\u00fchrlich die Methoden und Verfahren des registergest\\u00fctzten Zensus 2011; von der Datengewinnung und -aufbereitung bis hin zur Ergebniserstellung und Geheimhaltung. Der vorliegende Band wurde von den Statistischen \\u00c4mtern des Bundes und der L\\u00e4nder im Juni 2015 ver\\u00f6ffentlicht.", + "path": "https://www.destatis.de/DE/Publikationen/Thematisch/Bevoelkerung/Zensus/ZensusBuLaMethodenVerfahren5121105119004.pdf?__blob=publicationFile", + "licenses": [ + { + "name": "dl-by-de/2.0", + "title": "Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0", + "path": "www.govdata.de/dl-de/by-2-0", + "instruction": "Jede Nutzung ist unter den Bedingungen dieser \\u201eDatenlizenz Deutschland - Namensnennung - Version 2.0 zul\\u00e4ssig.\\nDie bereitgestellten Daten und Metadaten d\\u00fcrfen f\\u00fcr die kommerzielle und nicht kommerzielle Nutzung insbesondere:(1) vervielf\\u00e4ltigt, ausgedruckt, pr\\u00e4sentiert, ver\\u00e4ndert, bearbeitet sowie an Dritte \\u00fcbermittelt werden;\\n (2) mit eigenen Daten und Daten Anderer zusammengef\\u00fchrt und zu selbst\\u00e4ndigen neuen Datens\\u00e4tzen verbunden werden;\\n (3) in interne und externe Gesch\\u00e4ftsprozesse, Produkte und Anwendungen in \\u00f6ffentlichen und nicht \\u00f6ffentlichen elektronischen Netzwerken eingebunden werden.\\nBei der Nutzung ist sicherzustellen, dass folgende Angaben als Quellenvermerk enthalten sind:\\n(1) Bezeichnung des Bereitstellers nach dessen Ma\\u00dfgabe,\\n(2) der Vermerk Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0 oder dl-de/by-2-0 mit Verweis auf den Lizenztext unter www.govdata.de/dl-de/by-2-0 sowie\\n(3) einen Verweis auf den Datensatz (URI).Dies gilt nur soweit die datenhaltende Stelle die Angaben(1) bis (3) zum Quellenvermerk bereitstellt.\\nVer\\u00e4nderungen, Bearbeitungen, neue Gestaltungen oder sonstige Abwandlungen sind im Quellenvermerk mit dem Hinweis zu versehen, dass die Daten ge\\u00e4ndert wurden.", + "attribution": "\\u00a9 Statistisches Bundesamt, Wiesbaden 2015 (im Auftrag der Herausgebergemeinschaft)" + } + ] + }, + { + "title": "ERA5 global reanalysis", + "description": "ERA5 is the fifth generation ECMWF reanalysis for the global climate and weather for the past 4 to 7 decades. Currently data is available from 1950, split into Climate Data Store entries for 1950-1978 (preliminary back extension) and from 1979 onwards (final release plus timely updates, this page). ERA5 replaces the ERA-Interim reanalysis. See the online ERA5 documentation (https://confluence.ecmwf.int/display/CKB/ERA5%3A+data+documentation#ERA5:datadocumentation-Dataupdatefrequency) for more information.", + "path": "https://confluence.ecmwf.int/display/CKB/ERA5%3A+data+documentation#ERA5:datadocumentation-Dataupdatefrequency", + "licenses": [ + { + "name": "Licence to use Copernicus Products", + "title": "Licence to use Copernicus Products", + "path": "https://cds.climate.copernicus.eu/api/v2/terms/static/licence-to-use-copernicus-products.pdf", + "instruction": "This Licence is free of charge, worldwide, non-exclusive, royalty free and perpetual. Access to Copernicus Products is given for any purpose in so far as it is lawful, whereas use may include, but is not limited to: reproduction; distribution; communication to the public; adaptation, modification and combination with other data and information; or any combination of the foregoing", + "attribution": "\u00a9 Copernicus Climate Change Service (C3S) Climate Data Store" + } + ] + }, + { + "title": "Peta5 0 1 HD ser", + "description": "modelled Heat Demand distribution (in GJ per hectare grid cell) for service-sector heat demands for space heating, hot water and process heat for the year 2015 using HRE4 data and the combined top-down bottom-up approach of HRE4", + "path": "https://s-eenergies-open-data-euf.hub.arcgis.com/search", + "licenses": [ + { + "name": "Creative Commons Attribution 4.0 International", + "title": "CC BY 4.0", + "path": "https://creativecommons.org/licenses/by/4.0/", + "instruction": "You are free: To Share, To Adapt; As long as you: Attribute!", + "attribution": "\\u00a9 Flensburg, Halmstad and Aalborg universities" + } + ] + } + ], + "licenses": [ + { + "name": "ODbL-1.0", + "title": "Open Data Commons Open Database License 1.0", + "path": "https://opendatacommons.org/licenses/odbl/1.0/", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute, Share-Alike, Keep open!", + "attribution": "\u00a9 Reiner Lemoine Institut" + } + ], + "contributors": [ + { + "title": "nailend", + "email": "https://github.com/nailend", + "date": "2023-03-13", + "object": "metadata", + "comment": "Create metadata" + } + ], + "resources": [ + { + "profile": "tabular-data-resource", + "name": "demand.egon_etrago_heat_cts", + "path": "", + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": [ + { + "name": "bus_id", + "description": "MV bus_id (reference: demand.egon_cts_heat_demand_building_share.bus_id, grid.egon_hvmv_substation.bus_id, grid.egon_mv_grid_district.bus_id)", + "type": "integer", + "unit": "none" + }, + { + "name": "scn_name", + "description": "Scenario name (reference: demand.egon_cts_heat_demand_building_share.scenario)", + "type": "character varying", + "unit": "none" + }, + { + "name": "p_set", + "description": "Cts heat demand load timeseries for one year (8760 hours) in MWh", + "type": "array of real", + "unit": "MWh" + } + ], + "primaryKey": "id, scn_name" + }, + "dialect": { + "delimiter": "", + "decimalSeparator": "." + } + } + ], + "review": { + "path": "", + "badge": "" + }, + "metaMetadata": { + "metadataVersion": "OEP-1.4.1", + "metadataLicense": { + "name": "CC0-1.0", + "title": "Creative Commons Zero v1.0 Universal", + "path": "https://creativecommons.org/publicdomain/zero/1.0/" + } + }, + "_comment": { + "metadata": "Metadata documentation and explanation (https://github.com/OpenEnergyPlatform/oemetadata/blob/master/metadata/v141/metadata_key_description.md)", + "dates": "Dates and time must follow the ISO8601 including time zone (YYYY-MM-DD or YYYY-MM-DDThh:mm:ss\u00b1hh)", + "units": "Use a space between numbers and units (100 m)", + "languages": "Languages must follow the IETF (BCP47) format (en-GB, en-US, de-DE)", + "licenses": "License name must follow the SPDX License List (https://spdx.org/licenses/)", + "review": "Following the OEP Data Review (https://github.com/OpenEnergyPlatform/data-preprocessing/wiki)", + "none": "If not applicable use (none)" + } +} diff --git a/src/egon/data/metadata/demand.egon_heat_idp_pool.json b/src/egon/data/metadata/demand.egon_heat_idp_pool.json new file mode 100644 index 000000000..5f597c7f8 --- /dev/null +++ b/src/egon/data/metadata/demand.egon_heat_idp_pool.json @@ -0,0 +1,152 @@ +{ + "name": "demand.egon_heat_idp_pool", + "title": "eGon pool of residential intra-day heat demand profiles", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": "Pool of individual, normalized intra-day heat demand profiles of residential buildings. Timeseries only represent one day but are used to be merged for a whole year ", + "language": "undefined", + "keywords": [], + "publicationDate": "2022-09-14", + "context": { + "homepage": "https://ego-n.org/", + "documentation": "https://egon-data.readthedocs.io/en/latest/", + "sourceCode": "https://github.com/openego/eGon-data", + "contact": "https://ego-n.org/partners/", + "grantNo": "03EI1002", + "fundingAgency": "Bundesministerium für Wirtschaft und Klimaschutz", + "fundingAgencyLogo": "https://www.innovation-beratung-foerderung.de/INNO/Redaktion/DE/Bilder/Titelbilder/titel_foerderlogo_bmwi.jpg?__blob=normal&v=3", + "publisherLogo": "https://ego-n.org/images/eGon_logo_noborder_transbg.svg" + }, + "spatial": { + "location": "none", + "extent": "Germany", + "resolution": "" + }, + "temporal": { + "referenceDate": "2022-09-14", + "timeseries": { + "start": "2011-01-01 00:00", + "end": "2011-12-31 23:00", + "resolution": "1 h", + "alignment": "left", + "aggregationType": "sum" + } + }, + "sources": [ + { + "title": "eGon-data", + "description": "Workflow to download, process and generate data sets suitable for the further research conducted in the project eGon (https://ego-n.org/)", + "path": "https://github.com/openego/eGon-data", + "licenses": [ + { + "name": "AGPL-3.0 License", + "title": "GNU Affero General Public License v3.0", + "path": "https://www.gnu.org/licenses/agpl-3.0.de.html", + "instruction": "Permissions of this strongest copyleft license are conditioned on making available complete source code of licensed works and modifications, which include larger works using a licensed work, under the same license. Copyright and license notices must be preserved. Contributors provide an express grant of patent rights. When a modified version is used to provide a service over a network, the complete source code of the modified version must be made available.", + "attribution": "© eGon-data developers" + } + ] + }, + { + "title": "Data bundle for egon-data: A transparent and reproducible data processing pipeline for energy system modeling", + "description": "egon-data provides a transparent and reproducible open data based data processing pipeline for generating data models suitable for energy system modeling. The data is customized for the requirements of the research project eGon. The research project aims to develop tools for an open and cross-sectoral planning of transmission and distribution grids. For further information please visit the eGon project website or its Github repository. egon-data retrieves and processes data from several different external input sources. As not all data dependencies can be downloaded automatically from external sources we provide a data bundle to be downloaded by egon-data.", + "path": "https://doi.org/10.5281/zenodo.6630616", + "licenses": [ + { + "name": "CC-BY-4.0", + "title": "Creative Commons Attribution 4.0 International", + "path": "https://creativecommons.org/licenses/by/4.0/legalcode", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute.", + "attribution": "© eGon-data developers, © Leibniz Institute for Applied Geophysics, © Fraunhofer IEE, © BGR Hannover, © Danielle Schmidt, © Übertragungsnetzbetreiber, © DIW Berlin, © Reiner Lemoine Institut, © Statistisches Bundesamt, Wiesbaden 2008, © Statistische Ämter des Bundes und der Länder 2021" + } + ] + } + ], + "licenses": [ + { + "name": "CC-BY-4.0", + "title": "Creative Commons Attribution 4.0 International", + "path": "https://creativecommons.org/licenses/by/4.0/legalcode", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute.", + "attribution": "Europa-Universität Flensburg" + } + ], + "contributors": [ + { + "title": "ClaraBuettner", + "email": "https://github.com/ClaraBuettner", + "date": "2022-09-12", + "object": "metadata", + "comment": "Create metadata" + }, + { + "title": "nesnoj", + "email": "https://github.com/nesnoj", + "date": "2022-09-14", + "object": "metadata", + "comment": "Update metadata" + } + ], + "resources": [ + { + "profile": "tabular-data-resource", + "name": "demand.egon_heat_idp_pool", + "path": "", + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": [ + { + "name": "house", + "description": "Type of residential house, either SFH (single family house) or MFH (multi family house)", + "type": "str", + "unit": "none" + }, + { + "name": "idp", + "description": "Normalized intra-day-profile", + "type": "array of double precision", + "unit": "per unit" + }, + { + "name": "index", + "description": "Unique identifier", + "type": "integer", + "unit": "none" + }, + { + "name": "temperature_class", + "description": "Temperature class (considering the mean outside temeperature during the day)", + "type": "integer", + "unit": "none" + } + ], + "primaryKey": "index" + }, + "dialect": { + "delimiter": "", + "decimalSeparator": "" + } + } + ], + "review": { + "path": "", + "badge": "" + }, + "metaMetadata": { + "metadataVersion": "OEP-1.4.1", + "metadataLicense": { + "name": "CC0-1.0", + "title": "Creative Commons Zero v1.0 Universal", + "path": "https://creativecommons.org/publicdomain/zero/1.0/" + } + }, + "_comment": { + "metadata": "Metadata documentation and explanation (https://github.com/OpenEnergyPlatform/oemetadata/blob/master/metadata/v141/metadata_key_description.md)", + "dates": "Dates and time must follow the ISO8601 including time zone (YYYY-MM-DD or YYYY-MM-DDThh:mm:ss±hh)", + "units": "Use a space between numbers and units (100 m)", + "languages": "Languages must follow the IETF (BCP47) format (en-GB, en-US, de-DE)", + "licenses": "License name must follow the SPDX License List (https://spdx.org/licenses/)", + "review": "Following the OEP Data Review (https://github.com/OpenEnergyPlatform/data-preprocessing/wiki)", + "none": "If not applicable use (none)" + } +} diff --git a/src/egon/data/metadata/demand.egon_heat_timeseries_selected_profiles.json b/src/egon/data/metadata/demand.egon_heat_timeseries_selected_profiles.json new file mode 100644 index 000000000..ff1b05742 --- /dev/null +++ b/src/egon/data/metadata/demand.egon_heat_timeseries_selected_profiles.json @@ -0,0 +1,157 @@ +{ + "name": "demand.egon_heat_timeseries_selected_profiles", + "title": "eGon selected intra-day profiles per building", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": "List of selected intra-day-profiles for each residential building in Germany", + "language": [ + "en-EN", + "de-DE" + ], + "keywords": [], + "publicationDate": "2022-09-14", + "context": { + "homepage": "https://ego-n.org/", + "documentation": "https://egon-data.readthedocs.io/en/latest/", + "sourceCode": "https://github.com/openego/eGon-data", + "contact": "https://ego-n.org/partners/", + "grantNo": "03EI1002", + "fundingAgency": "Bundesministerium f\u00fcr Wirtschaft und Klimaschutz", + "fundingAgencyLogo": "https://www.innovation-beratung-foerderung.de/INNO/Redaktion/DE/Bilder/Titelbilder/titel_foerderlogo_bmwi.jpg?__blob=normal&v=3", + "publisherLogo": "https://ego-n.org/images/eGon_logo_noborder_transbg.svg" + }, + "spatial": { + "location": "none", + "extent": "Germany", + "resolution": "buildings" + }, + "temporal": { + "referenceDate": "2022-09-14", + "timeseries": {} + }, + "sources": [ + { + "title": "Data bundle for egon-data: A transparent and reproducible data processing pipeline for energy system modeling", + "description": "egon-data provides a transparent and reproducible open data based data processing pipeline for generating data models suitable for energy system modeling. The data is customized for the requirements of the research project eGon. The research project aims to develop tools for an open and cross-sectoral planning of transmission and distribution grids. For further information please visit the eGon project website or its Github repository. egon-data retrieves and processes data from several different external input sources. As not all data dependencies can be downloaded automatically from external sources we provide a data bundle to be downloaded by egon-data.", + "path": "https://doi.org/10.5281/zenodo.6630616", + "licenses": [ + { + "name": "CC-BY-4.0", + "title": "Creative Commons Attribution 4.0 International", + "path": "https://creativecommons.org/licenses/by/4.0/legalcode", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute.", + "attribution": "\u00a9 eGon-data developers, \u00a9 Leibniz Institute for Applied Geophysics, \u00a9 Fraunhofer IEE, \u00a9 BGR Hannover, \u00a9 Danielle Schmidt, \u00a9 \u00dcbertragungsnetzbetreiber, \u00a9 DIW Berlin, \u00a9 Reiner Lemoine Institut, \u00a9 Statistisches Bundesamt, Wiesbaden 2008, \u00a9 Statistische \u00c4mter des Bundes und der L\u00e4nder 2021" + } + ] + }, + { + "title": "eGon-data", + "description": "Workflow to download, process and generate data sets suitable for the further research conducted in the project eGon (https://ego-n.org/)", + "path": "https://github.com/openego/eGon-data", + "licenses": [ + { + "name": "AGPL-3.0 License", + "title": "GNU Affero General Public License v3.0", + "path": "https://www.gnu.org/licenses/agpl-3.0.de.html", + "instruction": "Permissions of this strongest copyleft license are conditioned on making available complete source code of licensed works and modifications, which include larger works using a licensed work, under the same license. Copyright and license notices must be preserved. Contributors provide an express grant of patent rights. When a modified version is used to provide a service over a network, the complete source code of the modified version must be made available.", + "attribution": "\u00a9 eGon-data developers" + } + ] + }, + { + "title": "OpenStreetMap Data Extracts (Geofabrik)", + "description": "Full data extract of OpenStreetMap data for defined spatial extent at ''referenceDate''", + "path": "https://download.geofabrik.de/europe/germany-210101.osm.pbf", + "licenses": [ + { + "name": "ODbL-1.0", + "title": "Open Data Commons Open Database License 1.0", + "path": "https://opendatacommons.org/licenses/odbl/1.0/", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute, Share-Alike, Keep open!", + "attribution": "\u00a9 OpenStreetMap contributors" + } + ] + } + ], + "licenses": [ + { + "name": "ODbL-1.0", + "title": "Open Data Commons Open Database License 1.0", + "path": "https://opendatacommons.org/licenses/odbl/1.0/", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute, Share-Alike, Keep open!", + "attribution": "\u00a9 Europa-Universit\u00e4t Flensburg" + } + ], + "contributors": [ + { + "title": "ClaraBuettner", + "email": "https://github.com/ClaraBuettner", + "date": "2022-09-12", + "object": "metadata", + "comment": "Create metadata" + }, + { + "title": "nesnoj", + "email": "https://github.com/nesnoj", + "date": "2022-09-14", + "object": "metadata", + "comment": "Update metadata" + } + ], + "resources": [ + { + "profile": "tabular-data-resource", + "name": "demand.egon_heat_timeseries_selected_profiles", + "path": "", + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": [ + { + "name": "building_id", + "description": "Index of the corresponding building", + "type": "integer", + "unit": "none" + }, + { + "name": "selected_idp_profiles", + "description": "List of selected profiles for each day in the year", + "type": "array of integer", + "unit": "none" + }, + { + "name": "zensus_population_id", + "description": "Index of corresponding census cell", + "type": "integer", + "unit": "none" + } + ], + "primaryKey": "building_id" + }, + "dialect": { + "delimiter": "", + "decimalSeparator": "" + } + } + ], + "review": { + "path": "", + "badge": "" + }, + "metaMetadata": { + "metadataVersion": "OEP-1.4.1", + "metadataLicense": { + "name": "CC0-1.0", + "title": "Creative Commons Zero v1.0 Universal", + "path": "https://creativecommons.org/publicdomain/zero/1.0/" + } + }, + "_comment": { + "metadata": "Metadata documentation and explanation (https://github.com/OpenEnergyPlatform/oemetadata/blob/master/metadata/v141/metadata_key_description.md)", + "dates": "Dates and time must follow the ISO8601 including time zone (YYYY-MM-DD or YYYY-MM-DDThh:mm:ss\u00b1hh)", + "units": "Use a space between numbers and units (100 m)", + "languages": "Languages must follow the IETF (BCP47) format (en-GB, en-US, de-DE)", + "licenses": "License name must follow the SPDX License List (https://spdx.org/licenses/)", + "review": "Following the OEP Data Review (https://github.com/OpenEnergyPlatform/data-preprocessing/wiki)", + "none": "If not applicable use (none)" + } +} diff --git a/src/egon/data/metadata/demand.egon_household_electricity_profile_in_census_cell.json b/src/egon/data/metadata/demand.egon_household_electricity_profile_in_census_cell.json new file mode 100644 index 000000000..3e0696b84 --- /dev/null +++ b/src/egon/data/metadata/demand.egon_household_electricity_profile_in_census_cell.json @@ -0,0 +1,218 @@ +{ + "name": "demand.egon_household_electricity_profile_in_census_cell", + "title": "eGon household electricity profiles in census cells", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": "Mapping table for residential electricity profiles to census cell including scaling factors for two scenarios (eGon2035, eGon100RE).", + "language": [ + "en-EN", + "de-DE" + ], + "keywords": [ + "" + ], + "publicationDate": "2022-09-14", + "context": { + "homepage": "https://ego-n.org/", + "documentation": "https://egon-data.readthedocs.io/en/latest/", + "sourceCode": "https://github.com/openego/eGon-data", + "contact": "https://ego-n.org/partners/", + "grantNo": "03EI1002", + "fundingAgency": "Bundesministerium f\u00fcr Wirtschaft und Klimaschutz", + "fundingAgencyLogo": "https://commons.wikimedia.org/wiki/File:BMWi_Logo_2021.svg#/media/File:BMWi_Logo_2021.svg", + "publisherLogo": "https://ego-n.org/images/eGon_logo_noborder_transbg.svg" + }, + "spatial": { + "location": "", + "extent": "Germany", + "resolution": "100 m x 100 m" + }, + "temporal": { + "referenceDate": "2022-09-14", + "timeseries": {} + }, + "sources": [ + { + "title": "Data bundle for egon-data: A transparent and reproducible data processing pipeline for energy system modeling", + "description": "egon-data provides a transparent and reproducible open data based data processing pipeline for generating data models suitable for energy system modeling. The data is customized for the requirements of the research project eGon. The research project aims to develop tools for an open and cross-sectoral planning of transmission and distribution grids. For further information please visit the eGon project website or its Github repository. egon-data retrieves and processes data from several different external input sources. As not all data dependencies can be downloaded automatically from external sources we provide a data bundle to be downloaded by egon-data.", + "path": "https://doi.org/10.5281/zenodo.6630616", + "licenses": [ + { + "name": "CC-BY-4.0", + "title": "Creative Commons Attribution 4.0 International", + "path": "https://creativecommons.org/licenses/by/4.0/legalcode", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute.", + "attribution": "\u00a9 eGon-data developers, \u00a9 Leibniz Institute for Applied Geophysics, \u00a9 Fraunhofer IEE, \u00a9 BGR Hannover, \u00a9 Danielle Schmidt, \u00a9 \u00dcbertragungsnetzbetreiber, \u00a9 DIW Berlin, \u00a9 Reiner Lemoine Institut, \u00a9 Statistisches Bundesamt, Wiesbaden 2008, \u00a9 Statistische \u00c4mter des Bundes und der L\u00e4nder 2021" + } + ] + }, + { + "title": "eGon-data", + "description": "Workflow to download, process and generate data sets suitable for the further research conducted in the project eGon (https://ego-n.org/)", + "path": "https://github.com/openego/eGon-data", + "licenses": [ + { + "name": "AGPL-3.0 License", + "title": "GNU Affero General Public License v3.0", + "path": "https://www.gnu.org/licenses/agpl-3.0.de.html", + "instruction": "Permissions of this strongest copyleft license are conditioned on making available complete source code of licensed works and modifications, which include larger works using a licensed work, under the same license. Copyright and license notices must be preserved. Contributors provide an express grant of patent rights. When a modified version is used to provide a service over a network, the complete source code of the modified version must be made available.", + "attribution": "\u00a9 eGon-data developers" + } + ] + }, + { + "title": "Verwaltungsgebiete 1:250 000 (Ebenen)", + "description": "Der Datenbestand umfasst s\u00e4mtliche Verwaltungseinheiten der hierarchischen Verwaltungsebenen vom Staat bis zu den Gemeinden mit ihren Grenzen, statistischen Schl\u00fcsselzahlen, Namen der Verwaltungseinheit sowie die spezifische Bezeichnung der Verwaltungsebene des jeweiligen Landes.", + "path": "https://daten.gdz.bkg.bund.de/produkte/vg/vg250_ebenen_0101/2020/vg250_01-01.geo84.shape.ebenen.zip", + "licenses": [ + { + "name": "dl-by-de/2.0", + "title": "Datenlizenz Deutschland \u2013 Namensnennung \u2013 Version 2.0", + "path": "www.govdata.de/dl-de/by-2-0", + "instruction": "Jede Nutzung ist unter den Bedingungen dieser \u201eDatenlizenz Deutschland - Namensnennung - Version 2.0 zul\u00e4ssig.Die bereitgestellten Daten und Metadaten d\u00fcrfen f\u00fcr die kommerzielle und nicht kommerzielle Nutzung insbesondere:(1) vervielf\u00e4ltigt, ausgedruckt, pr\u00e4sentiert, ver\u00e4ndert, bearbeitet sowie an Dritte \u00fcbermittelt werden; (2) mit eigenen Daten und Daten Anderer zusammengef\u00fchrt und zu selbst\u00e4ndigen neuen Datens\u00e4tzen verbunden werden; (3) in interne und externe Gesch\u00e4ftsprozesse, Produkte und Anwendungen in \u00f6ffentlichen und nicht \u00f6ffentlichen elektronischen Netzwerken eingebunden werden.Bei der Nutzung ist sicherzustellen, dass folgende Angaben als Quellenvermerk enthalten sind:(1) Bezeichnung des Bereitstellers nach dessen Ma\u00dfgabe,(2) der Vermerk Datenlizenz Deutschland \u2013 Namensnennung \u2013 Version 2.0 oder dl-de/by-2-0 mit Verweis auf den Lizenztext unter www.govdata.de/dl-de/by-2-0 sowie(3) einen Verweis auf den Datensatz (URI).Dies gilt nur soweit die datenhaltende Stelle die Angaben(1) bis (3) zum Quellenvermerk bereitstellt.Ver\u00e4nderungen, Bearbeitungen, neue Gestaltungen oder sonstige Abwandlungen sind im Quellenvermerk mit dem Hinweis zu versehen, dass die Daten ge\u00e4ndert wurden.", + "attribution": "\u00a9 Bundesamt f\u00fcr Kartographie und Geod\u00e4sie 2020 (Daten ver\u00e4ndert)" + } + ] + }, + { + "title": "OpenStreetMap Data Extracts (Geofabrik)", + "description": "Full data extract of OpenStreetMap data for defined spatial extent at 2022-01-01", + "path": "https://download.geofabrik.de/europe/germany-210101.osm.pbf", + "licenses": [ + { + "name": "ODbL-1.0", + "title": "Open Data Commons Open Database License 1.0", + "path": "https://opendatacommons.org/licenses/odbl/1.0/", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute, Share-Alike, Keep open!", + "attribution": "\u00a9 OpenStreetMap contributors" + } + ] + }, + { + "title": "DemandRegio", + "description": "Harmonisierung und Entwicklung von Verfahren zur regionalen und zeitlichen Aufl\u00f6sung von Energienachfragen", + "path": "https://doi.org/10.34805/ffe-119-20", + "license": [ + { + "name": "Creative Commons Attribution 4.0 International", + "title": "CC BY 4.0", + "path": "https://creativecommons.org/licenses/by/4.0/", + "instruction": "You are free: To Share, To Adapt; As long as you: Attribute!", + "attribution": "\u00a9 FZJ, TUB, FfE" + } + ] + } + ], + "licenses": [ + { + "name": "ODbL-1.0", + "title": "Open Data Commons Open Database License 1.0", + "path": "https://opendatacommons.org/licenses/odbl/1.0/", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute, Share-Alike, Keep open!", + "attribution": "\u00a9 Reiner Lemoine Institut" + } + ], + "contributors": [ + { + "title": "ClaraBuettner", + "email": "https://github.com/ClaraBuettner", + "date": "2022-09-12", + "object": "metadata", + "comment": "Create metadata" + }, + { + "title": "nailend", + "email": "https://github.com/nailend", + "date": "2022-09-12", + "object": "metadata", + "comment": "Updated metadata" + }, + { + "title": "nesnoj", + "email": "https://github.com/nesnoj", + "date": "2022-09-14", + "object": "metadata", + "comment": "Update metadata" + } + ], + "resources": [ + { + "profile": "tabular-data-resource", + "name": "demand.egon_household_electricity_profile_in_census_cell", + "path": "", + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": [ + { + "name": "cell_id", + "description": "Census cell id (reference to: society.destatis_zensus_population_per_ha_inside_germany.id)", + "type": "integer", + "unit": "none" + }, + { + "name": "grid_id", + "description": "Descriptive ID of census cell given by Statistisches Bundesamt (Destatis)", + "type": "character varying", + "unit": "none" + }, + { + "name": "cell_profile_ids", + "description": "Household profile ids", + "type": "array of integer", + "unit": "none" + }, + { + "name": "nuts3", + "description": "NUTS-3 ID of the census cell", + "type": "character varying", + "unit": "none" + }, + { + "name": "nuts1", + "description": "NUTS-1 ID of the census cell", + "type": "character varying", + "unit": "none" + }, + { + "name": "factor_2035", + "description": "Scaling factor for all profiles in the respective census cell for scenario eGon2035", + "type": "double precision", + "unit": "none" + }, + { + "name": "factor_2050", + "description": "Scaling factor for all profiles in the respective census cell for scenario eGon100RE", + "type": "double precision", + "unit": "none" + } + ], + "primaryKey": "cell_id" + }, + "dialect": { + "delimiter": "", + "decimalSeparator": "." + } + } + ], + "review": { + "path": "", + "badge": "" + }, + "metaMetadata": { + "metadataVersion": "OEP-1.4.1", + "metadataLicense": { + "name": "CC0-1.0", + "title": "Creative Commons Zero v1.0 Universal", + "path": "https://creativecommons.org/publicdomain/zero/1.0/" + } + }, + "_comment": { + "metadata": "Metadata documentation and explanation (https://github.com/OpenEnergyPlatform/oemetadata/blob/master/metadata/v141/metadata_key_description.md)", + "dates": "Dates and time must follow the ISO8601 including time zone (YYYY-MM-DD or YYYY-MM-DDThh:mm:ss\u00b1hh)", + "units": "Use a space between numbers and units (100 m)", + "languages": "Languages must follow the IETF (BCP47) format (en-GB, en-US, de-DE)", + "licenses": "License name must follow the SPDX License List (https://spdx.org/licenses/)", + "review": "Following the OEP Data Review (https://github.com/OpenEnergyPlatform/data-preprocessing/wiki)", + "none": "If not applicable use (none)" + } +} diff --git a/src/egon/data/metadata/demand.egon_household_electricity_profile_of_buildings.json b/src/egon/data/metadata/demand.egon_household_electricity_profile_of_buildings.json new file mode 100644 index 000000000..f302d0a27 --- /dev/null +++ b/src/egon/data/metadata/demand.egon_household_electricity_profile_of_buildings.json @@ -0,0 +1,170 @@ +{ + "name": "demand.egon_household_electricity_profile_of_buildings", + "title": "eGon household electricity profiles of buildings", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": "Mapping table for residential electricity profiles to buildings via project internal building_id and corresponding census cell_id. ", + "language": [ + "en-EN", + "de-DE" + ], + "keywords": [], + "publicationDate": "2022-09-14", + "context": { + "homepage": "https://ego-n.org/", + "documentation": "https://egon-data.readthedocs.io/en/latest/", + "sourceCode": "https://github.com/openego/eGon-data", + "contact": "https://ego-n.org/partners/", + "grantNo": "03EI1002", + "fundingAgency": "Bundesministerium f\u00fcr Wirtschaft und Klimaschutz", + "fundingAgencyLogo": "https://commons.wikimedia.org/wiki/File:BMWi_Logo_2021.svg#/media/File:BMWi_Logo_2021.svg", + "publisherLogo": "https://ego-n.org/images/eGon_logo_noborder_transbg.svg" + }, + "spatial": { + "location": "", + "extent": "Germany", + "resolution": "Buildings" + }, + "temporal": { + "referenceDate": "2022-09-14", + "timeseries": {} + }, + "sources": [ + { + "title": "Data bundle for egon-data: A transparent and reproducible data processing pipeline for energy system modeling", + "description": "egon-data provides a transparent and reproducible open data based data processing pipeline for generating data models suitable for energy system modeling. The data is customized for the requirements of the research project eGon. The research project aims to develop tools for an open and cross-sectoral planning of transmission and distribution grids. For further information please visit the eGon project website or its Github repository. egon-data retrieves and processes data from several different external input sources. As not all data dependencies can be downloaded automatically from external sources we provide a data bundle to be downloaded by egon-data.", + "path": "https://doi.org/10.5281/zenodo.6630616", + "licenses": [ + { + "name": "CC-BY-4.0", + "title": "Creative Commons Attribution 4.0 International", + "path": "https://creativecommons.org/licenses/by/4.0/legalcode", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute.", + "attribution": "\u00a9 eGon-data developers, \u00a9 Leibniz Institute for Applied Geophysics, \u00a9 Fraunhofer IEE, \u00a9 BGR Hannover, \u00a9 Danielle Schmidt, \u00a9 \u00dcbertragungsnetzbetreiber, \u00a9 DIW Berlin, \u00a9 Reiner Lemoine Institut, \u00a9 Statistisches Bundesamt, Wiesbaden 2008, \u00a9 Statistische \u00c4mter des Bundes und der L\u00e4nder 2021" + } + ] + }, + { + "title": "eGon-data", + "description": "Workflow to download, process and generate data sets suitable for the further research conducted in the project eGon (https://ego-n.org/)", + "path": "https://github.com/openego/eGon-data", + "licenses": [ + { + "name": "AGPL-3.0 License", + "title": "GNU Affero General Public License v3.0", + "path": "https://www.gnu.org/licenses/agpl-3.0.de.html", + "instruction": "Permissions of this strongest copyleft license are conditioned on making available complete source code of licensed works and modifications, which include larger works using a licensed work, under the same license. Copyright and license notices must be preserved. Contributors provide an express grant of patent rights. When a modified version is used to provide a service over a network, the complete source code of the modified version must be made available.", + "attribution": "\u00a9 eGon-data developers" + } + ] + }, + { + "title": "OpenStreetMap Data Extracts (Geofabrik)", + "description": "Full data extract of OpenStreetMap data for defined spatial extent at 2022-01-01", + "path": "https://download.geofabrik.de/europe/germany-210101.osm.pbf", + "licenses": [ + { + "name": "ODbL-1.0", + "title": "Open Data Commons Open Database License 1.0", + "path": "https://opendatacommons.org/licenses/odbl/1.0/", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute, Share-Alike, Keep open!", + "attribution": "\u00a9 OpenStreetMap contributors" + } + ] + } + ], + "licenses": [ + { + "name": "ODbL-1.0", + "title": "Open Data Commons Open Database License 1.0", + "path": "https://opendatacommons.org/licenses/odbl/1.0/", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute, Share-Alike, Keep open!", + "attribution": "\u00a9 Reiner Lemoine Institut" + } + ], + "contributors": [ + { + "title": "ClaraBuettner", + "email": "https://github.com/ClaraBuettner", + "date": "2022-09-12", + "object": "metadata", + "comment": "Create metadata" + }, + { + "title": "nailend", + "email": "https://github.com/nailend", + "date": "2022-09-12", + "object": "metadata", + "comment": "Updated metadata" + }, + { + "title": "nesnoj", + "email": "https://github.com/nesnoj", + "date": "2022-09-14", + "object": "metadata", + "comment": "Update metadata" + } + ], + "resources": [ + { + "profile": "tabular-data-resource", + "name": "demand.egon_household_electricity_profile_of_buildings", + "path": "", + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": [ + { + "name": "id", + "description": "Unique identifier", + "type": "integer", + "unit": "none" + }, + { + "name": "building_id", + "description": "Building id (reference to: openstreetmap.osm_buildings_residential.id, openstreetmap.osm_buildings_synthetic.id)", + "type": "integer", + "unit": "none" + }, + { + "name": "cell_id", + "description": "Census cell id (reference to: society.destatis_zensus_population_per_ha_inside_germany.id)", + "type": "integer", + "unit": "none" + }, + { + "name": "profile_id", + "description": "ID of iee household profiles (reference to: iee_household_load_profiles.type)", + "type": "character varying", + "unit": "none" + } + ], + "primaryKey": "id" + }, + "dialect": { + "delimiter": "", + "decimalSeparator": "." + } + } + ], + "review": { + "path": "", + "badge": "" + }, + "metaMetadata": { + "metadataVersion": "OEP-1.4.1", + "metadataLicense": { + "name": "CC0-1.0", + "title": "Creative Commons Zero v1.0 Universal", + "path": "https://creativecommons.org/publicdomain/zero/1.0/" + } + }, + "_comment": { + "metadata": "Metadata documentation and explanation (https://github.com/OpenEnergyPlatform/oemetadata/blob/master/metadata/v141/metadata_key_description.md)", + "dates": "Dates and time must follow the ISO8601 including time zone (YYYY-MM-DD or YYYY-MM-DDThh:mm:ss\u00b1hh)", + "units": "Use a space between numbers and units (100 m)", + "languages": "Languages must follow the IETF (BCP47) format (en-GB, en-US, de-DE)", + "licenses": "License name must follow the SPDX License List (https://spdx.org/licenses/)", + "review": "Following the OEP Data Review (https://github.com/OpenEnergyPlatform/data-preprocessing/wiki)", + "none": "If not applicable use (none)" + } +} diff --git a/src/egon/data/metadata/demand.iee_household_load_profiles.json b/src/egon/data/metadata/demand.iee_household_load_profiles.json new file mode 100644 index 000000000..093fc47dd --- /dev/null +++ b/src/egon/data/metadata/demand.iee_household_load_profiles.json @@ -0,0 +1,158 @@ +{ + "name": "demand.iee_household_load_profiles", + "title": "eGon pool of electricity household load profiles", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": "100.000 annual profiles in hourly resolution of electricity demand of private households for different household types (singles, couples, other) with varying number of elderly and children. The profiles were created using a bottom-up load profile generator by Fraunhofer IEE developed in the Bachelors thesis \"Auswirkungen verschiedener Haushaltslastprofile auf PV-Batterie-Systeme\" by Jonas Haack, Fachhochschule Flensburg, December 2012. The columns are named as follows: \"a\", e.g. P2a0000 is the first profile of a couples household with 2 children. See publication below for the list of prefixes. Values are given in Wh.", + "language": [ + "en-EN", + "de-DE" + ], + "keywords": [ + "" + ], + "publicationDate": "2022-09-14", + "context": { + "homepage": "https://ego-n.org/", + "documentation": "https://egon-data.readthedocs.io/en/latest/", + "sourceCode": "https://github.com/openego/eGon-data", + "contact": "https://ego-n.org/partners/", + "grantNo": "03EI1002", + "fundingAgency": "Bundesministerium f\u00fcr Wirtschaft und Klimaschutz", + "fundingAgencyLogo": "https://commons.wikimedia.org/wiki/File:BMWi_Logo_2021.svg#/media/File:BMWi_Logo_2021.svg", + "publisherLogo": "https://ego-n.org/images/eGon_logo_noborder_transbg.svg" + }, + "spatial": { + "location": "", + "extent": "Germany", + "resolution": "" + }, + "temporal": { + "referenceDate": "2022-06-10", + "timeseries": { + "start": "2016-01-01", + "end": "2016-12-31", + "resolution": "1 h", + "alignment": "left", + "aggregationType": "sum" + } + }, + "sources": [ + { + "title": "Data bundle for egon-data: A transparent and reproducible data processing pipeline for energy system modeling", + "description": "egon-data provides a transparent and reproducible open data based data processing pipeline for generating data models suitable for energy system modeling. The data is customized for the requirements of the research project eGon. The research project aims to develop tools for an open and cross-sectoral planning of transmission and distribution grids. For further information please visit the eGon project website or its Github repository. egon-data retrieves and processes data from several different external input sources. As not all data dependencies can be downloaded automatically from external sources we provide a data bundle to be downloaded by egon-data.", + "path": "https://doi.org/10.5281/zenodo.6630616", + "licenses": [ + { + "name": "CC-BY-4.0", + "title": "Creative Commons Attribution 4.0 International", + "path": "https://creativecommons.org/licenses/by/4.0/legalcode", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute.", + "attribution": "\u00a9 eGon-data developers, \u00a9 Leibniz Institute for Applied Geophysics, \u00a9 Fraunhofer IEE, \u00a9 BGR Hannover, \u00a9 Danielle Schmidt, \u00a9 \u00dcbertragungsnetzbetreiber, \u00a9 DIW Berlin, \u00a9 Reiner Lemoine Institut, \u00a9 Statistisches Bundesamt, Wiesbaden 2008, \u00a9 Statistische \u00c4mter des Bundes und der L\u00e4nder 2021" + } + ] + }, + { + "title": "eGon-data", + "description": "Workflow to download, process and generate data sets suitable for the further research conducted in the project eGon (https://ego-n.org/)", + "path": "https://github.com/openego/eGon-data", + "licenses": [ + { + "name": "AGPL-3.0 License", + "title": "GNU Affero General Public License v3.0", + "path": "https://www.gnu.org/licenses/agpl-3.0.de.html", + "instruction": "Permissions of this strongest copyleft license are conditioned on making available complete source code of licensed works and modifications, which include larger works using a licensed work, under the same license. Copyright and license notices must be preserved. Contributors provide an express grant of patent rights. When a modified version is used to provide a service over a network, the complete source code of the modified version must be made available.", + "attribution": "\u00a9 eGon-data developers" + } + ] + } + ], + "licenses": [ + { + "name": "ODbL-1.0", + "title": "Open Data Commons Open Database License 1.0", + "path": "https://opendatacommons.org/licenses/odbl/1.0/", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute, Share-Alike, Keep open!", + "attribution": "\u00a9 Reiner Lemoine Institut" + } + ], + "contributors": [ + { + "title": "ClaraBuettner", + "email": "https://github.com/ClaraBuettner", + "date": "2022-09-12", + "object": "metadata", + "comment": "Create metadata" + }, + { + "title": "nailend", + "email": "https://github.com/nailend", + "date": "2022-09-12", + "object": "metadata", + "comment": "Updated metadata" + }, + { + "title": "nesnoj", + "email": "https://github.com/nesnoj", + "date": "2022-09-14", + "object": "metadata", + "comment": "Update metadata" + } + ], + "resources": [ + { + "profile": "tabular-data-resource", + "name": "demand.iee_household_load_profiles", + "path": "", + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": [ + { + "name": "id", + "description": "Unique identifier", + "type": "integer", + "unit": "none" + }, + { + "name": "type", + "description": "Profile id", + "type": "character(8)", + "unit": "none" + }, + { + "name": "load_in_wh", + "description": "Residential demand load timeseries for one year (8760 hours) in Wh", + "type": "array of real", + "unit": "Wh" + } + ], + "primaryKey": "id" + }, + "dialect": { + "delimiter": "", + "decimalSeparator": "." + } + } + ], + "review": { + "path": "", + "badge": "" + }, + "metaMetadata": { + "metadataVersion": "OEP-1.4.1", + "metadataLicense": { + "name": "CC0-1.0", + "title": "Creative Commons Zero v1.0 Universal", + "path": "https://creativecommons.org/publicdomain/zero/1.0/" + } + }, + "_comment": { + "metadata": "Metadata documentation and explanation (https://github.com/OpenEnergyPlatform/oemetadata/blob/master/metadata/v141/metadata_key_description.md)", + "dates": "Dates and time must follow the ISO8601 including time zone (YYYY-MM-DD or YYYY-MM-DDThh:mm:ss\u00b1hh)", + "units": "Use a space between numbers and units (100 m)", + "languages": "Languages must follow the IETF (BCP47) format (en-GB, en-US, de-DE)", + "licenses": "License name must follow the SPDX License List (https://spdx.org/licenses/)", + "review": "Following the OEP Data Review (https://github.com/OpenEnergyPlatform/data-preprocessing/wiki)", + "none": "If not applicable use (none)" + } +} diff --git a/src/egon/data/metadata/grid.egon_mv_grid_district.json b/src/egon/data/metadata/grid.egon_mv_grid_district.json new file mode 100644 index 000000000..a8eb56501 --- /dev/null +++ b/src/egon/data/metadata/grid.egon_mv_grid_district.json @@ -0,0 +1,150 @@ +{ + "name": "grid.egon_mv_grid_district", + "title": "eGon MV grid districts", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": "Mapping table of bus_id to polygons shapes for MV grid districts.", + "language": "undefined", + "keywords": [], + "publicationDate": "2023-03-13", + "context": { + "homepage": "https://ego-n.org/", + "documentation": "https://egon-data.readthedocs.io/en/latest/", + "sourceCode": "https://github.com/openego/eGon-data", + "contact": "https://ego-n.org/partners/", + "grantNo": "03EI1002", + "fundingAgency": "Bundesministerium f\u00fcr Wirtschaft und Klimaschutz", + "fundingAgencyLogo": "https://commons.wikimedia.org/wiki/File:BMWi_Logo_2021.svg#/media/File:BMWi_Logo_2021.svg", + "publisherLogo": "https://ego-n.org/images/eGon_logo_noborder_transbg.svg" + }, + "spatial": { + "location": "", + "extent": "Germany", + "resolution": "MV Grids" + }, + "temporal": {}, + "sources": [ + { + "title": "Data bundle for egon-data: A transparent and reproducible data processing pipeline for energy system modeling", + "description": "egon-data provides a transparent and reproducible open data based data processing pipeline for generating data models suitable for energy system modeling. The data is customized for the requirements of the research project eGon. The research project aims to develop tools for an open and cross-sectoral planning of transmission and distribution grids. For further information please visit the eGon project website or its Github repository. egon-data retrieves and processes data from several different external input sources. As not all data dependencies can be downloaded automatically from external sources we provide a data bundle to be downloaded by egon-data.", + "path": "https://doi.org/10.5281/zenodo.6630616", + "licenses": [ + { + "name": "CC-BY-4.0", + "title": "Creative Commons Attribution 4.0 International", + "path": "https://creativecommons.org/licenses/by/4.0/legalcode", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute.", + "attribution": "\u00a9 eGon-data developers, \u00a9 Leibniz Institute for Applied Geophysics, \u00a9 Fraunhofer IEE, \u00a9 BGR Hannover, \u00a9 Danielle Schmidt, \u00a9 \u00dcbertragungsnetzbetreiber, \u00a9 DIW Berlin, \u00a9 Reiner Lemoine Institut, \u00a9 Statistisches Bundesamt, Wiesbaden 2008, \u00a9 Statistische \u00c4mter des Bundes und der L\u00e4nder 2021" + } + ] + }, + { + "title": "eGon-data", + "description": "Workflow to download, process and generate data sets suitable for the further research conducted in the project eGon (https://ego-n.org/)", + "path": "https://github.com/openego/eGon-data", + "licenses": [ + { + "name": "AGPL-3.0 License", + "title": "GNU Affero General Public License v3.0", + "path": "https://www.gnu.org/licenses/agpl-3.0.de.html", + "instruction": "Permissions of this strongest copyleft license are conditioned on making available complete source code of licensed works and modifications, which include larger works using a licensed work, under the same license. Copyright and license notices must be preserved. Contributors provide an express grant of patent rights. When a modified version is used to provide a service over a network, the complete source code of the modified version must be made available.", + "attribution": "\u00a9 eGon-data developers" + } + ] + }, + { + "title": "Verwaltungsgebiete 1:250 000 (Ebenen)", + "description": "Der Datenbestand umfasst s\u00e4mtliche Verwaltungseinheiten der hierarchischen Verwaltungsebenen vom Staat bis zu den Gemeinden mit ihren Grenzen, statistischen Schl\u00fcsselzahlen, Namen der Verwaltungseinheit sowie die spezifische Bezeichnung der Verwaltungsebene des jeweiligen Landes.", + "path": "https://daten.gdz.bkg.bund.de/produkte/vg/vg250_ebenen_0101/2020/vg250_01-01.geo84.shape.ebenen.zip", + "licenses": [ + { + "name": "dl-by-de/2.0", + "title": "Datenlizenz Deutschland \u2013 Namensnennung \u2013 Version 2.0", + "path": "www.govdata.de/dl-de/by-2-0", + "instruction": "Jede Nutzung ist unter den Bedingungen dieser \u201eDatenlizenz Deutschland - Namensnennung - Version 2.0 zul\u00e4ssig.Die bereitgestellten Daten und Metadaten d\u00fcrfen f\u00fcr die kommerzielle und nicht kommerzielle Nutzung insbesondere:(1) vervielf\u00e4ltigt, ausgedruckt, pr\u00e4sentiert, ver\u00e4ndert, bearbeitet sowie an Dritte \u00fcbermittelt werden; (2) mit eigenen Daten und Daten Anderer zusammengef\u00fchrt und zu selbst\u00e4ndigen neuen Datens\u00e4tzen verbunden werden; (3) in interne und externe Gesch\u00e4ftsprozesse, Produkte und Anwendungen in \u00f6ffentlichen und nicht \u00f6ffentlichen elektronischen Netzwerken eingebunden werden.Bei der Nutzung ist sicherzustellen, dass folgende Angaben als Quellenvermerk enthalten sind:(1) Bezeichnung des Bereitstellers nach dessen Ma\u00dfgabe,(2) der Vermerk Datenlizenz Deutschland \u2013 Namensnennung \u2013 Version 2.0 oder dl-de/by-2-0 mit Verweis auf den Lizenztext unter www.govdata.de/dl-de/by-2-0 sowie(3) einen Verweis auf den Datensatz (URI).Dies gilt nur soweit die datenhaltende Stelle die Angaben(1) bis (3) zum Quellenvermerk bereitstellt.Ver\u00e4nderungen, Bearbeitungen, neue Gestaltungen oder sonstige Abwandlungen sind im Quellenvermerk mit dem Hinweis zu versehen, dass die Daten ge\u00e4ndert wurden.", + "attribution": "\u00a9 Bundesamt f\u00fcr Kartographie und Geod\u00e4sie 2020 (Daten ver\u00e4ndert)" + } + ] + } + ], + "licenses": [ + { + "name": "ODbL-1.0", + "title": "Open Data Commons Open Database License 1.0", + "path": "https://opendatacommons.org/licenses/odbl/1.0/", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute, Share-Alike, Keep open!", + "attribution": "\u00a9 Reiner Lemoine Institut, \u00a9 Bundesamt f\u00fcr Kartographie und Geod\u00e4sie 2020 (Daten ver\u00e4ndert), \u00a9 eGon-data developers" + } + ], + "contributors": [ + { + "title": "nailend", + "email": "https://github.com/nailend", + "date": "2023-03-13", + "object": "metadata", + "comment": "create metadata" + } + ], + "resources": [ + { + "profile": "tabular-data-resource", + "name": "grid.egon_mv_grid_district", + "path": "", + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": [ + { + "name": "building_id", + "description": "Building id (reference to: openstreetmap.osm_buildings_filtered.id, openstreetmap.osm_buildings_synthetic.id)", + "type": "integer", + "unit": "none" + }, + { + "name": "bus_id", + "description": "MVGD bus id (reference to: grid.egon_hvmv_substation.bus_id, grid.egon_mv_grid_district.bus_id)", + "type": "integer", + "unit": "none" + }, + { + "name": "geom", + "description": "Polygon of mv grid district (CRS: ERTS89/EPSG:3035)", + "type": "geometry", + "unit": "None" + }, + { + "name": "area", + "description": "Area covered by mvgd", + "type": "double precision", + "unit": "m^2" + } + ], + "primaryKey": "bus_id" + }, + "dialect": { + "delimiter": "", + "decimalSeparator": "." + } + } + ], + "review": { + "path": "", + "badge": "" + }, + "metaMetadata": { + "metadataVersion": "OEP-1.4.1", + "metadataLicense": { + "name": "CC0-1.0", + "title": "Creative Commons Zero v1.0 Universal", + "path": "https://creativecommons.org/publicdomain/zero/1.0/" + } + }, + "_comment": { + "metadata": "Metadata documentation and explanation (https://github.com/OpenEnergyPlatform/oemetadata/blob/master/metadata/v141/metadata_key_description.md)", + "dates": "Dates and time must follow the ISO8601 including time zone (YYYY-MM-DD or YYYY-MM-DDThh:mm:ss\u00b1hh)", + "units": "Use a space between numbers and units (100 m)", + "languages": "Languages must follow the IETF (BCP47) format (en-GB, en-US, de-DE)", + "licenses": "License name must follow the SPDX License List (https://spdx.org/licenses/)", + "review": "Following the OEP Data Review (https://github.com/OpenEnergyPlatform/data-preprocessing/wiki)", + "none": "If not applicable use (none)" + } +} diff --git a/src/egon/data/metadata/openstreetmap.osm_buildings_filtered.json b/src/egon/data/metadata/openstreetmap.osm_buildings_filtered.json new file mode 100644 index 000000000..519c87941 --- /dev/null +++ b/src/egon/data/metadata/openstreetmap.osm_buildings_filtered.json @@ -0,0 +1,183 @@ +{ + "name": "openstreetmap.osm_buildings_filtered", + "title": "eGon filtered buildings from openstreetmap", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": "Tag filtered list of buildings from OpenStreetMap which might have electricity or heat demand - (c) OpenStreetMap contributors", + "language": "undefined", + "keywords": [], + "publicationDate": "2023-03-13", + "context": { + "homepage": "https://ego-n.org/", + "documentation": "https://egon-data.readthedocs.io/en/latest/", + "sourceCode": "https://github.com/openego/eGon-data", + "contact": "https://ego-n.org/partners/", + "grantNo": "03EI1002", + "fundingAgency": "Bundesministerium f\u00fcr Wirtschaft und Klimaschutz", + "fundingAgencyLogo": "https://commons.wikimedia.org/wiki/File:BMWi_Logo_2021.svg#/media/File:BMWi_Logo_2021.svg", + "publisherLogo": "https://ego-n.org/images/eGon_logo_noborder_transbg.svg" + }, + "spatial": { + "location": "", + "extent": "Germany", + "resolution": "Buildings" + }, + "temporal": { + "referenceDate": "2022-01-01", + "timeseries": {} + }, + "sources": [ + { + "title": "Data bundle for egon-data: A transparent and reproducible data processing pipeline for energy system modeling", + "description": "egon-data provides a transparent and reproducible open data based data processing pipeline for generating data models suitable for energy system modeling. The data is customized for the requirements of the research project eGon. The research project aims to develop tools for an open and cross-sectoral planning of transmission and distribution grids. For further information please visit the eGon project website or its Github repository. egon-data retrieves and processes data from several different external input sources. As not all data dependencies can be downloaded automatically from external sources we provide a data bundle to be downloaded by egon-data.", + "path": "https://doi.org/10.5281/zenodo.6630616", + "licenses": [ + { + "name": "CC-BY-4.0", + "title": "Creative Commons Attribution 4.0 International", + "path": "https://creativecommons.org/licenses/by/4.0/legalcode", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute.", + "attribution": "\u00a9 eGon-data developers, \u00a9 Leibniz Institute for Applied Geophysics, \u00a9 Fraunhofer IEE, \u00a9 BGR Hannover, \u00a9 Danielle Schmidt, \u00a9 \u00dcbertragungsnetzbetreiber, \u00a9 DIW Berlin, \u00a9 Reiner Lemoine Institut, \u00a9 Statistisches Bundesamt, Wiesbaden 2008, \u00a9 Statistische \u00c4mter des Bundes und der L\u00e4nder 2021" + } + ] + }, + { + "title": "eGon-data", + "description": "Workflow to download, process and generate data sets suitable for the further research conducted in the project eGon (https://ego-n.org/)", + "path": "https://github.com/openego/eGon-data", + "licenses": [ + { + "name": "AGPL-3.0 License", + "title": "GNU Affero General Public License v3.0", + "path": "https://www.gnu.org/licenses/agpl-3.0.de.html", + "instruction": "Permissions of this strongest copyleft license are conditioned on making available complete source code of licensed works and modifications, which include larger works using a licensed work, under the same license. Copyright and license notices must be preserved. Contributors provide an express grant of patent rights. When a modified version is used to provide a service over a network, the complete source code of the modified version must be made available.", + "attribution": "\u00a9 eGon-data developers" + } + ] + }, + { + "title": "OpenStreetMap Data Extracts (Geofabrik)", + "description": "Full data extract of OpenStreetMap data for defined spatial extent at 2022-01-01", + "path": "https://download.geofabrik.de/europe/germany-210101.osm.pbf", + "licenses": [ + { + "name": "ODbL-1.0", + "title": "Open Data Commons Open Database License 1.0", + "path": "https://opendatacommons.org/licenses/odbl/1.0/", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute, Share-Alike, Keep open!", + "attribution": "\u00a9 OpenStreetMap contributors" + } + ] + } + ], + "licenses": [ + { + "name": "ODbL-1.0", + "title": "Open Data Commons Open Database License 1.0", + "path": "https://opendatacommons.org/licenses/odbl/1.0/", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute, Share-Alike, Keep open!", + "attribution": "\u00a9 Reiner Lemoine Institut" + } + ], + "contributors": [ + { + "title": "nailend", + "email": "https://github.com/nailend", + "date": "2023-03-13", + "object": "metadata", + "comment": "Created metadata" + } + ], + "resources": [ + { + "profile": "tabular-data-resource", + "name": "openstreetmap.osm_buildings_filtered", + "path": "", + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": [ + { + "name": "osm_id", + "description": "Openstreetmap identifier", + "type": "bigint", + "unit": "none" + }, + { + "name": "amenitiy", + "description": "Type of amenitiy if given", + "type": "text", + "unit": "none" + }, + { + "name": "building", + "description": "Type of building", + "type": "text", + "unit": "none" + }, + { + "name": "name", + "description": "Name of the building if given", + "type": "text", + "unit": "none" + }, + { + "name": "geom_building", + "description": "Polygon of the building (CRS: ERTS89/EPSG:3035)", + "type": "geometry", + "unit": "none" + }, + { + "name": "area", + "description": "Surface area of building", + "type": "double precision", + "unit": "m^2" + }, + { + "name": "geom_point", + "description": "Centroid of the building (CRS: ERTS89/EPSG:3035)", + "type": "geometry", + "unit": "none" + }, + { + "name": "tags", + "description": "Opensteetmap tags assigned to the building", + "type": "hstore", + "unit": "none" + }, + { + "name": "id", + "description": "Unique identifyer and project internal building id (reference to: demand.egon_hp_capacity_buildings.building_id, demand.egon_cts_heat_demand_building_share.building_id, demand.egon_cts_electricity_demand_building_share.building_id, demand.egon_household_electricity_profile_of_buildings.building_id, demand.egon_building_heat_peak_loads.building_id, demand.egon_building_electricity_peak_loads.building_id, boundaries.egon_map_zensus_mvgd_buildings.building_id)", + "type": "integer", + "unit": "none" + } + ], + "primaryKey": "id" + }, + "dialect": { + "delimiter": "", + "decimalSeparator": "." + } + } + ], + "review": { + "path": "", + "badge": "" + }, + "metaMetadata": { + "metadataVersion": "OEP-1.4.1", + "metadataLicense": { + "name": "CC0-1.0", + "title": "Creative Commons Zero v1.0 Universal", + "path": "https://creativecommons.org/publicdomain/zero/1.0/" + } + }, + "_comment": { + "metadata": "Metadata documentation and explanation (https://github.com/OpenEnergyPlatform/oemetadata/blob/master/metadata/v141/metadata_key_description.md)", + "dates": "Dates and time must follow the ISO8601 including time zone (YYYY-MM-DD or YYYY-MM-DDThh:mm:ss\u00b1hh)", + "units": "Use a space between numbers and units (100 m)", + "languages": "Languages must follow the IETF (BCP47) format (en-GB, en-US, de-DE)", + "licenses": "License name must follow the SPDX License List (https://spdx.org/licenses/)", + "review": "Following the OEP Data Review (https://github.com/OpenEnergyPlatform/data-preprocessing/wiki)", + "none": "If not applicable use (none)" + } +} diff --git a/src/egon/data/metadata/openstreetmap.osm_buildings_synthetic.json b/src/egon/data/metadata/openstreetmap.osm_buildings_synthetic.json new file mode 100644 index 000000000..5957f61f3 --- /dev/null +++ b/src/egon/data/metadata/openstreetmap.osm_buildings_synthetic.json @@ -0,0 +1,188 @@ +{ + "name": "openstreetmap.osm_buildings_synthetic", + "title": "eGon synthetic residential buildings", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": " List of generated synthetic buildings ", + "language": [ + "en-EN", + "de-DE" + ], + "keywords": [], + "publicationDate": "2022-09-14", + "context": { + "homepage": "https://ego-n.org/", + "documentation": "https://egon-data.readthedocs.io/en/latest/", + "sourceCode": "https://github.com/openego/eGon-data", + "contact": "https://ego-n.org/partners/", + "grantNo": "03EI1002", + "fundingAgency": "Bundesministerium f\u00fcr Wirtschaft und Klimaschutz", + "fundingAgencyLogo": "https://commons.wikimedia.org/wiki/File:BMWi_Logo_2021.svg#/media/File:BMWi_Logo_2021.svg", + "publisherLogo": "https://ego-n.org/images/eGon_logo_noborder_transbg.svg" + }, + "spatial": { + "location": "", + "extent": "Germany", + "resolution": "Buildings" + }, + "temporal": { + "referenceDate": "2022-09-14", + "timeseries": {} + }, + "sources": [ + { + "title": "Data bundle for egon-data: A transparent and reproducible data processing pipeline for energy system modeling", + "description": "egon-data provides a transparent and reproducible open data based data processing pipeline for generating data models suitable for energy system modeling. The data is customized for the requirements of the research project eGon. The research project aims to develop tools for an open and cross-sectoral planning of transmission and distribution grids. For further information please visit the eGon project website or its Github repository. egon-data retrieves and processes data from several different external input sources. As not all data dependencies can be downloaded automatically from external sources we provide a data bundle to be downloaded by egon-data.", + "path": "https://doi.org/10.5281/zenodo.6630616", + "licenses": [ + { + "name": "CC-BY-4.0", + "title": "Creative Commons Attribution 4.0 International", + "path": "https://creativecommons.org/licenses/by/4.0/legalcode", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute.", + "attribution": "\u00a9 eGon-data developers, \u00a9 Leibniz Institute for Applied Geophysics, \u00a9 Fraunhofer IEE, \u00a9 BGR Hannover, \u00a9 Danielle Schmidt, \u00a9 \u00dcbertragungsnetzbetreiber, \u00a9 DIW Berlin, \u00a9 Reiner Lemoine Institut, \u00a9 Statistisches Bundesamt, Wiesbaden 2008, \u00a9 Statistische \u00c4mter des Bundes und der L\u00e4nder 2021" + } + ] + }, + { + "title": "eGon-data", + "description": "Workflow to download, process and generate data sets suitable for the further research conducted in the project eGon (https://ego-n.org/)", + "path": "https://github.com/openego/eGon-data", + "licenses": [ + { + "name": "AGPL-3.0 License", + "title": "GNU Affero General Public License v3.0", + "path": "https://www.gnu.org/licenses/agpl-3.0.de.html", + "instruction": "Permissions of this strongest copyleft license are conditioned on making available complete source code of licensed works and modifications, which include larger works using a licensed work, under the same license. Copyright and license notices must be preserved. Contributors provide an express grant of patent rights. When a modified version is used to provide a service over a network, the complete source code of the modified version must be made available.", + "attribution": "\u00a9 eGon-data developers" + } + ] + }, + { + "title": "OpenStreetMap Data Extracts (Geofabrik)", + "description": "Full data extract of OpenStreetMap data for defined spatial extent at 2022-01-01", + "path": "https://download.geofabrik.de/europe/germany-210101.osm.pbf", + "licenses": [ + { + "name": "ODbL-1.0", + "title": "Open Data Commons Open Database License 1.0", + "path": "https://opendatacommons.org/licenses/odbl/1.0/", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute, Share-Alike, Keep open!", + "attribution": "\u00a9 OpenStreetMap contributors" + } + ] + } + ], + "licenses": [ + { + "name": "ODbL-1.0", + "title": "Open Data Commons Open Database License 1.0", + "path": "https://opendatacommons.org/licenses/odbl/1.0/", + "instruction": "You are free: To Share, To Create, To Adapt; As long as you: Attribute, Share-Alike, Keep open!", + "attribution": "\u00a9 Reiner Lemoine Institut" + } + ], + "contributors": [ + { + "title": "ClaraBuettner", + "email": "https://github.com/ClaraBuettner", + "date": "2022-09-12", + "object": "metadata", + "comment": "Create metadata" + }, + { + "title": "nailend", + "email": "https://github.com/nailend", + "date": "2022-09-12", + "object": "metadata", + "comment": "Updated metadata" + }, + { + "title": "nesnoj", + "email": "https://github.com/nesnoj", + "date": "2022-09-14", + "object": "metadata", + "comment": "Update metadata" + } + ], + "resources": [ + { + "profile": "tabular-data-resource", + "name": "openstreetmap.osm_buildings_synthetic", + "path": "", + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": [ + { + "name": "id", + "description": "Unique identifyer and project internal building id (reference to: openstreetmap.osm_buildings_residential.id, demand.egon_household_electricity_profile_of_buildings.building_id)", + "type": "character varying", + "unit": "none" + }, + { + "name": "area", + "description": "Surface area of building", + "type": "real", + "unit": "m^2" + }, + { + "name": "building", + "description": "Type of building (residential or cts)", + "type": "character varying(11)", + "unit": "none" + }, + { + "name": "cell_id", + "description": "Census cell id (reference to: society.destatis_zensus_population_per_ha_inside_germany.id)", + "type": "character varying", + "unit": "none" + }, + { + "name": "geom_building", + "description": "Polygon of building (CRS: ERTS89/EPSG:3035)", + "type": "geometry", + "unit": "none" + }, + { + "name": "geom_point", + "description": "Centroid of building (CRS: ERTS89/EPSG:3035)", + "type": "geometry", + "unit": "none" + }, + { + "name": "n_amenities_inside", + "description": "Number of amenities inside the building (always zero if not cts building)", + "type": "integer", + "unit": "none" + } + ], + "primaryKey": "id" + }, + "dialect": { + "delimiter": "", + "decimalSeparator": "." + } + } + ], + "review": { + "path": "", + "badge": "" + }, + "metaMetadata": { + "metadataVersion": "OEP-1.4.1", + "metadataLicense": { + "name": "CC0-1.0", + "title": "Creative Commons Zero v1.0 Universal", + "path": "https://creativecommons.org/publicdomain/zero/1.0/" + } + }, + "_comment": { + "metadata": "Metadata documentation and explanation (https://github.com/OpenEnergyPlatform/oemetadata/blob/master/metadata/v141/metadata_key_description.md)", + "dates": "Dates and time must follow the ISO8601 including time zone (YYYY-MM-DD or YYYY-MM-DDThh:mm:ss\u00b1hh)", + "units": "Use a space between numbers and units (100 m)", + "languages": "Languages must follow the IETF (BCP47) format (en-GB, en-US, de-DE)", + "licenses": "License name must follow the SPDX License List (https://spdx.org/licenses/)", + "review": "Following the OEP Data Review (https://github.com/OpenEnergyPlatform/data-preprocessing/wiki)", + "none": "If not applicable use (none)" + } +} diff --git a/src/egon/data/metadata/society.destatis_zensus_population_per_ha.json b/src/egon/data/metadata/society.destatis_zensus_population_per_ha.json new file mode 100644 index 000000000..5e20057a6 --- /dev/null +++ b/src/egon/data/metadata/society.destatis_zensus_population_per_ha.json @@ -0,0 +1,142 @@ +{ + "name": "society.destatis_zensus_population_per_ha", + "title": "DESTATIS - Zensus 2011 - Population per hectar", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": "National census in Germany in 2011.", + "language": "undefined", + "keywords": [], + "publicationDate": "2022-08-06", + "context": { + "homepage": "https://ego-n.org/", + "documentation": "https://egon-data.readthedocs.io/en/latest/", + "sourceCode": "https://github.com/openego/eGon-data", + "contact": "https://ego-n.org/partners/", + "grantNo": "03EI1002", + "fundingAgency": "Bundesministerium f\\u00fcr Wirtschaft und Energie", + "fundingAgencyLogo": "https://www.innovation-beratung-foerderung.de/INNO/Redaktion/DE/Bilder/Titelbilder/titel_foerderlogo_bmwi.jpg?__blob=normal&v=3", + "publisherLogo": "https://ego-n.org/images/eGon_logo_noborder_transbg.svg" + }, + "spatial": { + "location": "", + "extent": "Germany", + "resolution": "1 ha" + }, + "temporal": {}, + "sources": [ + { + "title": "Statistisches Bundesamt (Destatis) - Ergebnisse des Zensus 2011 zum Download", + "description": "Als Download bieten wir Ihnen auf dieser Seite zus\\u00e4tzlich zur Zensusdatenbank CSV- und teilweise Excel-Tabellen mit umfassenden Personen-, Haushalts- und Familien- sowie Geb\\u00e4ude- und Wohnungsmerkmalen. Die Ergebnisse liegen auf Bundes-, L\\u00e4nder-, Kreis- und Gemeindeebene vor. Au\\u00dferdem sind einzelne Ergebnisse f\\u00fcr Gitterzellen verf\\u00fcgbar.", + "path": "https://www.zensus2011.de/DE/Home/Aktuelles/DemografischeGrunddaten.html", + "licenses": [ + { + "name": "dl-by-de/2.0", + "title": "Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0", + "path": "www.govdata.de/dl-de/by-2-0", + "instruction": "Jede Nutzung ist unter den Bedingungen dieser \\u201eDatenlizenz Deutschland - Namensnennung - Version 2.0 zul\\u00e4ssig.\\nDie bereitgestellten Daten und Metadaten d\\u00fcrfen f\\u00fcr die kommerzielle und nicht kommerzielle Nutzung insbesondere:(1) vervielf\\u00e4ltigt, ausgedruckt, pr\\u00e4sentiert, ver\\u00e4ndert, bearbeitet sowie an Dritte \\u00fcbermittelt werden;\\n (2) mit eigenen Daten und Daten Anderer zusammengef\\u00fchrt und zu selbst\\u00e4ndigen neuen Datens\\u00e4tzen verbunden werden;\\n (3) in interne und externe Gesch\\u00e4ftsprozesse, Produkte und Anwendungen in \\u00f6ffentlichen und nicht \\u00f6ffentlichen elektronischen Netzwerken eingebunden werden.\\nBei der Nutzung ist sicherzustellen, dass folgende Angaben als Quellenvermerk enthalten sind:\\n(1) Bezeichnung des Bereitstellers nach dessen Ma\\u00dfgabe,\\n(2) der Vermerk Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0 oder dl-de/by-2-0 mit Verweis auf den Lizenztext unter www.govdata.de/dl-de/by-2-0 sowie\\n(3) einen Verweis auf den Datensatz (URI).Dies gilt nur soweit die datenhaltende Stelle die Angaben(1) bis (3) zum Quellenvermerk bereitstellt.\\nVer\\u00e4nderungen, Bearbeitungen, neue Gestaltungen oder sonstige Abwandlungen sind im Quellenvermerk mit dem Hinweis zu versehen, dass die Daten ge\\u00e4ndert wurden.", + "attribution": "\\u00a9 Statistische \\u00c4mter des Bundes und der L\\u00e4nder 2014" + } + ] + }, + { + "title": "Dokumentation - Zensus 2011 - Methoden und Verfahren", + "description": "Diese Publikation beschreibt ausf\\u00fchrlich die Methoden und Verfahren des registergest\\u00fctzten Zensus 2011; von der Datengewinnung und -aufbereitung bis hin zur Ergebniserstellung und Geheimhaltung. Der vorliegende Band wurde von den Statistischen \\u00c4mtern des Bundes und der L\\u00e4nder im Juni 2015 ver\\u00f6ffentlicht.", + "path": "https://www.destatis.de/DE/Publikationen/Thematisch/Bevoelkerung/Zensus/ZensusBuLaMethodenVerfahren5121105119004.pdf?__blob=publicationFile", + "licenses": [ + { + "name": "dl-by-de/2.0", + "title": "Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0", + "path": "www.govdata.de/dl-de/by-2-0", + "instruction": "Jede Nutzung ist unter den Bedingungen dieser \\u201eDatenlizenz Deutschland - Namensnennung - Version 2.0 zul\\u00e4ssig.\\nDie bereitgestellten Daten und Metadaten d\\u00fcrfen f\\u00fcr die kommerzielle und nicht kommerzielle Nutzung insbesondere:(1) vervielf\\u00e4ltigt, ausgedruckt, pr\\u00e4sentiert, ver\\u00e4ndert, bearbeitet sowie an Dritte \\u00fcbermittelt werden;\\n (2) mit eigenen Daten und Daten Anderer zusammengef\\u00fchrt und zu selbst\\u00e4ndigen neuen Datens\\u00e4tzen verbunden werden;\\n (3) in interne und externe Gesch\\u00e4ftsprozesse, Produkte und Anwendungen in \\u00f6ffentlichen und nicht \\u00f6ffentlichen elektronischen Netzwerken eingebunden werden.\\nBei der Nutzung ist sicherzustellen, dass folgende Angaben als Quellenvermerk enthalten sind:\\n(1) Bezeichnung des Bereitstellers nach dessen Ma\\u00dfgabe,\\n(2) der Vermerk Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0 oder dl-de/by-2-0 mit Verweis auf den Lizenztext unter www.govdata.de/dl-de/by-2-0 sowie\\n(3) einen Verweis auf den Datensatz (URI).Dies gilt nur soweit die datenhaltende Stelle die Angaben(1) bis (3) zum Quellenvermerk bereitstellt.\\nVer\\u00e4nderungen, Bearbeitungen, neue Gestaltungen oder sonstige Abwandlungen sind im Quellenvermerk mit dem Hinweis zu versehen, dass die Daten ge\\u00e4ndert wurden.", + "attribution": "\\u00a9 Statistisches Bundesamt, Wiesbaden 2015 (im Auftrag der Herausgebergemeinschaft)" + } + ] + } + ], + "licenses": [ + { + "name": "dl-by-de/2.0", + "title": "Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0", + "path": "www.govdata.de/dl-de/by-2-0", + "instruction": "Jede Nutzung ist unter den Bedingungen dieser \\u201eDatenlizenz Deutschland - Namensnennung - Version 2.0 zul\\u00e4ssig.\\nDie bereitgestellten Daten und Metadaten d\\u00fcrfen f\\u00fcr die kommerzielle und nicht kommerzielle Nutzung insbesondere:(1) vervielf\\u00e4ltigt, ausgedruckt, pr\\u00e4sentiert, ver\\u00e4ndert, bearbeitet sowie an Dritte \\u00fcbermittelt werden;\\n (2) mit eigenen Daten und Daten Anderer zusammengef\\u00fchrt und zu selbst\\u00e4ndigen neuen Datens\\u00e4tzen verbunden werden;\\n (3) in interne und externe Gesch\\u00e4ftsprozesse, Produkte und Anwendungen in \\u00f6ffentlichen und nicht \\u00f6ffentlichen elektronischen Netzwerken eingebunden werden.\\nBei der Nutzung ist sicherzustellen, dass folgende Angaben als Quellenvermerk enthalten sind:\\n(1) Bezeichnung des Bereitstellers nach dessen Ma\\u00dfgabe,\\n(2) der Vermerk Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0 oder dl-de/by-2-0 mit Verweis auf den Lizenztext unter www.govdata.de/dl-de/by-2-0 sowie\\n(3) einen Verweis auf den Datensatz (URI).Dies gilt nur soweit die datenhaltende Stelle die Angaben(1) bis (3) zum Quellenvermerk bereitstellt.\\nVer\\u00e4nderungen, Bearbeitungen, neue Gestaltungen oder sonstige Abwandlungen sind im Quellenvermerk mit dem Hinweis zu versehen, dass die Daten ge\\u00e4ndert wurden.", + "attribution": "\\u00a9 Statistische \\u00c4mter des Bundes und der L\\u00e4nder 2014; \\u00a9 Statistisches Bundesamt, Wiesbaden 2015 (Daten ver\\u00e4ndert)" + } + ], + "contributors": [ + { + "title": "nailend", + "email": "https://github.com/nailend", + "date": "2023-03-13", + "object": "metadata", + "comment": "Create metadata" + } + ], + "resources": [ + { + "profile": "tabular-data-resource", + "name": "society.destatis_zensus_population_per_ha", + "path": "", + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": [ + { + "name": "id", + "description": "Unique identifier", + "type": "integer", + "unit": "none" + }, + { + "name": "grid_id", + "description": "Grid number of source", + "type": "string", + "unit": "none" + }, + { + "name": "population", + "description": "Number of registred residents", + "type": "integer", + "unit": "resident" + }, + { + "name": "geom_point", + "description": "Geometry centroid of zensus cell (CRS: ERTS89/EPSG:3035)", + "type": "Geometry", + "unit": "none" + }, + { + "name": "geom", + "description": "Geometry of zensus cell (CRS: ERTS89/EPSG:3035)", + "type": "Geometry", + "unit": "none" + } + ], + "primaryKey": "id" + }, + "dialect": { + "delimiter": "", + "decimalSeparator": "." + } + } + ], + "review": { + "path": "", + "badge": "" + }, + "metaMetadata": { + "metadataVersion": "OEP-1.4.1", + "metadataLicense": { + "name": "CC0-1.0", + "title": "Creative Commons Zero v1.0 Universal", + "path": "https://creativecommons.org/publicdomain/zero/1.0/" + } + }, + "_comment": { + "metadata": "Metadata documentation and explanation (https://github.com/OpenEnergyPlatform/oemetadata/blob/master/metadata/v141/metadata_key_description.md)", + "dates": "Dates and time must follow the ISO8601 including time zone (YYYY-MM-DD or YYYY-MM-DDThh:mm:ss\u00b1hh)", + "units": "Use a space between numbers and units (100 m)", + "languages": "Languages must follow the IETF (BCP47) format (en-GB, en-US, de-DE)", + "licenses": "License name must follow the SPDX License List (https://spdx.org/licenses/)", + "review": "Following the OEP Data Review (https://github.com/OpenEnergyPlatform/data-preprocessing/wiki)", + "none": "If not applicable use (none)" + } +} diff --git a/src/egon/data/metadata/society.egon_destatis_zensus_apartment_building_population_per_ha.json b/src/egon/data/metadata/society.egon_destatis_zensus_apartment_building_population_per_ha.json new file mode 100644 index 000000000..a48513071 --- /dev/null +++ b/src/egon/data/metadata/society.egon_destatis_zensus_apartment_building_population_per_ha.json @@ -0,0 +1,154 @@ +{ + "name": "society.egon_destatis_zensus_apartment_building_population_per_ha", + "title": "eGon - Zensus 2011 - Apartments, Buildings, Population per hectar", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": "Apartments, Buildings, Population from national census in Germany in 2011.", + "language": "undefined", + "keywords": [], + "publicationDate": "2023-03-13", + "context": { + "homepage": "https://ego-n.org/", + "documentation": "https://egon-data.readthedocs.io/en/latest/", + "sourceCode": "https://github.com/openego/eGon-data", + "contact": "https://ego-n.org/partners/", + "grantNo": "03EI1002", + "fundingAgency": "Bundesministerium f\\u00fcr Wirtschaft und Energie", + "fundingAgencyLogo": "https://www.innovation-beratung-foerderung.de/INNO/Redaktion/DE/Bilder/Titelbilder/titel_foerderlogo_bmwi.jpg?__blob=normal&v=3", + "publisherLogo": "https://ego-n.org/images/eGon_logo_noborder_transbg.svg" + }, + "spatial": { + "location": "", + "extent": "Germany", + "resolution": "1 ha" + }, + "temporal": {}, + "sources": [ + { + "title": "Statistisches Bundesamt (Destatis) - Ergebnisse des Zensus 2011 zum Download", + "description": "Als Download bieten wir Ihnen auf dieser Seite zus\\u00e4tzlich zur Zensusdatenbank CSV- und teilweise Excel-Tabellen mit umfassenden Personen-, Haushalts- und Familien- sowie Geb\\u00e4ude- und Wohnungsmerkmalen. Die Ergebnisse liegen auf Bundes-, L\\u00e4nder-, Kreis- und Gemeindeebene vor. Au\\u00dferdem sind einzelne Ergebnisse f\\u00fcr Gitterzellen verf\\u00fcgbar.", + "path": "https://www.zensus2011.de/DE/Home/Aktuelles/DemografischeGrunddaten.html", + "licenses": [ + { + "name": "dl-by-de/2.0", + "title": "Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0", + "path": "www.govdata.de/dl-de/by-2-0", + "instruction": "Jede Nutzung ist unter den Bedingungen dieser \\u201eDatenlizenz Deutschland - Namensnennung - Version 2.0 zul\\u00e4ssig.\\nDie bereitgestellten Daten und Metadaten d\\u00fcrfen f\\u00fcr die kommerzielle und nicht kommerzielle Nutzung insbesondere:(1) vervielf\\u00e4ltigt, ausgedruckt, pr\\u00e4sentiert, ver\\u00e4ndert, bearbeitet sowie an Dritte \\u00fcbermittelt werden;\\n (2) mit eigenen Daten und Daten Anderer zusammengef\\u00fchrt und zu selbst\\u00e4ndigen neuen Datens\\u00e4tzen verbunden werden;\\n (3) in interne und externe Gesch\\u00e4ftsprozesse, Produkte und Anwendungen in \\u00f6ffentlichen und nicht \\u00f6ffentlichen elektronischen Netzwerken eingebunden werden.\\nBei der Nutzung ist sicherzustellen, dass folgende Angaben als Quellenvermerk enthalten sind:\\n(1) Bezeichnung des Bereitstellers nach dessen Ma\\u00dfgabe,\\n(2) der Vermerk Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0 oder dl-de/by-2-0 mit Verweis auf den Lizenztext unter www.govdata.de/dl-de/by-2-0 sowie\\n(3) einen Verweis auf den Datensatz (URI).Dies gilt nur soweit die datenhaltende Stelle die Angaben(1) bis (3) zum Quellenvermerk bereitstellt.\\nVer\\u00e4nderungen, Bearbeitungen, neue Gestaltungen oder sonstige Abwandlungen sind im Quellenvermerk mit dem Hinweis zu versehen, dass die Daten ge\\u00e4ndert wurden.", + "attribution": "\\u00a9 Statistische \\u00c4mter des Bundes und der L\\u00e4nder 2014" + } + ] + }, + { + "title": "Dokumentation - Zensus 2011 - Methoden und Verfahren", + "description": "Diese Publikation beschreibt ausf\\u00fchrlich die Methoden und Verfahren des registergest\\u00fctzten Zensus 2011; von der Datengewinnung und -aufbereitung bis hin zur Ergebniserstellung und Geheimhaltung. Der vorliegende Band wurde von den Statistischen \\u00c4mtern des Bundes und der L\\u00e4nder im Juni 2015 ver\\u00f6ffentlicht.", + "path": "https://www.destatis.de/DE/Publikationen/Thematisch/Bevoelkerung/Zensus/ZensusBuLaMethodenVerfahren5121105119004.pdf?__blob=publicationFile", + "licenses": [ + { + "title": "Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0", + "path": "www.govdata.de/dl-de/by-2-0", + "instruction": "Jede Nutzung ist unter den Bedingungen dieser \\u201eDatenlizenz Deutschland - Namensnennung - Version 2.0 zul\\u00e4ssig.\\nDie bereitgestellten Daten und Metadaten d\\u00fcrfen f\\u00fcr die kommerzielle und nicht kommerzielle Nutzung insbesondere:(1) vervielf\\u00e4ltigt, ausgedruckt, pr\\u00e4sentiert, ver\\u00e4ndert, bearbeitet sowie an Dritte \\u00fcbermittelt werden;\\n (2) mit eigenen Daten und Daten Anderer zusammengef\\u00fchrt und zu selbst\\u00e4ndigen neuen Datens\\u00e4tzen verbunden werden;\\n (3) in interne und externe Gesch\\u00e4ftsprozesse, Produkte und Anwendungen in \\u00f6ffentlichen und nicht \\u00f6ffentlichen elektronischen Netzwerken eingebunden werden.\\nBei der Nutzung ist sicherzustellen, dass folgende Angaben als Quellenvermerk enthalten sind:\\n(1) Bezeichnung des Bereitstellers nach dessen Ma\\u00dfgabe,\\n(2) der Vermerk Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0 oder dl-de/by-2-0 mit Verweis auf den Lizenztext unter www.govdata.de/dl-de/by-2-0 sowie\\n(3) einen Verweis auf den Datensatz (URI).Dies gilt nur soweit die datenhaltende Stelle die Angaben(1) bis (3) zum Quellenvermerk bereitstellt.\\nVer\\u00e4nderungen, Bearbeitungen, neue Gestaltungen oder sonstige Abwandlungen sind im Quellenvermerk mit dem Hinweis zu versehen, dass die Daten ge\\u00e4ndert wurden.", + "attribution": "\\u00a9 Statistisches Bundesamt, Wiesbaden 2015 (im Auftrag der Herausgebergemeinschaft)", + "name": "dl-by-de/2.0" + } + ] + } + ], + "licenses": [ + { + "name": "dl-by-de/2.0", + "title": "Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0", + "path": "www.govdata.de/dl-de/by-2-0", + "instruction": "Jede Nutzung ist unter den Bedingungen dieser \\u201eDatenlizenz Deutschland - Namensnennung - Version 2.0 zul\\u00e4ssig.\\nDie bereitgestellten Daten und Metadaten d\\u00fcrfen f\\u00fcr die kommerzielle und nicht kommerzielle Nutzung insbesondere:(1) vervielf\\u00e4ltigt, ausgedruckt, pr\\u00e4sentiert, ver\\u00e4ndert, bearbeitet sowie an Dritte \\u00fcbermittelt werden;\\n (2) mit eigenen Daten und Daten Anderer zusammengef\\u00fchrt und zu selbst\\u00e4ndigen neuen Datens\\u00e4tzen verbunden werden;\\n (3) in interne und externe Gesch\\u00e4ftsprozesse, Produkte und Anwendungen in \\u00f6ffentlichen und nicht \\u00f6ffentlichen elektronischen Netzwerken eingebunden werden.\\nBei der Nutzung ist sicherzustellen, dass folgende Angaben als Quellenvermerk enthalten sind:\\n(1) Bezeichnung des Bereitstellers nach dessen Ma\\u00dfgabe,\\n(2) der Vermerk Datenlizenz Deutschland \\u2013 Namensnennung \\u2013 Version 2.0 oder dl-de/by-2-0 mit Verweis auf den Lizenztext unter www.govdata.de/dl-de/by-2-0 sowie\\n(3) einen Verweis auf den Datensatz (URI).Dies gilt nur soweit die datenhaltende Stelle die Angaben(1) bis (3) zum Quellenvermerk bereitstellt.\\nVer\\u00e4nderungen, Bearbeitungen, neue Gestaltungen oder sonstige Abwandlungen sind im Quellenvermerk mit dem Hinweis zu versehen, dass die Daten ge\\u00e4ndert wurden.", + "attribution": "\\u00a9 Statistische \\u00c4mter des Bundes und der L\\u00e4nder 2014; \\u00a9 Statistisches Bundesamt, Wiesbaden 2015 (Daten ver\\u00e4ndert)" + } + ], + "contributors": [ + { + "title": "nailend", + "email": "https://github.com/nailend", + "date": "2023-03-13", + "object": "metadata", + "comment": "Create metadata" + } + ], + "resources": [ + { + "profile": "tabular-data-resource", + "name": "society.egon_destatis_zensus_apartment_building_population_per_ha", + "path": "", + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": [ + { + "name": "grid_id", + "description": "Grid number of source", + "type": "string", + "unit": "none" + }, + { + "name": "zensus_population_id", + "description": "Unique identifier", + "type": "integer", + "unit": "none" + }, + { + "name": "building_count", + "description": "Number of buildings within zensus cell ", + "type": "smallint", + "unit": "none" + }, + { + "name": "apartment_count", + "description": "Number of appartments within zensus cells", + "type": "smallint", + "unit": "none" + }, + { + "name": "population", + "description": "Number of registred residents", + "type": "integer", + "unit": "resident" + }, + { + "name": "geom", + "description": "Geometry of zensus cell (CRS: ERTS89/EPSG:3035)", + "type": "Geometry", + "unit": "none" + }, + { + "name": "geom_point", + "description": "Geometry centroid of zensus cell (CRS: ERTS89/EPSG:3035)", + "type": "Geometry", + "unit": "none" + } + ], + "primaryKey": "none" + }, + "dialect": { + "delimiter": "", + "decimalSeparator": "." + } + } + ], + "metaMetadata": { + "metadataVersion": "OEP-1.4.1", + "metadataLicense": { + "name": "CC0-1.0", + "title": "Creative Commons Zero v1.0 Universal", + "path": "https://creativecommons.org/publicdomain/zero/1.0/" + } + }, + "review": { + "path": "", + "badge": "" + }, + "_comment": { + "metadata": "Metadata documentation and explanation (https://github.com/OpenEnergyPlatform/oemetadata/blob/master/metadata/v141/metadata_key_description.md)", + "dates": "Dates and time must follow the ISO8601 including time zone (YYYY-MM-DD or YYYY-MM-DDThh:mm:ss\u00b1hh)", + "units": "Use a space between numbers and units (100 m)", + "languages": "Languages must follow the IETF (BCP47) format (en-GB, en-US, de-DE)", + "licenses": "License name must follow the SPDX License List (https://spdx.org/licenses/)", + "review": "Following the OEP Data Review (https://github.com/OpenEnergyPlatform/data-preprocessing/wiki)", + "none": "If not applicable use (none)" + } +}