23. Wrappers example

In this notebook, we’ll explore the wrappers defined by Sinergym and how to use them. You can also create your own wrappers by inheriting from gym.Wrapper or its variants.

[1]:

import gymnasium as gym
import numpy as np
import sinergym
from sinergym.utils.wrappers import *

23.1. Multi-Objective Wrapper

MO-Gymnasium is an open-source Python library for developing and comparing multi-objective reinforcement learning algorithms. These environments return a reward vector instead of a scalar value, one for each objective.

To be as general as possible, it could be beneficial for Sinergym to also provide that reward vector. This way, Sinergym would be compatible with both multi-objective algorithms and algorithms that work with a traditional reward value.

We can transform the returned reward into a vector using the following wrapper:

[2]:

env=gym.make('Eplus-5zone-hot-discrete-v1')
env=MultiObjectiveReward(env,reward_terms=['energy_term','comfort_term'])

#==============================================================================================#
[ENVIRONMENT] (INFO) : Creating Gymnasium environment.
[ENVIRONMENT] (INFO) : Name: 5zone-hot-discrete-v1
#==============================================================================================#
[MODELING] (INFO) : Experiment working directory created.
[MODELING] (INFO) : Working directory: /workspaces/sinergym/examples/Eplus-env-5zone-hot-discrete-v1-res1
[MODELING] (INFO) : Model Config is correct.
[MODELING] (INFO) : Update building model Output:Variable with variable names.
[MODELING] (INFO) : Update building model Output:Meter with meter names.
[MODELING] (INFO) : Runperiod established.
[MODELING] (INFO) : Episode length (seconds): 31536000.0
[MODELING] (INFO) : timestep size (seconds): 900.0
[MODELING] (INFO) : timesteps per episode: 35040
[REWARD] (INFO) : Reward function initialized.
[ENVIRONMENT] (INFO) : Environment created successfully.
[WRAPPER DiscretizeEnv] (INFO) : New Discrete Space and mapping: Discrete(10)
[WRAPPER DiscretizeEnv] (INFO) : Make sure that the action space is compatible and contained in the original environment.
[WRAPPER DiscretizeEnv] (INFO) : Wrapper initialized
[WRAPPER MultiObjectiveReward] (INFO) : wrapper initialized.

Ensure that reward_terms are available in the info dict returned in the environment’s step method. Otherwise, an execution error will occur. By default, Sinergym environments return all reward terms specified in the used reward class in the info dict, so if the objective exists in the reward term, you shouldn’t encounter any problems.

[3]:

env.reset()
action = env.action_space.sample()
obs, reward, terminated, truncated, info = env.step(action)
env.close()
print(reward)

#----------------------------------------------------------------------------------------------#
[ENVIRONMENT] (INFO) : Starting a new episode.
[ENVIRONMENT] (INFO) : Episode 1: 5zone-hot-discrete-v1
#----------------------------------------------------------------------------------------------#
[MODELING] (INFO) : Episode directory created.
[MODELING] (INFO) : Weather file USA_AZ_Davis-Monthan.AFB.722745_TMY3.epw used.
[MODELING] (INFO) : Adapting weather to building model.
[ENVIRONMENT] (INFO) : Saving episode output path.
[ENVIRONMENT] (INFO) : Episode 1 started.
[SIMULATOR] (INFO) : handlers initialized.
[SIMULATOR] (INFO) : handlers are ready.
[SIMULATOR] (INFO) : System is ready.

/usr/local/lib/python3.12/dist-packages/gymnasium/spaces/box.py:240: UserWarning: WARN: Casting input x to numpy array.
  gym.logger.warn("Casting input x to numpy array.")

[ENVIRONMENT] (INFO) : Environment closed. [5zone-hot-discrete-v1]
Simulation Progress [Episode 1]: 100%|██████████| 100/100 [00:06<00:00, 21.85%/s, 100% completed][-0.00589738497079933, -0.09533249490526607]

23.2. Previous Observations Wrappers

This Wrapper will add the previous timestep’s observation values to the current environment observation. You can select the variables you want to track its previous observation values. The observation space will be updated with the new dimension.

[4]:

env=gym.make('Eplus-5zone-hot-discrete-v1')
env = PreviousObservationWrapper(env, previous_variables=[
        'htg_setpoint',
        'clg_setpoint',
        'air_temperature'])

#==============================================================================================#
[ENVIRONMENT] (INFO) : Creating Gymnasium environment.
[ENVIRONMENT] (INFO) : Name: 5zone-hot-discrete-v1
Simulation Progress [Episode 1]: 100%|██████████| 100/100 [00:06<00:00, 21.85%/s, 100% completed]#==============================================================================================#
[MODELING] (INFO) : Experiment working directory created.
[MODELING] (INFO) : Working directory: /workspaces/sinergym/examples/Eplus-env-5zone-hot-discrete-v1-res2
[MODELING] (INFO) : Model Config is correct.
[MODELING] (INFO) : Update building model Output:Variable with variable names.
[MODELING] (INFO) : Update building model Output:Meter with meter names.
[MODELING] (INFO) : Runperiod established.
[MODELING] (INFO) : Episode length (seconds): 31536000.0
[MODELING] (INFO) : timestep size (seconds): 900.0
[MODELING] (INFO) : timesteps per episode: 35040
[REWARD] (INFO) : Reward function initialized.
[ENVIRONMENT] (INFO) : Environment created successfully.
[WRAPPER DiscretizeEnv] (INFO) : New Discrete Space and mapping: Discrete(10)
[WRAPPER DiscretizeEnv] (INFO) : Make sure that the action space is compatible and contained in the original environment.
[WRAPPER DiscretizeEnv] (INFO) : Wrapper initialized
Simulation Progress [Episode 1]: 100%|██████████| 100/100 [00:06<00:00, 16.44%/s, 100% completed]
[WRAPPER PreviousObservationWrapper] (INFO) : Wrapper initialized.

You can see that the observation values have been updated:

[5]:

env.reset()
obs,_,_,_,_=env.step(env.action_space.sample())
obs_dict=dict(zip(env.get_wrapper_attr('observation_variables'),obs))
env.close()
print('NEW OBSERVATION: ',obs_dict)

#----------------------------------------------------------------------------------------------#
[ENVIRONMENT] (INFO) : Starting a new episode.
[ENVIRONMENT] (INFO) : Episode 1: 5zone-hot-discrete-v1
#----------------------------------------------------------------------------------------------#
[MODELING] (INFO) : Episode directory created.
[MODELING] (INFO) : Weather file USA_AZ_Davis-Monthan.AFB.722745_TMY3.epw used.
[MODELING] (INFO) : Adapting weather to building model.
[ENVIRONMENT] (INFO) : Saving episode output path.
[ENVIRONMENT] (INFO) : Episode 1 started.
[SIMULATOR] (INFO) : handlers initialized.
[SIMULATOR] (INFO) : handlers are ready.
[SIMULATOR] (INFO) : System is ready.

/usr/local/lib/python3.12/dist-packages/gymnasium/spaces/box.py:240: UserWarning: WARN: Casting input x to numpy array.
  gym.logger.warn("Casting input x to numpy array.")

[ENVIRONMENT] (INFO) : Environment closed. [5zone-hot-discrete-v1]
Simulation Progress [Episode 1]: 100%|██████████| 100/100 [00:05<00:00, 22.84%/s, 100% completed]NEW OBSERVATION:  {'month': np.float32(1.0), 'day_of_month': np.float32(1.0), 'hour': np.float32(0.0), 'outdoor_temperature': np.float32(4.8), 'outdoor_humidity': np.float32(61.0), 'wind_speed': np.float32(4.65), 'wind_direction': np.float32(160.0), 'diffuse_solar_radiation': np.float32(0.0), 'direct_solar_radiation': np.float32(0.0), 'htg_setpoint': np.float32(12.8), 'clg_setpoint': np.float32(40.0), 'air_temperature': np.float32(19.809336), 'air_humidity': np.float32(27.848707), 'people_occupant': np.float32(0.0), 'co2_emission': np.float32(0.0), 'HVAC_electricity_demand_rate': np.float32(117.9477), 'total_electricity_HVAC': np.float32(106152.93), 'htg_setpoint_previous': np.float32(12.8), 'clg_setpoint_previous': np.float32(40.0), 'air_temperature_previous': np.float32(19.95039)}

23.3. Datetime Wrapper

This wrapper will replace the day value with the is_weekend flag, and hour and month with sin and cos values. The observation space is also automatically updated.

[6]:

env=gym.make('Eplus-5zone-hot-discrete-v1')
env = DatetimeWrapper(env)

#==============================================================================================#
[ENVIRONMENT] (INFO) : Creating Gymnasium environment.
[ENVIRONMENT] (INFO) : Name: 5zone-hot-discrete-v1
Simulation Progress [Episode 1]: 100%|██████████| 100/100 [00:05<00:00, 22.84%/s, 100% completed]#==============================================================================================#
[MODELING] (INFO) : Experiment working directory created.
[MODELING] (INFO) : Working directory: /workspaces/sinergym/examples/Eplus-env-5zone-hot-discrete-v1-res3
[MODELING] (INFO) : Model Config is correct.
[MODELING] (INFO) : Update building model Output:Variable with variable names.
[MODELING] (INFO) : Update building model Output:Meter with meter names.
[MODELING] (INFO) : Runperiod established.
[MODELING] (INFO) : Episode length (seconds): 31536000.0
[MODELING] (INFO) : timestep size (seconds): 900.0
[MODELING] (INFO) : timesteps per episode: 35040
[REWARD] (INFO) : Reward function initialized.
[ENVIRONMENT] (INFO) : Environment created successfully.
[WRAPPER DiscretizeEnv] (INFO) : New Discrete Space and mapping: Discrete(10)
[WRAPPER DiscretizeEnv] (INFO) : Make sure that the action space is compatible and contained in the original environment.
[WRAPPER DiscretizeEnv] (INFO) : Wrapper initialized
Simulation Progress [Episode 1]: 100%|██████████| 100/100 [00:05<00:00, 18.53%/s, 100% completed]
[WRAPPER DatetimeWrapper] (INFO) : Wrapper initialized.

Specifically, this wrapper removes the observation variables month, day, and hour, and adds month_sin, month_cos, is_weekend, hour_sin, and hour_cos instead:

[7]:

env.reset()
obs,_,_,_,_=env.step(env.action_space.sample())
obs_dict=dict(zip(env.get_wrapper_attr('observation_variables'),obs))
env.close()
print('NEW OBSERVATION: ',obs_dict)

#----------------------------------------------------------------------------------------------#
[ENVIRONMENT] (INFO) : Starting a new episode.
[ENVIRONMENT] (INFO) : Episode 1: 5zone-hot-discrete-v1
#----------------------------------------------------------------------------------------------#
[MODELING] (INFO) : Episode directory created.
[MODELING] (INFO) : Weather file USA_AZ_Davis-Monthan.AFB.722745_TMY3.epw used.
[MODELING] (INFO) : Adapting weather to building model.
[ENVIRONMENT] (INFO) : Saving episode output path.
[ENVIRONMENT] (INFO) : Episode 1 started.
[SIMULATOR] (INFO) : handlers initialized.
[SIMULATOR] (INFO) : handlers are ready.
[SIMULATOR] (INFO) : System is ready.

/usr/local/lib/python3.12/dist-packages/gymnasium/spaces/box.py:240: UserWarning: WARN: Casting input x to numpy array.
  gym.logger.warn("Casting input x to numpy array.")

[ENVIRONMENT] (INFO) : Environment closed. [5zone-hot-discrete-v1]
Simulation Progress [Episode 1]: 100%|██████████| 100/100 [00:05<00:00, 23.21%/s, 100% completed]NEW OBSERVATION:  {'month_cos': np.float64(1.0), 'month_sin': np.float64(0.0), 'is_weekend': np.float64(0.0), 'hour_cos': np.float64(1.0), 'hour_sin': np.float64(0.0), 'outdoor_temperature': np.float64(4.800000190734863), 'outdoor_humidity': np.float64(61.0), 'wind_speed': np.float64(4.650000095367432), 'wind_direction': np.float64(160.0), 'diffuse_solar_radiation': np.float64(0.0), 'direct_solar_radiation': np.float64(0.0), 'htg_setpoint': np.float64(12.800000190734863), 'clg_setpoint': np.float64(40.0), 'air_temperature': np.float64(19.809335708618164), 'air_humidity': np.float64(27.84870719909668), 'people_occupant': np.float64(0.0), 'co2_emission': np.float64(0.0), 'HVAC_electricity_demand_rate': np.float64(117.94770050048828), 'total_electricity_HVAC': np.float64(106152.9296875)}

23.4. Environment Action Normalization Wrapper

Here’s an example of how to normalize a previous continuous environment action space. If we don’t define the range values, it will default to the range [-1,1]:

[8]:

# We will create a continuous environment
env=gym.make('Eplus-5zone-hot-continuous-v1')
print('ORIGINAL ACTION SPACE: ',env.get_wrapper_attr('action_space'))
# NORMALIZATION
# Apply the normalize action wrapper
env=NormalizeAction(env,normalize_range=(-1.0,1.0))
print('WRAPPED ACTION SPACE: ',env.get_wrapper_attr('action_space'))
env.reset()
for i in range(5):
    action=env.action_space.sample()
    print('Normalized action: ',action)
    _,_,_,_,info=env.step(action)
    print('Action done in simulator: ', info['action'])
env.close()

#==============================================================================================#
[ENVIRONMENT] (INFO) : Creating Gymnasium environment.
[ENVIRONMENT] (INFO) : Name: 5zone-hot-continuous-v1
Simulation Progress [Episode 1]: 100%|██████████| 100/100 [00:05<00:00, 23.21%/s, 100% completed]#==============================================================================================#
[MODELING] (INFO) : Experiment working directory created.
[MODELING] (INFO) : Working directory: /workspaces/sinergym/examples/Eplus-env-5zone-hot-continuous-v1-res2
[MODELING] (INFO) : Model Config is correct.
[MODELING] (INFO) : Update building model Output:Variable with variable names.
[MODELING] (INFO) : Update building model Output:Meter with meter names.
[MODELING] (INFO) : Runperiod established.
[MODELING] (INFO) : Episode length (seconds): 31536000.0
[MODELING] (INFO) : timestep size (seconds): 900.0
[MODELING] (INFO) : timesteps per episode: 35040
[REWARD] (INFO) : Reward function initialized.
[ENVIRONMENT] (INFO) : Environment created successfully.
Simulation Progress [Episode 1]: 100%|██████████| 100/100 [00:05<00:00, 18.69%/s, 100% completed]
ORIGINAL ACTION SPACE:  Box([12.   23.25], [23.25 30.  ], (2,), float32)
[WRAPPER NormalizeAction] (INFO) : New normalized action Space: Box(-1.0, 1.0, (2,), float32)
[WRAPPER NormalizeAction] (INFO) : Wrapper initialized
WRAPPED ACTION SPACE:  Box(-1.0, 1.0, (2,), float32)
#----------------------------------------------------------------------------------------------#
[ENVIRONMENT] (INFO) : Starting a new episode.
[ENVIRONMENT] (INFO) : Episode 1: 5zone-hot-continuous-v1
#----------------------------------------------------------------------------------------------#
[MODELING] (INFO) : Episode directory created.
[MODELING] (INFO) : Weather file USA_AZ_Davis-Monthan.AFB.722745_TMY3.epw used.
[MODELING] (INFO) : Adapting weather to building model.
[ENVIRONMENT] (INFO) : Saving episode output path.
[ENVIRONMENT] (INFO) : Episode 1 started.
[SIMULATOR] (INFO) : handlers initialized.
[SIMULATOR] (INFO) : handlers are ready.
[SIMULATOR] (INFO) : System is ready.
Normalized action:  [-0.01846057  0.9679283 ]
Action done in simulator:  [17.52116  29.891758]
Normalized action:  [-0.09003034  0.29940483]
Action done in simulator:  [17.11858 27.63549]
Normalized action:  [0.7780604  0.41575432]
Action done in simulator:  [22.00159  28.028172]
Normalized action:  [-0.33482978 -0.15819682]
Action done in simulator:  [15.741583 26.091085]
Normalized action:  [ 0.7994888 -0.8743588]
Action done in simulator:  [22.122124 23.67404 ]
[ENVIRONMENT] (INFO) : Environment closed. [5zone-hot-continuous-v1]
Simulation Progress [Episode 1]: 100%|██████████| 100/100 [00:05<00:00, 22.73%/s, 100% completed]

23.5. Environment Discretization Wrapper

Here’s an example of how to discretize a previous continuous environment. We’ll need the new discrete action space and an action mapping function whose output matches the original unwrapped environment action space:

[9]:

# We will create a continuous environment
env=gym.make('Eplus-5zone-hot-continuous-v1')
print('ORIGINAL ACTION SPACE: ',env.get_wrapper_attr('action_space'))
print('IS DISCRETE?: ',env.get_wrapper_attr('is_discrete'))
# DISCRETIZATION
# Defining new discrete space and action mapping function
new_discrete_space = gym.spaces.Discrete(10) # Action values [0,9]
def action_mapping_function(action):
    mapping = {
        0: [15, 30], # These lists matches with original action space
        1: [16, 29],
        2: [17, 28],
        3: [18, 27],
        4: [19, 26],
        5: [20, 25],
        6: [21, 24],
        7: [22, 23],
        8: [22, 22.5],
        9: [21, 22.5]
    }

    return mapping[action]
# Apply the discretize wrapper
env=DiscretizeEnv(env,discrete_space=new_discrete_space,action_mapping=action_mapping_function)
print('WRAPPED ACTION SPACE: ',env.get_wrapper_attr('action_space'))
print('IS DISCRETE?: ',env.get_wrapper_attr('is_discrete'))
env.reset()
for i in range(5):
    action=env.action_space.sample()
    print('ACTION DISCRETE: ',action)
    _,_,_,_,info=env.step(action)
    print('Action done in simulator: ', info['action'])
env.close()

#==============================================================================================#
[ENVIRONMENT] (INFO) : Creating Gymnasium environment.
[ENVIRONMENT] (INFO) : Name: 5zone-hot-continuous-v1
Simulation Progress [Episode 1]: 100%|██████████| 100/100 [00:05<00:00, 22.73%/s, 100% completed]#==============================================================================================#
[MODELING] (INFO) : Experiment working directory created.
[MODELING] (INFO) : Working directory: /workspaces/sinergym/examples/Eplus-env-5zone-hot-continuous-v1-res3
[MODELING] (INFO) : Model Config is correct.
[MODELING] (INFO) : Update building model Output:Variable with variable names.
[MODELING] (INFO) : Update building model Output:Meter with meter names.
[MODELING] (INFO) : Runperiod established.
[MODELING] (INFO) : Episode length (seconds): 31536000.0
[MODELING] (INFO) : timestep size (seconds): 900.0
[MODELING] (INFO) : timesteps per episode: 35040
[REWARD] (INFO) : Reward function initialized.
[ENVIRONMENT] (INFO) : Environment created successfully.
Simulation Progress [Episode 1]: 100%|██████████| 100/100 [00:05<00:00, 17.84%/s, 100% completed]
ORIGINAL ACTION SPACE:  Box([12.   23.25], [23.25 30.  ], (2,), float32)
IS DISCRETE?:  False
[WRAPPER DiscretizeEnv] (INFO) : New Discrete Space and mapping: Discrete(10)
[WRAPPER DiscretizeEnv] (INFO) : Make sure that the action space is compatible and contained in the original environment.
[WRAPPER DiscretizeEnv] (INFO) : Wrapper initialized
WRAPPED ACTION SPACE:  Discrete(10)
IS DISCRETE?:  True
#----------------------------------------------------------------------------------------------#
[ENVIRONMENT] (INFO) : Starting a new episode.
[ENVIRONMENT] (INFO) : Episode 1: 5zone-hot-continuous-v1
#----------------------------------------------------------------------------------------------#
[MODELING] (INFO) : Episode directory created.
[MODELING] (INFO) : Weather file USA_AZ_Davis-Monthan.AFB.722745_TMY3.epw used.
[MODELING] (INFO) : Adapting weather to building model.
[ENVIRONMENT] (INFO) : Saving episode output path.
[ENVIRONMENT] (INFO) : Episode 1 started.
[SIMULATOR] (INFO) : handlers initialized.
[SIMULATOR] (INFO) : handlers are ready.
[SIMULATOR] (INFO) : System is ready.
ACTION DISCRETE:  0
Action done in simulator:  [15, 30]
ACTION DISCRETE:  7
[ENVIRONMENT] (WARNING) : Step: The action [22, 23] is not correct for the Action Space Box([12.   23.25], [23.25 30.  ], (2,), float32)
Action done in simulator:  [22, 23]
ACTION DISCRETE:  9
[ENVIRONMENT] (WARNING) : Step: The action [21, 22.5] is not correct for the Action Space Box([12.   23.25], [23.25 30.  ], (2,), float32)

/usr/local/lib/python3.12/dist-packages/gymnasium/spaces/box.py:240: UserWarning: WARN: Casting input x to numpy array.
  gym.logger.warn("Casting input x to numpy array.")

Action done in simulator:  [21, 22.5]
ACTION DISCRETE:  6
Action done in simulator:  [21, 24]
ACTION DISCRETE:  4
Action done in simulator:  [19, 26]
[ENVIRONMENT] (INFO) : Environment closed. [5zone-hot-continuous-v1]
Simulation Progress [Episode 1]: 100%|██████████| 100/100 [00:05<00:00, 23.09%/s, 100% completed]

As seen in the output, the input is an int, but a list with the original action space is created in the simulator.

23.6. Discrete Incremental Wrapper

This wrapper updates an environment for an incremental setpoint action space. It converts the environment into a discrete environment with an action mapping function and action space depending on the step and delta specified. The action is added to the current setpoint values instead of overwriting the latest action. Thus, the action is the current setpoint values with the increase, not the discrete value action that defines the increment/decrement itself.

[10]:

env=gym.make('Eplus-5zone-hot-continuous-v1')
print('ORIGINAL ACTION SPACE: ',env.get_wrapper_attr('action_space'))
env = DiscreteIncrementalWrapper(
        env,initial_values=[21.0,25.0], delta_temp=2, step_temp=0.5)
print('WRAPPED ACTION SPACE: ',env.get_wrapper_attr('action_space'))
print('WRAPPED ACTION MAPPING: ',env.get_wrapper_attr('action_mapping'))

#==============================================================================================#
[ENVIRONMENT] (INFO) : Creating Gymnasium environment.
[ENVIRONMENT] (INFO) : Name: 5zone-hot-continuous-v1
Simulation Progress [Episode 1]: 100%|██████████| 100/100 [00:05<00:00, 23.09%/s, 100% completed]#==============================================================================================#
[MODELING] (INFO) : Experiment working directory created.
[MODELING] (INFO) : Working directory: /workspaces/sinergym/examples/Eplus-env-5zone-hot-continuous-v1-res4
[MODELING] (INFO) : Model Config is correct.
[MODELING] (INFO) : Update building model Output:Variable with variable names.
[MODELING] (INFO) : Update building model Output:Meter with meter names.
[MODELING] (INFO) : Runperiod established.
[MODELING] (INFO) : Episode length (seconds): 31536000.0
[MODELING] (INFO) : timestep size (seconds): 900.0
[MODELING] (INFO) : timesteps per episode: 35040
[REWARD] (INFO) : Reward function initialized.
[ENVIRONMENT] (INFO) : Environment created successfully.
Simulation Progress [Episode 1]: 100%|██████████| 100/100 [00:05<00:00, 18.80%/s, 100% completed]
ORIGINAL ACTION SPACE:  Box([12.   23.25], [23.25 30.  ], (2,), float32)
[WRAPPER DiscreteIncrementalWrapper] (INFO) : New incremental action mapping: 17
[WRAPPER DiscreteIncrementalWrapper] (INFO) : {0: [0.0, 0.0], 1: [np.float64(0.5), 0.0], 2: [np.float64(1.0), 0.0], 3: [np.float64(1.5), 0.0], 4: [np.float64(2.0), 0.0], 5: [np.float64(-0.5), 0.0], 6: [np.float64(-1.0), 0.0], 7: [np.float64(-1.5), 0.0], 8: [np.float64(-2.0), 0.0], 9: [0.0, np.float64(0.5)], 10: [0.0, np.float64(1.0)], 11: [0.0, np.float64(1.5)], 12: [0.0, np.float64(2.0)], 13: [0.0, np.float64(-0.5)], 14: [0.0, np.float64(-1.0)], 15: [0.0, np.float64(-1.5)], 16: [0.0, np.float64(-2.0)]}
[WRAPPER DiscreteIncrementalWrapper] (INFO) : Wrapper initialized
WRAPPED ACTION SPACE:  Discrete(17)
WRAPPED ACTION MAPPING:  <bound method DiscreteIncrementalWrapper.action_mapping of <DiscreteIncrementalWrapper<OrderEnforcing<PassiveEnvChecker<EplusEnv<Eplus-5zone-hot-continuous-v1>>>>>>

The maximum and minimum values for creating the action mapping are read from the environment action space, ensuring that the setpoint increments and decrements do not exceed the agreed limits. The delta and step values are used to determine how the discrete space of these increments and decrements will be constructed. Here’s an example of how it works:

[11]:

env.reset()
print('CURRENT SETPOINTS VALUES: ', env.get_wrapper_attr('current_setpoints'))
for i in range(5):
    action=env.action_space.sample()
    _,_,_,_,info=env.step(action)
    print('Action number ',i,': ',env.get_wrapper_attr('action_mapping')(action))
    print('Setpoints update: ', info['action'])
env.close()

#----------------------------------------------------------------------------------------------#
[ENVIRONMENT] (INFO) : Starting a new episode.
[ENVIRONMENT] (INFO) : Episode 1: 5zone-hot-continuous-v1
#----------------------------------------------------------------------------------------------#
[MODELING] (INFO) : Episode directory created.
[MODELING] (INFO) : Weather file USA_AZ_Davis-Monthan.AFB.722745_TMY3.epw used.
[MODELING] (INFO) : Adapting weather to building model.
[ENVIRONMENT] (INFO) : Saving episode output path.
[ENVIRONMENT] (INFO) : Episode 1 started.
[SIMULATOR] (INFO) : handlers initialized.
[SIMULATOR] (INFO) : handlers are ready.
[SIMULATOR] (INFO) : System is ready.

/usr/local/lib/python3.12/dist-packages/gymnasium/spaces/box.py:240: UserWarning: WARN: Casting input x to numpy array.
  gym.logger.warn("Casting input x to numpy array.")

CURRENT SETPOINTS VALUES:  [21.0, 25.0]
Action number  0 :  [0.0, 0.0]
Setpoints update:  [np.float64(21.0), np.float64(25.0)]
Action number  1 :  [0.0, np.float64(0.5)]
Setpoints update:  [np.float64(21.0), np.float64(25.5)]
Action number  2 :  [0.0, 0.0]
Setpoints update:  [np.float64(21.0), np.float64(25.5)]
Action number  3 :  [0.0, np.float64(-2.0)]
Setpoints update:  [np.float64(21.0), np.float64(23.5)]
Action number  4 :  [np.float64(-0.5), 0.0]
Setpoints update:  [np.float64(20.5), np.float64(23.5)]
[ENVIRONMENT] (INFO) : Environment closed. [5zone-hot-continuous-v1]
Simulation Progress [Episode 1]: 100%|██████████| 100/100 [00:05<00:00, 21.37%/s, 100% completed]

23.7. Normalization Wrapper

This wrapper is used to transform the observation received from the simulator into values between -1 and 1. It’s based on the dynamic normalization wrapper of Gymnasium. Initially, it may not be precise, and the values may often be out of range, so use this wrapper with caution.

[12]:

#Original env
env=gym.make('Eplus-5zone-hot-discrete-v1')
env = NormalizeObservation(
        env=env)

#==============================================================================================#
[ENVIRONMENT] (INFO) : Creating Gymnasium environment.
[ENVIRONMENT] (INFO) : Name: 5zone-hot-discrete-v1
Simulation Progress [Episode 1]: 100%|██████████| 100/100 [00:05<00:00, 21.37%/s, 100% completed]#==============================================================================================#
[MODELING] (INFO) : Experiment working directory created.
[MODELING] (INFO) : Working directory: /workspaces/sinergym/examples/Eplus-env-5zone-hot-discrete-v1-res4
[MODELING] (INFO) : Model Config is correct.
[MODELING] (INFO) : Update building model Output:Variable with variable names.
[MODELING] (INFO) : Update building model Output:Meter with meter names.
[MODELING] (INFO) : Runperiod established.
[MODELING] (INFO) : Episode length (seconds): 31536000.0
[MODELING] (INFO) : timestep size (seconds): 900.0
[MODELING] (INFO) : timesteps per episode: 35040
[REWARD] (INFO) : Reward function initialized.
[ENVIRONMENT] (INFO) : Environment created successfully.
[WRAPPER DiscretizeEnv] (INFO) : New Discrete Space and mapping: Discrete(10)
[WRAPPER DiscretizeEnv] (INFO) : Make sure that the action space is compatible and contained in the original environment.
[WRAPPER DiscretizeEnv] (INFO) : Wrapper initialized
Simulation Progress [Episode 1]: 100%|██████████| 100/100 [00:05<00:00, 17.29%/s, 100% completed]
[WRAPPER NormalizeObservation] (INFO) : Wrapper initialized.

In the following code, you can see how the specified variables have been correctly normalized:

[13]:

env.reset()
obs,_,_,_,_=env.step(env.action_space.sample())
obs_dict=dict(zip(env.get_wrapper_attr('observation_variables'),obs))
env.close()
print('OBSERVATION WITH NORMALIZATION: ',obs_dict)

#----------------------------------------------------------------------------------------------#
[ENVIRONMENT] (INFO) : Starting a new episode.
[ENVIRONMENT] (INFO) : Episode 1: 5zone-hot-discrete-v1
#----------------------------------------------------------------------------------------------#
[MODELING] (INFO) : Episode directory created.
[MODELING] (INFO) : Weather file USA_AZ_Davis-Monthan.AFB.722745_TMY3.epw used.
[MODELING] (INFO) : Adapting weather to building model.
[ENVIRONMENT] (INFO) : Saving episode output path.
[ENVIRONMENT] (INFO) : Episode 1 started.
[SIMULATOR] (INFO) : handlers initialized.
[SIMULATOR] (INFO) : handlers are ready.
[SIMULATOR] (INFO) : System is ready.
[WRAPPER NormalizeObservation] (INFO) : Saving normalization calibration data.

/usr/local/lib/python3.12/dist-packages/gymnasium/spaces/box.py:240: UserWarning: WARN: Casting input x to numpy array.
  gym.logger.warn("Casting input x to numpy array.")

[ENVIRONMENT] (INFO) : Environment closed. [5zone-hot-discrete-v1]
[WRAPPER NormalizeObservation] (INFO) : Saving normalization calibration data.
Simulation Progress [Episode 1]: 100%|██████████| 100/100 [00:06<00:00, 19.93%/s, 100% completed]OBSERVATION WITH NORMALIZATION:  {'month': np.float64(0.00499968750430059), 'day_of_month': np.float64(0.00499968750430059), 'hour': np.float64(0.0), 'outdoor_temperature': np.float64(0.9875907976185373), 'outdoor_humidity': np.float64(-0.9745626607331893), 'wind_speed': np.float64(0.9973969815420386), 'wind_direction': np.float64(0.9908525387095581), 'diffuse_solar_radiation': np.float64(0.0), 'direct_solar_radiation': np.float64(0.0), 'htg_setpoint': np.float64(0.007049581561774207), 'clg_setpoint': np.float64(0.0070688585864955075), 'air_temperature': np.float64(-0.44169349819860654), 'air_humidity': np.float64(0.1687716004048815), 'people_occupant': np.float64(0.0), 'co2_emission': np.float64(0.0), 'HVAC_electricity_demand_rate': np.float64(0.007070813620268407), 'total_electricity_HVAC': np.float64(0.0070710678115464875)}

23.8. Logging and storing data with logger wrappers

23.8.1. LoggerWrapper layer

This wrapper uses Sinergym’s logger storage class to capture the interaction flow with the environment (LoggerStorage), accumulating all the information. The class used by the wrapper can be replaced with a different back-end. It can then be combined with various wrappers to output the stored data, such as CSVLogger or WandBLogger. For more information about the Sinergym Logger, visit Logging System Overview, Logger Wrappers and an example about custom loggers.

[14]:

env=gym.make('Eplus-5zone-hot-discrete-v1')
env=LoggerWrapper(env, storage_class=LoggerStorage)

#==============================================================================================#
[ENVIRONMENT] (INFO) : Creating Gymnasium environment.
[ENVIRONMENT] (INFO) : Name: 5zone-hot-discrete-v1
Simulation Progress [Episode 1]: 100%|██████████| 100/100 [00:06<00:00, 19.93%/s, 100% completed]#==============================================================================================#
[MODELING] (INFO) : Experiment working directory created.
[MODELING] (INFO) : Working directory: /workspaces/sinergym/examples/Eplus-env-5zone-hot-discrete-v1-res5
[MODELING] (INFO) : Model Config is correct.
[MODELING] (INFO) : Update building model Output:Variable with variable names.
[MODELING] (INFO) : Update building model Output:Meter with meter names.
[MODELING] (INFO) : Runperiod established.
[MODELING] (INFO) : Episode length (seconds): 31536000.0
[MODELING] (INFO) : timestep size (seconds): 900.0
[MODELING] (INFO) : timesteps per episode: 35040
[REWARD] (INFO) : Reward function initialized.
[ENVIRONMENT] (INFO) : Environment created successfully.
[WRAPPER DiscretizeEnv] (INFO) : New Discrete Space and mapping: Discrete(10)
[WRAPPER DiscretizeEnv] (INFO) : Make sure that the action space is compatible and contained in the original environment.
[WRAPPER DiscretizeEnv] (INFO) : Wrapper initialized
Simulation Progress [Episode 1]: 100%|██████████| 100/100 [00:06<00:00, 15.43%/s, 100% completed]
[WRAPPER LoggerWrapper] (INFO) : Wrapper initialized.

This wrapper enables the use of a LoggerStorage instance within the environment class and automatically captures interaction data while actions are sent by an agent. At each reset, the data from this class is cleared to start the next episode. The idea is to combine it with other output loggers like those listed below:

23.8.2. LoggerCSV layer

[15]:

env=CSVLogger(env)

env.reset()
truncated = terminated = False
current_month = 0
while not (terminated or truncated):
    a = env.action_space.sample()
    _,_,terminated,truncated,_=env.step(a)
env.close()

[WRAPPER CSVLogger] (INFO) : Wrapper initialized.
#----------------------------------------------------------------------------------------------#
[ENVIRONMENT] (INFO) : Starting a new episode.
[ENVIRONMENT] (INFO) : Episode 1: 5zone-hot-discrete-v1
#----------------------------------------------------------------------------------------------#
[MODELING] (INFO) : Episode directory created.
[MODELING] (INFO) : Weather file USA_AZ_Davis-Monthan.AFB.722745_TMY3.epw used.
[MODELING] (INFO) : Adapting weather to building model.
[ENVIRONMENT] (INFO) : Saving episode output path.
[ENVIRONMENT] (INFO) : Episode 1 started.
[SIMULATOR] (INFO) : handlers initialized.
[SIMULATOR] (INFO) : handlers are ready.
[SIMULATOR] (INFO) : System is ready.

/usr/local/lib/python3.12/dist-packages/gymnasium/spaces/box.py:240: UserWarning: WARN: Casting input x to numpy array.
  gym.logger.warn("Casting input x to numpy array.")

[WRAPPER CSVLogger] (INFO) : Environment closed, data updated in monitor and progress.csv.
Simulation Progress [Episode 1]: 100%|██████████| 100/100 [00:17<00:00,  5.58%/s, 100% completed]
[ENVIRONMENT] (INFO) : Environment closed. [5zone-hot-discrete-v1]

Once the LoggerWrapper is applied, this wrapper can be used to output episode data through Sinergym’s output, along with summary metrics added to CSV files. More details on this structure can be found in OutputFormat. Sinergym will raise an error if this wrapper is used without first enabling LoggerWrapper or a custom logger.

23.8.3. WandBLogger layer

[16]:

# env=WandBLogger(env = Env,
#                 entity = <wandb_account_entity>,
#                 project_name = <wandb_project_name>,
#                 run_name = <run_name>
#                 group = 'Notebook_example',
#                 tags: ['tag1','tag2'],
#                 save_code = False,
#                 dump_frequency = 1000,
#                 artifact_save = True,
#                 artifact_type = 'output',
#                 excluded_info_keys = ['reward',
#                                   'action',
#                                   'timestep',
#                                   'month',
#                                   'day',
#                                   'hour',
#                                   'time_elapsed(hours)',
#                                   'reward_weight',
#                                   'is_raining'],
#                 excluded_episode_summary_keys = ['terminated',
#                                              'truncated']):

# env.reset()
# truncated = terminated = False
# current_month = 0
# while not (terminated or truncated):
#     a = env.action_space.sample()
#     _,_,terminated,truncated,_=env.step(a)
# env.close()

Similar to CSVLogger, this wrapper requires the environment to have been previously encapsulated by a LoggerWrapper or any custom logger. The user must have a pre-existing Weights & Biases account and correctly specify the fields. This wrapper does not override CSVLogger; both can be applied simultaneously without issue.

23.9. Multi Observation Wrapper

This wrapper stacks the observations received in a history queue (the size can be customized).

[17]:

#Original environment
env=gym.make('Eplus-5zone-hot-discrete-v1')
obs, info=env.reset()
print('BEFORE MULTI OBSERVATION: ',obs)

#Multi Observation environment
env=MultiObsWrapper(env, n=5, flatten=True)
obs, info=env.reset()

#==============================================================================================#
[ENVIRONMENT] (INFO) : Creating Gymnasium environment.
[ENVIRONMENT] (INFO) : Name: 5zone-hot-discrete-v1
#==============================================================================================#
[MODELING] (INFO) : Experiment working directory created.
[MODELING] (INFO) : Working directory: /workspaces/sinergym/examples/Eplus-env-5zone-hot-discrete-v1-res6
[MODELING] (INFO) : Model Config is correct.
[MODELING] (INFO) : Update building model Output:Variable with variable names.
[MODELING] (INFO) : Update building model Output:Meter with meter names.
[MODELING] (INFO) : Runperiod established.
[MODELING] (INFO) : Episode length (seconds): 31536000.0
[MODELING] (INFO) : timestep size (seconds): 900.0
[MODELING] (INFO) : timesteps per episode: 35040
[REWARD] (INFO) : Reward function initialized.
[ENVIRONMENT] (INFO) : Environment created successfully.
[WRAPPER DiscretizeEnv] (INFO) : New Discrete Space and mapping: Discrete(10)
[WRAPPER DiscretizeEnv] (INFO) : Make sure that the action space is compatible and contained in the original environment.
[WRAPPER DiscretizeEnv] (INFO) : Wrapper initialized
#----------------------------------------------------------------------------------------------#
[ENVIRONMENT] (INFO) : Starting a new episode.
[ENVIRONMENT] (INFO) : Episode 1: 5zone-hot-discrete-v1
#----------------------------------------------------------------------------------------------#
[MODELING] (INFO) : Episode directory created.
[MODELING] (INFO) : Weather file USA_AZ_Davis-Monthan.AFB.722745_TMY3.epw used.
[MODELING] (INFO) : Adapting weather to building model.
[ENVIRONMENT] (INFO) : Saving episode output path.
[ENVIRONMENT] (INFO) : Episode 1 started.
[SIMULATOR] (INFO) : handlers initialized.
[SIMULATOR] (INFO) : handlers are ready.
[SIMULATOR] (INFO) : System is ready.
BEFORE MULTI OBSERVATION:  [1.0000000e+00 1.0000000e+00 0.0000000e+00 4.4000001e+00 6.5000000e+01
 3.8750000e+00 1.4500000e+02 0.0000000e+00 0.0000000e+00 1.2800000e+01
 4.0000000e+01 1.9950390e+01 2.7784170e+01 0.0000000e+00 0.0000000e+00
 1.1794770e+02 1.0615293e+05]
[WRAPPER MultiObsWrapper] (INFO) : Wrapper initialized.
Simulation Progress [Episode 1]: 100%|██████████| 100/100 [00:06<00:00, 21.29%/s, 100% completed]#----------------------------------------------------------------------------------------------#
[ENVIRONMENT] (INFO) : Starting a new episode.
[ENVIRONMENT] (INFO) : Episode 2: 5zone-hot-discrete-v1
Simulation Progress [Episode 1]: 100%|██████████| 100/100 [00:06<00:00, 21.29%/s, 100% completed]#----------------------------------------------------------------------------------------------#
[MODELING] (INFO) : Episode directory created.
[MODELING] (INFO) : Weather file USA_AZ_Davis-Monthan.AFB.722745_TMY3.epw used.
[MODELING] (INFO) : Adapting weather to building model.
[ENVIRONMENT] (INFO) : Saving episode output path.
Simulation Progress [Episode 1]: 100%|██████████| 100/100 [00:06<00:00, 15.82%/s, 100% completed]
[ENVIRONMENT] (INFO) : Episode 2 started.

[18]:

print('AFTER MULTI OBSERVATION: ',obs)
env.close()

AFTER MULTI OBSERVATION:  [1.0000000e+00 1.0000000e+00 0.0000000e+00 5.1999998e+00 5.7000000e+01
 5.4250002e+00 1.7500000e+02 0.0000000e+00 0.0000000e+00 1.2800000e+01
 4.0000000e+01 1.9671045e+01 2.7883675e+01 0.0000000e+00 0.0000000e+00
 1.1794770e+02 1.0615293e+05 1.0000000e+00 1.0000000e+00 0.0000000e+00
 5.1999998e+00 5.7000000e+01 5.4250002e+00 1.7500000e+02 0.0000000e+00
 0.0000000e+00 1.2800000e+01 4.0000000e+01 1.9671045e+01 2.7883675e+01
 0.0000000e+00 0.0000000e+00 1.1794770e+02 1.0615293e+05 1.0000000e+00
 1.0000000e+00 0.0000000e+00 5.1999998e+00 5.7000000e+01 5.4250002e+00
 1.7500000e+02 0.0000000e+00 0.0000000e+00 1.2800000e+01 4.0000000e+01
 1.9671045e+01 2.7883675e+01 0.0000000e+00 0.0000000e+00 1.1794770e+02
 1.0615293e+05 1.0000000e+00 1.0000000e+00 0.0000000e+00 5.1999998e+00
 5.7000000e+01 5.4250002e+00 1.7500000e+02 0.0000000e+00 0.0000000e+00
 1.2800000e+01 4.0000000e+01 1.9671045e+01 2.7883675e+01 0.0000000e+00
 0.0000000e+00 1.1794770e+02 1.0615293e+05 1.0000000e+00 1.0000000e+00
 0.0000000e+00 5.1999998e+00 5.7000000e+01 5.4250002e+00 1.7500000e+02
 0.0000000e+00 0.0000000e+00 1.2800000e+01 4.0000000e+01 1.9671045e+01
 2.7883675e+01 0.0000000e+00 0.0000000e+00 1.1794770e+02 1.0615293e+05]
[ENVIRONMENT] (INFO) : Environment closed. [5zone-hot-discrete-v1]

23.10. Nesting wrappers

All wrappers available in Sinergym are stackable, organized in layers. However, the order in which these layers are applied can affect the final result, depending on the wrappers being used.

For instance, activating the logger before normalizing differs from doing it in the reverse order. In the first case, the data will be logged without normalization, even though the agent will operate in a normalized environment. In the second case, the logger will capture the normalized values since it encapsulates the normalization applied by the previous layer.

An example of how to nest wrappers is shown below:

[19]:

env = gym.make('Eplus-5zone-hot-continuous-v1')
env = MultiObjectiveReward(
    env=env,
    reward_terms=[
        'energy_term',
        'comfort_term'])
env = PreviousObservationWrapper(env, previous_variables=[
    'htg_setpoint',
    'clg_setpoint',
    'air_temperature'])
env = DatetimeWrapper(env)
env = DiscreteIncrementalWrapper(
    env,initial_values=[21.0,25.0], delta_temp=2, step_temp=0.5)
env = NormalizeObservation(
    env=env)
env = LoggerWrapper(env=env)
env = MultiObsWrapper(env=env, n=5, flatten=True)

#==============================================================================================#
[ENVIRONMENT] (INFO) : Creating Gymnasium environment.
[ENVIRONMENT] (INFO) : Name: 5zone-hot-continuous-v1
#==============================================================================================#
[MODELING] (INFO) : Experiment working directory created.
[MODELING] (INFO) : Working directory: /workspaces/sinergym/examples/Eplus-env-5zone-hot-continuous-v1-res5
[MODELING] (INFO) : Model Config is correct.
[MODELING] (INFO) : Update building model Output:Variable with variable names.
[MODELING] (INFO) : Update building model Output:Meter with meter names.
[MODELING] (INFO) : Runperiod established.
[MODELING] (INFO) : Episode length (seconds): 31536000.0
[MODELING] (INFO) : timestep size (seconds): 900.0
[MODELING] (INFO) : timesteps per episode: 35040
[REWARD] (INFO) : Reward function initialized.
[ENVIRONMENT] (INFO) : Environment created successfully.
[WRAPPER MultiObjectiveReward] (INFO) : wrapper initialized.
[WRAPPER PreviousObservationWrapper] (INFO) : Wrapper initialized.
[WRAPPER DatetimeWrapper] (INFO) : Wrapper initialized.
[WRAPPER DiscreteIncrementalWrapper] (INFO) : New incremental action mapping: 17
[WRAPPER DiscreteIncrementalWrapper] (INFO) : {0: [0.0, 0.0], 1: [np.float64(0.5), 0.0], 2: [np.float64(1.0), 0.0], 3: [np.float64(1.5), 0.0], 4: [np.float64(2.0), 0.0], 5: [np.float64(-0.5), 0.0], 6: [np.float64(-1.0), 0.0], 7: [np.float64(-1.5), 0.0], 8: [np.float64(-2.0), 0.0], 9: [0.0, np.float64(0.5)], 10: [0.0, np.float64(1.0)], 11: [0.0, np.float64(1.5)], 12: [0.0, np.float64(2.0)], 13: [0.0, np.float64(-0.5)], 14: [0.0, np.float64(-1.0)], 15: [0.0, np.float64(-1.5)], 16: [0.0, np.float64(-2.0)]}
[WRAPPER DiscreteIncrementalWrapper] (INFO) : Wrapper initialized
[WRAPPER NormalizeObservation] (INFO) : Wrapper initialized.
[WRAPPER LoggerWrapper] (INFO) : Wrapper initialized.
[WRAPPER MultiObsWrapper] (INFO) : Wrapper initialized.

Now we simply use the environment with the wrappers, for example:

[20]:

for i in range(1):
    obs, info = env.reset()
    truncated = terminated = False
    current_month = 0
    while not (terminated or truncated):
        a = env.action_space.sample()
        obs, reward, terminated, truncated, info = env.step(a)
        if info['month'] != current_month:  # display results every month
            current_month = info['month']
            print('Reward: ', reward, info)
env.close()

#----------------------------------------------------------------------------------------------#
[ENVIRONMENT] (INFO) : Starting a new episode.
[ENVIRONMENT] (INFO) : Episode 1: 5zone-hot-continuous-v1
#----------------------------------------------------------------------------------------------#
[MODELING] (INFO) : Episode directory created.
[MODELING] (INFO) : Weather file USA_AZ_Davis-Monthan.AFB.722745_TMY3.epw used.
[MODELING] (INFO) : Adapting weather to building model.
[ENVIRONMENT] (INFO) : Saving episode output path.
[ENVIRONMENT] (INFO) : Episode 1 started.
[SIMULATOR] (INFO) : handlers initialized.
[SIMULATOR] (INFO) : handlers are ready.
[SIMULATOR] (INFO) : System is ready.
[WRAPPER NormalizeObservation] (INFO) : Saving normalization calibration data.
Reward:  [-0.00589738497079933, -0.09533249490526607] {'time_elapsed(hours)': 0.5, 'month': 1, 'day': 1, 'hour': 0, 'is_raining': False, 'action': [np.float64(21.0), np.float64(24.5)], 'timestep': 1, 'reward': -0.10122987987606541, 'energy_term': -0.00589738497079933, 'comfort_term': -0.09533249490526607, 'reward_weight': 0.5, 'abs_energy_penalty': -117.9476994159866, 'abs_comfort_penalty': -0.19066498981053215, 'total_power_demand': 117.9476994159866, 'total_temperature_violation': 0.19066498981053215}

/usr/local/lib/python3.12/dist-packages/gymnasium/spaces/box.py:240: UserWarning: WARN: Casting input x to numpy array.
  gym.logger.warn("Casting input x to numpy array.")

Simulation Progress [Episode 1]:  10%|█         | 10/100 [00:01<00:18,  4.85%/s, 10% completed] Reward:  [-0.00589738497079933, 0.0] {'time_elapsed(hours)': 744.375, 'month': 2, 'day': 1, 'hour': 0, 'is_raining': False, 'action': [np.float64(18.25), np.float64(23.5)], 'timestep': 2976, 'reward': -0.00589738497079933, 'energy_term': -0.00589738497079933, 'comfort_term': 0.0, 'reward_weight': 0.5, 'abs_energy_penalty': -117.9476994159866, 'abs_comfort_penalty': 0, 'total_power_demand': 117.9476994159866, 'total_temperature_violation': 0.0}
Simulation Progress [Episode 1]:  17%|█▋        | 17/100 [00:03<00:14,  5.90%/s, 17% completed]Reward:  [-0.00589738497079933, 0.0] {'time_elapsed(hours)': 1416.25, 'month': 3, 'day': 1, 'hour': 0, 'is_raining': False, 'action': [np.float64(13.5), np.float64(23.25)], 'timestep': 5664, 'reward': -0.00589738497079933, 'energy_term': -0.00589738497079933, 'comfort_term': 0.0, 'reward_weight': 0.5, 'abs_energy_penalty': -117.9476994159866, 'abs_comfort_penalty': 0, 'total_power_demand': 117.9476994159866, 'total_temperature_violation': 0.0}
Simulation Progress [Episode 1]:  26%|██▌       | 26/100 [00:04<00:12,  6.13%/s, 26% completed]Reward:  [-0.1517230445551448, 0.0] {'time_elapsed(hours)': 2160.25, 'month': 4, 'day': 1, 'hour': 0, 'is_raining': False, 'action': [np.float64(22.25), np.float64(23.75)], 'timestep': 8640, 'reward': -0.1517230445551448, 'energy_term': -0.1517230445551448, 'comfort_term': 0.0, 'reward_weight': 0.5, 'abs_energy_penalty': -3034.460891102896, 'abs_comfort_penalty': 0, 'total_power_demand': 3034.460891102896, 'total_temperature_violation': 0.0}
Simulation Progress [Episode 1]:  34%|███▍      | 34/100 [00:06<00:12,  5.11%/s, 34% completed]Reward:  [-0.00986094443919355, 0.0] {'time_elapsed(hours)': 2880.25, 'month': 5, 'day': 1, 'hour': 0, 'is_raining': False, 'action': [np.float64(17.0), np.float64(30.0)], 'timestep': 11520, 'reward': -0.00986094443919355, 'energy_term': -0.00986094443919355, 'comfort_term': 0.0, 'reward_weight': 0.5, 'abs_energy_penalty': -197.218888783871, 'abs_comfort_penalty': 0, 'total_power_demand': 197.218888783871, 'total_temperature_violation': 0.0}
Simulation Progress [Episode 1]:  42%|████▏     | 42/100 [00:07<00:10,  5.49%/s, 42% completed]Reward:  [-0.041639635243045196, 0.0] {'time_elapsed(hours)': 3624.25, 'month': 6, 'day': 1, 'hour': 0, 'is_raining': False, 'action': [np.float64(17.75), np.float64(29.0)], 'timestep': 14496, 'reward': -0.041639635243045196, 'energy_term': -0.041639635243045196, 'comfort_term': 0.0, 'reward_weight': 0.5, 'abs_energy_penalty': -832.7927048609039, 'abs_comfort_penalty': 0, 'total_power_demand': 832.7927048609039, 'total_temperature_violation': 0.0}
Simulation Progress [Episode 1]:  51%|█████     | 51/100 [00:09<00:09,  5.03%/s, 51% completed]Reward:  [-0.03291880417306371, -0.05343972414400788] {'time_elapsed(hours)': 4344.25, 'month': 7, 'day': 1, 'hour': 0, 'is_raining': False, 'action': [np.float64(12.0), np.float64(25.25)], 'timestep': 17376, 'reward': -0.0863585283170716, 'energy_term': -0.03291880417306371, 'comfort_term': -0.05343972414400788, 'reward_weight': 0.5, 'abs_energy_penalty': -658.3760834612742, 'abs_comfort_penalty': -0.10687944828801577, 'total_power_demand': 658.3760834612742, 'total_temperature_violation': 0.10687944828801577}
Simulation Progress [Episode 1]:  59%|█████▉    | 59/100 [00:10<00:07,  5.52%/s, 59% completed]Reward:  [-0.039679044554484136, -0.2110973093444386] {'time_elapsed(hours)': 5088.25, 'month': 8, 'day': 1, 'hour': 0, 'is_raining': False, 'action': [np.float64(17.0), np.float64(24.75)], 'timestep': 20352, 'reward': -0.2507763538989227, 'energy_term': -0.039679044554484136, 'comfort_term': -0.2110973093444386, 'reward_weight': 0.5, 'abs_energy_penalty': -793.5808910896826, 'abs_comfort_penalty': -0.4221946186888772, 'total_power_demand': 793.5808910896826, 'total_temperature_violation': 0.4221946186888772}
Simulation Progress [Episode 1]:  67%|██████▋   | 67/100 [00:12<00:05,  5.62%/s, 67% completed]Reward:  [-0.031145555987538456, -0.10638397813522893] {'time_elapsed(hours)': 5832.25, 'month': 9, 'day': 1, 'hour': 0, 'is_raining': False, 'action': [np.float64(14.5), np.float64(23.75)], 'timestep': 23328, 'reward': -0.1375295341227674, 'energy_term': -0.031145555987538456, 'comfort_term': -0.10638397813522893, 'reward_weight': 0.5, 'abs_energy_penalty': -622.9111197507691, 'abs_comfort_penalty': -0.21276795627045786, 'total_power_demand': 622.9111197507691, 'total_temperature_violation': 0.21276795627045786}
Simulation Progress [Episode 1]:  76%|███████▌  | 76/100 [00:13<00:04,  5.61%/s, 76% completed]Reward:  [-0.034473965270484005, 0.0] {'time_elapsed(hours)': 6552.25, 'month': 10, 'day': 1, 'hour': 0, 'is_raining': False, 'action': [np.float64(14.25), np.float64(26.25)], 'timestep': 26208, 'reward': -0.034473965270484005, 'energy_term': -0.034473965270484005, 'comfort_term': 0.0, 'reward_weight': 0.5, 'abs_energy_penalty': -689.4793054096801, 'abs_comfort_penalty': 0, 'total_power_demand': 689.4793054096801, 'total_temperature_violation': 0.0}
Simulation Progress [Episode 1]:  84%|████████▍ | 84/100 [00:15<00:02,  6.13%/s, 84% completed]Reward:  [-0.00589738497079933, 0.0] {'time_elapsed(hours)': 7296.25, 'month': 11, 'day': 1, 'hour': 0, 'is_raining': False, 'action': [np.float64(17.0), np.float64(27.25)], 'timestep': 29184, 'reward': -0.00589738497079933, 'energy_term': -0.00589738497079933, 'comfort_term': 0.0, 'reward_weight': 0.5, 'abs_energy_penalty': -117.9476994159866, 'abs_comfort_penalty': 0, 'total_power_demand': 117.9476994159866, 'total_temperature_violation': 0.0}
Simulation Progress [Episode 1]:  92%|█████████▏| 92/100 [00:16<00:01,  5.72%/s, 92% completed]Reward:  [-0.00589738497079933, 0.0] {'time_elapsed(hours)': 8016.25, 'month': 12, 'day': 1, 'hour': 0, 'is_raining': False, 'action': [np.float64(14.75), np.float64(23.5)], 'timestep': 32064, 'reward': -0.00589738497079933, 'energy_term': -0.00589738497079933, 'comfort_term': 0.0, 'reward_weight': 0.5, 'abs_energy_penalty': -117.9476994159866, 'abs_comfort_penalty': 0, 'total_power_demand': 117.9476994159866, 'total_temperature_violation': 0.0}
Simulation Progress [Episode 1]: 100%|██████████| 100/100 [00:20<00:00,  4.94%/s, 100% completed]
[ENVIRONMENT] (INFO) : Environment closed. [5zone-hot-continuous-v1]
[WRAPPER NormalizeObservation] (INFO) : Saving normalization calibration data.