21. DRL usage example
In this notebook example, Stable Baselines 3 has been used to train and to load an agent. However, Sinergym is completely agnostic to any DRL algorithm (although there are custom callbacks for SB3 specifically) and can be used with any DRL library that works with gymnasium environments.
21.1. Training a model
We are going to rely on the script available in the repository root called DRL_battery.py. This script applies all the possibilities that Sinergym has to work with deep reinforcement learning algorithms and set parameters to everything so that we can define the training options from the execution of the script easily by a JSON file.
For more information about how run DRL_battery.py, please, see Train a model.
[2]:
import sys
from datetime import datetime
import gymnasium as gym
import numpy as np
import wandb
from stable_baselines3 import *
from stable_baselines3.common.callbacks import CallbackList
from stable_baselines3.common.logger import HumanOutputFormat, Logger
from stable_baselines3.common.monitor import Monitor
import sinergym
import sinergym.utils.gcloud as gcloud
from sinergym.utils.callbacks import *
from sinergym.utils.constants import *
from sinergym.utils.logger import CSVLogger, WandBOutputFormat
from sinergym.utils.rewards import *
from sinergym.utils.wrappers import *
First let’s define some variables for the execution.
[3]:
# Environment ID
environment = "Eplus-demo-v1"
# Training episodes
episodes = 4
#Name of the experiment
experiment_date = datetime.today().strftime('%Y-%m-%d_%H:%M')
experiment_name = 'SB3_DQN-' + environment + \
'-episodes-' + str(episodes)
experiment_name += '_' + experiment_date
We can combine this experiment executions with Weights&Biases in order to host all information extracted. With wandb, it’s possible to track and visualize all DRL training process in real time, register hyperparameters and details of each experiment, save artifacts such as models and sinergym output, and compare between different executions.
[4]:
# Create wandb.config object in order to log all experiment params
experiment_params = {
'sinergym-version': sinergym.__version__,
'python-version': sys.version
}
experiment_params.update({'environment':environment,
'episodes':episodes,
'algorithm':'SB3_DQN'})
# Get wandb init params (you have to specify your own project and entity)
wandb_params = {"project": 'sinergym',
"entity": 'alex_ugr'}
# Init wandb entry
run = wandb.init(
name=experiment_name + '_' + wandb.util.generate_id(),
config=experiment_params,
** wandb_params
)
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb: Currently logged in as: alex_ugr. Use `wandb login --relogin` to force relogin
Tracking run with wandb version 0.14.0
Run data is saved locally in
/workspaces/sinergym/examples/wandb/run-20230405_145126-gxs4rau3
View project at https://wandb.ai/alex_ugr/sinergym
View run at https://wandb.ai/alex_ugr/sinergym/runs/gxs4rau3
Now we are ready to create the Gymnasium Environment. Here we use the environment name defined, remember that you can change default environment configuration. We will create a eval_env too in order to interact in the evaluation episodes. We can overwrite the env name with experiment name if we want.
[5]:
env = gym.make(environment, env_name=experiment_name)
eval_env = gym.make(environment, env_name=experiment_name+'_EVALUATION')
[2023-04-05 14:51:32,107] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_MainThread_ROOT INFO:Updating idf ExternalInterface object if it is not present...
[2023-04-05 14:51:32,109] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_MainThread_ROOT INFO:Updating idf Site:Location and SizingPeriod:DesignDay(s) to weather and ddy file...
[2023-04-05 14:51:32,112] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_MainThread_ROOT INFO:Updating idf OutPut:Variable and variables XML tree model for BVCTB connection.
[2023-04-05 14:51:32,114] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_MainThread_ROOT INFO:Setting up extra configuration in building model if exists...
[2023-04-05 14:51:32,116] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_MainThread_ROOT INFO:Setting up action definition in building model if exists...
[2023-04-05 14:51:33,584] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_EVALUATION_MainThread_ROOT INFO:Updating idf ExternalInterface object if it is not present...
[2023-04-05 14:51:33,586] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_EVALUATION_MainThread_ROOT INFO:Updating idf Site:Location and SizingPeriod:DesignDay(s) to weather and ddy file...
[2023-04-05 14:51:33,590] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_EVALUATION_MainThread_ROOT INFO:Updating idf OutPut:Variable and variables XML tree model for BVCTB connection.
[2023-04-05 14:51:33,592] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_EVALUATION_MainThread_ROOT INFO:Setting up extra configuration in building model if exists...
[2023-04-05 14:51:33,593] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_EVALUATION_MainThread_ROOT INFO:Setting up action definition in building model if exists...
We can also add a Wrapper to the environment, we are going to use a Logger (extension of gym.Wrapper) this is used to monitor and log the interactions with the environment and save the data into a CSV. Files generated will be stored as artifact in wandb too.
[6]:
env = LoggerWrapper(env)
At this point, we have the environment set up and ready to be used. We are going to create our learning model (Stable Baselines 3 DQN), but we can use any other algorithm.
[7]:
model = DQN('MlpPolicy', env, verbose=1)
Using cpu device
Wrapping the env with a `Monitor` wrapper
Wrapping the env in a DummyVecEnv.
Now we need to calculate the number of timesteps of each episode for the evaluation. Evaluation will execute the current model during a number of episodes determined to decide if it is the best current version of the model at that point of the training. Output generated will be stored in wandb server too.
[8]:
n_timesteps_episode = env.simulator._eplus_one_epi_len / \
env.simulator._eplus_run_stepsize
We are going to use the LoggerEval callback to print and save the best model evaluated during training.
[9]:
callbacks = []
# Set up Evaluation and saving best model
eval_callback = LoggerEvalCallback(
eval_env,
best_model_save_path=eval_env.simulator._env_working_dir_parent +
'/best_model/',
log_path=eval_env.simulator._env_working_dir_parent +
'/best_model/',
eval_freq=n_timesteps_episode * 2,
deterministic=True,
render=False,
n_eval_episodes=1)
callbacks.append(eval_callback)
callback = CallbackList(callbacks)
In order to track all the training process in wandb, it is necessary to create a callback with a compatible wandb output format (which call wandb log method in the learning algorithm process).
[10]:
# wandb logger and setting in SB3
logger = Logger(
folder=None,
output_formats=[
HumanOutputFormat(
sys.stdout,
max_length=120),
WandBOutputFormat()])
model.set_logger(logger)
# Append callback
log_callback = LoggerCallback()
callbacks.append(log_callback)
callback = CallbackList(callbacks)
This is the number of total time steps for the training.
[11]:
timesteps = episodes * n_timesteps_episode
Now is time to train the model with the callbacks defined earlier. This may take a few minutes, depending on your computer.
[12]:
model.learn(
total_timesteps=timesteps,
callback=callback,
log_interval=1)
[2023-04-05 14:51:53,742] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_MainThread_ROOT INFO:Creating new EnergyPlus simulation episode...
[2023-04-05 14:51:53,957] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_MainThread_ROOT INFO:EnergyPlus working directory is in /workspaces/sinergym/examples/Eplus-env-SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51-res1/Eplus-env-sub_run1
/usr/local/lib/python3.10/dist-packages/opyplus/weather_data/weather_data.py:493: FutureWarning: the 'line_terminator'' keyword is deprecated, use 'lineterminator' instead.
epw_content = self._headers_to_epw(use_datetimes=use_datetimes) + df.to_csv(
[2023-04-05 14:52:35,576] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_MainThread_ROOT INFO:EnergyPlus episode completed successfully.
[2023-04-05 14:52:35,578] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_MainThread_ROOT INFO:Creating new EnergyPlus simulation episode...
[2023-04-05 14:52:35,781] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_MainThread_ROOT INFO:EnergyPlus working directory is in /workspaces/sinergym/examples/Eplus-env-SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51-res1/Eplus-env-sub_run2
------------------------------------------------------------------------------------
| action/ | |
| Cooling_Setpoint_RL | 25.5 |
| Heating_Setpoint_RL | 19.1 |
| action_simulation/ | |
| Cooling_Setpoint_RL | 25.5 |
| Heating_Setpoint_RL | 19.1 |
| episode/ | |
| comfort_violation_time(%) | 52.5 |
| cumulative_comfort_penalty | -2.65e+04 |
| cumulative_power | 2.06e+08 |
| cumulative_power_penalty | -2.06e+04 |
| cumulative_reward | -23582.973 |
| ep_length | 35040 |
| mean_comfort_penalty | -0.758 |
| mean_power | 5.89e+03 |
| mean_power_penalty | -0.589 |
| mean_reward | -0.67303 |
| observation/ | |
| Facility Total HVAC Electricity Demand Rate(Whole Building) | 5.89e+03 |
| People Air Temperature(SPACE1-1 PEOPLE 1) | 21.2 |
| Site Diffuse Solar Radiation Rate per Area(Environment) | 73.6 |
| Site Direct Solar Radiation Rate per Area(Environment) | 115 |
| Site Outdoor Air Drybulb Temperature(Environment) | 11.2 |
| Site Outdoor Air Relative Humidity(Environment) | 71.3 |
| Site Wind Direction(Environment) | 195 |
| Site Wind Speed(Environment) | 3.87 |
| Zone Air Relative Humidity(SPACE1-1) | 43 |
| Zone Air Temperature(SPACE1-1) | 21.2 |
| Zone People Occupant Count(SPACE1-1) | 3.18 |
| Zone Thermal Comfort Clothing Value(SPACE1-1 PEOPLE 1) | 0.662 |
| Zone Thermal Comfort Fanger Model PPD(SPACE1-1 PEOPLE 1) | 31.7 |
| Zone Thermal Comfort Mean Radiant Temperature(SPACE1-1 PEOPLE 1) | 21.4 |
| Zone Thermostat Cooling Setpoint Temperature(SPACE1-1) | 25.5 |
| Zone Thermostat Heating Setpoint Temperature(SPACE1-1) | 19.1 |
| day | 15.7 |
| hour | 11.5 |
| month | 6.53 |
| year | 1.99e+03 |
| rollout/ | |
| ep_len_mean | 3.5e+04 |
| ep_rew_mean | -2.36e+04 |
| exploration_rate | 0.05 |
| time/ | |
| episodes | 1 |
| fps | 815 |
| time_elapsed | 42 |
| total_timesteps | 35040 |
------------------------------------------------------------------------------------
[2023-04-05 14:53:29,396] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_MainThread_ROOT INFO:EnergyPlus episode completed successfully.
[2023-04-05 14:53:29,398] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_MainThread_ROOT INFO:Creating new EnergyPlus simulation episode...
[2023-04-05 14:53:29,543] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_MainThread_ROOT INFO:EnergyPlus working directory is in /workspaces/sinergym/examples/Eplus-env-SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51-res1/Eplus-env-sub_run3
[2023-04-05 14:53:30,304] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_EVALUATION_MainThread_ROOT INFO:Creating new EnergyPlus simulation episode...
[2023-04-05 14:53:30,455] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_EVALUATION_MainThread_ROOT INFO:EnergyPlus working directory is in /workspaces/sinergym/examples/Eplus-env-SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_EVALUATION-res1/Eplus-env-sub_run1
[2023-04-05 14:54:01,956] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_EVALUATION_MainThread_ROOT INFO:EnergyPlus episode completed successfully.
[2023-04-05 14:54:01,959] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_EVALUATION_MainThread_ROOT INFO:Creating new EnergyPlus simulation episode...
[2023-04-05 14:54:02,117] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_EVALUATION_MainThread_ROOT INFO:EnergyPlus working directory is in /workspaces/sinergym/examples/Eplus-env-SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_EVALUATION-res1/Eplus-env-sub_run2
Eval num_timesteps=70080, episode_reward=-22366.66 +/- 0.00
Episode length: 35040.00 +/- 0.00
New best mean reward!
-------------------------------------------------------------------------------------
| action/ | |
| Cooling_Setpoint_RL | 23.7 |
| Heating_Setpoint_RL | 20.6 |
| action_simulation/ | |
| Cooling_Setpoint_RL | 23.7 |
| Heating_Setpoint_RL | 20.6 |
| episode/ | |
| comfort_violation_time(%) | 47.8 |
| cumulative_comfort_penalty | -1.95e+04 |
| cumulative_power | 2.62e+08 |
| cumulative_power_penalty | -2.62e+04 |
| cumulative_reward | -22837.932 |
| ep_length | 35040 |
| mean_comfort_penalty | -0.556 |
| mean_power | 7.47e+03 |
| mean_power_penalty | -0.747 |
| mean_reward | -0.65176743 |
| eval/ | |
| comfort_penalty | -1.37e+04 |
| comfort_violation(%) | 33.4 |
| mean_ep_length | 3.5e+04 |
| mean_power_consumption | 3.1e+08 |
| mean_rewards | -22366.656 |
| power_penalty | -3.1e+04 |
| std_rewards | 0.0 |
| observation/ | |
| Facility Total HVAC Electricity Demand Rate(Whole Building) | 7.47e+03 |
| People Air Temperature(SPACE1-1 PEOPLE 1) | 21.5 |
| Site Diffuse Solar Radiation Rate per Area(Environment) | 73.6 |
| Site Direct Solar Radiation Rate per Area(Environment) | 115 |
| Site Outdoor Air Drybulb Temperature(Environment) | 11.2 |
| Site Outdoor Air Relative Humidity(Environment) | 71.3 |
| Site Wind Direction(Environment) | 195 |
| Site Wind Speed(Environment) | 3.87 |
| Zone Air Relative Humidity(SPACE1-1) | 42.1 |
| Zone Air Temperature(SPACE1-1) | 21.5 |
| Zone People Occupant Count(SPACE1-1) | 3.18 |
| Zone Thermal Comfort Clothing Value(SPACE1-1 PEOPLE 1) | 0.662 |
| Zone Thermal Comfort Fanger Model PPD(SPACE1-1 PEOPLE 1) | 27 |
| Zone Thermal Comfort Mean Radiant Temperature(SPACE1-1 PEOPLE 1) | 21.6 |
| Zone Thermostat Cooling Setpoint Temperature(SPACE1-1) | 23.7 |
| Zone Thermostat Heating Setpoint Temperature(SPACE1-1) | 20.6 |
| day | 15.7 |
| hour | 11.5 |
| month | 6.53 |
| year | 1.99e+03 |
| rollout/ | |
| ep_len_mean | 3.5e+04 |
| ep_rew_mean | -2.32e+04 |
| exploration_rate | 0.05 |
| time/ | |
| episodes | 2 |
| fps | 542 |
| time_elapsed | 129 |
| total_timesteps | 70080 |
| train/ | |
| learning_rate | 0.0001 |
| loss | 93.4 |
| n_updates | 5019 |
-------------------------------------------------------------------------------------
[2023-04-05 14:55:12,660] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_MainThread_ROOT INFO:EnergyPlus episode completed successfully.
[2023-04-05 14:55:12,663] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_MainThread_ROOT INFO:Creating new EnergyPlus simulation episode...
[2023-04-05 14:55:12,822] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_MainThread_ROOT INFO:EnergyPlus working directory is in /workspaces/sinergym/examples/Eplus-env-SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51-res1/Eplus-env-sub_run4
------------------------------------------------------------------------------------
| action/ | |
| Cooling_Setpoint_RL | 23.3 |
| Heating_Setpoint_RL | 21 |
| action_simulation/ | |
| Cooling_Setpoint_RL | 23.3 |
| Heating_Setpoint_RL | 21 |
| episode/ | |
| comfort_violation_time(%) | 36 |
| cumulative_comfort_penalty | -1.63e+04 |
| cumulative_power | 3.04e+08 |
| cumulative_power_penalty | -3.04e+04 |
| cumulative_reward | -23348.816 |
| ep_length | 35040 |
| mean_comfort_penalty | -0.466 |
| mean_power | 8.66e+03 |
| mean_power_penalty | -0.866 |
| mean_reward | -0.6663475 |
| observation/ | |
| Facility Total HVAC Electricity Demand Rate(Whole Building) | 8.66e+03 |
| People Air Temperature(SPACE1-1 PEOPLE 1) | 21.9 |
| Site Diffuse Solar Radiation Rate per Area(Environment) | 73.6 |
| Site Direct Solar Radiation Rate per Area(Environment) | 115 |
| Site Outdoor Air Drybulb Temperature(Environment) | 11.2 |
| Site Outdoor Air Relative Humidity(Environment) | 71.3 |
| Site Wind Direction(Environment) | 195 |
| Site Wind Speed(Environment) | 3.87 |
| Zone Air Relative Humidity(SPACE1-1) | 41.4 |
| Zone Air Temperature(SPACE1-1) | 21.9 |
| Zone People Occupant Count(SPACE1-1) | 3.18 |
| Zone Thermal Comfort Clothing Value(SPACE1-1 PEOPLE 1) | 0.662 |
| Zone Thermal Comfort Fanger Model PPD(SPACE1-1 PEOPLE 1) | 23.5 |
| Zone Thermal Comfort Mean Radiant Temperature(SPACE1-1 PEOPLE 1) | 21.8 |
| Zone Thermostat Cooling Setpoint Temperature(SPACE1-1) | 23.3 |
| Zone Thermostat Heating Setpoint Temperature(SPACE1-1) | 21 |
| day | 15.7 |
| hour | 11.5 |
| month | 6.53 |
| year | 1.99e+03 |
| rollout/ | |
| ep_len_mean | 3.5e+04 |
| ep_rew_mean | -2.33e+04 |
| exploration_rate | 0.05 |
| time/ | |
| episodes | 3 |
| fps | 526 |
| time_elapsed | 199 |
| total_timesteps | 105120 |
| train/ | |
| learning_rate | 0.0001 |
| loss | 19.8 |
| n_updates | 13779 |
------------------------------------------------------------------------------------
[2023-04-05 14:56:23,839] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_MainThread_ROOT INFO:EnergyPlus episode completed successfully.
[2023-04-05 14:56:23,840] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_MainThread_ROOT INFO:Creating new EnergyPlus simulation episode...
[2023-04-05 14:56:23,986] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_MainThread_ROOT INFO:EnergyPlus working directory is in /workspaces/sinergym/examples/Eplus-env-SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51-res1/Eplus-env-sub_run5
[2023-04-05 14:56:29,946] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_EVALUATION_MainThread_ROOT INFO:EnergyPlus episode completed successfully.
[2023-04-05 14:56:29,948] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_EVALUATION_MainThread_ROOT INFO:Creating new EnergyPlus simulation episode...
[2023-04-05 14:56:30,097] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_EVALUATION_MainThread_ROOT INFO:EnergyPlus working directory is in /workspaces/sinergym/examples/Eplus-env-SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_EVALUATION-res1/Eplus-env-sub_run3
[2023-04-05 14:57:02,000] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_EVALUATION_MainThread_ROOT INFO:EnergyPlus episode completed successfully.
[2023-04-05 14:57:02,001] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_EVALUATION_MainThread_ROOT INFO:Creating new EnergyPlus simulation episode...
[2023-04-05 14:57:02,153] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_EVALUATION_MainThread_ROOT INFO:EnergyPlus working directory is in /workspaces/sinergym/examples/Eplus-env-SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_EVALUATION-res1/Eplus-env-sub_run4
Eval num_timesteps=140160, episode_reward=-24046.03 +/- 0.00
Episode length: 35040.00 +/- 0.00
------------------------------------------------------------------------------------
| action/ | |
| Cooling_Setpoint_RL | 23.9 |
| Heating_Setpoint_RL | 20.3 |
| action_simulation/ | |
| Cooling_Setpoint_RL | 23.9 |
| Heating_Setpoint_RL | 20.3 |
| episode/ | |
| comfort_violation_time(%) | 35 |
| cumulative_comfort_penalty | -1.92e+04 |
| cumulative_power | 2.9e+08 |
| cumulative_power_penalty | -2.9e+04 |
| cumulative_reward | -24122.254 |
| ep_length | 35040 |
| mean_comfort_penalty | -0.548 |
| mean_power | 8.29e+03 |
| mean_power_penalty | -0.829 |
| mean_reward | -0.6884205 |
| eval/ | |
| comfort_penalty | -1.85e+04 |
| comfort_violation(%) | 35.6 |
| mean_ep_length | 3.5e+04 |
| mean_power_consumption | 2.96e+08 |
| mean_rewards | -24046.03 |
| power_penalty | -2.96e+04 |
| std_rewards | 0.0 |
| observation/ | |
| Facility Total HVAC Electricity Demand Rate(Whole Building) | 8.29e+03 |
| People Air Temperature(SPACE1-1 PEOPLE 1) | 21.7 |
| Site Diffuse Solar Radiation Rate per Area(Environment) | 73.6 |
| Site Direct Solar Radiation Rate per Area(Environment) | 115 |
| Site Outdoor Air Drybulb Temperature(Environment) | 11.2 |
| Site Outdoor Air Relative Humidity(Environment) | 71.3 |
| Site Wind Direction(Environment) | 195 |
| Site Wind Speed(Environment) | 3.87 |
| Zone Air Relative Humidity(SPACE1-1) | 41.9 |
| Zone Air Temperature(SPACE1-1) | 21.7 |
| Zone People Occupant Count(SPACE1-1) | 3.18 |
| Zone Thermal Comfort Clothing Value(SPACE1-1 PEOPLE 1) | 0.662 |
| Zone Thermal Comfort Fanger Model PPD(SPACE1-1 PEOPLE 1) | 25.4 |
| Zone Thermal Comfort Mean Radiant Temperature(SPACE1-1 PEOPLE 1) | 21.7 |
| Zone Thermostat Cooling Setpoint Temperature(SPACE1-1) | 23.9 |
| Zone Thermostat Heating Setpoint Temperature(SPACE1-1) | 20.3 |
| day | 15.7 |
| hour | 11.5 |
| month | 6.53 |
| year | 1.99e+03 |
| rollout/ | |
| ep_len_mean | 3.5e+04 |
| ep_rew_mean | -2.35e+04 |
| exploration_rate | 0.05 |
| time/ | |
| episodes | 4 |
| fps | 453 |
| time_elapsed | 309 |
| total_timesteps | 140160 |
| train/ | |
| learning_rate | 0.0001 |
| loss | 26.9 |
| n_updates | 22539 |
------------------------------------------------------------------------------------
[12]:
<stable_baselines3.dqn.dqn.DQN at 0x7f2498717550>
Now, we save the current model (model version when training has finished).
[13]:
model.save(env.simulator._env_working_dir_parent + '/' + experiment_name)
And as always, remember to close the environment.
[14]:
env.close()
/usr/local/lib/python3.10/dist-packages/numpy/core/fromnumeric.py:3464: RuntimeWarning: Mean of empty slice.
return _methods._mean(a, axis=axis, dtype=dtype,
/usr/local/lib/python3.10/dist-packages/numpy/core/_methods.py:192: RuntimeWarning: invalid value encountered in scalar divide
ret = ret.dtype.type(ret / rcount)
/usr/local/lib/python3.10/dist-packages/numpy/core/_methods.py:269: RuntimeWarning: Degrees of freedom <= 0 for slice
ret = _var(a, axis=axis, dtype=dtype, out=out, ddof=ddof,
/usr/local/lib/python3.10/dist-packages/numpy/core/_methods.py:226: RuntimeWarning: invalid value encountered in divide
arrmean = um.true_divide(arrmean, div, out=arrmean,
/usr/local/lib/python3.10/dist-packages/numpy/core/_methods.py:261: RuntimeWarning: invalid value encountered in scalar divide
ret = ret.dtype.type(ret / rcount)
[2023-04-05 15:01:11,136] EPLUS_ENV_SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_MainThread_ROOT INFO:EnergyPlus simulation closed successfully.
We have to upload all Sinergym output as wandb artifact. This output include all sinergym_output (and LoggerWrapper CSV files) and models generated in training and evaluation episodes.
[15]:
artifact = wandb.Artifact(
name="training",
type="experiment1")
artifact.add_dir(
env.simulator._env_working_dir_parent,
name='training_output/')
artifact.add_dir(
eval_env.simulator._env_working_dir_parent,
name='evaluation_output/')
run.log_artifact(artifact)
# wandb has finished
run.finish()
wandb: Adding directory to artifact (/workspaces/sinergym/examples/Eplus-env-SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51-res1)... Done. 0.1s
wandb: Adding directory to artifact (/workspaces/sinergym/examples/Eplus-env-SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_EVALUATION-res1)... Done. 0.1s
Waiting for W&B process to finish... (success).
wandb: Network error (TransientError), entering retry loop.
wandb: Network error (TransientError), entering retry loop.
wandb: Network error (TransientError), entering retry loop.
Run history:
| action/Cooling_Setpoint_RL | █▂▁▃ |
| action/Heating_Setpoint_RL | ▁▆█▆ |
| action_simulation/Cooling_Setpoint_RL | █▂▁▃ |
| action_simulation/Heating_Setpoint_RL | ▁▆█▆ |
| episode/comfort_violation_time(%) | █▆▁▁ |
| episode/cumulative_comfort_penalty | ▁▆█▆ |
| episode/cumulative_power | ▁▅█▇ |
| episode/cumulative_power_penalty | █▄▁▂ |
| episode/cumulative_reward | ▄█▅▁ |
| episode/ep_length | ▁▁▁▁ |
| episode/mean_comfort_penalty | ▁▆█▆ |
| episode/mean_power | ▁▅█▇ |
| episode/mean_power_penalty | █▄▁▂ |
| episode/mean_reward | ▄█▅▁ |
| eval/comfort_penalty | █▁ |
| eval/comfort_violation(%) | ▁█ |
| eval/mean_ep_length | ▁▁ |
| eval/mean_power_consumption | █▁ |
| eval/mean_rewards | █▁ |
| eval/power_penalty | ▁█ |
| eval/std_rewards | ▁▁ |
| observation/Facility Total HVAC Electricity Demand Rate(Whole Building) | ▁▅█▇ |
| observation/People Air Temperature(SPACE1-1 PEOPLE 1) | ▁▄█▆ |
| observation/Site Diffuse Solar Radiation Rate per Area(Environment) | ▁▁▁▁ |
| observation/Site Direct Solar Radiation Rate per Area(Environment) | ▁▁▁▁ |
| observation/Site Outdoor Air Drybulb Temperature(Environment) | ▁▁▁▁ |
| observation/Site Outdoor Air Relative Humidity(Environment) | ▁▁▁▁ |
| observation/Site Wind Direction(Environment) | ▁▁▁▁ |
| observation/Site Wind Speed(Environment) | ▁▁▁▁ |
| observation/Zone Air Relative Humidity(SPACE1-1) | █▄▁▃ |
| observation/Zone Air Temperature(SPACE1-1) | ▁▄█▆ |
| observation/Zone People Occupant Count(SPACE1-1) | ▁▁▁▁ |
| observation/Zone Thermal Comfort Clothing Value(SPACE1-1 PEOPLE 1) | ▁▁▁▁ |
| observation/Zone Thermal Comfort Fanger Model PPD(SPACE1-1 PEOPLE 1) | █▄▁▃ |
| observation/Zone Thermal Comfort Mean Radiant Temperature(SPACE1-1 PEOPLE 1) | ▁▄█▆ |
| observation/Zone Thermostat Cooling Setpoint Temperature(SPACE1-1) | █▂▁▃ |
| observation/Zone Thermostat Heating Setpoint Temperature(SPACE1-1) | ▁▆█▆ |
| observation/day | ▁▁▁▁ |
| observation/hour | ▁▁▁▁ |
| observation/month | ▁▁▁▁ |
| observation/year | ▁▁▁▁ |
| rollout/ep_len_mean | ▁▁▁▁ |
| rollout/ep_rew_mean | ▁█▇▃ |
| rollout/exploration_rate | ▁▁▁▁ |
| time/episodes | ▁▃▆█ |
| time/fps | █▃▂▁ |
| time/time_elapsed | ▁▃▅█ |
| time/total_timesteps | ▁▃▆█ |
| train/learning_rate | ▁▁▁ |
| train/loss | █▁▂ |
| train/n_updates | ▁▅█ |
Run summary:
| action/Cooling_Setpoint_RL | 23.92586 |
| action/Heating_Setpoint_RL | 20.33833 |
| action_simulation/Cooling_Setpoint_RL | 23.92586 |
| action_simulation/Heating_Setpoint_RL | 20.33833 |
| episode/comfort_violation_time(%) | 35.0 |
| episode/cumulative_comfort_penalty | -19195.31635 |
| episode/cumulative_power | 290491842.64102 |
| episode/cumulative_power_penalty | -29049.18426 |
| episode/cumulative_reward | -24122.25391 |
| episode/ep_length | 35040 |
| episode/mean_comfort_penalty | -0.54781 |
| episode/mean_power | 8290.29231 |
| episode/mean_power_penalty | -0.82903 |
| episode/mean_reward | -0.68842 |
| eval/comfort_penalty | -18462.58008 |
| eval/comfort_violation(%) | 35.63356 |
| eval/mean_ep_length | 35040.0 |
| eval/mean_power_consumption | 296288163.17097 |
| eval/mean_rewards | -24046.0293 |
| eval/power_penalty | -29628.81632 |
| eval/std_rewards | 0.0 |
| observation/Facility Total HVAC Electricity Demand Rate(Whole Building) | 8290.1535 |
| observation/People Air Temperature(SPACE1-1 PEOPLE 1) | 21.68222 |
| observation/Site Diffuse Solar Radiation Rate per Area(Environment) | 73.57191 |
| observation/Site Direct Solar Radiation Rate per Area(Environment) | 115.15166 |
| observation/Site Outdoor Air Drybulb Temperature(Environment) | 11.24046 |
| observation/Site Outdoor Air Relative Humidity(Environment) | 71.29315 |
| observation/Site Wind Direction(Environment) | 194.72517 |
| observation/Site Wind Speed(Environment) | 3.86764 |
| observation/Zone Air Relative Humidity(SPACE1-1) | 41.94364 |
| observation/Zone Air Temperature(SPACE1-1) | 21.68282 |
| observation/Zone People Occupant Count(SPACE1-1) | 3.17908 |
| observation/Zone Thermal Comfort Clothing Value(SPACE1-1 PEOPLE 1) | 0.66228 |
| observation/Zone Thermal Comfort Fanger Model PPD(SPACE1-1 PEOPLE 1) | 25.36044 |
| observation/Zone Thermal Comfort Mean Radiant Temperature(SPACE1-1 PEOPLE 1) | 21.71455 |
| observation/Zone Thermostat Cooling Setpoint Temperature(SPACE1-1) | 23.92591 |
| observation/Zone Thermostat Heating Setpoint Temperature(SPACE1-1) | 20.3383 |
| observation/day | 15.72055 |
| observation/hour | 11.5 |
| observation/month | 6.52603 |
| observation/year | 1991.0 |
| rollout/ep_len_mean | 35040.0 |
| rollout/ep_rew_mean | -23472.99229 |
| rollout/exploration_rate | 0.05 |
| time/episodes | 4 |
| time/fps | 453 |
| time/time_elapsed | 309 |
| time/total_timesteps | 140160 |
| train/learning_rate | 0.0001 |
| train/loss | 26.87911 |
| train/n_updates | 22539 |
View run SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_nt18lsm5 at: https://wandb.ai/alex_ugr/sinergym/runs/gxs4rau3
Synced 5 W&B file(s), 0 media file(s), 180 artifact file(s) and 0 other file(s)
Synced 5 W&B file(s), 0 media file(s), 180 artifact file(s) and 0 other file(s)
Find logs at:
./wandb/run-20230405_145126-gxs4rau3/logsWe have all the experiments results in our local computer, but we can see the execution in wandb too:
If we check our projects, we can see the execution allocated:
Hyperparameters tracked in the training experiment:
Artifacts registered (if evaluation is enabled, best model is registered too):
Visualization of metrics in real time:
21.2. Loading a model
We are going to rely on the script available in the repository root called load_agent.py. This script applies all the possibilities that Sinergym has to work with deep reinforcement learning models loaded and set parameters to everything so that we can define the load options from the execution of the script easily by a JSON file.
For more information about how run load_agent.py, please, see Load a trained model.
First we define the Sinergym environment ID where we want to check the loaded agent and the name of the evaluation experiment.
[16]:
# Environment ID
environment = "Eplus-demo-v1"
# Episodes
episodes=5
# Evaluation name
evaluation_date = datetime.today().strftime('%Y-%m-%d_%H:%M')
evaluation_name = 'SB3_DQN-EVAL-' + environment + \
'-episodes-' + str(episodes)
evaluation_name += '_' + evaluation_date
We can also use wandb here. We can allocate this evaluation of a loaded model in other project in order to not merge experiments.
[17]:
# Create wandb.config object in order to log all experiment params
experiment_params = {
'sinergym-version': sinergym.__version__,
'python-version': sys.version
}
experiment_params.update({'environment':environment,
'episodes':episodes,
'algorithm':'SB3_DQN'})
# Get wandb init params (you have to specify your own project and entity)
wandb_params = {"project": 'sinergym_evaluations',
"entity": 'alex_ugr'}
# Init wandb entry
run = wandb.init(
name=experiment_name + '_' + wandb.util.generate_id(),
config=experiment_params,
** wandb_params
)
Tracking run with wandb version 0.14.0
Run data is saved locally in
/workspaces/sinergym/examples/wandb/run-20230405_150935-2wfphha2
View project at https://wandb.ai/alex_ugr/sinergym_evaluations
We make the gymnasium environment and wrap with LoggerWrapper. We can use the evaluation experiment name to rename the environment.
[20]:
env=gym.make(environment, env_name=evaluation_name)
env=LoggerWrapper(env)
[2023-04-05 15:12:10,652] EPLUS_ENV_SB3_DQN-EVAL-Eplus-demo-v1-episodes-5_2023-04-05_15:09_MainThread_ROOT INFO:Updating idf ExternalInterface object if it is not present...
[2023-04-05 15:12:10,654] EPLUS_ENV_SB3_DQN-EVAL-Eplus-demo-v1-episodes-5_2023-04-05_15:09_MainThread_ROOT INFO:Updating idf Site:Location and SizingPeriod:DesignDay(s) to weather and ddy file...
[2023-04-05 15:12:10,659] EPLUS_ENV_SB3_DQN-EVAL-Eplus-demo-v1-episodes-5_2023-04-05_15:09_MainThread_ROOT INFO:Updating idf OutPut:Variable and variables XML tree model for BVCTB connection.
[2023-04-05 15:12:10,663] EPLUS_ENV_SB3_DQN-EVAL-Eplus-demo-v1-episodes-5_2023-04-05_15:09_MainThread_ROOT INFO:Setting up extra configuration in building model if exists...
[2023-04-05 15:12:10,664] EPLUS_ENV_SB3_DQN-EVAL-Eplus-demo-v1-episodes-5_2023-04-05_15:09_MainThread_ROOT INFO:Setting up action definition in building model if exists...
We load the Stable Baselines 3 DQN model using the model allocated in our local computer, although we can use a remote model allocated in wandb from other training experiment.
[21]:
# get wandb artifact path (to load model)
load_artifact_entity = 'alex_ugr'
load_artifact_project = 'sinergym'
load_artifact_name = 'training'
load_artifact_tag = 'latest'
load_artifact_model_path = 'evaluation_output/best_model/model.zip'
wandb_path = load_artifact_entity + '/' + load_artifact_project + \
'/' + load_artifact_name + ':' + load_artifact_tag
# Download artifact
artifact = run.use_artifact(wandb_path)
artifact.get_path(load_artifact_model_path).download('.')
# Set model path to local wandb file downloaded
model_path = './' + load_artifact_model_path
model = DQN.load(model_path)
As we can see, The wandb model we want to load can come from an artifact of an different entity or project from the one we are using to register the evaluation of the loaded model, as long as it is accessible. The next step is use the model to predict actions and interact with the environment in order to collect data to evaluate the model.
[22]:
for i in range(episodes):
obs, info = env.reset()
rewards = []
terminated = False
current_month = 0
while not terminated:
a, _ = model.predict(obs)
obs, reward, terminated, truncated, info = env.step(a)
rewards.append(reward)
if info['month'] != current_month:
current_month = info['month']
print(info['month'], sum(rewards))
print(
'Episode ',
i,
'Mean reward: ',
np.mean(rewards),
'Cumulative reward: ',
sum(rewards))
env.close()
[2023-04-05 15:12:27,791] EPLUS_ENV_SB3_DQN-EVAL-Eplus-demo-v1-episodes-5_2023-04-05_15:09_MainThread_ROOT INFO:Creating new EnergyPlus simulation episode...
[2023-04-05 15:12:27,939] EPLUS_ENV_SB3_DQN-EVAL-Eplus-demo-v1-episodes-5_2023-04-05_15:09_MainThread_ROOT INFO:EnergyPlus working directory is in /workspaces/sinergym/examples/Eplus-env-SB3_DQN-EVAL-Eplus-demo-v1-episodes-5_2023-04-05_15:09-res1/Eplus-env-sub_run1
/usr/local/lib/python3.10/dist-packages/opyplus/weather_data/weather_data.py:493: FutureWarning: the 'line_terminator'' keyword is deprecated, use 'lineterminator' instead.
epw_content = self._headers_to_epw(use_datetimes=use_datetimes) + df.to_csv(
1 -1.086220480544118
2 -2079.9961147920526
3 -4054.5450027104594
4 -5315.115280552358
5 -6240.707702132893
6 -7147.282200268969
7 -9757.11094202821
8 -12816.141002414372
9 -15759.877949459922
10 -18249.178074198662
11 -19008.92750593897
12 -20059.684052467783
1 -22170.235119703077
Episode 0 Mean reward: -0.6327121894892542 Cumulative reward: -22170.235119703077
[2023-04-05 15:13:01,011] EPLUS_ENV_SB3_DQN-EVAL-Eplus-demo-v1-episodes-5_2023-04-05_15:09_MainThread_ROOT INFO:EnergyPlus episode completed successfully.
[2023-04-05 15:13:01,013] EPLUS_ENV_SB3_DQN-EVAL-Eplus-demo-v1-episodes-5_2023-04-05_15:09_MainThread_ROOT INFO:Creating new EnergyPlus simulation episode...
[2023-04-05 15:13:01,189] EPLUS_ENV_SB3_DQN-EVAL-Eplus-demo-v1-episodes-5_2023-04-05_15:09_MainThread_ROOT INFO:EnergyPlus working directory is in /workspaces/sinergym/examples/Eplus-env-SB3_DQN-EVAL-Eplus-demo-v1-episodes-5_2023-04-05_15:09-res1/Eplus-env-sub_run2
1 -1.086220480544118
2 -2077.808749644109
3 -4069.9817270164053
4 -5325.590232162119
5 -6251.094400409479
6 -7142.7185322162595
7 -9777.3107632979
8 -12818.25361887402
9 -15762.374127346286
10 -18231.94068105046
11 -18987.961511158555
12 -20041.518570795673
1 -22144.44512276465
Episode 1 Mean reward: -0.6319761735948897 Cumulative reward: -22144.44512276465
[2023-04-05 15:13:31,919] EPLUS_ENV_SB3_DQN-EVAL-Eplus-demo-v1-episodes-5_2023-04-05_15:09_MainThread_ROOT INFO:EnergyPlus episode completed successfully.
[2023-04-05 15:13:31,921] EPLUS_ENV_SB3_DQN-EVAL-Eplus-demo-v1-episodes-5_2023-04-05_15:09_MainThread_ROOT INFO:Creating new EnergyPlus simulation episode...
[2023-04-05 15:13:32,073] EPLUS_ENV_SB3_DQN-EVAL-Eplus-demo-v1-episodes-5_2023-04-05_15:09_MainThread_ROOT INFO:EnergyPlus working directory is in /workspaces/sinergym/examples/Eplus-env-SB3_DQN-EVAL-Eplus-demo-v1-episodes-5_2023-04-05_15:09-res1/Eplus-env-sub_run3
1 -1.086220480544118
2 -2074.74448257394
3 -4062.5949917628845
4 -5325.70363538299
5 -6253.469406019579
6 -7148.541871566801
7 -9784.916120573987
8 -12838.678311807513
9 -15794.291411067137
10 -18276.527032414844
11 -19042.489923154084
12 -20085.940757588887
1 -22178.803167409744
Episode 2 Mean reward: -0.6329567113986868 Cumulative reward: -22178.803167409744
[2023-04-05 15:14:02,027] EPLUS_ENV_SB3_DQN-EVAL-Eplus-demo-v1-episodes-5_2023-04-05_15:09_MainThread_ROOT INFO:EnergyPlus episode completed successfully.
[2023-04-05 15:14:02,029] EPLUS_ENV_SB3_DQN-EVAL-Eplus-demo-v1-episodes-5_2023-04-05_15:09_MainThread_ROOT INFO:Creating new EnergyPlus simulation episode...
[2023-04-05 15:14:02,171] EPLUS_ENV_SB3_DQN-EVAL-Eplus-demo-v1-episodes-5_2023-04-05_15:09_MainThread_ROOT INFO:EnergyPlus working directory is in /workspaces/sinergym/examples/Eplus-env-SB3_DQN-EVAL-Eplus-demo-v1-episodes-5_2023-04-05_15:09-res1/Eplus-env-sub_run4
1 -1.086220480544118
2 -2082.9054717823406
3 -4074.250922934873
4 -5327.335935890106
5 -6251.275680234238
6 -7147.772973553305
7 -9767.521129024877
8 -12840.88671128354
9 -15811.114469435519
10 -18272.994929371645
11 -19031.05416218702
12 -20078.863700569666
1 -22181.401316477346
Episode 3 Mean reward: -0.6330308594885183 Cumulative reward: -22181.401316477346
[2023-04-05 15:14:32,352] EPLUS_ENV_SB3_DQN-EVAL-Eplus-demo-v1-episodes-5_2023-04-05_15:09_MainThread_ROOT INFO:EnergyPlus episode completed successfully.
[2023-04-05 15:14:32,355] EPLUS_ENV_SB3_DQN-EVAL-Eplus-demo-v1-episodes-5_2023-04-05_15:09_MainThread_ROOT INFO:Creating new EnergyPlus simulation episode...
[2023-04-05 15:14:32,509] EPLUS_ENV_SB3_DQN-EVAL-Eplus-demo-v1-episodes-5_2023-04-05_15:09_MainThread_ROOT INFO:EnergyPlus working directory is in /workspaces/sinergym/examples/Eplus-env-SB3_DQN-EVAL-Eplus-demo-v1-episodes-5_2023-04-05_15:09-res1/Eplus-env-sub_run5
1 -1.086220480544118
2 -2084.652790803984
3 -4070.5495742366775
4 -5324.967427767579
5 -6250.029044962512
6 -7148.441029556015
7 -9778.087495747093
8 -12814.333239737103
9 -15767.066370294466
10 -18255.85311251997
11 -19016.03275121007
12 -20063.851931359102
1 -22163.97043890765
Episode 4 Mean reward: -0.6325334029368671 Cumulative reward: -22163.97043890765
[2023-04-05 15:15:02,528] EPLUS_ENV_SB3_DQN-EVAL-Eplus-demo-v1-episodes-5_2023-04-05_15:09_MainThread_ROOT INFO:EnergyPlus simulation closed successfully.
Finally, we register the evaluation data in wandb as an artifact to save it.
[25]:
artifact = wandb.Artifact(
name="evaluating",
type="evaluation1")
artifact.add_dir(
env.simulator._env_working_dir_parent,
name='evaluation_output/')
run.log_artifact(artifact)
# wandb has finished
run.finish()
wandb: Adding directory to artifact (/workspaces/sinergym/examples/Eplus-env-SB3_DQN-EVAL-Eplus-demo-v1-episodes-5_2023-04-05_15:09-res1)... Done. 0.1s
Waiting for W&B process to finish... (success).
View run SB3_DQN-Eplus-demo-v1-episodes-4_2023-04-05_14:51_1h487a7p at: https://wandb.ai/alex_ugr/sinergym_evaluations/runs/2wfphha2
Synced 5 W&B file(s), 0 media file(s), 101 artifact file(s) and 0 other file(s)
Synced 5 W&B file(s), 0 media file(s), 101 artifact file(s) and 0 other file(s)
Find logs at:
./wandb/run-20230405_150935-2wfphha2/logsWe have the loaded model results in our local computer, but we can see the execution in wandb too:
If we check the wandb project list, we can see that sinergym_evaluations project has a new run:
Hyperparameters tracked in the evaluation experiment and we can see the previous training artifact used to load the model:
Artifact registered with Sinergym Output (and CSV files generated with the Logger Wrapper):
