[Question] Configuring Experiment

[mbignotti]

In Hydra docs, they described a nice pattern for Configuring Experiments, which is especially useful for data science.
I would like to ask, how would you go about repeating a similar pattern with Hydra-Zen?

Thanks a lot!

[rsokl]

Thanks for asking this @mbignotti , I will try to find time in the next couple of days to answer 😄

[jgbos]

First, you may have realized I deleted a comment where I accidently replied to @rsokl comment, this replaces that with a fix to a user error I had.

---

Thanks for asking this @mbignotti. This is definitely something we should have in our docs. @rsokl I'll take a cut at this using our latest features zen and store. Here's the code:

from dataclasses import dataclass

from hydra_zen import MISSING, make_config, store, to_yaml, zen

# Here we utilize hydra-zen's `store` to
# add configurations to the Hydra ConfigStore

# First create stores for each group
db_store = store(group="db")
server_store = store(group="server")
experiment_store = store(group="experiment")


# add to named configurations for the `db` group


@db_store(name="mysql")
@dataclass
class MySQL:
    name: str = "mysql"


@db_store(name="sqlite")
@dataclass
class SQLITE:
    name: str = "sqlite"


# add to named configurations for the `server` group


@server_store(name="apache")
@dataclass
class Apache:
    name: str = "apache"
    port: int = 80


@server_store(name="nginx")
@dataclass
class NGINX:
    name: str = "nginx"
    port: int = 80


# utilize `make_config` to easily create configurations with
# hydra defaults

Config = make_config(
    hydra_defaults=["_self_", {"db": "sqlite"}, {"server": "apache"}],
    db=MISSING,
    server=MISSING,
)


# the experiment configs:
# - must inherit from `Config` 
# - must set `package="_global_"`

experiment_store(
    make_config(
        hydra_defaults=["_self_", {"override /db": "sqlite"}],
        server=dict(port=8080),
        bases=(Config,),
    ),
    name="aplite",
    package="_global_",
)


experiment_store(
    make_config(
        hydra_defaults=[
            "_self_",
            {"override /db": "sqlite"},
            {"override /server": "nginx"},
        ],
        server=dict(port=8080),
        bases=(Config,)
    ),
    name="nglite",
    package="_global_"
)


# put main configuration in the store
store(
    Config,
    name="config",
)

# the task function

def task(db, server):
    print("db:")
    print(to_yaml(db))

    print("server:")
    print(to_yaml(server))


if __name__ == "__main__":
    store.add_to_hydra_store()
    zen(task).hydra_main(config_path=None, config_name="config", version_base="1.2")

You should be able to run the following experiments:

$ python my_app.py
db:
name: sqlite

server:
name: apache
port: 80

$ python my_app.py db=mysql server.port=8080
db:
name: mysql

server:
name: apache
port: 8080

$ python my_app.py  +experiment=aplite
db:
name: sqlite

server:
name: apache
port: 8080

$ python my_app.py --multirun +experiment=aplite,nglite
[2023-01-17 10:45:25,609][HYDRA] Launching 2 jobs locally
[2023-01-17 10:45:25,609][HYDRA]        #0 : +experiment=aplite
db:
name: sqlite

server:
name: apache
port: 8080

[2023-01-17 10:45:25,713][HYDRA]        #1 : +experiment=nglite
db:
name: sqlite

server:
name: nginx
port: 8080

[jgbos]

Also, I wanted to show using pure dataclasses, but you can easily just use make_config (or builds for targeted configs):

db_store(make_config(name="mysql"), name="mysql")
...

server_store(make_config(name="apache", port=80), name="apache")
...

[rsokl]

Also, I wanted to show using pure dataclasses, but you can easily just use make_config (or builds for targeted configs):

Another neat trick that store provides, which is a bit subtle, is that you can avoid having to specify name twice by telling the store how to infer the store-entry's name from the config:

I.e.

# will infer store-entry name from .name attr of config
auto_name_store = store(name=lambda cfg: cfg.name)

# First create stores for each group
db_store = auto_name_store(group="db")
server_store = auto_name_store(group="server")
experiment_store = store(group="experiment")


# add to named configurations for the `db` group
db_store(make_config(name="mysql"))
db_store(make_config(name="sqlite"))

# add to named configurations for the `server` group
server_store(make_config(name="apache", port=80))
server_store(make_config(name="nginx", port=80))

[mbignotti]

Hi @jgbos, @rsokl,
Thanks a lot for the clean and exhaustive answers.
I think this is a really neat pattern. It should definitely make it to the docs to convince even more people to start using Hydra-Zen!
Thanks again!

[rsokl]

It should definitely make it to the docs

Done! 😄 https://mit-ll-responsible-ai.github.io/hydra-zen/changes.html#documentation-2023-01-22