Problem

If you are going to do a ClickHouse node with multiple block storage, do not specify disk path and <path> in the same directory.

<storage_configuration>
    <disks>
        <disk_hdd>
            <path>/var/lib/clickhouse-hdd/</path>
        </disk_hdd>
        <disk_ssd>
            <path>/var/lib/clickhouse-ssd/</path>
        </disk_ssd>
    </disks>
    <policies>
        <moving_to_ssd>
            <volumes>
                <hot>
                    <disk>disk_ssd</disk>
                </hot>
                <cold>
                    <disk>disk_hdd</disk>
                </cold>
            </volumes>
            <move_factor>0.2</move_factor>
        </moving_to_ssd>
    </policies>
</storage_configuration>
<!-- Path to data directory, with trailing slash. -->
<path>/var/lib/clickhouse-hdd/</path>

CREATE TABLE stripcash.events
(
    `type` UInt64,
    `eventAt` DateTime,
    `eventDate` Date DEFAULT toDate(clickAt),
)
ENGINE = MergeTree()
PARTITION BY toYYYYMM(eventDate)
SETTINGS storage_policy = 'moving_to_ssd'

Reason

If you start server with the config you don’t get any notice and errors, but after a while when you start using the table and reload the server. You get an error:

2020.06.15 05:54:59.867745 [ 195173 ] {} <Error> Application: DB::Exception: Part `201811_0_2998_6` was found on disk `default` which is not defined in the storage policy: Cannot attach table `default`.`events` from metadata file

ClickHouse thinks that parts which are found in /var/lib/clickhouse-hdd/ belongs to default disk in <path> section. But events table doesn’t have default disk in the storage policy.

Solution

1. Create directories /var/lib/clickhouse-hdd/hdd and /var/lib/clickhouse-hdd/default (make sure the owner of the directories is clickhouse)
2. Change the config

<storage_configuration>
    <disks>
        <disk_hdd>
            <path>/var/lib/clickhouse-hdd/hdd/</path>
        </disk_hdd>
        <disk_ssd>
            <path>/var/lib/clickhouse-ssd/</path>
        </disk_ssd>
    </disks>
    <policies>
        <moving_to_ssd>
            <volumes>
                <hot>
                    <disk>disk_ssd</disk>
                </hot>
                <cold>
                    <disk>disk_hdd</disk>
                </cold>
            </volumes>
            <move_factor>0.2</move_factor>
        </moving_to_ssd>
    </policies>
</storage_configuration>
<!-- Path to data directory, with trailing slash. -->
<path>/var/lib/clickhouse-hdd/default/</path>

3. Move everything from /var/lib/clickhouse-hdd/ to /var/lib/clickhouse-hdd/default except hdd и default
4. Move /var/lib/clickhouse-hdd/default/data/events to /var/lib/clickhouse-hdd/hdd/data/events
5. Run the server.

In this case, all parts belong to moving_to_ssd policies will be in the right place.

ttps://github.com/ClickHouse/ClickHouse/issues/11678