Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MDEV-19191 Partial support of foreign keys in partitioned tables #3513

Open
wants to merge 13 commits into
base: main
Choose a base branch
from

Conversation

midenok
Copy link
Contributor

@midenok midenok commented Sep 12, 2024

The patch adds the ability to run foreign keys in partitioned tables with
limitations.

Example:

  create or replace table t1 (id int primary key) engine innodb;
  create or replace table t2 (fk int references t1(id)) engine innodb
  partition by hash (fk) partitions 2;

Limitations:

  1. Foreign keys cannot refer partitioned table even SYSTEM_TIME-partitioned
  still. create_foreign_keys() at InnoDB layer receives Foreign_key about
  referenced table T, but no such table exists in InnoDB layer, only partition
  tables T#P#p0, T#P#p1, ..., T#P#pn. Finding out it is SYSTEM_TIME partitioning
  and current partition at that point is impossible without modification of
  SYS_TABLES or opening TABLE object of referenced table. Both should be avoided
  as this is superseded by MDEV-12483.

  2. CASCADE and SET NULL actions are disabled in partitioned foreign table as
  these actions update row data and this is the subject of row placement into
  another partition but it cannot be done in InnoDB layer. DELETE CASCADE for
  SYSTEM_TIME partitioning requires the row to be moved from current to history
  partition.

The task is basically divided into 3 parts:

  1. Remove prohibiting code, allow FKs to be created for partitions;
  2. Allow partitioned FKs at SQL layer for such functions as SHOW CREATE;
  3. Implement correct handling of FKs when partitioning configuration changes.

1. Remove prohibiting code, allow FKs to be created for partitions

  In SYS_FOREIGN table foreign key records are unique by ID which was taken from
  constraint name or automatically generated. Normally foreign ID and constraint
  name are identical, but that's not the case for partitioned table as InnoDB
  holds foreign keys per dict_table_t object and each partition is a different
  table for InnoDB. So for each foreign key in SQL layer there is a set of foreign
  keys in InnoDB layer per each partition (except SYSTEM_TIME partitioning where
  we keep foreign keys only for current partition). To constitute unique foreign
  ID at InnoDB layer we concatenate constraint name with partition suffix, the one
  what is added to partition table name beginning with #P# and optionally
  containing #SP# for subpartitions. Constraint name and partitioning suffix are
  separated by \xFF character which is the safe character code non-clashing with
  identifier character set.

  When we return back foreign ID to SQL layer this partitioning suffix is stripped
  off the constraint name and SQL output receives the name similar to
  non-partitioned table.

  User may see a bit more truthful version of foreign ID in
  INFORMATION_SCHEMA.INNODB_SYS_FOREIGN with \xFF replaced by ':' and #P# or
  everything starting from #P# and ending by #SP# chopped out. So he may see
  corresponding partition name or subpartition name in ID of INNODB_SYS_FOREIGN.

2. Allow partitioned FKs at SQL layer for such functions as SHOW CREATE

  Through standard handler interface get_foreign_key_list() foreign keys are
  returned to SQL layer. For SYSTEM_TIME partitioning from current partition, for
  any other partitioning from first read-marked partition.

3. Implement correct handling of FKs when partitioning configuration changes

  ALTER operations such as ADD PARTITION, DROP PARTITION, REMOVE PARTITIONING,
  etc. are reflected into correct configuration of foreign keys in InnoDB.
  Handling of foreign key ID for temporary tables in ALTER was done based on
  MDEV-28933.

When there are GCC-incompatible compiler flags dtrace fails like this:

gcc: error: unrecognized command-line option ‘-fno-limit-debug-info’
gcc: error: unrecognized command-line option ‘-mbranches-within-32B-boundaries’
"gcc .dtrace-temp.3fd6bacf.c" failed
Usage /usr/bin/dtrace [--help] [-h | -G] [-C [-I<Path>]] -s File.d [-o <File>]
Return true for temporary partitions.
  Based on Marko Makela's patch 4a591d4:

  dict_get_referenced_table(): Let the caller convert names when needed.

  row_rename_table_for_mysql(): Specify the treatment of FOREIGN KEY
  constraints in a 4-valued enum parameter. In cases where FOREIGN KEY
  constraints cannot exist (partitioned tables, or internal tables of
  FULLTEXT INDEX), we can use the mode RENAME_IGNORE_FK.
  The mod RENAME_REBUILD is for any DDL operation that rebuilds the
  table inside InnoDB, such as TRUNCATE and native ALTER TABLE
  (or OPTIMIZE TABLE). The mode RENAME_ALTER_COPY is used solely
  during non-native ALTER TABLE in ha_innobase::rename_table().
  Normal ha_innobase::rename_table() will use the mode RENAME_FK.

  CREATE OR REPLACE will rename the old table (if one exists) along
  with its FOREIGN KEY constraints into a temporary name. The replacement
  table will be initially created with another temporary name.
  Unlike in ALTER TABLE, all FOREIGN KEY constraints must be renamed
  and not inherited as part of these operations, using the mode RENAME_FK.

  dict_get_referenced_table(): Let the callers convert names when needed.

  create_table_info_t::create_foreign_keys(): CREATE OR REPLACE creates
  the replacement table with a temporary name table, so for
  self-references foreign->referenced_table will be a table with
  temporary name and charset conversion must be skipped for it.
All InnoDB internal SQL functions should be moved to sql_funcs.h
Use temporary constraint names for temporary tables. The constraints
are not added to cache (skipped in dict_table_rename_in_cache()).

The scheme for temporary constraint names is as follows:

    for old table: db_name/\xFFconstraint_name
    for new table: db_name/\xFF\xFFconstraint_name

normalize_table_name_c_low(): wrong comparison "less than FN_REFLEN -
1". Somewhere array of FN_REFLEN includes the trailing 0, somewhere
array of FN_REFLEN + 1 includes trailing 0, but nowhere array of
FN_REFLEN - 1 must include trailing 0.

Change from original MDEV-28933 fix:

As temporary FK is now required for any ALTER operation (especially
partitioning operations) row_rename_table_for_mysql() does FK rename
on both RENAME_FK and RENAME_ALTER_COPY (see do_rename_fk).
The patch adds the ability to run foreign keys in partitioned tables with
limitations.

Example:

  create or replace table t1 (id int primary key) engine innodb;
  create or replace table t2 (fk int references t1(id)) engine innodb
  partition by hash (fk) partitions 2;

Limitations:

  1. Foreign keys cannot refer partitioned table even SYSTEM_TIME-partitioned
  still. create_foreign_keys() at InnoDB layer receives Foreign_key about
  referenced table T, but no such table exists in InnoDB layer, only partition
  tables T#P#p0, T#P#p1, ..., T#P#pn. Finding out it is SYSTEM_TIME partitioning
  and current partition at that point is impossible without modification of
  SYS_TABLES or opening TABLE object of referenced table. Both should be avoided
  as this is superseded by MDEV-12483.

  2. CASCADE and SET NULL actions are disabled in partitioned foreign table as
  these actions update row data and this is the subject of row placement into
  another partition but it cannot be done in InnoDB layer. DELETE CASCADE for
  SYSTEM_TIME partitioning requires the row to be moved from current to history
  partition.

The task is basically divided into 3 parts:

  1. Remove prohibiting code, allow FKs to be created for partitions;
  2. Allow partitioned FKs at SQL layer for such functions as SHOW CREATE;
  3. Implement correct handling of FKs when partitioning configuration changes.

1. Remove prohibiting code, allow FKs to be created for partitions

  In SYS_FOREIGN table foreign key records are unique by ID which was taken from
  constraint name or automatically generated. Normally foreign ID and constraint
  name are identical, but that's not the case for partitioned table as InnoDB
  holds foreign keys per dict_table_t object and each partition is a different
  table for InnoDB. So for each foreign key in SQL layer there is a set of foreign
  keys in InnoDB layer per each partition (except SYSTEM_TIME partitioning where
  we keep foreign keys only for current partition). To constitute unique foreign
  ID at InnoDB layer we concatenate constraint name with partition suffix, the one
  what is added to partition table name beginning with #P# and optionally
  containing #SP# for subpartitions. Constraint name and partitioning suffix are
  separated by \xFF character which is the safe character code non-clashing with
  identifier character set.

  When we return back foreign ID to SQL layer this partitioning suffix is stripped
  off the constraint name and SQL output receives the name similar to
  non-partitioned table.

  User may see a bit more truthful version of foreign ID in
  INFORMATION_SCHEMA.INNODB_SYS_FOREIGN with \xFF replaced by ':' and #P# or
  everything starting from #P# and ending by #SP# chopped out. So he may see
  corresponding partition name or subpartition name in ID of INNODB_SYS_FOREIGN.

2. Allow partitioned FKs at SQL layer for such functions as SHOW CREATE

  Through standard handler interface get_foreign_key_list() foreign keys are
  returned to SQL layer. For SYSTEM_TIME partitioning from current partition, for
  any other partitioning from first read-marked partition.

3. Implement correct handling of FKs when partitioning configuration changes

  ALTER operations such as ADD PARTITION, DROP PARTITION, REMOVE PARTITIONING,
  etc. are reflected into correct configuration of foreign keys in InnoDB.
  Handling of foreign key ID for temporary tables in ALTER was done based on
  MDEV-28933.
This commit is not intended to be pushed into main branch. Its mission
is to simplify the review process. The comments here are in different
style: // and without indentation. The reviewer may ask for some of
them to be included into the task code. In that case they will be
reformatted and moved to the task commit.
Related to FIXME here. Now check_bulk_buffer() returns false.

3389                    if (auto t= trx->check_bulk_buffer(index->table)) {
3390                            /* MDEV-25036 FIXME:
3391                            row_ins_check_foreign_constraint() check
3392                            should be done before buffering the insert
3393                            operation. */
3394                            ut_ad(index->table->skip_alter_undo
3395                                  || !trx->check_foreigns);
3396                            return t->bulk_insert_buffered(*entry, *index, trx);
3397                    }
@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants