TiDB 4.0.9 Release Notes

Release date: December 21, 2020

TiDB version: 4.0.9

Compatibility Changes

• TiDB

  • Deprecate the enable-streaming configuration item #21055

• TiKV

  • Reduce I/O and mutex contention when encryption at rest is enabled. The change is backwardly incompatible. If users need to downgrade the cluster to a version earlier than v4.0.9, security.encryption.enable-file-dictionary-log must be disabled and TiKV must be restarted before the downgrade. #9195

New Features

• TiFlash

  • Support storing the latest data of the storage engine on multiple disks (experimental)

• TiDB Dashboard

  • Support displaying and sorting by all fields in the SQL Statements page #749
  • Support zooming and panning the topology graph #772
  • Support displaying the disk usage information in the SQL Statements and Slow Queries pages #777
  • Support exporting list data in the SQL Statements and Slow Queries pages #778
  • Support customizing the Prometheus address #808
  • Add a page for cluster statistics #815
  • Add more time-related fields in the Slow Queries details #810

Improvements

• TiDB

  • Avoid the (index) merge join in a heuristical way when converting equal conditions to other conditions #21146
  • Differentiate the types of user variables #21107
  • Support setting the GOGC variable in the configuration file #20922
  • Make the dumped binary time (Timestamp and Datetime) more compatible with MySQL #21135
  • Provide an error message for statements that use the LOCK IN SHARE MODE syntax #21005
  • Avoid outputting unnecessary warnings or errors when folding constants in shortcut-able expressions #21040
  • Raise an error when preparing the LOAD DATA statement #21199
  • Ignore the attribute of the integer zero-fill size when changing the integer column types #20986
  • Add the executor-related runtime information of DML statements in the result of EXPLAIN ANALYZE #21066
  • Disallow multiple updates on the primary key in a singe SQL statements #21113
  • Add a monitoring metric for the connection idle time #21301
  • Temporarily enable the slow log when the runtime/trace tool is running #20578

• TiKV

  • Add the tag to trace the source of the split command #8936
  • Support dynamically changing the pessimistic-txn.pipelined configuration #9100
  • Reduce the impact on performance when running Backup & Restore and TiDB Lightning #9098
  • Add monitoring metrics for the ingesting SST errors #9096
  • Prevent the leader from being hibernated when some peers still need to replicate logs #9093
  • Increase the success rate of the pipelined pessimistic locking #9086
  • Change the default value of apply-max-batch-size and store-max-batch-size to 1024 #9020
  • Add the max-background-flushes configuration item #8947
  • Disable force-consistency-checks by default to improve performance #9029
  • Offload the queries on the Region size from pd heartbeat worker to split check worker #9185

• PD

  • Check the TiKV cluster version when a TiKV stores become Tombstone, which prevents users from enabling incompatible features during the process of downgrade or upgrade #3213
  • Disallow the TiKV store of a lower version to change from Tombstone back to Up #3206

• TiDB Dashboard

  • Keep expanding when "Expand" is clicked for SQL statements #775
  • Open detail pages in new windows for SQL Statements and Slow Queries #816
  • Improve descriptions for time-related fields in Slow Queries details #817
  • Display detailed error messages #794

• TiFlash

  • Reduce the latency of replica reads
  • Refine TiFlash's error messages
  • Limit the memory usage of cache data when the data volume is huge
  • Add a monitoring metric for the number of coprocessor tasks being handled

• Tools

  • Backup & Restore (BR)

    • Disallow the ambiguous --checksum false argument in the command line, which does not correctly disable checksum. Only --checksum=false is accepted. #588
    • Support changing the PD configuration temporarily so that PD can recover the original configuration after BR accidentally exists #596
    • Support analyzing tables after restore #622
    • Retry for the read index not ready and proposal in merging mode errors #626

  • TiCDC

    • Add an alert for enabling TiKV's Hibernate Region feature #1120
    • Reduce memory usage in the schema storage #1127
    • Add the feature of unified sorter, which accelerates replication when the data size of the incremental scan is large (experimental) #1122
    • Support configuring the maximum message size and the maximum message batch in the TiCDC Open Protocol message (only for Kafka sink) #1079

  • Dumpling

    • Retry dumping data on failed chunks #182
    • Support configuring both the -F and -r arguments at the same time #177
    • Exclude system databases in --filter by default #194
    • Support the --transactional-consistency parameter and support rebuilding MySQL connections during retry #199
    • Support using the -c,--compress parameter to specify the compression algorithm used by Dumpling. An empty string means no compression. #202

  • TiDB Lightning

    • Filter out all system schemas by default #459
    • Support setting a default value for the auto-random primary key for the Local-backend or Importer-backend #457
    • Use range properties to make the range split more precise in Local-backend #422
    • Support a human-readable format (such as "2.5 GiB") in tikv-importer.region-split-size, mydumper.read-block-size, mydumper.batch-size, and mydumper.max-region-size #471

  • TiDB Binlog

    • Exit the Drainer process with the non-zero code if the upstream PD is down or if applying DDL or DML statements to the downstream fails #1012

Bug Fixes

• TiDB

  • Fix the issue of incorrect results when using a prefix index with the OR condition #21287
  • Fix a bug that might cause panic when automatic retry is enabled #21285
  • Fix a bug that occurs when checking partition definition according to column type #21273
  • Fix a bug that the value type of the partition expression is not consistent with the partition column type #21136
  • Fix a bug that the hash-type partition does not check whether the partition name is unique #21257
  • Fix the wrong results returned after inserting a value of the non-INT type into the hash partitioned table #21238
  • Fix the unexpected error when using index join in the INSERT statement in some cases #21249
  • Fix the issue that the BigInt unsigned column value in the CASE WHEN operator is incorrectly converted to the BigInt signed value #21236
  • Fix a bug that index hash join and index merge join do not consider collation #21219
  • Fix a bug that the partitioned table does not consider collation in the CREATE TABLE and SELECT syntax #21181
  • Fix the issue that the query result of slow_query might miss some rows #21211
  • Fix the issue that DELETE might not delete data correctly when the database name is not in a pure lower representation #21206
  • Fix a bug that causes schema change after DML operations #21050
  • Fix the bug that the coalesced column cannot be queried when using join #21021
  • Fix the wrong results of some semi-join queries #21019
  • Fix the issue that the table lock does not take effect on the UPDATE statement #21002
  • Fix the issue of stack overflow that occurs when building the recursive view #21001
  • Fix the unexpected result returned when performing index merge join operations on outer join #20954
  • Fix the issue that sometimes a transaction that has an undetermined result might be treated as failed #20925
  • Fix the issue that EXPLAIN FOR CONNECTION cannot show the last query plan #21315
  • Fix the issue that when Index Merge is used in a transaction with the Read Committed isolation level, the result might be incorrect #21253
  • Fix the auto-ID allocation failure caused by the transaction retry after the write conflict #21079
  • Fix the issue that JSON data cannot be correctly imported to TiDB using LOAD DATA #21074
  • Fix the issue that the default value of newly added Enum-type columns is incorrect #20998
  • Fix the issue that the adddate function inserts invalid characters #21176
  • Fix the issue that the wrong PointGet plan generated in some situations causes wrong results #21244
  • Ignore the conversion of daylight saving time in the ADD_DATE function to be compatible with MySQL #20888
  • Fix a bug that prevents inserting strings with trailing spaces that exceed varchar or char's length constraint #21282
  • Fix a bug that does not converting the integer from [1, 69] to [2001, 2069] or from [70, 99] to [1970, 1999] when comparing int with year #21283
  • Fix the panic caused by the overflowing result of the sum() function when calculating the Double type field #21272
  • Fix a bug that DELETE fails to add lock on the unique key #20705
  • Fix a bug that snapshot reads hits the lock cache #21539
  • Fix an issue of potential memory leak after reading a lot of data in a long-lived transaction #21129
  • Fix the issue that omitting the table alias in a subquery will have a syntax error returned #20367
  • Fix the issue that when the argument of the IN function in a query is the time type, the query might return an incorrect result #21290

• TiKV

  • Fix the issue that Coprocessor might return wrong results when there are more than 255 columns #9131
  • Fix the issue that Region Merge might cause data loss during network partition #9108
  • Fix the issue that the ANALYZE statement might cause panic when using the latin1 character set #9082
  • Fix the wrong results returned when converting the numeric type to the time type #9031
  • Fix a bug that TiDB Lightning fails to ingest SST files to TiKV with the Importer-backend or Local-backend when Transparent Data Encryption (TDE) is enabled #8995
  • Fix the invalid advertise-status-addr value (0.0.0.0) #9036
  • Fix the issue that an error is returned indicating that a key exists when this key is locked and deleted in a committed transaction #8930
  • Fix the issue that the RocksDB cache mapping error causes data corruption #9029
  • Fix a bug that Follower Read might return stale data after the leader is transferred #9240
  • Fix the issue that stale old values might be read in the pessimistic lock #9282
  • Fix a bug that replica read might get stale data after the leader transfer #9240
  • Fix the issue of TiKV crash that occurs when receiving SIGPROF after profiling #9229

• PD

  • Fix the issue that the leader roles specified using placement rules do not take effect in some cases #3208
  • Fix the issue that the trace-region-flow value is unexpectedly set to false #3120
  • Fix a bug that the service safepoint with infinite Time To Live (TTL) does not work #3143

• TiDB Dashboard

  • Fix a display issue of time in the Chinese language #755
  • Fix a bug that the browser compatibility notice does not work #776
  • Fix the issue that the transaction start_ts is incorrectly displayed in some scenarios #793
  • Fix the issue that some SQL texts are incorrectly formatted #805

• TiFlash

  • Fix the issue that INFORMATION_SCHEMA.CLUSTER_HARDWARE might contain the information of disks that are not in use
  • Fix the issue that the estimate on memory usage of Delta Cache is smaller than the actual usage
  • Fix the memory leak caused by thread information statistics

• Tools

  • Backup & Restore (BR)

    • Fix the failure caused by special characters in S3 secret access keys #617

  • TiCDC

    • Fix the issue that multiple owners might exist when the owner campaign key is deleted #1104
    • Fix a bug that TiCDC might fail to continue replicating data when a TiKV node crashes or recovers from a crash. This bug only exists in v4.0.8. #1198
    • Fix the issue that the metadata is repeatedly flushed to etcd before a table is initialized #1191
    • Fix an issue of replication interruption caused by early GC or the latency of updating TableInfo when the schema storage caches TiDB tables #1114
    • Fix the issue that the schema storage costs too much memory when DDL operations are frequent #1127
    • Fix the goroutine leak when a changefeed is paused or stopped #1075
    • Increase the maximum retry timeout to 600 seconds in Kafka producer to prevent replication interruption caused by the service or network jitter in the downstream Kafka #1118
    • Fix a bug that the Kafka batch size does not take effect #1112
    • Fix a bug that some tables' row change might be lost when the network between TiCDC and PD has jitter and when there are paused changefeeds being resumed at the same time #1213
    • Fix a bug that the TiCDC process might exit when the network between TiCDC and PD is not stable #1218
    • Use a singleton PD client in TiCDC and fix a bug that TiCDC closes PD client by accident which causes replication block #1217
    • Fix a bug that the TiCDC owner might consume too much memory in the etcd watch client #1224

  • Dumpling

    • Fix the issue that Dumpling might get blocked when its connection to the MySQL database server is closed #190

  • TiDB Lightning

    • Fix the issue that keys are encoded using the wrong field information #437
    • Fix the issue that GC life time TTL does not take effect #448
    • Fix the issue that causes panic when manually stops the running TiDB Lightning in the Local-backend mode #484