Apache IoTDB 1.3.2

Features & Improvements

• Storage Module: Performance improvement in the insertRecords interface for writing
• Query Module: New Explain Analyze statement added (monitoring the time spent on each stage of a single SQL execution)
• Query Module: New UDAF (User-Defined Aggregate Function) framework added
• Query Module: New MaxBy/MinBy functions added, supporting the retrieval of maximum/minimum values along with the corresponding timestamps
• Query Module: Performance improvement in value filtering queries
• Data Synchronization: Path matching supports path pattern
• Data Synchronization: Supports metadata synchronization (including time series and related attributes, permissions, etc.)
• Stream Processing: Added Alter Pipe statement, supporting hot updates of plugins for Pipe tasks
• System Module: System data point count statistics now include statistics for data imported by loading TsFile
• Scripts and Tools: New local upgrade backup tool added (backing up original data through hard links)
• Scripts and Tools: New export-data/import-data scripts added, supporting data export in CSV, TsFile formats or SQL statements
• Scripts and Tools: Windows environment now supports distinguishing ConfigNode, DataNode, and Cli by window name
  ...

Bugs

• Optimize the error message when a NullPointerException (NPE) occurs due to a timeout when dropping a database.
• Add logs for notifyLeaderReady, notifyLeaderChanged, and procedure worker.
• Add compatibility handling for existing erroneous data during file merging.
• Fix the deadlock issue caused by flushing empty files during querying.
• Fix the issue where Ratis becomes unresponsive during read, write, and delete operations.
• Fix the concurrent bug in load and merge operations.
• Fix the issue where the system's compression ratio is recorded as a negative number in the file for certain scenarios.
• Fix the ConcurrentModificationException issue during memory estimation for merge tasks.
• Fix potential deadlocks that may occur when writing, automatically creating, and deleting databases concurrently.
  ...

What's Changed

• Delete write mem control parameters by @l2280212 in #12007
• Pipe: Ensure at-least-once semantic by forcibly starting PipeHistoricalDataRegionTsFileExtractor after pipe restart by @Caideyipi in #12010
• Separate the synchronized block of AbstractCompactionEstimator file info cache by @shuwenwei in #12019
• Pipe: using clusterId to judge whether the target cluster is source cluster by @xuanronaldo in #11994
• Storage Engine: fix wal delete thread exception because multi thread safety question by @caozj1011 in #12020
• PBTree: Fix Dead Lock and Refactor write/update inteface by @bigreybear in #11985
• Remove sql merge by @shuwenwei in #12021
• Load: Trigger last flush time map & tsfile resource degrading after successful load by @SteveYurongSu in #12024
• feat(jdbc):getDouble add type check by @CritasWang in #12023
• Remove infinite retry logic for replica inconsistency to avoid potential problems by @OneSizeFitsQuorum in #12028
• Pipe: Support "sink.host" and "connector.host" key in receiver IP specification by @Caideyipi in #12017
• Fixed Ratis query not retrying when DataNode restarts by @OneSizeFitsQuorum in #12029
• Pipe: support replace and modify mode for alter pipe sql by @VGalaxies in #12018
• Storage Engine: put DeviceCache into the total memory of the storage engine by @caozj1011 in #12016
• Bump jackson.version from 2.15.2 to 2.15.3 by @dependabot in #12014
• IT framework supports restarting cluster by @l2280212 in #12022
• Fix typo for parameter dn_multi_dir_strategy by @OneSizeFitsQuorum in #12033
• Upgrade github action version to fix the warning by @HTHou in #12032
• Introduce UDAF(User-Defined Aggregation Functions) by @Sh-Zh-7 in #12005
• Pipe: Move parsing event logic into PipeProcessorSubtask to separate CPU intensive and IO intensive tasks by @DanielWang2035 in #11965
• DataNodeDevicePathCache should use free memory of memtable by @HTHou in #12034
• Pipe: Revert "Pipe: stop pipe using restarting strategy to unpin the wal's reference count to avoid WAL stacking (#11971)" to avoid unnecessary pipe drop during subtask exception handling by @Caideyipi in #12031
• Pipe: Removed mayLinkedTsFileCountReachDangerousThreshold judgement from stuck restart logic by @Caideyipi in #12041
• Pipe: fixed the bug that processor dies when encountered memory shortage for tablets in a period of time by @Caideyipi in #12042
• Pipe: Reduced pipe logs and controlled how often the log is printed below a certain frequency by @Caideyipi in #11973
• [IOTDB-6296] Fix memory leak in MQTTService by @JackieTien97 in #12046
• Pipe: Pattern parsing pruning: When pattern is at the level below database, the parsing logic can be skipped if tsfiles / tablets completely match with the pattern by @Caideyipi in #12049
• Bump org.apache.commons:commons-compress from 1.21 to 1.26.0 by @dependabot in #12058
• Bump ip from 2.0.0 to 2.0.1 in /iotdb-connector/grafana-plugin by @dependabot in #12057
• Pipe: fixed potential lose point bug caused by cancelled flush of historical extractor by @Caideyipi in #12056
• Fix it which use 'merge' command by @shuwenwei in #12060
• Pipe: Fixed the bug that CN cannot sense drop pipe failure in meta sync and may lead to constantly skip of drop pipe by @Caideyipi in #12059
• [IOTDB-6297] Optimize the distribute plan in the situation of aggregation with align by device by @Beyyes in #12043
• [IOTDB-6298] Fix number overflow in group by time interval by @JackieTien97 in #12065
• [IOTDB-6299] Fix bug in merging overlapped data process caused by filter & offset push down by @liuminghui233 in #12068
• Introduce Aggregation Function MinBy by @lancelly in #12071
• Add recover IT for Procedure, and delete ProcedureStore by @liyuheng55555 in #12045
• PBTree: Implement dual-buffer container for MNode management by @linxt20 in #12048
• Optimize flushing memtable check by @HTHou in #12081
• [IOTDB-6302] Enhance the support of ISO_LOCAL_DATE_TIME timestamp format by @HTHou in #12086
• IT: revert some changes in restart IT to make the error log clear when failed to restart by @l2280212 in #12066
• change ci to github ci by @CritasWang in #12073
• feat(bat): add title and set format by @CritasWang in #12069
• Fix dead lock in PBTree.SchemaFile by @bigreybear in #12079
• fix.root privileges lost and grant role. by @ColinLeeo in #12091
• [IOTDB-6300] Support place time column at any column index in insert statement by @JackieTien97 in #12074
• Make Drop database timeout return message more detailed by @Cpaulyz in #12082
• Fix VolatileSubtreeIterator missing update buffer if all nodes in new buffer are not qualified by @Cpaulyz in #12093
• Pbtree: MNode iterating with merge sort upon disk and buffer by @linxt20 in #12077
• Add FileUtils.moveFileSafe function by @liyuheng55555 in #12094
• Add logs for data partition allocation by @CRZbulabula in #12090
• fix: IoTDBPreparedStatement implement addBatch function by @CritasWang in #12095
• [IOTDB-6303] Correct the default path of user, role and procedure folder by @CRZbulabula in #12089
• Cache clusterId in IoTDBConfig by @liyuheng55555 in #12053
• Implement new plan optimizer: PredicatePushDown by @liuminghui233 in #11859
• Optimize ConfigNode ConsensusManager init logic by @OneSizeFitsQuorum in #12098
• Do not wait to retry when configLeader exists by @Pengzna in #12075
• Faster repair data by @shuwenwei in #12039
• Bump org.scala-lang:scala-library from 2.12.18 to 2.12.19 by @dependabot in #12104
• Bump flink.version from 1.17.0 to 1.17.2 by @dependabot in #12103
• [IOTDB-6305] Add show current_timestamp statement by @yx-zhang in #12106
• [IOTDB-6306] Fix the issue that UDTF with boolean type is not supported in filter by @lancelly in #12105
• [IOTDB-6301] Optimize insert first by @CRZbulabula in #12080
• Add remove TsFile for DataRegion by @HeimingZ in #12126
• Fix fail to read memTable ids from the wal file by @HTHou in #12132
• Pipe Subscription: initialize the subscription client RPC payload by @VGalaxies in #12121
• Fix some ignored ITs of udf lib by @HTHou in #12135
• Fix compaction schedule task manager cannot stop and modify some output in log by @shuwenwei in #12125
• [IOTDB-6304] Support EXPLAN ANALYZE in query. by @ycycse in #12100
• Pipe: fixed the bug that released PipeRawTabletInsertionEvent's tablet is still referenced and causing OOM by @Caideyipi in #12117
• flink-iotdb-connector: IoTDBSink throws NPE by @xuanronaldo in #12133
• [IOTDB-6297] Optimize the distribute plan in aggregation align by device when some device cross data regions by @Beyyes in #12102
• Set ConfigNode list early during DataNode registration by @liyuheng55555 in #12137
• Revert "change ci to github ci" by @HTHou in #12140
• Ignore log: memtableId -9223372036854775808 not found in MemTableId2Info by @SteveYurongSu in #12141
• [IOTDB-6309] Lager interval for auto ratis leader balance by @CRZbulabula in #12116
• Fixed ConfigNode startup error in SimpleConsensus by @OneSizeFitsQuorum in #12143
• Pipe meta sync: support meta data sync between different clusters by @SteveYurongSu in #11794
• Remove dependency on org.openjdk.jol.jol-core and io.airlift.slice by @JackieTien97 in #12111
• fix bug : the change logic error of the CachedMNodeContainerIterator by @linxt20 in #12147
• Pipe: fix PipeRealtimeDataRegionExtractor cannot fetch time partition bound by @VGalaxies in #12118
• Fix bug: parent procedure should not execute before sub-procedure finished by @liyuheng55555 in #12134
• Pipe: fix legacy receiver's unsafe execution race problem (fix IoTDBPipeDataSinkIT.testLegacyConnector) by @SteveYurongSu in #12152
• Pipe IT: add parameter "connector.batch.enable" = "false" for IoTDBPipeNullValueIT to temp fix CI by @SteveYurongSu in #12153
• Fix StorageEngine recover slowly by @shuwenwei in #12151
• Pipe: Fixed potential lose point bug when an unclosed tsfile reaches isEventTimeOverlappedWithTimeRange() judgement (CI fails on testInsertNullValueTemplate) by @Caideyipi in #12156
• [IOTDB-6301] Optimize insert first by @BUAAserein in #12136
• Pipe: support IoTDB-style pattern by @DanielWang2035 in #12085
• Pipe Meta: Completed the logical view related logics & some bug fixes in IT and dirs by @Caideyipi in #12159
• UDF: add "timestampPrecision" on UDFParameters by @caozj1011 in #12158
• Fix cpp client IT on windows by @HTHou in #12162
• Pipe: Refactored IT to complement necessary flush to all the pipe ITs by @Caideyipi in #12164
• [IOTDB-6310] Optimize for query resource init by @JackieTien97 in #12160
• fix error log caused by ClosedByInterruptException in compaction module by @shuwenwei in #12142
• Refactor IQueryExecution to support Table Model in the future by @JackieTien97 in #12157
• fix(jdbc): when catalog is "Apache IoTDB", ignore it by @CritasWang in #12166
• Print native request api in datanode_slow_sql.log by @Beyyes in #12155
• Bump google.golang.org/protobuf from 1.31.0 to 1.33.0 in /iotdb-connector/grafana-plugin by @dependabot in #12168
• Random schedule compaction of bound data regions in one compaction schedule worker by @shuwenwei in #12167
• Fixed logical view creation with "into" pattern failure problem by @Caideyipi in #12169
• Support schema region snapshot parser. by @ColinLeeo in #12145
• Pipe: implemented sdt-sampling-processor & refactored down-sampling-processor by @xuanronaldo in #12114
• UDF: add an udf function for envelope demodulation analysis by @caozj1011 in #12146
• Pipe: Fixed create logical view idempotency problem & some IT check coverage missing by @Caideyipi in #12170
• Integration test cluster stop faster by @liyuheng55555 in #12110
• Fix Concurrent bug between NodeSupplier.close() and its run() by @JackieTien97 in #12175
• Fix wal delete sort method by @HeimingZ in #12176
• Fixed simple consensus recover problem by @Caideyipi in #12173
• Update iotdb-doap.rdf to update the info in https://projects.apache.org/project.html?iotdb by @HTHou in #12179
• Thrift zero-copy optimization for TElasticFramedTransport by @MrQuansy in #12050
• Optimize Count/Delete Timeseries root.** or root.db.** by @Cpaulyz in #12174
• Subscription: Subscription Management On ConfigNode by @MiniSho in #12149
• Bump follow-redirects from 1.15.4 to 1.15.6 in /iotdb-connector/grafana-plugin by @dependabot in #12182
• Bump org.apache.tomcat.embed:tomcat-embed-core from 9.0.83 to 9.0.86 by @dependabot in #12181
• Bump com.nimbusds:nimbus-jose-jwt from 9.34 to 9.37.2 by @dependabot in #12180
• Fix deadlock when flush an empty TsFile and query concurrently by @HTHou in #12171
• Update master verison to 1.3.2-SNAPSHOT by @HTHou in #12185
• Fixed the bug that TimeoutChangeableTFastFramedTransport is unused so that timeout changing is disabled in thrift transferring by @Caideyipi in #12186
• [IOTDB-6312] Throw correct error msg while using wrong password by @JackieTien97 in #12191
• Fix flush command failed when dataregion using ratis consensus by @HTHou in #12194
• Add more query metrics for estimated memory and max overlapped page by @Beyyes in #12172
• Change default threshold of slow query from 30s to 10s by @Beyyes in #12187
• Fix NPE in SessionConnection while first node in nodeurls is unavailable by @JackieTien97 in #12196
• Pipe: fix unable to handle timeseries already existed exception caused by CreateMultiTimeSeriesStatement & add throwing-exception-processor for auto recovery show cases by @SteveYurongSu in #12198
• Append time column at the end of input columns in mappable UDTF by @Sh-Zh-7 in #12199
• [IOTDB-6313] Fix uncorrect implementation in FILL(LINEAR) by @JackieTien97 in #12200
• [IOTDB-6314] Treat ExplainAnalyze as a query by @ycycse in #12204
• Add Query Metric for TsFileResourceManager by @JackieTien97 in #12205
• Bump black from 22.3.0 to 24.3.0 in /iotdb-client/client-py by @dependabot in #12207
• Bump org.springframework:spring-web from 5.3.30 to 5.3.33 by @dependabot in #12084
• [IOTDB-6301] Optimize insert first for single replica by @BUAAserein in #12209
• Replace string device id with interface IDeviceID by @shuwenwei in #12177
• Pipe: avoid executing too many PipeMetaSyncProcedure after system reboot by @SteveYurongSu in #12213
• Remove isClusterMode configuration by @HTHou in #12193
• Bump org.apache.commons:commons-configuration2 from 2.9.0 to 2.10.1 by @dependabot in #12214
• Refactor Procedure recover framework by @liyuheng55555 in #12202
• Display 'UNKNOWN' in SHOW FUNCTIONS when CN accidentally drop DN's function. by @Sh-Zh-7 in #12208
• Left rotate time column to the end of input columns in UDAF by @Sh-Zh-7 in #12201
• Introducing new features to IoTDB JVM GC options by @Pengzna in #12088
• Pipe: filtered empty tsFiles which should not be parsed or reported by @Caideyipi in #12216
• support config snapshot parser by @ColinLeeo in #12211
• Pipe Subscription: initialize the subscription agent on DN by @VGalaxies in #12138
• Fix NPE problem in EXPLAIN ANALYZE by @ycycse in #12218
• Allow series data type not consistent in compaction by @shuwenwei in #12203
• Remove duplicate lookups in dictionary encoder by @MrQuansy in #12224
• fix: consider more general type of nodes above TopK-Node as parent by @ycycse in #12225
• Pipe: Introduced aggregate processor as the universal aggregation framework by @Caideyipi in #12210
• Fix procedure concurrency problem caused by non-atomic procedure Id generation by @liyuheng55555 in #12229
• fix cn parser and code style. by @ColinLeeo in #12231
• UDF: Fix envelope function execution error without frequency parameter by @caozj1011 in #12233
• Log: add logs for notifyLeaderReady, notifyLeaderChanged and procedure workers by @SteveYurongSu in #12235
• Use heartbeat to update DataNode's ConfigNodeList by @liyuheng55555 in #12232
• [RatisConsensus] Remove JavaUtils dependencies by @SzyWilliam in #12236
• Warn user if they use GraalVM by @liyuheng55555 in #12238
• Region migration improvement by @liyuheng55555 in #12165
• Remove -XX:-UseBiasedLock in IoTDB JVM options by @Pengzna in #12239
• feat: Refactored the visibility of some methods to make it possible to add BasicAuthorizer implementations in other packages. by @chrisdutz in #12195
• Optimize SeriesScanUtil by memorizing the order time and satisfied information for each Seq and Unseq Resource by @JackieTien97 in #12227
• Subscription: fix some issues on CN by @DanielWang2035 in #12217
• change cli classpath by @CritasWang in #12237
• Fix C++ client build problem by @liyuheng55555 in #12241
• [RatisConsensus] Bump ratis snapshot version to support force snapshot by @SzyWilliam in #12242
• Optimize String value type infer logic by @HTHou in #12223
• IoTConsensus: Skip retry sending batch caused by TApplicationException by @BUAAserein in #12243
• Bump jackson.version from 2.15.3 to 2.15.4 by @dependabot in #12190
• Pipe: fix deadlock caused by PipeResourceManagerHolder. and PipePeriodicalJobExecutor by @SteveYurongSu in #12245
• [IOTDB-6316] Fix potential concurrent bug between compaction and load by @JackieTien97 in #12244
• Subscription: stop meta sync properly when no topic/consumer group by @DanielWang2035 in #12247
• Subscription: initialize the subscription session and consumer & Pipe: fix data points loss caused by unclosed tsfile listened by historical extractor by @VGalaxies in #12228
• Fix uncompress byte buffer in RepairDataFileScanUtil by @shuwenwei in #12250
• Pipe: Support mod transfer in pipe tsFiles & schema region snapshot transferring logic by @Caideyipi in #11998
• fix: consider more general type of nodes in optimization above LimitNode by @ycycse in #12240
• Forcibly take snapshot when upgrade procedure WAL by @liyuheng55555 in #12252
• Modify compaction task priority comparator by @shuwenwei in #12255
• Add the import and export function of SQl files on the basis of CSV import and export scripts by @2b3c511 in #12248
• invert insertRecords to insertTablets by @l2280212 in #12113
• Fix the mistaken argument in LZ4Uncompressor by @jt2594838 in #12249
• Subscription: fix some issues on DN and session & improve IT by @VGalaxies in #12254
• Pipe: Added handshake logic when config receiver is reset due to receiver side DataNode redirection & Enhance IT & Refactor by @Caideyipi in #12256
• Region migration related work by @liyuheng55555 in #12246
• [IOTDB-6307] GCR algorithm for multi-database by @CRZbulabula in #12183
• Region migration related work by @liyuheng55555 in #12259
• [IOTDB-6308] CFD algorithm for multi-database by @CRZbulabula in #12184

Full Changelog: v1.3.1...v1.3.2