725 Commits

Author SHA1 Message Date
ViacheslavKlimov
aa0179303c Subscribe to Zookeeper events only after application starts
Sometimes ZK events came before the app start, causing repartition change events, while event listeners are not yet initialized by Spring
2025-02-27 12:45:22 +02:00
ViacheslavKlimov
e7ed98910a Merge branch 'master' of github.com:thingsboard/thingsboard into edqs 2025-02-25 11:10:31 +02:00
Igor Kulikov
5cf26d4851 Update license header 2025-02-25 09:39:16 +02:00
Igor Kulikov
b7750bc342 Version set to 4.0.0-SNAPSHOT 2025-02-25 09:28:44 +02:00
Igor Kulikov
6a0fff0737 Version set to 3.9.1-RC 2025-02-25 09:27:01 +02:00
ViacheslavKlimov
a6a6112ecc Cleanup for EDQS 2025-02-24 12:11:05 +02:00
ViacheslavKlimov
278ec36f37 Fixes for EDQS 2025-02-21 15:48:33 +02:00
ViacheslavKlimov
965210f17b Multiple improvements for EDQS 2025-02-21 14:39:26 +02:00
ViacheslavKlimov
347cb5bc36 EDQS code cleanup 2025-02-21 11:48:21 +02:00
ViacheslavKlimov
a48144d2bb EDQS code cleanup 2025-02-20 17:34:28 +02:00
ViacheslavKlimov
92b83335c0 Merge branch 'master' of github.com:thingsboard/thingsboard into edqs 2025-02-20 12:25:17 +02:00
ViacheslavKlimov
7519889c1a Refactoring for EDQS repartitioning 2025-02-20 12:25:08 +02:00
ViacheslavKlimov
ded6daf2b3 Kafka states restore improvements 2025-02-19 15:44:05 +02:00
ViacheslavKlimov
6dd08276f8 Merge branch 'rc' of github.com:thingsboard/thingsboard 2025-02-14 13:45:35 +02:00
ViacheslavKlimov
bb7ea08e59 EDQS: refactoring, OOM handling, healthcheck 2025-02-14 12:08:05 +02:00
Volodymyr Babak
1b4930ebaf Introduced separate count for edge consumers 2025-02-10 18:51:13 +02:00
Volodymyr Babak
3860d79613 Improved Edge handling in case connect/disconnect. Speed up initial setup 2025-02-10 18:42:25 +02:00
dashevchenko
b2bf5b9f65 configurable edqs sync batch sizes 2025-02-05 12:20:23 +02:00
dashevchenko
0e9d0b3ba3 fixed EntityServiceTest 2025-02-03 18:27:36 +02:00
ViacheslavKlimov
c8db304ce6 Optimized postgres sync (#3208)
* optimized postgres sync

* Fix consumer stopping when stopWhenRead enabled

* Fix NPE on EDQS repartitioning

* fixed EntityServiceTest

* Fix EDQS yml props

---------

Co-authored-by: ViacheslavKlimov <viacheslavklimov11@gmail.com>
2025-01-30 12:43:14 +02:00
Viacheslav Klimov
86b5378d59 EDQS (#3196)
* Experiments with CSV

* CSV Loader v1

* EDQ tests

* Volatile variables instead of final

* Improvements

* updated loader with new entities

* Fix double memory usage issue

* Basic data structures and load

* Minor improvements

* Snappy + Large String reuse

* added EntityFields classes for each entity

* Basic implementation

* Minor improvements to KeyFilters

* implemented RepositoryUtils.checkKeyFilters

* Generic query implementation

* New structure

* Refactoring and few processors implementation

* extended DeviceData with shared/client attributes and device profile

* Minor refactoring of attribute scopes

* DeviceTypeFilter support

* Strong types of fields for each entity data class

* DeviceType and AssetType filters

* EntityView and Edge queries

* Relations Query

* Relation Query Implementation

* Update EDQS module version

* Sync with EDQS via Kafka

* EDQS: major refactoring

* EDQS API requests via Kafka

* EDQS: full sync with the database

* Refactoring for EDQS sync

* EDQS: major refactoring and new features

* EDQS refactoring, count query support, fix tests

* EDQS: refactoring for query processors

* Fix EDQS pom version

* Cleanup edqs.yml

* EDQS: tenant partitioning strategy; refactoring

* EDQS: latest events queue

* EDQS: support for monolith setup; RocksDB; other improvements

* EDQS: merge sync and events topics, introduce state topic

* EDQS: dynamic repartitioning

* implemented entity data query filters for edqs

* EdqsEntityQueryControllerTest - use in-memory queue

* edqs-filter fixes, added test

* EDQS: blob entity support

* EdqsEntityQueryControllerTest - use in-memory queue

* Use DummyEdqsService when disabled

* Fixes for EDQS

* Refactoring for EDQS tests

* Fix edqs requests partitioning

* EDQS: Fix for attributes handling

* Fix attributes saving in EntityServiceTest

* EDQS: refactoring, fixes

* Minor refactoring for query processor

* added ownerName/ownerType support

* fixed relation query processor

* fixed EntityServiceTest

* refactoring

* added support for parentId for relation query result

* Get rid of EntityNameFetcher

* Add fixme for relation query processor

* db restore with select all edqs fields

* fixed entity deletion

* fixed FieldUtils with new EntityFields

* dao method renamed

* EDQS: instance groups with same partitions; automatic sync; multiple fixes

* Refactoring for EDQS sync

* EDQS: refactoring

* Fix startup with Kafka

* fixed EntityQueryControllerTest

* fixed EdqsEntityServiceTest

* Separate queue admin for EDQS request template

* Implement new EDQS partitioning strategy

* EDQS: multiple fixes and refactoring

* Add mock EdqsRocksDb beans to tests

* added edqs stats for inmemory/grafana

* fixed filter tests

* Update todos

* Refactoring for QueueConfig

* Improvements and refactoring for EDQS consumers

* implemented TODOs

* test fixes

* Consume state topic up to end offsets

* edqs stats refactoring

* EDQS: cleanup on partitions removal; refactoring

* EDQS: minor refactoring

* EDQS: remove CSV loader

---------

Co-authored-by: Andrii Shvaika <ashvayka@thingsboard.io>
Co-authored-by: dashevchenko <dshevchenko@thingsboard.io>
2025-01-27 17:41:12 +02:00
ViacheslavKlimov
d4d3240f78 Merge branch 'master' of github.com:thingsboard/thingsboard into develop/4.0 2024-12-23 11:15:16 +02:00
ViacheslavKlimov
1ea5fdf7fe Merge branch 'master' of github.com:thingsboard/thingsboard into feature/tbmsg-refactoring 2024-12-23 10:51:11 +02:00
ViacheslavKlimov
e4cc2b4f9f Introduce TbMsgTransformer 2024-12-13 13:23:15 +02:00
ViacheslavKlimov
e920525456 Merge branch 'master' of github.com:thingsboard/thingsboard into develop/4.0 2024-12-05 11:26:34 +02:00
Igor Kulikov
77d2826f96 Version set to 4.0.0-SNAPSHOT 2024-12-04 16:53:19 +02:00
Igor Kulikov
190bba72f9 Temp Version set to 3.9.0-RC 2024-12-04 16:52:14 +02:00
Igor Kulikov
8ac0468edd Version set to 4.0.0-SNAPSHOT 2024-12-04 16:51:03 +02:00
Igor Kulikov
cca4ac3629 Version set to 3.9.0-RC 2024-12-04 16:44:31 +02:00
Dmytro Skarzhynets
68aaf3c918 Merge branch 'master' into safe-scheduler
# Conflicts:
#	common/util/src/main/java/org/thingsboard/common/util/ThingsBoardExecutors.java
2024-12-02 17:43:07 +02:00
Andrii Landiak
988dfce500 Refactor KafkaEdgeTopicsCleanUpService to properly clean up both deleted edges and expired topics 2024-11-29 15:18:27 +02:00
Andrii Landiak
63b53a4a45 Final improvement after review 2024-11-28 16:51:55 +02:00
Andrii Landiak
76bed44705 Minor improvement 2024-11-28 15:16:20 +02:00
Andrii Landiak
b796355554 Merge remote-tracking branch 'upstream/master' into feature/edge-kafka-events 2024-11-26 09:42:31 +02:00
Andrii Landiak
c388034d01 Delete EdgeGrpcSession interface: use abstract instead 2024-11-22 10:26:06 +02:00
Andrii Landiak
b981281205 Fix tests 2024-11-01 11:37:11 +02:00
Andrii Landiak
50a55d15a9 Add poll properties for edge-event kafka poll 2024-10-31 18:12:16 +02:00
Andrii Landiak
38eeada8ec Implement clean up service to clear empty edge topics 2024-10-30 12:29:07 +02:00
ViacheslavKlimov
12ca1739d4 Remove all queue types except Kafka 2024-10-23 15:30:07 +03:00
Andrii Landiak
3ecbd3258e Introduce kafka approach to work with edge-event instead of postgres 2024-10-22 12:38:57 +03:00
ViacheslavKlimov
2c0b7eb2b3 Deprecate all queue types except Kafka and in-memory 2024-10-18 14:11:59 +03:00
Igor Kulikov
fdbe59c610 Version set to 3.9.0-SNAPSHOT 2024-10-14 18:32:35 +03:00
Igor Kulikov
d11f866856 Version set to 3.8.1-RC 2024-10-14 18:27:21 +03:00
Igor Kulikov
b4124f7c4c Merge with RC 2024-09-26 11:10:56 +03:00
Igor Kulikov
4d705f26fd Version set to 3.8.0-RC 2024-09-26 11:09:26 +03:00
Igor Kulikov
3dadc34b5b Version set to 3.9.0-SNAPSHOT 2024-09-26 10:38:27 +03:00
Igor Kulikov
025fb1ca8c Version set to 3.8.0 2024-09-26 09:46:57 +03:00
Dmytro Skarzhynets
bbe328d158 Implemented safe scheduled thread pool 2024-09-18 19:46:17 +03:00
ViacheslavKlimov
ac460e8c3b Merge remote-tracking branch 'origin/hotfix/3.7' 2024-09-17 12:45:27 +03:00
YevhenBondarenko
f10f63bb8a fixed concurrent modification in TopicService 2024-09-17 11:25:50 +02:00