735 Commits

Author SHA1 Message Date
ViacheslavKlimov
43f9083f70 Merge branch 'master' of github.com:thingsboard/thingsboard into edqs 2025-02-28 17:12:33 +02:00
Andrii Shvaika
62da129d6b Review comments fixed 2025-02-27 16:46:55 +02:00
ViacheslavKlimov
16169dc728 Improve ZkDiscoveryService shutdown 2025-02-27 15:58:26 +02:00
ViacheslavKlimov
aa0179303c Subscribe to Zookeeper events only after application starts
Sometimes ZK events came before the app start, causing repartition change events, while event listeners are not yet initialized by Spring
2025-02-27 12:45:22 +02:00
ViacheslavKlimov
44439584f4 Merge branch 'master' of github.com:thingsboard/thingsboard into feature/calculated-fields 2025-02-25 11:49:56 +02:00
ViacheslavKlimov
e7ed98910a Merge branch 'master' of github.com:thingsboard/thingsboard into edqs 2025-02-25 11:10:31 +02:00
Igor Kulikov
5cf26d4851 Update license header 2025-02-25 09:39:16 +02:00
ViacheslavKlimov
ab9bef4d90 Fix HashPartitionServiceTest 2025-02-24 12:19:33 +02:00
ViacheslavKlimov
a6a6112ecc Cleanup for EDQS 2025-02-24 12:11:05 +02:00
ViacheslavKlimov
278ec36f37 Fixes for EDQS 2025-02-21 15:48:33 +02:00
ViacheslavKlimov
965210f17b Multiple improvements for EDQS 2025-02-21 14:39:26 +02:00
ViacheslavKlimov
347cb5bc36 EDQS code cleanup 2025-02-21 11:48:21 +02:00
ViacheslavKlimov
a48144d2bb EDQS code cleanup 2025-02-20 17:34:28 +02:00
ViacheslavKlimov
8d161382e6 Merge branch 'master' of github.com:thingsboard/thingsboard into feature/calculated-fields 2025-02-20 14:08:43 +02:00
ViacheslavKlimov
92b83335c0 Merge branch 'master' of github.com:thingsboard/thingsboard into edqs 2025-02-20 12:25:17 +02:00
ViacheslavKlimov
7519889c1a Refactoring for EDQS repartitioning 2025-02-20 12:25:08 +02:00
ViacheslavKlimov
ed6ca38524 Fix locking in QueueStateService 2025-02-19 18:54:50 +02:00
Andrii Shvaika
535730b61a Minor renaming due to code review 2025-02-19 18:23:30 +02:00
ViacheslavKlimov
2b3af16530 CF states restore improvements 2025-02-19 17:27:12 +02:00
ViacheslavKlimov
5dc6991907 Kafka states restore improvements 2025-02-19 15:51:15 +02:00
ViacheslavKlimov
ded6daf2b3 Kafka states restore improvements 2025-02-19 15:44:05 +02:00
ViacheslavKlimov
ffc88eeb2d CF Kafka states cleanup 2025-02-18 15:28:46 +02:00
Andrii Shvaika
3cdc933b67 Merge improvements 2025-02-18 13:02:48 +02:00
ViacheslavKlimov
6dd08276f8 Merge branch 'rc' of github.com:thingsboard/thingsboard 2025-02-14 13:45:35 +02:00
ViacheslavKlimov
bb7ea08e59 EDQS: refactoring, OOM handling, healthcheck 2025-02-14 12:08:05 +02:00
ViacheslavKlimov
f342bd989e Minor improvements 2025-02-13 15:31:09 +02:00
ViacheslavKlimov
677385e8ba CF states restore from Kafka 2025-02-13 13:01:17 +02:00
Andrii Shvaika
d75bc3ac28 Minor improvements 2025-02-11 12:47:37 +02:00
Volodymyr Babak
1b4930ebaf Introduced separate count for edge consumers 2025-02-10 18:51:13 +02:00
Volodymyr Babak
3860d79613 Improved Edge handling in case connect/disconnect. Speed up initial setup 2025-02-10 18:42:25 +02:00
Andrii Shvaika
5efb94dc7a Refactoring 2025-02-10 15:58:01 +02:00
Andrii Landiak
b54e0cd59c Rename tbEdgeEventNotificationsTopic 2025-02-07 16:12:53 +02:00
Andrii Landiak
8de861f897 Improve customization for edge-event topic 2025-02-07 16:11:33 +02:00
Andrii Shvaika
407511be52 Cache implementation 2025-02-05 12:51:50 +02:00
dashevchenko
b2bf5b9f65 configurable edqs sync batch sizes 2025-02-05 12:20:23 +02:00
Andrii Landiak
e3a72fec09 Refactor TopicService. Change ymls to be able to configure notification topic names 2025-02-04 12:39:06 +02:00
dashevchenko
0e9d0b3ba3 fixed EntityServiceTest 2025-02-03 18:27:36 +02:00
ViacheslavKlimov
c8db304ce6 Optimized postgres sync (#3208)
* optimized postgres sync

* Fix consumer stopping when stopWhenRead enabled

* Fix NPE on EDQS repartitioning

* fixed EntityServiceTest

* Fix EDQS yml props

---------

Co-authored-by: ViacheslavKlimov <viacheslavklimov11@gmail.com>
2025-01-30 12:43:14 +02:00
IrynaMatveieva
71bb1c0250 merge with upstream 2025-01-28 09:51:30 +02:00
Viacheslav Klimov
86b5378d59 EDQS (#3196)
* Experiments with CSV

* CSV Loader v1

* EDQ tests

* Volatile variables instead of final

* Improvements

* updated loader with new entities

* Fix double memory usage issue

* Basic data structures and load

* Minor improvements

* Snappy + Large String reuse

* added EntityFields classes for each entity

* Basic implementation

* Minor improvements to KeyFilters

* implemented RepositoryUtils.checkKeyFilters

* Generic query implementation

* New structure

* Refactoring and few processors implementation

* extended DeviceData with shared/client attributes and device profile

* Minor refactoring of attribute scopes

* DeviceTypeFilter support

* Strong types of fields for each entity data class

* DeviceType and AssetType filters

* EntityView and Edge queries

* Relations Query

* Relation Query Implementation

* Update EDQS module version

* Sync with EDQS via Kafka

* EDQS: major refactoring

* EDQS API requests via Kafka

* EDQS: full sync with the database

* Refactoring for EDQS sync

* EDQS: major refactoring and new features

* EDQS refactoring, count query support, fix tests

* EDQS: refactoring for query processors

* Fix EDQS pom version

* Cleanup edqs.yml

* EDQS: tenant partitioning strategy; refactoring

* EDQS: latest events queue

* EDQS: support for monolith setup; RocksDB; other improvements

* EDQS: merge sync and events topics, introduce state topic

* EDQS: dynamic repartitioning

* implemented entity data query filters for edqs

* EdqsEntityQueryControllerTest - use in-memory queue

* edqs-filter fixes, added test

* EDQS: blob entity support

* EdqsEntityQueryControllerTest - use in-memory queue

* Use DummyEdqsService when disabled

* Fixes for EDQS

* Refactoring for EDQS tests

* Fix edqs requests partitioning

* EDQS: Fix for attributes handling

* Fix attributes saving in EntityServiceTest

* EDQS: refactoring, fixes

* Minor refactoring for query processor

* added ownerName/ownerType support

* fixed relation query processor

* fixed EntityServiceTest

* refactoring

* added support for parentId for relation query result

* Get rid of EntityNameFetcher

* Add fixme for relation query processor

* db restore with select all edqs fields

* fixed entity deletion

* fixed FieldUtils with new EntityFields

* dao method renamed

* EDQS: instance groups with same partitions; automatic sync; multiple fixes

* Refactoring for EDQS sync

* EDQS: refactoring

* Fix startup with Kafka

* fixed EntityQueryControllerTest

* fixed EdqsEntityServiceTest

* Separate queue admin for EDQS request template

* Implement new EDQS partitioning strategy

* EDQS: multiple fixes and refactoring

* Add mock EdqsRocksDb beans to tests

* added edqs stats for inmemory/grafana

* fixed filter tests

* Update todos

* Refactoring for QueueConfig

* Improvements and refactoring for EDQS consumers

* implemented TODOs

* test fixes

* Consume state topic up to end offsets

* edqs stats refactoring

* EDQS: cleanup on partitions removal; refactoring

* EDQS: minor refactoring

* EDQS: remove CSV loader

---------

Co-authored-by: Andrii Shvaika <ashvayka@thingsboard.io>
Co-authored-by: dashevchenko <dshevchenko@thingsboard.io>
2025-01-27 17:41:12 +02:00
Andrii Shvaika
d3278f05bb WIP: Cluster mode refactoring 2025-01-27 17:34:35 +02:00
IrynaMatveieva
0e1cd69e34 updated cf consumer 2025-01-27 15:55:40 +02:00
IrynaMatveieva
5641626443 added processNotification impl 2025-01-24 10:53:13 +02:00
IrynaMatveieva
69dc8b4b5d merge with upstream 2025-01-23 15:56:25 +02:00
IrynaMatveieva
2cc0d6f513 added implementations for consumer/producer methods 2025-01-23 15:49:31 +02:00
Andrii Shvaika
4c71b9d5f6 WIP: Cluster mode implementation 2025-01-23 13:56:44 +02:00
Andrii Shvaika
069725a2b9 WIP Refactoring of the cluster mode 2025-01-23 10:43:26 +02:00
Andrii Shvaika
6b9d374a5f Tmp commit for merge 2025-01-22 12:23:42 +02:00
YevhenBondarenko
933d1447d2 refactoring due to comments (removed serviceId duplications in ToUsageStatsServiceMsgPack) 2025-01-14 12:08:53 +01:00
YevhenBondarenko
87f0483019 merge with master 2025-01-13 19:27:17 +01:00