58 Commits

Author SHA1 Message Date
Volodymyr Babak
cc5d9dbb2c Added sync of fat edge consumer group into per edge group. Fixed clean up edge topics 2025-04-11 12:44:09 +03:00
ViacheslavKlimov
b40fa86bac Refactoring and fixes for CF lifecycle events handling 2025-03-13 17:35:00 +02:00
ViacheslavKlimov
43f9083f70 Merge branch 'master' of github.com:thingsboard/thingsboard into edqs 2025-02-28 17:12:33 +02:00
ViacheslavKlimov
44439584f4 Merge branch 'master' of github.com:thingsboard/thingsboard into feature/calculated-fields 2025-02-25 11:49:56 +02:00
ViacheslavKlimov
e7ed98910a Merge branch 'master' of github.com:thingsboard/thingsboard into edqs 2025-02-25 11:10:31 +02:00
Igor Kulikov
5cf26d4851 Update license header 2025-02-25 09:39:16 +02:00
Andrii Shvaika
83b338c697 Entity lifecycle implementation 2025-01-31 13:54:53 +02:00
Andrii Shvaika
5aabbd0f1e Link dispatch implementation 2025-01-29 15:37:36 +02:00
Viacheslav Klimov
86b5378d59 EDQS (#3196)
* Experiments with CSV

* CSV Loader v1

* EDQ tests

* Volatile variables instead of final

* Improvements

* updated loader with new entities

* Fix double memory usage issue

* Basic data structures and load

* Minor improvements

* Snappy + Large String reuse

* added EntityFields classes for each entity

* Basic implementation

* Minor improvements to KeyFilters

* implemented RepositoryUtils.checkKeyFilters

* Generic query implementation

* New structure

* Refactoring and few processors implementation

* extended DeviceData with shared/client attributes and device profile

* Minor refactoring of attribute scopes

* DeviceTypeFilter support

* Strong types of fields for each entity data class

* DeviceType and AssetType filters

* EntityView and Edge queries

* Relations Query

* Relation Query Implementation

* Update EDQS module version

* Sync with EDQS via Kafka

* EDQS: major refactoring

* EDQS API requests via Kafka

* EDQS: full sync with the database

* Refactoring for EDQS sync

* EDQS: major refactoring and new features

* EDQS refactoring, count query support, fix tests

* EDQS: refactoring for query processors

* Fix EDQS pom version

* Cleanup edqs.yml

* EDQS: tenant partitioning strategy; refactoring

* EDQS: latest events queue

* EDQS: support for monolith setup; RocksDB; other improvements

* EDQS: merge sync and events topics, introduce state topic

* EDQS: dynamic repartitioning

* implemented entity data query filters for edqs

* EdqsEntityQueryControllerTest - use in-memory queue

* edqs-filter fixes, added test

* EDQS: blob entity support

* EdqsEntityQueryControllerTest - use in-memory queue

* Use DummyEdqsService when disabled

* Fixes for EDQS

* Refactoring for EDQS tests

* Fix edqs requests partitioning

* EDQS: Fix for attributes handling

* Fix attributes saving in EntityServiceTest

* EDQS: refactoring, fixes

* Minor refactoring for query processor

* added ownerName/ownerType support

* fixed relation query processor

* fixed EntityServiceTest

* refactoring

* added support for parentId for relation query result

* Get rid of EntityNameFetcher

* Add fixme for relation query processor

* db restore with select all edqs fields

* fixed entity deletion

* fixed FieldUtils with new EntityFields

* dao method renamed

* EDQS: instance groups with same partitions; automatic sync; multiple fixes

* Refactoring for EDQS sync

* EDQS: refactoring

* Fix startup with Kafka

* fixed EntityQueryControllerTest

* fixed EdqsEntityServiceTest

* Separate queue admin for EDQS request template

* Implement new EDQS partitioning strategy

* EDQS: multiple fixes and refactoring

* Add mock EdqsRocksDb beans to tests

* added edqs stats for inmemory/grafana

* fixed filter tests

* Update todos

* Refactoring for QueueConfig

* Improvements and refactoring for EDQS consumers

* implemented TODOs

* test fixes

* Consume state topic up to end offsets

* edqs stats refactoring

* EDQS: cleanup on partitions removal; refactoring

* EDQS: minor refactoring

* EDQS: remove CSV loader

---------

Co-authored-by: Andrii Shvaika <ashvayka@thingsboard.io>
Co-authored-by: dashevchenko <dshevchenko@thingsboard.io>
2025-01-27 17:41:12 +02:00
IrynaMatveieva
0e1cd69e34 updated cf consumer 2025-01-27 15:55:40 +02:00
IrynaMatveieva
a652b31d7f implementation processing notification 2025-01-24 15:41:00 +02:00
IrynaMatveieva
5641626443 added processNotification impl 2025-01-24 10:53:13 +02:00
Andrii Shvaika
4c71b9d5f6 WIP: Cluster mode implementation 2025-01-23 13:56:44 +02:00
IrynaMatveieva
2080b439a7 added entity added/deleted events handling 2024-11-28 11:28:51 +02:00
IrynaMatveieva
31103e90e3 added implementation to handle update entity profiles events 2024-11-22 17:17:59 +02:00
IrynaMatveieva
3072861a8f implemented methods for calculated field update/delete in cluster service 2024-11-13 17:42:55 +02:00
Andrii Landiak
5602807dbe
Introduce Edge queue to handle edge notification messages 2024-08-29 12:51:20 +03:00
Andrii Landiak
c6ee0bff19
Add edgeIdsCache to decrease number of calls to db for findRelatedEdgeByEntityId 2024-08-27 12:41:39 +03:00
IrynaMatveieva
74ac5a949f added rule engine controller and send rest api call reply node 2024-05-14 10:05:41 +03:00
ViacheslavKlimov
55dc23b9ff Merge with hotfix/3.6.2 2024-02-09 11:38:40 +02:00
ViacheslavKlimov
a571153b7c Single update message for multiple queues 2024-01-24 18:08:01 +02:00
Andrii Shvaika
a3471c367a Merge PR#9968 2024-01-16 11:35:31 +02:00
Igor Kulikov
c5a72ed8df Update license header to 2024 year. 2024-01-09 10:46:16 +02:00
Volodymyr Babak
d329df0323 Merge remote-tracking branch 'upstream/master' into feature/broadcast-logs-improvement 2024-01-02 19:37:39 +02:00
YevhenBondarenko
316a09854c implemented device actor delete msg and fixed assign to tenant 2023-11-12 22:15:57 +01:00
Andrew Shvayka
3ec2e0956f
Merge pull request #9185 from AndriiLandiak/feature/edge-sync-improvement
Edge Synchronization improvement using EdgeId
2023-10-31 17:54:28 +02:00
dashevchenko
86f8965eab fixed thingsboard.yml 2023-10-30 12:21:30 +02:00
dashevchenko
62703d3583 merged with upstream 2023-10-30 12:19:24 +02:00
Andrii Shvaika
d78bb26a73 Merge with hotfix/3.6 2023-10-26 17:11:29 +03:00
Andrii Shvaika
694b6bf851 Refactoring: new data structures and message flow
Local Subscription Service refactoring

Refactoring of the proto structures

Bug fixing after refactoring

Minor improvements and bug fixes

Improvements to avoid additional DB calls

Bug fixes after refactoring

Cluster mode improvements
2023-10-26 17:07:16 +03:00
ViacheslavKlimov
cf951ee2b2 Multiple improvements for Rule Engine consumer services 2023-10-19 11:09:34 +03:00
Andrii Landiak
61c9eb977f Merge remote-tracking branch 'upstream/develop/3.6' into feature/broadcast-logs-improvement 2023-09-13 11:57:17 +03:00
Andrii Landiak
e2520ae0f5 Provide edgeId for EdgeEventSourcingListener to handle cloud entities updated on Edge and process to other Edges 2023-09-05 09:38:48 +03:00
Andrii Landiak
85361c609a EntityStateListener: process entity broadcasting service to handle all event 2023-08-21 15:32:19 +03:00
Andrii Shvaika
ca6095402f Merge with master 2023-08-17 12:19:02 +03:00
ViacheslavKlimov
f32e2f6fde Refactoring after review 2023-08-10 12:03:15 +03:00
Andrii Landiak
8faa1410b6
Edge crud notification: implement event publisher strategy to process pubsub model for detecting changes in entities
* Improve edge notification for entities' CRUD operations. Use service layer to notify instead of TbService

* Improve queue service, delete unused class for edge event updates

* Improve alarm delete and add handle fox delete dao event notification

* Refactoring: provide notification for relations and alarms. Improve logic and bad edge event type using

* Add entity type to SaveEvent to process correct message type to edge

* Improve relation service publish event

* Introduce EdgeEventSourcing service instead of saving edge events on controller/service layers. Part #2

* Improved stability of device edge test

* Push credential updated event only in case update

* Add tenantId to saveUser signature to send correct notification for listener

* Fix tests to send correct notification msg to edge

* Fix tests with correct action type

* Add delete msg to edge for customer

* Refactor ActionEntityEvent to use lombok builder

* Remove unnecessary comments

* Added edgeSynchronizationManager into BaseAlarmProcessor and BaseRelationProcessor

* Fixed license header

* Remove notification to edge from Version Control Service

* Fixed alarm del processing - find related edges inside edge processor

* Fix controller test for publish event to listener if entity was deleted

* Added check for edge imitator messages during login as tenant admin

* Refactoring: Added filtering of relation on EdgeEventSourcingListener

* Refactored to be in sync with PE

* Refactored edge test to be in sync with PE edge test changes

* EdgeControllerTest - moved await block into separate method to reuse it

* Fixed EdgeControllerTest

* Fixed testAssignEdgeToCustomerFromDifferentTenant test

* testSyncEdge - make stable

* Refacroting - update utils method name to pop* in EdgeControllerTest

* testSyncEdge - fixed order and nubmer of edge events

* testGetEdgeEvents - check by pop items, and not by index to improve stability on slow machines

* testGetEdgeEvents - added check that list is empty

* Removed test debug output

* EntityServiceTest - Fixed compilation error after merge

* Improve service layer event publisher to process each notification and validate in listener

* Improve BaseAlarmService to send notification to listener

* Fix asset-device notification action to send delete to all edges

* Delete unnecessary usage of sendMsgToEdge

* Improve processEntityNotification to be in sync with changed needed for PE

* Pull request review - minor refactoring

* Fix tests after review-refactoring

* Refactor tests to be in sync with PE

* Fixed repeated update - added check for old_edge_event table existance before migration

* DeviceEdgeProcessor - do edgeSynchronizationManager as soon as possible to avoid unnecessary downlinks

* BaseEdgeProcessor - refactoring and remove duplicate methods. Introduce EdgeEventType.isAllEdgesRelated

* Organize imports

* Improve Edge test: add sync completed message to await

* Minor refactoring for EdgeProcessor notification: asset and device

* EdgeEventSourcingListener - updated logging to avoid null pointer exception

* BaseAlarmService - added check for alarm to avoid NPE. EdgeEventSourcingListener - added try/catch blocks

* EdgeEventSourcingListener - fixed error message log level

---------

Co-authored-by: Volodymyr Babak <volodymyr.babak@gmail.com>
2023-08-07 13:21:02 +03:00
ViacheslavKlimov
c087359077 Move messages to other queue on deletion; improvements 2023-08-01 13:57:32 +03:00
ViacheslavKlimov
5862b417aa Add custom topic properties configuration 2023-07-28 12:06:04 +03:00
YevhenBondarenko
fbf9773a61 optimize imports 2023-05-02 10:57:58 +02:00
ViacheslavKlimov
fccd60601f RemoteNotificationRuleProcessor 2023-04-04 15:05:56 +03:00
ViacheslavKlimov
bbaf2a5054 Notification rule trigger via ClusterService 2023-04-04 12:58:47 +03:00
Igor Kulikov
cbdc4517e4 Update license header year to 2023 2023-01-31 10:43:56 +02:00
Volodymyr Babak
faaf07b1ea Edge sync functionality - added cluster support 2022-08-16 17:34:02 +03:00
Volodymyr Babak
17ec3f41f4 Merge remote-tracking branch 'origin/develop/3.4' into feature/edge-converters-integration 2022-06-20 22:06:05 +03:00
Volodymyr Babak
e184d1df60 Merge remote-tracking branch 'origin/develop/3.4' into feature/edge-converters-integration 2022-06-02 16:45:40 +03:00
Volodymyr Babak
4a77cc27cb Renaming ..ToEdgeService to ..ToEdge 2022-05-26 12:24:41 +03:00
Andrii Shvaika
7071c7a6bc Implementation of the Git Queue service client 2022-05-20 17:33:24 +03:00
YevhenBondarenko
321730ed02 added upgrade for queues 2022-05-11 23:02:03 +02:00
YevhenBondarenko
f74f3a9293 created TbQueueService and improvements 2022-05-02 08:23:16 +02:00