347 Commits

Author SHA1 Message Date
ViacheslavKlimov
965210f17b Multiple improvements for EDQS 2025-02-21 14:39:26 +02:00
ViacheslavKlimov
45a3730708 Fixes for CF 2025-02-20 16:19:13 +02:00
ViacheslavKlimov
7519889c1a Refactoring for EDQS repartitioning 2025-02-20 12:25:08 +02:00
Andrii Landiak
44a7bf82c4 Fix ComponentLifecycleMsg: message and builder 2025-02-17 18:00:12 +02:00
Andrii Shvaika
5efb94dc7a Refactoring 2025-02-10 15:58:01 +02:00
Andrii Shvaika
83b338c697 Entity lifecycle implementation 2025-01-31 13:54:53 +02:00
Andrii Shvaika
5aabbd0f1e Link dispatch implementation 2025-01-29 15:37:36 +02:00
Viacheslav Klimov
86b5378d59 EDQS (#3196)
* Experiments with CSV

* CSV Loader v1

* EDQ tests

* Volatile variables instead of final

* Improvements

* updated loader with new entities

* Fix double memory usage issue

* Basic data structures and load

* Minor improvements

* Snappy + Large String reuse

* added EntityFields classes for each entity

* Basic implementation

* Minor improvements to KeyFilters

* implemented RepositoryUtils.checkKeyFilters

* Generic query implementation

* New structure

* Refactoring and few processors implementation

* extended DeviceData with shared/client attributes and device profile

* Minor refactoring of attribute scopes

* DeviceTypeFilter support

* Strong types of fields for each entity data class

* DeviceType and AssetType filters

* EntityView and Edge queries

* Relations Query

* Relation Query Implementation

* Update EDQS module version

* Sync with EDQS via Kafka

* EDQS: major refactoring

* EDQS API requests via Kafka

* EDQS: full sync with the database

* Refactoring for EDQS sync

* EDQS: major refactoring and new features

* EDQS refactoring, count query support, fix tests

* EDQS: refactoring for query processors

* Fix EDQS pom version

* Cleanup edqs.yml

* EDQS: tenant partitioning strategy; refactoring

* EDQS: latest events queue

* EDQS: support for monolith setup; RocksDB; other improvements

* EDQS: merge sync and events topics, introduce state topic

* EDQS: dynamic repartitioning

* implemented entity data query filters for edqs

* EdqsEntityQueryControllerTest - use in-memory queue

* edqs-filter fixes, added test

* EDQS: blob entity support

* EdqsEntityQueryControllerTest - use in-memory queue

* Use DummyEdqsService when disabled

* Fixes for EDQS

* Refactoring for EDQS tests

* Fix edqs requests partitioning

* EDQS: Fix for attributes handling

* Fix attributes saving in EntityServiceTest

* EDQS: refactoring, fixes

* Minor refactoring for query processor

* added ownerName/ownerType support

* fixed relation query processor

* fixed EntityServiceTest

* refactoring

* added support for parentId for relation query result

* Get rid of EntityNameFetcher

* Add fixme for relation query processor

* db restore with select all edqs fields

* fixed entity deletion

* fixed FieldUtils with new EntityFields

* dao method renamed

* EDQS: instance groups with same partitions; automatic sync; multiple fixes

* Refactoring for EDQS sync

* EDQS: refactoring

* Fix startup with Kafka

* fixed EntityQueryControllerTest

* fixed EdqsEntityServiceTest

* Separate queue admin for EDQS request template

* Implement new EDQS partitioning strategy

* EDQS: multiple fixes and refactoring

* Add mock EdqsRocksDb beans to tests

* added edqs stats for inmemory/grafana

* fixed filter tests

* Update todos

* Refactoring for QueueConfig

* Improvements and refactoring for EDQS consumers

* implemented TODOs

* test fixes

* Consume state topic up to end offsets

* edqs stats refactoring

* EDQS: cleanup on partitions removal; refactoring

* EDQS: minor refactoring

* EDQS: remove CSV loader

---------

Co-authored-by: Andrii Shvaika <ashvayka@thingsboard.io>
Co-authored-by: dashevchenko <dshevchenko@thingsboard.io>
2025-01-27 17:41:12 +02:00
Andrii Shvaika
d3278f05bb WIP: Cluster mode refactoring 2025-01-27 17:34:35 +02:00
Andrii Shvaika
6b9d374a5f Tmp commit for merge 2025-01-22 12:23:42 +02:00
IrynaMatveieva
4ba131ad85 changed partititoning implementation 2025-01-06 14:15:32 +02:00
IrynaMatveieva
a9f39e4917 added cache class 2024-12-24 12:50:27 +02:00
IrynaMatveieva
1482d1c4bb added logic to avoid looping 2024-12-23 17:00:10 +02:00
ViacheslavKlimov
ac61d7c526 TbMsg: add resetRuleNodeId and copyMetaData 2024-12-19 16:09:36 +02:00
YevhenBondarenko
e0fd611c76 used TbMsgProto instead of ByteString 2024-12-13 14:35:02 +01:00
ViacheslavKlimov
1b6256a9e8 Refactor TbMsg copying 2024-12-13 15:18:37 +02:00
ViacheslavKlimov
0b3ffb5b4a Refactor TbMsg transforming 2024-12-13 14:21:55 +02:00
ViacheslavKlimov
e4cc2b4f9f Introduce TbMsgTransformer 2024-12-13 13:23:15 +02:00
ViacheslavKlimov
cbd2e012dd Refactor TbMsg.newMsg usages 2024-12-13 13:05:54 +02:00
ViacheslavKlimov
6c76e0b2fe TbMsg refactoring to use builders 2024-12-13 12:38:56 +02:00
Andrii Landiak
3ecbd3258e Introduce kafka approach to work with edge-event instead of postgres 2024-10-22 12:38:57 +03:00
YevhenBondarenko
48b5cadb4a created separate converter methow for gateway telemetry 2024-09-13 15:30:37 +02:00
Viacheslav Klimov
2f501c628d
Merge pull request #11521 from AndriiLandiak/feature/edge-queue
Introduce Edge queue to handle edge notification messages
2024-09-03 15:25:18 +03:00
Andrii Landiak
5602807dbe
Introduce Edge queue to handle edge notification messages 2024-08-29 12:51:20 +03:00
Viacheslav Klimov
a003ca4de5
Merge branch 'master' into http_transport_large_request_fix 2024-08-22 14:46:58 +03:00
ViacheslavKlimov
152e6f13da Merge remote-tracking branch 'origin/master' into feature/entities-version 2024-08-09 13:08:26 +03:00
dashevchenko
936bb82334 updated transport http limits, added logging for initialized filters 2024-08-07 19:03:05 +03:00
Viacheslav Klimov
af5f501cd4
Merge pull request #11347 from volodymyr-babak/edge-misc-fixes
Edge: Enhanced Credential Handling and Sync Improvements
2024-08-07 13:38:40 +03:00
ViacheslavKlimov
7007563627 Merge remote-tracking branch 'origin/hotfix/3.7' into master-hotfix 2024-08-06 19:46:05 +03:00
ViacheslavKlimov
5759610340 Rate limit for WS subscriptions 2024-08-05 17:22:58 +03:00
ViacheslavKlimov
a5a92d9334 Merge remote-tracking branch 'origin/master' into feature/entities-version 2024-08-03 14:33:21 +03:00
Volodymyr Babak
8cc1bccc5b Merge remote-tracking branch 'upstream/master' into edge-misc-fixes 2024-08-02 12:50:54 +03:00
Volodymyr Babak
cfca80defe Remove Edge request messages - send multiple data messages in single rpc message 2024-08-02 12:48:13 +03:00
dashevchenko
4169da4c08 configurable max payload size by url pattern 2024-08-01 16:10:39 +03:00
Andrii Landiak
5ca6ad03e3 CE: optimize java imports 2024-08-01 11:12:10 +03:00
dashevchenko
02f7001102 added spring security for http transport, added large RequestSizeFilter for http transport 2024-07-31 16:46:18 +03:00
Daria Shevchenko
15cb12e48f
HTTP API: validate payload size (#11265)
* added check for large ToDeviceRpcResponseMsg, ToServerRpcRequestMsg messages on default transport level

* moved http request size check to controller

* minor refactoring

* test fixed

* updated error messages, updated url patterns to single one, added yml property to http transport

* updated swagger docs, added to rpc filter "/api/rpc/**" and "/api/plugins/rpc/**" endpoints

* remove redundant set

* fixed large request check for DeviceApiController in microservice architecture

* renamed yml parameter and filter
2024-07-30 14:04:53 +03:00
Viacheslav Klimov
5809d014de
Queue option to duplicate msgs to all partitions (#10815)
* Queue config option to duplicate msgs to all partitions

* UI for queue config option to duplicate msgs to all partitions

* Fixes for msg duplication; add tests

* Introduce TbRuleEngineProducerService

* Use TbRuleEngineProducerService in TbClusterService

* Fix DefaultTbClusterServiceTest

* Add correlationId to TbMsg

* Refactor TbMsg

* Support msg duplication for all msg types
2024-05-22 12:11:32 +03:00
ViacheslavKlimov
b16d8484e9 Minor refactoring for notifications consumers 2024-05-08 14:50:11 +03:00
Oleksandra Matviienko
63df92e995 JUnit5 migration in common. Junit-platform.properties and logback-test.xml where added to java resource directories if needed.
Signed-off-by: Oleksandra Matviienko <al.zzzeebra@gmail.com>
2024-04-26 23:55:17 +02:00
Andrew Shvayka
aecf654d65
Merge pull request #10201 from thingsboard/feature/housekeeper
Housekeeping service
2024-04-26 13:37:06 +03:00
Sergey Matvienko
e532e0f2a9 fixed import org.bouncycastle.pqc.legacy.math.linearalgebra.ByteUtils, test added for EncryptionUtil 2024-04-19 23:49:35 +02:00
Sergey Matvienko
a3e7e0b4a1 major dependency update 2024-04-18 17:11:29 +02:00
ViacheslavKlimov
be3e78044c Merge remote-tracking branch 'origin/master' into feature/housekeeper 2024-03-21 11:52:07 +02:00
ViacheslavKlimov
4ed952a9d6 Merge branch 'master' into develop/3.7 2024-03-19 15:44:23 +02:00
ViacheslavKlimov
22f5573e8d Housekeeper client for all service types 2024-03-13 19:59:13 +02:00
Dmytro Skarzhynets
c05d543b0a Add test to ensure message timestamp is used as event timestamp when metadata ts property is not present 2024-02-02 16:03:35 +02:00
Dmytro Skarzhynets
eadcf916f5 Add optimization using separate service which routes activity actions to device state service directly, when running in monolith. Refactor rule node tests 2024-01-26 16:15:11 +02:00
YevhenBondarenko
60be820915 merge with master 2024-01-15 18:51:00 +01:00
Igor Kulikov
c5a72ed8df Update license header to 2024 year. 2024-01-09 10:46:16 +02:00