596 Commits

Author SHA1 Message Date
ViacheslavKlimov
965210f17b Multiple improvements for EDQS 2025-02-21 14:39:26 +02:00
ViacheslavKlimov
347cb5bc36 EDQS code cleanup 2025-02-21 11:48:21 +02:00
ViacheslavKlimov
a48144d2bb EDQS code cleanup 2025-02-20 17:34:28 +02:00
ViacheslavKlimov
92b83335c0 Merge branch 'master' of github.com:thingsboard/thingsboard into edqs 2025-02-20 12:25:17 +02:00
ViacheslavKlimov
7519889c1a Refactoring for EDQS repartitioning 2025-02-20 12:25:08 +02:00
ViacheslavKlimov
ded6daf2b3 Kafka states restore improvements 2025-02-19 15:44:05 +02:00
ViacheslavKlimov
6dd08276f8 Merge branch 'rc' of github.com:thingsboard/thingsboard 2025-02-14 13:45:35 +02:00
ViacheslavKlimov
bb7ea08e59 EDQS: refactoring, OOM handling, healthcheck 2025-02-14 12:08:05 +02:00
Volodymyr Babak
1b4930ebaf Introduced separate count for edge consumers 2025-02-10 18:51:13 +02:00
Volodymyr Babak
3860d79613 Improved Edge handling in case connect/disconnect. Speed up initial setup 2025-02-10 18:42:25 +02:00
dashevchenko
b2bf5b9f65 configurable edqs sync batch sizes 2025-02-05 12:20:23 +02:00
dashevchenko
0e9d0b3ba3 fixed EntityServiceTest 2025-02-03 18:27:36 +02:00
ViacheslavKlimov
c8db304ce6 Optimized postgres sync (#3208)
* optimized postgres sync

* Fix consumer stopping when stopWhenRead enabled

* Fix NPE on EDQS repartitioning

* fixed EntityServiceTest

* Fix EDQS yml props

---------

Co-authored-by: ViacheslavKlimov <viacheslavklimov11@gmail.com>
2025-01-30 12:43:14 +02:00
Viacheslav Klimov
86b5378d59 EDQS (#3196)
* Experiments with CSV

* CSV Loader v1

* EDQ tests

* Volatile variables instead of final

* Improvements

* updated loader with new entities

* Fix double memory usage issue

* Basic data structures and load

* Minor improvements

* Snappy + Large String reuse

* added EntityFields classes for each entity

* Basic implementation

* Minor improvements to KeyFilters

* implemented RepositoryUtils.checkKeyFilters

* Generic query implementation

* New structure

* Refactoring and few processors implementation

* extended DeviceData with shared/client attributes and device profile

* Minor refactoring of attribute scopes

* DeviceTypeFilter support

* Strong types of fields for each entity data class

* DeviceType and AssetType filters

* EntityView and Edge queries

* Relations Query

* Relation Query Implementation

* Update EDQS module version

* Sync with EDQS via Kafka

* EDQS: major refactoring

* EDQS API requests via Kafka

* EDQS: full sync with the database

* Refactoring for EDQS sync

* EDQS: major refactoring and new features

* EDQS refactoring, count query support, fix tests

* EDQS: refactoring for query processors

* Fix EDQS pom version

* Cleanup edqs.yml

* EDQS: tenant partitioning strategy; refactoring

* EDQS: latest events queue

* EDQS: support for monolith setup; RocksDB; other improvements

* EDQS: merge sync and events topics, introduce state topic

* EDQS: dynamic repartitioning

* implemented entity data query filters for edqs

* EdqsEntityQueryControllerTest - use in-memory queue

* edqs-filter fixes, added test

* EDQS: blob entity support

* EdqsEntityQueryControllerTest - use in-memory queue

* Use DummyEdqsService when disabled

* Fixes for EDQS

* Refactoring for EDQS tests

* Fix edqs requests partitioning

* EDQS: Fix for attributes handling

* Fix attributes saving in EntityServiceTest

* EDQS: refactoring, fixes

* Minor refactoring for query processor

* added ownerName/ownerType support

* fixed relation query processor

* fixed EntityServiceTest

* refactoring

* added support for parentId for relation query result

* Get rid of EntityNameFetcher

* Add fixme for relation query processor

* db restore with select all edqs fields

* fixed entity deletion

* fixed FieldUtils with new EntityFields

* dao method renamed

* EDQS: instance groups with same partitions; automatic sync; multiple fixes

* Refactoring for EDQS sync

* EDQS: refactoring

* Fix startup with Kafka

* fixed EntityQueryControllerTest

* fixed EdqsEntityServiceTest

* Separate queue admin for EDQS request template

* Implement new EDQS partitioning strategy

* EDQS: multiple fixes and refactoring

* Add mock EdqsRocksDb beans to tests

* added edqs stats for inmemory/grafana

* fixed filter tests

* Update todos

* Refactoring for QueueConfig

* Improvements and refactoring for EDQS consumers

* implemented TODOs

* test fixes

* Consume state topic up to end offsets

* edqs stats refactoring

* EDQS: cleanup on partitions removal; refactoring

* EDQS: minor refactoring

* EDQS: remove CSV loader

---------

Co-authored-by: Andrii Shvaika <ashvayka@thingsboard.io>
Co-authored-by: dashevchenko <dshevchenko@thingsboard.io>
2025-01-27 17:41:12 +02:00
ViacheslavKlimov
d4d3240f78 Merge branch 'master' of github.com:thingsboard/thingsboard into develop/4.0 2024-12-23 11:15:16 +02:00
ViacheslavKlimov
e4cc2b4f9f Introduce TbMsgTransformer 2024-12-13 13:23:15 +02:00
ViacheslavKlimov
e920525456 Merge branch 'master' of github.com:thingsboard/thingsboard into develop/4.0 2024-12-05 11:26:34 +02:00
Dmytro Skarzhynets
68aaf3c918 Merge branch 'master' into safe-scheduler
# Conflicts:
#	common/util/src/main/java/org/thingsboard/common/util/ThingsBoardExecutors.java
2024-12-02 17:43:07 +02:00
Andrii Landiak
988dfce500 Refactor KafkaEdgeTopicsCleanUpService to properly clean up both deleted edges and expired topics 2024-11-29 15:18:27 +02:00
Andrii Landiak
63b53a4a45 Final improvement after review 2024-11-28 16:51:55 +02:00
Andrii Landiak
76bed44705 Minor improvement 2024-11-28 15:16:20 +02:00
Andrii Landiak
b796355554 Merge remote-tracking branch 'upstream/master' into feature/edge-kafka-events 2024-11-26 09:42:31 +02:00
Andrii Landiak
c388034d01 Delete EdgeGrpcSession interface: use abstract instead 2024-11-22 10:26:06 +02:00
Andrii Landiak
b981281205 Fix tests 2024-11-01 11:37:11 +02:00
Andrii Landiak
50a55d15a9 Add poll properties for edge-event kafka poll 2024-10-31 18:12:16 +02:00
Andrii Landiak
38eeada8ec Implement clean up service to clear empty edge topics 2024-10-30 12:29:07 +02:00
ViacheslavKlimov
12ca1739d4 Remove all queue types except Kafka 2024-10-23 15:30:07 +03:00
Andrii Landiak
3ecbd3258e Introduce kafka approach to work with edge-event instead of postgres 2024-10-22 12:38:57 +03:00
ViacheslavKlimov
2c0b7eb2b3 Deprecate all queue types except Kafka and in-memory 2024-10-18 14:11:59 +03:00
Dmytro Skarzhynets
bbe328d158 Implemented safe scheduled thread pool 2024-09-18 19:46:17 +03:00
ViacheslavKlimov
ac460e8c3b Merge remote-tracking branch 'origin/hotfix/3.7' 2024-09-17 12:45:27 +03:00
YevhenBondarenko
f10f63bb8a fixed concurrent modification in TopicService 2024-09-17 11:25:50 +02:00
ViacheslavKlimov
c4d0867635 Merge branch 'hotfix/3.7' of github.com:thingsboard/thingsboard into master-hotfix-3.7 2024-09-10 11:42:09 +03:00
ThingsBoard Bot
189778efb9 License header format 2024-09-09 14:57:22 +00:00
Andrew Shvayka
e286a051df
Merge pull request #8696 from driesva/rabbitmq-max-poll-messages
fix: Slow RabbitMQ message polling
2024-09-09 17:45:30 +03:00
Dmytro Skarzhynets
39049eecd9 Remove redundant logging of thread name and scheduler name 2024-09-04 17:11:53 +03:00
Dmytro Skarzhynets
f6a1c14353 Fix last activity not being reported when activity reporting period ends 2024-09-04 16:45:58 +03:00
Andrii Landiak
5602807dbe
Introduce Edge queue to handle edge notification messages 2024-08-29 12:51:20 +03:00
Andrii Landiak
c6ee0bff19
Add edgeIdsCache to decrease number of calls to db for findRelatedEdgeByEntityId 2024-08-27 12:41:39 +03:00
ViacheslavKlimov
92cabebb04 Fix unrecognized Kafka config 'session.timeout.ms' 2024-08-14 11:52:13 +03:00
Andrii Landiak
5ca6ad03e3 CE: optimize java imports 2024-08-01 11:12:10 +03:00
ViacheslavKlimov
88af44870e Check for queueName emptiness on partition resolve 2024-07-04 14:22:32 +03:00
Andrew Shvayka
a8dee2643b
Merge pull request #10728 from smatvienko-tb/feature/kafka-consumer-group-per-partition
Rule engine: Kafka consumer group per partition
2024-05-22 18:11:25 +03:00
Sergey Matvienko
11a8e5c7eb Merge remote-tracking branch 'ce/master' into feature/kafka-consumer-group-per-partition
# Conflicts:
#	application/src/main/java/org/thingsboard/server/service/queue/ruleengine/TbQueueConsumerTask.java
#	application/src/main/java/org/thingsboard/server/service/queue/ruleengine/TbRuleEngineQueueConsumerManager.java
#	application/src/test/java/org/thingsboard/server/service/queue/ruleengine/TbRuleEngineQueueConsumerManagerTest.java
#	common/queue/src/main/java/org/thingsboard/server/queue/kafka/TbKafkaAdmin.java
2024-05-22 14:37:46 +02:00
Andrew Shvayka
1579e1e1a3
Merge pull request #10757 from smatvienko-tb/feature/kafka-confluent-out-of-the-box
Kafka Confluent compatibility out of the box: segment.bytes:52428800
2024-05-22 12:26:40 +03:00
Viacheslav Klimov
5809d014de
Queue option to duplicate msgs to all partitions (#10815)
* Queue config option to duplicate msgs to all partitions

* UI for queue config option to duplicate msgs to all partitions

* Fixes for msg duplication; add tests

* Introduce TbRuleEngineProducerService

* Use TbRuleEngineProducerService in TbClusterService

* Fix DefaultTbClusterServiceTest

* Add correlationId to TbMsg

* Refactor TbMsg

* Support msg duplication for all msg types
2024-05-22 12:11:32 +03:00
Sergey Matvienko
5e3ff6fe20 minor cleanup 2024-05-22 09:02:01 +02:00
ViacheslavKlimov
1a264dafa7 Fix TbKafkaSettingsTest 2024-05-16 14:34:50 +03:00
ViacheslavKlimov
e8d14f37cb Improvements for Kafka admin creation 2024-05-16 13:24:49 +03:00
Sergey Matvienko
258c26ed04 Kafka groupId syncOffsets from a fat group to a single-partition group 2024-05-15 16:23:20 +02:00