56 Commits

Author SHA1 Message Date
Viacheslav Klimov
e054802fe2
Merge pull request #13456 from thingsboard/rc
rc
2025-05-28 12:17:27 +03:00
Andrii Landiak
edd7d6392a Improvement after review 2025-04-24 16:52:31 +03:00
ViacheslavKlimov
f8bf512a0a Introduce PartitionedQueueResponseTemplate with consumer per partition; use it for EDQS requests processing 2025-04-15 11:51:45 +03:00
Dmytro Skarzhynets
e2cf33e6cb
Save attributes strategies: merge with master 2025-03-07 10:23:52 +02:00
Dmytro Skarzhynets
b3e4be6bda
Save attributes strategies: merge with master 2025-03-03 11:00:34 +02:00
ViacheslavKlimov
43f9083f70 Merge branch 'master' of github.com:thingsboard/thingsboard into edqs 2025-02-28 17:12:33 +02:00
Andrii Shvaika
62da129d6b Review comments fixed 2025-02-27 16:46:55 +02:00
ViacheslavKlimov
e7ed98910a Merge branch 'master' of github.com:thingsboard/thingsboard into edqs 2025-02-25 11:10:31 +02:00
Igor Kulikov
5cf26d4851 Update license header 2025-02-25 09:39:16 +02:00
Viacheslav Klimov
86b5378d59 EDQS (#3196)
* Experiments with CSV

* CSV Loader v1

* EDQ tests

* Volatile variables instead of final

* Improvements

* updated loader with new entities

* Fix double memory usage issue

* Basic data structures and load

* Minor improvements

* Snappy + Large String reuse

* added EntityFields classes for each entity

* Basic implementation

* Minor improvements to KeyFilters

* implemented RepositoryUtils.checkKeyFilters

* Generic query implementation

* New structure

* Refactoring and few processors implementation

* extended DeviceData with shared/client attributes and device profile

* Minor refactoring of attribute scopes

* DeviceTypeFilter support

* Strong types of fields for each entity data class

* DeviceType and AssetType filters

* EntityView and Edge queries

* Relations Query

* Relation Query Implementation

* Update EDQS module version

* Sync with EDQS via Kafka

* EDQS: major refactoring

* EDQS API requests via Kafka

* EDQS: full sync with the database

* Refactoring for EDQS sync

* EDQS: major refactoring and new features

* EDQS refactoring, count query support, fix tests

* EDQS: refactoring for query processors

* Fix EDQS pom version

* Cleanup edqs.yml

* EDQS: tenant partitioning strategy; refactoring

* EDQS: latest events queue

* EDQS: support for monolith setup; RocksDB; other improvements

* EDQS: merge sync and events topics, introduce state topic

* EDQS: dynamic repartitioning

* implemented entity data query filters for edqs

* EdqsEntityQueryControllerTest - use in-memory queue

* edqs-filter fixes, added test

* EDQS: blob entity support

* EdqsEntityQueryControllerTest - use in-memory queue

* Use DummyEdqsService when disabled

* Fixes for EDQS

* Refactoring for EDQS tests

* Fix edqs requests partitioning

* EDQS: Fix for attributes handling

* Fix attributes saving in EntityServiceTest

* EDQS: refactoring, fixes

* Minor refactoring for query processor

* added ownerName/ownerType support

* fixed relation query processor

* fixed EntityServiceTest

* refactoring

* added support for parentId for relation query result

* Get rid of EntityNameFetcher

* Add fixme for relation query processor

* db restore with select all edqs fields

* fixed entity deletion

* fixed FieldUtils with new EntityFields

* dao method renamed

* EDQS: instance groups with same partitions; automatic sync; multiple fixes

* Refactoring for EDQS sync

* EDQS: refactoring

* Fix startup with Kafka

* fixed EntityQueryControllerTest

* fixed EdqsEntityServiceTest

* Separate queue admin for EDQS request template

* Implement new EDQS partitioning strategy

* EDQS: multiple fixes and refactoring

* Add mock EdqsRocksDb beans to tests

* added edqs stats for inmemory/grafana

* fixed filter tests

* Update todos

* Refactoring for QueueConfig

* Improvements and refactoring for EDQS consumers

* implemented TODOs

* test fixes

* Consume state topic up to end offsets

* edqs stats refactoring

* EDQS: cleanup on partitions removal; refactoring

* EDQS: minor refactoring

* EDQS: remove CSV loader

---------

Co-authored-by: Andrii Shvaika <ashvayka@thingsboard.io>
Co-authored-by: dashevchenko <dshevchenko@thingsboard.io>
2025-01-27 17:41:12 +02:00
ViacheslavKlimov
e920525456 Merge branch 'master' of github.com:thingsboard/thingsboard into develop/4.0 2024-12-05 11:26:34 +02:00
ViacheslavKlimov
12ca1739d4 Remove all queue types except Kafka 2024-10-23 15:30:07 +03:00
Dmytro Skarzhynets
bbe328d158 Implemented safe scheduled thread pool 2024-09-18 19:46:17 +03:00
ViacheslavKlimov
c4d0867635 Merge branch 'hotfix/3.7' of github.com:thingsboard/thingsboard into master-hotfix-3.7 2024-09-10 11:42:09 +03:00
ThingsBoard Bot
189778efb9 License header format 2024-09-09 14:57:22 +00:00
Andrew Shvayka
e286a051df
Merge pull request #8696 from driesva/rabbitmq-max-poll-messages
fix: Slow RabbitMQ message polling
2024-09-09 17:45:30 +03:00
Dmytro Skarzhynets
f6a1c14353 Fix last activity not being reported when activity reporting period ends 2024-09-04 16:45:58 +03:00
Andrii Landiak
c6ee0bff19
Add edgeIdsCache to decrease number of calls to db for findRelatedEdgeByEntityId 2024-08-27 12:41:39 +03:00
ViacheslavKlimov
92cabebb04 Fix unrecognized Kafka config 'session.timeout.ms' 2024-08-14 11:52:13 +03:00
Andrew Shvayka
a8dee2643b
Merge pull request #10728 from smatvienko-tb/feature/kafka-consumer-group-per-partition
Rule engine: Kafka consumer group per partition
2024-05-22 18:11:25 +03:00
Sergey Matvienko
258c26ed04 Kafka groupId syncOffsets from a fat group to a single-partition group 2024-05-15 16:23:20 +02:00
Sergey Matvienko
34cfd588b1 mvn license:format 2024-05-13 17:26:59 +02:00
Sergey Matvienko
d89b14e359 Kafka Confluent compatibility out of the box: segment.bytes:26214400 increased segment.bytes:52428800 2024-05-10 11:45:52 +02:00
Sergey Matvienko
07de3b975f TbKafkaAdmin WIP 2024-05-08 13:58:21 +02:00
Oleksandra Matviienko
63df92e995 JUnit5 migration in common. Junit-platform.properties and logback-test.xml where added to java resource directories if needed.
Signed-off-by: Oleksandra Matviienko <al.zzzeebra@gmail.com>
2024-04-26 23:55:17 +02:00
Igor Kulikov
c5a72ed8df Update license header to 2024 year. 2024-01-09 10:46:16 +02:00
Andrii Shvaika
694b6bf851 Refactoring: new data structures and message flow
Local Subscription Service refactoring

Refactoring of the proto structures

Bug fixing after refactoring

Minor improvements and bug fixes

Improvements to avoid additional DB calls

Bug fixes after refactoring

Cluster mode improvements
2023-10-26 17:07:16 +03:00
Andrew Shvayka
f618599d15
Merge pull request #8998 from smatvienko-tb/feature/kafka-producer-append-debug-headers
TbKafkaProducerTemplate will add analytics headers for each message on debug or trace log level
2023-07-31 17:26:01 +03:00
YevhenBondarenko
ac2aac8aa7 refactored due to comments 2023-07-31 13:51:03 +03:00
YevhenBondarenko
948f517898 added zk restart node tests 2023-07-31 13:50:57 +03:00
Sergey Matvienko
c3e9ab5991 TbKafkaProducerTemplate will add headers for each message when log level: DEBUG - producerId and thread name; TRACE - stacktrace first 10-2=8 lines 2023-07-26 20:45:42 +02:00
driesva
ee45763ceb fix: Slow RabbitMQ message polling
Configurable RabbitMQ polling in order that more messages can be obtained from the queue during a poll.

Fixes #8445
2023-06-02 09:11:49 +02:00
YevhenBondarenko
fbf9773a61 optimize imports 2023-05-02 10:57:58 +02:00
Sergey Matvienko
26ed90c534 kafka settings: kafka yaml parameters added for microservices same as thingsboard.yml . PropertyUtils added, duplicated code refactored to getProps. TB_QUEUE_KAFKA_REQUEST_TIMEOUT_MS and TB_QUEUE_KAFKA_SESSION_TIMEOUT_MS moved from other to main properties. Other yaml props is deprecated. other-inline added TB_QUEUE_KAFKA_OTHER_PROPERTIES. Tests added. 2023-04-18 10:54:10 +02:00
Sergey Matvienko
e621a21df3 kafka settings: test added, refactored to use configureSSL exact once 2023-04-18 10:54:10 +02:00
Igor Kulikov
cbdc4517e4 Update license header year to 2023 2023-01-31 10:43:56 +02:00
ShvaykaD
f5e20b5a3e move default queue names and topics to DataConstants 2023-01-26 13:37:19 +02:00
Sergey Matvienko
cffd0a3a1e QueueKey toString for better logging experience 2022-06-21 18:12:29 +03:00
Sergey Matvienko
f8a6751182 InMemoryStorage performance improved. Many test cases added since it is essential piece of code. 2022-04-15 16:46:19 +03:00
Sergey Matvienko
b9b4d06376 DefaultInMemoryStorageTest test added on Poll before improvement 2022-04-15 16:46:15 +03:00
Sergey Matvienko
d18533a88f InMemoryStorage extracted 2022-04-15 16:46:09 +03:00
Sergey Matvienko
9a2bc5ab9d InMemoryStorage refactored from the static singleton to the Spring Bean 2022-04-14 13:20:34 +03:00
Igor Kulikov
da4c742872 Update license year to 2022 2022-01-17 14:07:46 +02:00
Sergey Matvienko
eab54435eb fixed license header for InMemoryStorageTest 2021-10-28 15:03:13 +03:00
Sergey Matvienko
d98419106b InMemoryStorage: added getLagTotal to be able to await while queue have any messages (no guarantee that messages was processed). 2021-10-18 16:47:43 +03:00
Igor Kulikov
b173e0799c Update OS package plugin version. Code cleanup. 2021-09-07 11:47:43 +03:00
Sergey Matvienko
bec228bc83 test: removed unnecessary stubbing on DefaultTbQueueRequestTemplateTest 2021-06-17 13:42:37 +03:00
Sergey Matvienko
aaedb9e879 DefaultTbQueueRequestTemplateTest updated licence header for CE 2021-06-17 13:35:20 +03:00
Sergey Matvienko
ed44ac8f19 test fixed for DefaultTbQueueRequestTemplateTest 2021-06-17 13:27:19 +03:00
Sergey Matvienko
9daa43a115 queue request template: sleep on exception shortened according to the stopwatch. test adjusted 2021-06-17 13:02:15 +03:00