35 Commits

Author SHA1 Message Date
Volodymyr Babak
3288ec257f EDQS SQL like query updates to be in sync with legacy 2025-06-17 12:37:24 +03:00
Volodymyr Babak
dd9d954c2f EDQS - added SQL like style for filter contains/starts with/ends with 2025-06-16 16:17:13 +03:00
ViacheslavKlimov
52d5e3f611 Merge branch 'rc' of github.com:thingsboard/thingsboard into edqs-improvements 2025-05-27 15:42:52 +03:00
ViacheslavKlimov
069f052122 Refactoring and improvements for EDQS 2025-05-27 15:38:32 +03:00
Igor Kulikov
5fbb3194a7 Version 4.0.1-RC 2025-04-22 10:46:16 +03:00
ViacheslavKlimov
f8bf512a0a Introduce PartitionedQueueResponseTemplate with consumer per partition; use it for EDQS requests processing 2025-04-15 11:51:45 +03:00
ViacheslavKlimov
b806a41e62 More stats for EDQS 2025-04-10 14:02:56 +03:00
ViacheslavKlimov
59fa19d1c8 Change EDQS consumers poll interval from 125 ms to 25 ms 2025-04-10 11:40:19 +03:00
ViacheslavKlimov
16eb027956 EDQS: configurable string compression threshold; fix stats service condition 2025-04-09 14:52:11 +03:00
ViacheslavKlimov
a91d9faad9 Fix EDQS tests init 2025-04-09 12:43:35 +03:00
ViacheslavKlimov
c50d7a3988 EDQS consumer management refactoring 2025-03-25 16:30:21 +02:00
ViacheslavKlimov
a5404c2b45 EDQS: don't use consumer group management for state topics; single consumer group for requests topic; reduce events topic retention to 24 hours 2025-03-24 17:26:01 +02:00
ViacheslavKlimov
9ddd22edcf EDQS: don't use Kafka manual partitions assignment 2025-03-20 16:12:42 +02:00
dashevchenko
bf2f4c3151 renamed package to tb-edqs 2025-03-20 11:54:34 +02:00
Igor Kulikov
3b4ee01aac Version set to 4.0.0-RC 2025-03-14 15:08:04 +02:00
dashevchenko
b4898568d1 removed edqs stats logs, fixed edqs apiUsage name 2025-03-11 15:12:01 +02:00
Viacheslav Klimov
5422a4545f
Merge branch 'edqs' into msa-edqs 2025-03-03 14:52:37 +02:00
ViacheslavKlimov
0115ac231b Fix license headers 2025-02-27 16:04:56 +02:00
dashevchenko
6c8887995a fixed license, included edqs tests to test suite 2025-02-25 13:23:26 +02:00
dashevchenko
f1e42aeeb0 merged with edqs branch 2025-02-25 11:25:10 +02:00
ViacheslavKlimov
e7ed98910a Merge branch 'master' of github.com:thingsboard/thingsboard into edqs 2025-02-25 11:10:31 +02:00
ViacheslavKlimov
a6a6112ecc Cleanup for EDQS 2025-02-24 12:11:05 +02:00
dashevchenko
93e191856f merged with edqs branch 2025-02-24 12:03:49 +02:00
dashevchenko
fb161730fb added msa tests for edqs 2025-02-21 18:38:26 +02:00
ViacheslavKlimov
965210f17b Multiple improvements for EDQS 2025-02-21 14:39:26 +02:00
ViacheslavKlimov
347cb5bc36 EDQS code cleanup 2025-02-21 11:48:21 +02:00
ViacheslavKlimov
a48144d2bb EDQS code cleanup 2025-02-20 17:34:28 +02:00
dashevchenko
e1e6f7c98b merged with opensource 2025-02-19 14:59:03 +02:00
dashevchenko
46005a957a fixed test 2025-02-19 14:46:39 +02:00
dashevchenko
2431fcf6c5 fixed RepositoryUtilsTest 2025-02-19 12:18:44 +02:00
ViacheslavKlimov
2a003709e0 Fix tests 2025-02-19 11:21:01 +02:00
dashevchenko
52a6540e5a added docker image for edqs 2025-02-14 15:12:00 +02:00
dashevchenko
0e9d0b3ba3 fixed EntityServiceTest 2025-02-03 18:27:36 +02:00
ViacheslavKlimov
c8db304ce6 Optimized postgres sync (#3208)
* optimized postgres sync

* Fix consumer stopping when stopWhenRead enabled

* Fix NPE on EDQS repartitioning

* fixed EntityServiceTest

* Fix EDQS yml props

---------

Co-authored-by: ViacheslavKlimov <viacheslavklimov11@gmail.com>
2025-01-30 12:43:14 +02:00
Viacheslav Klimov
86b5378d59 EDQS (#3196)
* Experiments with CSV

* CSV Loader v1

* EDQ tests

* Volatile variables instead of final

* Improvements

* updated loader with new entities

* Fix double memory usage issue

* Basic data structures and load

* Minor improvements

* Snappy + Large String reuse

* added EntityFields classes for each entity

* Basic implementation

* Minor improvements to KeyFilters

* implemented RepositoryUtils.checkKeyFilters

* Generic query implementation

* New structure

* Refactoring and few processors implementation

* extended DeviceData with shared/client attributes and device profile

* Minor refactoring of attribute scopes

* DeviceTypeFilter support

* Strong types of fields for each entity data class

* DeviceType and AssetType filters

* EntityView and Edge queries

* Relations Query

* Relation Query Implementation

* Update EDQS module version

* Sync with EDQS via Kafka

* EDQS: major refactoring

* EDQS API requests via Kafka

* EDQS: full sync with the database

* Refactoring for EDQS sync

* EDQS: major refactoring and new features

* EDQS refactoring, count query support, fix tests

* EDQS: refactoring for query processors

* Fix EDQS pom version

* Cleanup edqs.yml

* EDQS: tenant partitioning strategy; refactoring

* EDQS: latest events queue

* EDQS: support for monolith setup; RocksDB; other improvements

* EDQS: merge sync and events topics, introduce state topic

* EDQS: dynamic repartitioning

* implemented entity data query filters for edqs

* EdqsEntityQueryControllerTest - use in-memory queue

* edqs-filter fixes, added test

* EDQS: blob entity support

* EdqsEntityQueryControllerTest - use in-memory queue

* Use DummyEdqsService when disabled

* Fixes for EDQS

* Refactoring for EDQS tests

* Fix edqs requests partitioning

* EDQS: Fix for attributes handling

* Fix attributes saving in EntityServiceTest

* EDQS: refactoring, fixes

* Minor refactoring for query processor

* added ownerName/ownerType support

* fixed relation query processor

* fixed EntityServiceTest

* refactoring

* added support for parentId for relation query result

* Get rid of EntityNameFetcher

* Add fixme for relation query processor

* db restore with select all edqs fields

* fixed entity deletion

* fixed FieldUtils with new EntityFields

* dao method renamed

* EDQS: instance groups with same partitions; automatic sync; multiple fixes

* Refactoring for EDQS sync

* EDQS: refactoring

* Fix startup with Kafka

* fixed EntityQueryControllerTest

* fixed EdqsEntityServiceTest

* Separate queue admin for EDQS request template

* Implement new EDQS partitioning strategy

* EDQS: multiple fixes and refactoring

* Add mock EdqsRocksDb beans to tests

* added edqs stats for inmemory/grafana

* fixed filter tests

* Update todos

* Refactoring for QueueConfig

* Improvements and refactoring for EDQS consumers

* implemented TODOs

* test fixes

* Consume state topic up to end offsets

* edqs stats refactoring

* EDQS: cleanup on partitions removal; refactoring

* EDQS: minor refactoring

* EDQS: remove CSV loader

---------

Co-authored-by: Andrii Shvaika <ashvayka@thingsboard.io>
Co-authored-by: dashevchenko <dshevchenko@thingsboard.io>
2025-01-27 17:41:12 +02:00