Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Ts-1083] raw-parsed-toggle #3

Open
wants to merge 17 commits into
base: dev
Choose a base branch
from
Open

[Ts-1083] raw-parsed-toggle #3

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
fef22fb
added setting CodecUsePinAttributes
Dmitriy-Yugay Jul 19, 2022
9f201d1
added setting responseFormats
Dmitriy-Yugay Jul 20, 2022
69ea990
changed cradleApiVersion to 3.1.2
Dmitriy-Yugay Jul 20, 2022
81f51f2
minor fix
Dmitriy-Yugay Jul 22, 2022
69a5b77
added possibility of setting responseFormats from yaml config file
Dmitriy-Yugay Jul 22, 2022
bf53cc9
added possibility of setting responseFormats from yaml config file
Dmitriy-Yugay Jul 22, 2022
e87cefd
Update README.md
Dmitriy-Yugay Jul 22, 2022
659a499
Update README.md
Dmitriy-Yugay Jul 22, 2022
502b0f1
minor fixes
Dmitriy-Yugay Aug 1, 2022
26235ed
updated common version to 3.39.3
Dmitriy-Yugay Aug 9, 2022
9638f38
minor fixes
Dmitriy-Yugay Aug 19, 2022
97b0140
added local ResponseFormat
Dmitriy-Yugay Aug 24, 2022
1818e7b
minor fix
Dmitriy-Yugay Aug 29, 2022
ef66ecc
added logging to format converter
Dmitriy-Yugay Aug 30, 2022
9d613ed
added throwing exception when unrecognized response format
Dmitriy-Yugay Aug 30, 2022
67c4449
minor fix
Dmitriy-Yugay Aug 30, 2022
980d6d7
minor fix #2
Dmitriy-Yugay Aug 30, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 1 addition & 3 deletions build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,8 +53,6 @@ repositories {
dependencies {
api platform('com.exactpro.th2:bom:3.2.0')

implementation 'org.slf4j:slf4j-api'
implementation 'org.slf4j:slf4j-log4j12'

implementation 'com.exactpro.th2:common:3.39.3'

Expand All @@ -68,7 +66,7 @@ dependencies {
}
implementation 'com.exactpro.th2:grpc-data-provider:1.1.0-TS-1083-response-formats-2759013731-SNAPSHOT'

implementation 'io.github.microutils:kotlin-logging:2.1.23'
implementation 'io.github.microutils:kotlin-logging:2.1.14'

implementation('io.prometheus:simpleclient') {
because('need add custom metrics to provider')
Expand Down
2 changes: 1 addition & 1 deletion src/main/kotlin/com/exactpro/th2/lwdataprovider/RequestContext.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@ package com.exactpro.th2.lwdataprovider
import com.exactpro.cradle.messages.StoredMessage
import com.exactpro.th2.common.grpc.Message
import com.exactpro.th2.common.grpc.RawMessage
import com.exactpro.th2.dataprovider.grpc.MessageSearchRequest.ResponseFormat
import com.exactpro.th2.lwdataprovider.entities.responses.LastScannedObjectInfo
import com.exactpro.th2.lwdataprovider.entities.internal.ResponseFormat
import io.prometheus.client.Histogram
import mu.KotlinLogging
import java.time.Instant
Expand Down
2 changes: 1 addition & 1 deletion src/main/kotlin/com/exactpro/th2/lwdataprovider/configuration/Configuration.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@

package com.exactpro.th2.lwdataprovider.configuration

import com.exactpro.th2.dataprovider.grpc.MessageSearchRequest.ResponseFormat
import com.exactpro.th2.lwdataprovider.entities.internal.ResponseFormat
import java.util.*


Expand Down
2 changes: 1 addition & 1 deletion src/main/kotlin/com/exactpro/th2/lwdataprovider/db/CradleMessageExtractor.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,11 +23,11 @@ import com.exactpro.cradle.messages.StoredMessageId
import com.exactpro.th2.common.grpc.MessageGroupBatch
import com.exactpro.th2.common.grpc.RawMessage
import com.exactpro.th2.common.message.plusAssign
import com.exactpro.th2.dataprovider.grpc.MessageSearchRequest.ResponseFormat
import com.exactpro.th2.lwdataprovider.MessageRequestContext
import com.exactpro.th2.lwdataprovider.RabbitMqDecoder
import com.exactpro.th2.lwdataprovider.RequestedMessageDetails
import com.exactpro.th2.lwdataprovider.configuration.Configuration
import com.exactpro.th2.lwdataprovider.entities.internal.ResponseFormat
import mu.KotlinLogging
import kotlin.concurrent.withLock
import kotlin.system.measureTimeMillis
Expand Down
8 changes: 8 additions & 0 deletions src/main/kotlin/com/exactpro/th2/lwdataprovider/entities/internal/ResponseFormat.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
package com.exactpro.th2.lwdataprovider.entities.internal
Dmitriy-Yugay marked this conversation as resolved.
Show resolved Hide resolved

enum class ResponseFormat {
ALL,
BASE_64,
PARSED,

}
2 changes: 1 addition & 1 deletion src/main/kotlin/com/exactpro/th2/lwdataprovider/entities/requests/GetMessageRequest.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
package com.exactpro.th2.lwdataprovider.entities.requests

import com.exactpro.th2.common.grpc.MessageID
import com.exactpro.th2.dataprovider.grpc.MessageSearchRequest.ResponseFormat
import com.exactpro.th2.lwdataprovider.entities.internal.ResponseFormat
import com.exactpro.th2.lwdataprovider.grpc.toStoredMessageId


Expand Down
5 changes: 3 additions & 2 deletions src/main/kotlin/com/exactpro/th2/lwdataprovider/entities/requests/SseMessageSearchRequest.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,11 @@ import com.exactpro.cradle.Direction
import com.exactpro.cradle.TimeRelation
import com.exactpro.cradle.messages.StoredMessageId
import com.exactpro.th2.dataprovider.grpc.MessageSearchRequest
import com.exactpro.th2.dataprovider.grpc.MessageSearchRequest.ResponseFormat
import com.exactpro.th2.dataprovider.grpc.MessageStreamPointer
import com.exactpro.th2.lwdataprovider.entities.exceptions.InvalidRequestException
import com.exactpro.th2.lwdataprovider.entities.internal.ResponseFormat
import com.exactpro.th2.lwdataprovider.grpc.toInstant
import com.exactpro.th2.lwdataprovider.grpc.toLocalResponseFormats
import com.exactpro.th2.lwdataprovider.grpc.toProviderMessageStreams
import com.exactpro.th2.lwdataprovider.grpc.toProviderRelation
import com.exactpro.th2.lwdataprovider.grpc.toStoredMessageId
Expand Down Expand Up @@ -103,7 +104,7 @@ data class SseMessageSearchRequest(
attachedEvents = false, // disabled
lookupLimitDays = null,
onlyRaw = false, // NOT SUPPORTED in GRPC
responseFormats = grpcRequest.responseFormatsList
responseFormats = grpcRequest.responseFormatsList.toLocalResponseFormats()
)

private fun checkEndTimestamp() {
Expand Down
2 changes: 1 addition & 1 deletion src/main/kotlin/com/exactpro/th2/lwdataprovider/grpc/GrpcMessageRequestContext.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,12 @@ package com.exactpro.th2.lwdataprovider.grpc
import com.exactpro.cradle.messages.StoredMessage
import com.exactpro.th2.common.grpc.Message
import com.exactpro.th2.common.grpc.RawMessage
import com.exactpro.th2.dataprovider.grpc.MessageSearchRequest.ResponseFormat
import com.exactpro.th2.dataprovider.grpc.MessageSearchResponse
import com.exactpro.th2.dataprovider.grpc.MessageStreamPointers
import com.exactpro.th2.lwdataprovider.GrpcResponseHandler
import com.exactpro.th2.lwdataprovider.MessageRequestContext
import com.exactpro.th2.lwdataprovider.RequestedMessageDetails
import com.exactpro.th2.lwdataprovider.entities.internal.ResponseFormat
import com.exactpro.th2.lwdataprovider.entities.responses.LastScannedObjectInfo
import com.exactpro.th2.lwdataprovider.producers.GrpcMessageProducer
import java.util.concurrent.ConcurrentHashMap
Expand Down
16 changes: 16 additions & 0 deletions src/main/kotlin/com/exactpro/th2/lwdataprovider/grpc/GrpcUtils.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,26 @@ import com.exactpro.th2.common.grpc.Direction
import com.exactpro.th2.common.grpc.MessageID
import com.exactpro.th2.dataprovider.grpc.MessageStream
import com.exactpro.th2.dataprovider.grpc.TimeRelation
import com.exactpro.th2.lwdataprovider.entities.internal.ResponseFormat
import com.exactpro.th2.lwdataprovider.entities.requests.ProviderMessageStream
import com.google.protobuf.Timestamp
import java.time.Instant

fun List<com.exactpro.th2.dataprovider.grpc.MessageSearchRequest.ResponseFormat>?.toLocalResponseFormats() : List<ResponseFormat> {
val list : MutableList<ResponseFormat> = ArrayList()
if (this.isNullOrEmpty()) {
list.add(ResponseFormat.ALL)
} else
for (i in this) {
Dmitriy-Yugay marked this conversation as resolved.
Show resolved Hide resolved
when (i.name) {
Dmitriy-Yugay marked this conversation as resolved.
Show resolved Hide resolved
ResponseFormat.ALL.name -> list.add(ResponseFormat.ALL)
ResponseFormat.BASE_64.name -> list.add(ResponseFormat.BASE_64)
ResponseFormat.PARSED.name -> list.add(ResponseFormat.PARSED)
}
}
return list.toList()
}

fun Timestamp.toInstant() : Instant = Instant.ofEpochSecond(this.seconds, this.nanos.toLong())

fun TimeRelation?.toProviderRelation(): com.exactpro.cradle.TimeRelation {
Expand Down
2 changes: 1 addition & 1 deletion src/main/kotlin/com/exactpro/th2/lwdataprovider/handlers/SearchMessagesHandler.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,12 @@ import com.exactpro.cradle.messages.StoredMessageId
import com.exactpro.th2.lwdataprovider.MessageRequestContext
import com.exactpro.th2.lwdataprovider.configuration.Configuration
import com.exactpro.th2.lwdataprovider.db.CradleMessageExtractor
import com.exactpro.th2.lwdataprovider.entities.internal.ResponseFormat
import com.exactpro.th2.lwdataprovider.entities.requests.GetMessageRequest
import com.exactpro.th2.lwdataprovider.entities.requests.SseMessageSearchRequest
import mu.KotlinLogging
import java.util.concurrent.ExecutorService
import kotlin.math.max
import com.exactpro.th2.dataprovider.grpc.MessageSearchRequest.ResponseFormat

class SearchMessagesHandler(
private val cradleMsgExtractor: CradleMessageExtractor,
Expand Down
2 changes: 1 addition & 1 deletion src/main/kotlin/com/exactpro/th2/lwdataprovider/http/SseRequestContext.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@ package com.exactpro.th2.lwdataprovider.http
import com.exactpro.cradle.messages.StoredMessage
import com.exactpro.th2.common.grpc.Message
import com.exactpro.th2.common.grpc.RawMessage
import com.exactpro.th2.dataprovider.grpc.MessageSearchRequest.ResponseFormat
import com.exactpro.th2.lwdataprovider.*
import com.exactpro.th2.lwdataprovider.entities.internal.ResponseFormat
import com.exactpro.th2.lwdataprovider.entities.responses.Event
import com.exactpro.th2.lwdataprovider.entities.responses.LastScannedObjectInfo
import com.exactpro.th2.lwdataprovider.producers.MessageProducer53
Expand Down
6 changes: 3 additions & 3 deletions src/main/kotlin/com/exactpro/th2/lwdataprovider/producers/GrpcMessageProducer.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,10 @@ import com.exactpro.th2.common.grpc.ConnectionID
import com.exactpro.th2.common.grpc.MessageID
import com.exactpro.th2.dataprovider.grpc.MessageGroupItem
import com.exactpro.th2.dataprovider.grpc.MessageGroupResponse
import com.exactpro.th2.dataprovider.grpc.MessageSearchRequest.ResponseFormat.BASE_64
import com.exactpro.th2.dataprovider.grpc.MessageSearchRequest.ResponseFormat.PARSED
import com.exactpro.th2.dataprovider.grpc.MessageSearchRequest.ResponseFormat.ALL
import com.exactpro.th2.lwdataprovider.RequestedMessageDetails
import com.exactpro.th2.lwdataprovider.entities.internal.ResponseFormat.ALL
import com.exactpro.th2.lwdataprovider.entities.internal.ResponseFormat.BASE_64
import com.exactpro.th2.lwdataprovider.entities.internal.ResponseFormat.PARSED
import com.google.protobuf.Timestamp
import java.time.Instant

Expand Down
9 changes: 5 additions & 4 deletions src/main/kotlin/com/exactpro/th2/lwdataprovider/producers/MessageProducer53.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,14 +19,15 @@ package com.exactpro.th2.lwdataprovider.producers
import com.exactpro.th2.common.grpc.Message
import com.exactpro.th2.common.message.addFields
import com.exactpro.th2.common.message.messageType
import com.exactpro.th2.dataprovider.grpc.MessageSearchRequest.ResponseFormat
import com.exactpro.th2.lwdataprovider.CustomJsonFormatter
import com.exactpro.th2.lwdataprovider.RequestedMessageDetails
import com.exactpro.th2.lwdataprovider.entities.internal.ResponseFormat
import com.exactpro.th2.lwdataprovider.entities.internal.ResponseFormat.ALL
import com.exactpro.th2.lwdataprovider.entities.internal.ResponseFormat.BASE_64
import com.exactpro.th2.lwdataprovider.entities.internal.ResponseFormat.PARSED
import com.exactpro.th2.lwdataprovider.entities.responses.ProviderMessage53
import java.util.Base64
import com.exactpro.th2.dataprovider.grpc.MessageSearchRequest.ResponseFormat.PARSED
import com.exactpro.th2.dataprovider.grpc.MessageSearchRequest.ResponseFormat.BASE_64
import com.exactpro.th2.dataprovider.grpc.MessageSearchRequest.ResponseFormat.ALL


@Deprecated("for 5.3 messages")
class MessageProducer53 {
Expand Down