Skip to content

Latest commit

 

History

History
152 lines (94 loc) · 3.23 KB

08.2-avro-schema.md

File metadata and controls

152 lines (94 loc) · 3.23 KB

<< back to main index

Lab: Avro Schema

Overview

Use Avro schema with Kafka

Duration

30 minutes

References

Step-1: Create customer topic

Use command line or Kafka Manager to create a customer topic

This is using confluent kafka

# be in confluent directory
$   cd  ~/apps/confluent

$   ./bin/kafka-topics  --bootstrap-server localhost:9092   \
       --create --topic customer --replication-factor 1  --partitions 5

Step-2: Inspect Avro Schema

Avro schema file : src/main/java/x/lab08_avro/customer_v1.avsc

{"namespace": "x.lab08_avro",
 "type": "record",
 "name": "Customer",
 "fields": [
     {"name": "id", "type": "int"},
     {"name": "name", "type": "string"},
     {"name": "email", "type": "string"}
 ]
}

Step-3: Install AVRO Tools

If you don't have Avro tools, download them as follows:

$   mkdir  -p  ~/apps/avro
$   wget -O ~/apps/avro/avro-tools.jar  \
          https://dlcdn.apache.org/avro/stable/java/avro-tools-1.12.0.jar
$   wget -O ~/apps/avro/avro.jar   \
         https://dlcdn.apache.org/avro/stable/java/avro-1.12.0.jar

Verify Avro tools directory:

$   ls -lh  ~/apps/avro

You will see downloaded jar files

-rw-rw-r-- 1 sujee sujee 596K Jul 26 13:08 avro.jar
-rw-rw-r-- 1 sujee sujee  53M Jul 26 13:08 avro-tools.jar

Step-4: Compile Avro file

We are going to compile the avro schema into a Java file

$   cd  ~/dev/kafka-labs   # be in project root directory

$   java -jar ~/apps/avro/avro-tools.jar  \
         compile schema \
         src/main/java/x/lab08_avro/customer_v1.avsc \
         src/main/java/

This will generate Customer.java file.

Note: To force compile, try deleting the Customer.java file and run the compile again

==> Inspect the file here : src/main/java/x/lab08_avro/Customer.java

Step-5: Compile

In Eclipse you can force compile the project like this : Project --> Clean --> kafka-api

Make sure the compile is clean with no errors.

Also do a maven compile as follows

$   cd  ~/dev/kafka-labs   # be in project root directory

$   mvn clean package  -DskipTests

Make sure the compile is clean with no errors.

Step-6: Run the Consumer

Inspect Consumer: src/main/java/x/lab08_avro/AvroConsumer.java

Run the consumer.

Note: You will need schema registry running to make this work

Step-7: Run the Producer

Inspect producer: src/main/java/x/lab08_avro/AvroProducer.java

Run the producer.

Note: You will need schema registry running to make this work

==> Observe events going from producer to consumer

Step-8: Check Schema Registry

Query Schema Registry as follows

$   curl --silent -X GET http://localhost:8081/schemas/

Better formatting with jq

$   curl --silent -X GET http://localhost:8081/schemas/   | jq

Note: if jq is mising, you can install it as follows:

$   sudo apt install -y jq

Step-9: Shutdown Confluent Stack

Once you are done, you can shutdown the Confluent stack