in consumer git Linux producer protocol buffer rabbitmq ~ read.

Protocol Buffers and RabbitMQ on Ubuntu 14.04 Part 1

This article is the first post in series:

Introduction

In the distributed systems or systems is designed with micro services, to communicate between other systems is important. In the previous article, I am shown to you use JSON format to transfer data with RabbitMQ. In this article, I will show to you how to use Protocol Buffers to encode, decode data and transfer data with RabbitMQ as Queue in your system.

Protocol buffers are a flexible, efficient, automated mechanism for serializing structured data – think XML, but smaller, faster, and simpler. It is developed by Google. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages. You can even update your data structure without breaking deployed programs that are compiled against the “old” format.

To use Protocol Buffers in my systems, I will:

  • Step 1: Install Protocol buffer compiler on Ubuntu 14.04
  • Step 2: Define message structure formats in a .proto file.
  • Step 3: Use the protocol buffer compiler
  • Step 4: Write Producer use the Python protocol buffer API to encode and send messages to RabbitMQ server
  • Step 5: Write Consumer use the Python protocol buffer API to receive and decode messages from RabbitMQ server

Prerequisites

You must have RabbitMQ in your systems. If you do not have RabbitMQ, Please read this post and install it.

Step 1: Install Protocol buffer compiler on Ubuntu 14.04

  • Installing gcc compiler on Ubuntu
sudo apt-get update  
sudo apt-get install build-essential

#check gcc  
gcc -v  
  • Download and Extract the protocol buffer archive
wget https://github.com/google/protobuf/releases/download/v2.6.1/protobuf-2.6.1.tar.gz  
tar -xzvf protobuf-2.6.1.tar.gz  
  • Install Protocol buffer compiler
cd protobuf-2.6.1  
sudo ./configure  
make  
make check  
sudo make install  
protoc –version  

Sometimes the latest version of protocol version does not load up. So we can do it manually by this command:

sudo ldconfig  
protoc –version  

Step 2: Define message structure formats in a .proto file

In this step, You need to define your data which you want to send to Rabbit. I will define a message for User information and a message for ListUsers.

mkdir protocol_buffers_python  
cd protocol_buffers_python  
vim users.proto  

And content of users.proto :
 

package tutorial;  
message User {  
     required string name = 1;  
     required int32 id = 2;  
     optional string email = 3;  
}

message ListUsers {  
     repeated User user = 1;  
}

As you can see, the syntax is similar to C++ or Java. Let’s go through each part of the file and see what it does.

The .proto file starts with a package declaration, which helps to prevent naming conflicts between different projects. In Python, packages are normally determined by directory structure, so the package you define in your .proto file will have no effect on the generated code. However, you should still declare one to avoid name collisions in the Protocol Buffers name space as well as in non-Python languages

Next, you have your message definitions. A message is just an aggregate containing a set of typed fields. Many standard simple data types are available as field types, including bool, int32, float, double, and string.

The ” = 1″, ” = 2″ markers on each element identify the unique “tag” that field uses in the binary encoding. Tag numbers 1-15 require one less byte to encode than higher numbers, so as an optimization you can decide to use those tags for the commonly used or repeated elements, leaving tags 16 and higher for less-commonly used optional elements.

Each field must be annotated with one of the following modifiers:

  • required: a value for the field must be provided, otherwise the message will be considered “uninitialized”. Serializing an uninitialized message will raise an exception. Parsing an uninitialized message will fail. Other than this, a required field behaves exactly like an optional field.
  • optional: the field may or may not be set. If an optional field value isn’t set, a default value is used.
  • repeated: the field may be repeated any number of times (including zero). The order of the repeated values will be preserved in the protocol buffer. Think of repeated fields as dynamically sized arrays.

You’ll find a complete guide to writing .proto files – including all the possible field types – in the Protocol Buffer Language Guide

Step 3: Use the protocol buffer compiler

Now that you have a .proto, the next thing you need to do is generate the classes you’ll need to read and write ListUsers message. To compiler you can type:

protoc -I=$SRC_DIR –python_out=$DST_DIR $SRC_DIR/users.proto  
  • $SRC_DIR : where your application’s source code lives
  • $DST_DIR: where you want the generated code to go

To me, it is:

protoc -I=/home/ngoccuong/working/source_code/protocol_buffers_python \  
--python_out=/home/ngoccuong/working/source_code/protocol_buffers_python \ 
/home/ngoccuong/working/source_code/protocol_buffers_python/users.proto

Because I want Python classes, I use the --python_out option – similar options are provided for other supported languages. With Java, it is –-java_out.

This generates users_pb2.py in your specified destination directory.

In this article, I will finish at step 3. We have installed protocol buffer compiler and use it to compiler .proto file. In the next article, I will continue with* Step 4* and step 5 to work between Protocol buffer and RabbitMQ.

You can see full source code on:

https://github.com/cuongbangoc/protocolbufferspython

comments powered by Disqus