Table of Contents
It takes time, struggle to understand what you need in a field, it takes time…
1. Introduction: Why do we need blockchain?
"Unlike today’s systems, where a participant’s private programs are used to update their private ledgers, a blockchain system has shared programs to update shared ledgers."
1.1. Traditional Ledger Inefficiencies
Modern technology has taken this process from stone tablets and paper folders to hard drives and cloud platforms, but the underlying structure is the same.
1.1.1. 1. Separated Ledger – Fragmented System
In traditional business networks, each participant has their own way of managing their own separate records of transactions. This fragmentations creates inefficiencies and vulnerabilities, since there is no convinence way for different ledger to communicate.
- Verification
For example, each time an asset is sold, both sides requires to have their own provenance (history of ownership) established anew. Not only it is a laborious process, but also there is no efficient way for a single party to prove whether a record is valid.
- Manual Processes
Many processes, like contract signing and execution, are still done manually, which is slow and inefficient.
1.1.2. 2. Fault Tolerance – Replicas
Each database in a traditional system contains unique information and represents a single point of failure, If one part of the system fails or is compromised, it can have widespread implications. For that, many companies build distributed systems to store data in replica nodes, aiming to improve throughput and help fault tolerance, however, despite the data redundancy, this system still cannot achieve Byzantine fault tolerance.
1.2. Blockchain Advantages
1.2.1. 1. Distributed Ledger
n a blockchain network, every participant has a replicated copy of the ledger. This ensures consistency in records and eliminates the need for separate record-keeping and reconciliation.
1.2.2. 2. Immutable and Shared Processes
1.2.3. 3. Streamlining Processes (i.e., Human)
Thanks to Cryptography, blockchain can significantly reduce the time required for processing transactions and verifying assets, as everything is updated in real time on the ledger and is visible to all participants. (Smart Contracts) By streamlining processes and eliminating intermediaries, blockchain can significantly reduce transactional and operational costs.
1.3. "No Extra Data" in Permissioned Blockchains
As I mentioned before, in traditional distributed systems, each company might maintain numerous replicas of its data for fault tolerance, leading to significant redundancy. With a permissioned blockchain, each member company would only need to maintain a single copy of the ledger, which includes the necessary data from all participants. This approach effectively reduces one of the biggest issue of blockchain system - data redudency (No much extra space comparing to traditional distributed system), while still providing those core advantages of blockchain.
1.4. Scaling Issue of Blockchain
One of the challenges of blockchain technology, particularly public blockchains, is scalability in terms of transaction speed and throughput. In a permissioned blockchain, since the number of nodes is limited and known, it's easier to implement efficient consensus mechanisms that can handle a higher throughput of transactions, making them more suitable for industrial applications. But still comparing to traditional storage systems, that the gap remains too big for blockchains to be disruptive to incumbent database systems.
If these factors are well-managed, blockchains could indeed revolutionize how companies manage and process data, leading to more streamlined, secure, and efficient operational models.
2. Fabric
Being a permissioned blockchain, Fabric offers more control over network participants and transactions, which can lead to more consistent benchmarking conditions. VMs virtualize the hardware, whereas Docker containers virtualize the operating system.
curl -sSLO https://raw.githubusercontent.com/hyperledger/fabric/main/scripts/install-fabric.sh && chmod +x install-fabric.sh
2.1. Concepts of Fabric Network
2.1.1. Channels
Channels are a private layer of communication between specific network members (peers). Channels can be used only by organizations that are invited to the channel, and are invisible to other members of the network. Each channel has a separate blockchain ledger.
2.1.2. Chaincode
To sign a transaction, each organization needs to invoke and execute the chaincode on their peer, which then signs the output of the transaction. If the output is consistent and has been signed by enough organizations, the transaction can be committed to the ledger.
By default, endorsement policies are specified in the chaincode definition, which is agreed to by channel members and then committed to a channel. (that is, one endorsement policy covers all of the state associated with a chaincode). Please notice that, for now, up till fabric v2.5, the chaincode endorsement policy only supports inside single organization. Which means all peers inside a single organization will be treat as a single vote.
End users interact with the blockchain ledger by invoking smart contracts. In Hyperledger Fabric, smart contracts are deployed in packages referred to as chaincode. Organizations that want to validate transactions or query the ledger need to install a chaincode on their peers (nodes). After a chaincode has been installed on the peers joined to a channel, channel members can deploy the chaincode to the channel and use the smart contracts in the chaincode to create or update assets on the channel ledger.
Chaincode is a program, written in Go, Node.js, or Java that implements a prescribed interface. Chaincode runs in a secured Docker container isolated from the endorsing peer process. Chaincode initializes and manages ledger state through transactions submitted by applications.
Channel members come to agreement using the following four steps. Not every organization on a channel needs to complete each step.
- 1. Package the chaincode
This step can be completed by one organization or by each organization.
- 2. Install the chaincode on peers:
Every organization that will use the chaincode to endorse a transaction or query the ledger needs to complete this step.
- 3. Approve a chaincode definition for your organization:
Every organization that will use the chaincode needs to complete this step. The chaincode definition needs to be approved by a sufficient number of organizations to satisfy the channel’s LifecycleEndorsement policy (a majority, by default) before the chaincode can be started on the channel.
- Code Inspection:
Organizations may choose to inspect the source code to ensure it's not malicious and adheres to best practices. While this can be cumbersome, it's a critical step for ensuring trust in the network.
- Testing:
Organizations might run the chaincode in a controlled environment to test its functionality and ensure it behaves as expected.
- Security Audits:
In some cases, organizations might perform security audits on the chaincode, either internally or through third parties.
In a consortium where members have established trust, the process might be less stringent compared to a more open network.
- Code Inspection:
- 4. Commit the chaincode definition to the channel:
The commit transaction needs to be submitted by one organization once the required number of organizations on the channel have approved. The submitter first collects endorsements from enough peers of the organizations that have approved, and then submits the transaction to commit the chaincode definition.
Please notice that in Ethereum, smart contracts are stored on the blockchain itself. Every node on the Ethereum network has a copy of the entire blockchain, including all the smart contracts.
In contrast, Hyperledger Fabric stores chaincode on peer nodes, not on the blockchain. The chaincode is executed on these peer nodes, and the results (transactions) are then recorded on the ledger. This approach allows for greater flexibility and efficiency, especially in a permissioned blockchain setting where trust among participants is higher. And in addition, even though a organization modify the chaincode and wishing to act maliciously, as long as that org do not control major number of peers, the system remains safe.
2.1.3. Ledger
A ledger consists of two distinct, though related, parts – a world state and a blockchain, blockchain determines world state.
- World State
A database that holds current values of a set of ledger states. Ledger states are, by default, expressed as key-value pairs. The world state makes it easy for a program to directly access the current value of a state rather than having to calculate it by traversing the entire transaction log. The world state is implemented as a database. This makes a lot of sense because a database provides a rich set of operators for the efficient storage and retrieval of states.
Reading data from the World State in Hyperledger Fabric does not require endorsements, and is a local operation for each peer. If an organization alters its local World State, it will face inconsistencies and operational issues, but this will not impact the network's overall integrity, as the other peers' World States and the shared ledger remain unaffected.
- Merkle Patricia Trie (State Trie)
In public blockchains (e.g., Ethereum here), since the risks of a node act dishonestly is increased, it uses a data structure called a "state trie" (Merkle Patricia Trie) to securitly store the entire state of the network (chain), the state trie is a complex, cryptographic tree-like structure, which has following features:
- 1. Efficiency
The trie structure allows for efficient lookups and updates. Changes to any account only require updating the path from that account's leaf to the root, not the entire trie. In the trie, navigating between nodes are based on the value of the account address.
- 2. Security:
The cryptographic nature of the trie ensures that the state is tamper-evident. Any change in a leaf (account state) changes the root hash.
- 3. Merkle Proofs:
The trie enables Merkle proofs, allowing nodes to verify that a particular piece of data is included in the trie without needing the entire trie. (i.e., two leafs and their hashed root)
- Merkle Patricia Trie (State Trie)
Whereas the world state contains a set of facts relating to the current state of a set of business objects, the blockchain is an historical record of the facts about how these objects arrived at their current states.
- Understanding the use of blockchain: Traditional Distributed Systems v.s. Blockchain Systems
- Traditional Distributed Systems Without Blockchain:
In a conventional distributed database system, data is shared across multiple nodes, but there is often a reliance on some level of trust and centralized control mechanisms. These systems generally handle faults well (system failures), but they are not inherently designed to be resistant to Byzantine faults, where nodes may act maliciously or unpredictably. In Hyperledger fabric, even though transaction verification often do not require querying blockchain, blockchain here still brings a high-level resilience against Byzantine faults: Honest nodes in the network will reject any blocks or transactions that do not meet the network's rules, preventing malicious actors from causing harm even if they can tamper with their local copy of the world state or ledger
Thoughts on blockchain provides a basic for others like consensus algorithms and cryptography to take effect and build an Immutable Decentralized Ledger
- Traditional Distributed Systems Without Blockchain:
2.1.4. Peers and Orderers
Link for the official fabric doc. Many distributed blockchains, such as Ethereum and Bitcoin, are not permissioned, which means that any node can participate in the consensus process. These systems rely on probabilistic consensus algorithms which eventually guarantee ledger consistency to a high degree of probability, which are still vulnerable to divergent ledgers. (also known as a ledger "fork") where different participants in the network have a different view of the accepted order of transactions. Public blockchain trying to solving this problem by creating a system where the cost of cheating (successfuly forking) outweighs the potential benefits significantly.
- Fabric Gateway
Fabric Gateway is a service, introduced in Hyperledger Fabric v2.4 peers, that provides a simplified, minimal API for submitting transactions to a Fabric network.
Fabric's design relies on deterministic consensus algorithms, any block validated by the honest peer is guaranteed to be final and correct. It features a node called an orderer (it's also known as an "ordering node") that does this transaction ordering, which along with other orderer nodes forms an ordering service.
A client application sends a transaction proposal to the Fabric Gateway service to evaluate a transaction proposal. The gateway forwards the transaction to peers in the organizations required by the endorsement policy. These endorsing peers run the transaction and return the transaction response to the gateway service, they do not apply the proposed update to their copy of the ledger at this time. Note that a peer endorses a proposal response by adding its digital signature, and signing the entire payload using its private key. This endorsement can be subsequently used to prove that this organization’s peer generated a particular response.
- Endorsing Peers
A peer that commits transactions can also be an endorsing peer, but it doesn't have to be. All specified in the policy. If there are discrepancies in the results from different endorsing peers (which would be unusual in a correctly functioning network), the Gateway Service would identify the transaction as invalid. Normally, the results should be consistent if the chaincode is deterministic and the network is functioning correctly.
With successful completion of the first proposal phase, the client application has received an endorsed transaction proposal response from the Fabric Gateway service for signing. After signing and they collectively satisfy the endorsement policies, the gateway service forwards the transaction to the ordering service.
- Ordering Nodes
- Consensus Challenge in a Distributed Network
In a distributed network like a blockchain, nodes (peers) are spread out and do not necessarily have a synchronized view of all transactions at any given moment. If each node were to independently determine the order of transactions, inconsistencies could arise between different nodes' versions of the ledger. This is especially true in networks where transactions are frequent and nodes are geographically dispersed.
- Order Service
The ordering service in Hyperledger Fabric addresses this challenge. It is a separate component whose primary function is to receive transactions that have been endorsed (approved) by peers, then establish a consensus on the order of these transactions. By doing this, the ordering service ensures that all transactions are sequenced in a consistent manner across the entire network. After the blocks are created, the ordering service (nodes) distributes them to all peer nodes in the network. Each peer node then independently validates the block (which includes a set of ordered transactions) and, upon successful validation, adds the block to its copy of the ledger.
For example, the different ordering nodes would use the Raft consensus algorithm to come to agreement on the order of transactions across the network.
For an endorsed transaction, the gateway service forwards the transaction to the ordering service, which orders it with other endorsed transactions, and packages them all into a block. It's worth noting that the sequencing of transactions in a block is not necessarily the same as the order received by the ordering service, since there can be multiple ordering service nodes that receive transactions at approximately the same time. (determined by the most common order) What's important is that the ordering service puts the transactions into a strict order, and peers will use this order when validating and committing transactions.
Whereas peers execute smart contracts (chaincode) and process transactions, orderers most definitely do not. Every authorized transaction that arrives at an orderer is then mechanically packaged into a block — the orderer makes no judgement as to the content of a transaction. (except for channel configuration transactions)
- Consensus Challenge in a Distributed Network
The third and final phase of the transaction workflow is the distribution of ordered transactions from orderers to peers. Each peer then validates each transaction, in the correct order, and ensures that each transaction has been consistently endorsed by all required organizations. Only then does the peer commit the block to its copy of the channel ledger.
- No Re-execution of Chaincode
During this phase, the peers do not re-execute the chaincode. Instead, they use the results of the execution that were endorsed in the first step. The peers check whether the transaction has met the endorsement policy requirements – specifically, whether it has been endorsed by the required organizations/peers. They also perform MVCC checks to ensure that there have been no changes in the state of the ledger that would invalidate the transaction since the endorsements were received.
3. Fabric Workloads
3.1. Using the Fabric test network
3.1.1. 1. Bring up the test network
./network.sh up
- 1). createOrgs
Before brings up the network, we should first create organizations that will participates.
cryptogen generate --config=./organizations/cryptogen/crypto-config-org1.yaml --output="organizations"
cryptogen is an utility for generating Hyperledger Fabric key material. It is provided as a means of preconfiguring a network for testing purposes. It would normally not be used in the operation of a production network.
- The cryptogen tool reads the provided YAML configuration file
For each entity defined in the configuration file (like peers, users, and orderers), cryptogen generates the necessary cryptographic material. This includes:
- X.509 Certificates:
Used for TLS (Transport Layer Security) and client authentication. Each entity in the network (peer, orderer, CA, etc.) needs a certificate to participate securely in the network.
- Private and Public Keys:
Each certificate is paired with a private key, which is used for signing and encryption purposes. In the case of fabric network, generated files should be located in organizations/peerOrganizations and organizations/ordererOrganizations.
- X.509 Certificates:
- The cryptogen tool reads the provided YAML configuration file
- 2). start the docker container
COMPOSE_FILES="-f compose/${COMPOSE_FILE_BASE} -f compose/${CONTAINER_CLI}/${CONTAINER_CLI}-${COMPOSE_FILE_BASE}" DOCKER_SOCK="${DOCKER_SOCK}" ${CONTAINER_CLI_COMPOSE} ${COMPOSE_FILES} up -d 2>&1
Docker Compose efficiently starts and manages multiple Docker containers based on the configurations provided in a docker-compose.yaml file. It streamlines the process of deploying multi-container applications. The DOCKERSOCK is used for explicit communication with Docker daemon. Which is usually refers to var/run/docker.sock
3.1.2. 2. Create the channel
./network.sh createChannel
- 1). configtxgen
The configtxgen command allows users to create and inspect channel config related artifacts.
configtxgen -profile ChannelUsingRaft -outputBlock ./channel-artifacts/${CHANNEL_NAME}.block -channelID $CHANNEL_NAME
Basically, what this command will do is generate artifacts (i.e., the genesis block file) based on the config file in configtx/configtx.yaml. The term "creating the genesis block" refers to the process of generating the first block in the blockchain from the configurations specified in configtx.yaml. The file configtx.yaml includes definitions for various elements such as:
- Organizations
Information about each participating organization, their MSP (Membership Service Provider) details, etc.
- Orderer Settings
Configuration for the ordering service, which might include the type of consensus (like Raft), and settings related to batching, etc.
- Channel Configurations:
Specific settings for channels, including the policies for governance the participants, and the endorsement
The generated .block file includes static, fundamental information like orderer endpoints and network policies, it does not include dynamic details like peer endpoints, which are managed outside the genesis block through other mechanisms in Hyperledger Fabric.
Please note that this step aim to establish trust among multiple organizations by listing locations of each organizations's root CA in Profile section of configtx.yaml.
- Organizations
- 2). osnadmin
The osnadmin channel command allows administrators to perform channel-related operations on an orderer.
osnadmin channel join --channelID ${channel_name} --config-block ./channel-artifacts/${channel_name}.block -o localhost:7053 --ca-file "$ORDERER_CA" --client-cert "$ORDERER_ADMIN_TLS_SIGN_CERT" --client-key "$ORDERER_ADMIN_TLS_PRIVATE_KEY" >> log.txt 2>&1
osnadmin is primarily used for managing channels on the orderer side, like join the orderer to the channel, this usually involved installing the genesis block to the orderers.
- 3). peer channel
The peer channel command allows administrators to perform channel related operations on a peer, such as joining a channel or listing the channels to which a peer is joined.
peer channel join -b $BLOCKFILE >&log.txt
Be aware that before you call join, you should set up the global environment variables such as key, certificate of that organization. This also involved install the genesis block to the peers.
3.1.3. 3. Deploying a smart contract (chaincode) to the channel
- 1). Package the smart contract
peer lifecycle chaincode package basic.tar.gz --path ../asset-transfer-basic/chaincode-go/ --lang golang --label basic_1.0
- 2). Install the chaincode package
peer lifecycle chaincode install basic.tar.gz
- 3). Approve a chaincode definition
peer lifecycle chaincode approveformyorg
- 4). Committing the chaincode definition to the channel
peer lifecycle chaincode querycommitted --channelID mychannel --name basic