# Welcome! Hi, we are **Oracle Labs** - professional Chainlink Node Operator and service provider.\ \ *We’ve made our mission to help you build the future of Web3 by enabling next-generation hybrid smart-contracts and dApps with Chainlink products.* **What we do?** * We operate highly reliable Chainlink oracle infrastructure on various L1s and L2s * We help our clients develop and host external adapters / initiators, run jobs, connect their data to blockchain networks to monetize it and build hybrid smart contacts with Chainlink services * We also do web2/web3 development and data analytics **Our expertise:** * Chainlink oracle node hosting * DevOpsSec * Consulting and development (External Adapters, Chainlink Jobs, Smart Contracts) ## Our services:
Chainlink Node Operator/pages/D9RMFHI2twxJd29OHbph/files/rnIfSOAFHtcNyE1uomJv
Consulting & Integrations/pages/fj1JN2DJStxfpc1Z7lEr/files/mZ8mUBr5lTfn6rdU1HBX
### Useful links: Website: [https://www.oraclelabs.link
](https://www.oraclelabs.link/)Github: \ Telegram: \ Twitter: \ Discord: # Why Oracle Labs? We’ve made our mission to help you build the future of Web3 by enabling next-generation hybrid smart-contracts with Chainlink products. ## **High Availability** We operate a triple redundant cloud infrastructure for Chainlink and Blockchain nodes with guaranteed uptime. Our cloud operator has fault tolerant infrastructure, developed by most full-stack engineering teams worldwide. ## **Security** Your data is safe due to the best DevOpsSec practices ## **Reliability** Data parsed through a Decentralized Oracle Network (DON) comes from multiple sources to ensure its integrity and validity. # Chainlink node service provider Providing mainnet and testnet oracle node services for direct request jobs (anyAPI), market and data feeds (FluxMonitoring), data monetization (for Data Providers). # Consulting & development Advice on the operation of Chainlink nodes and smart contacts development Our services: * Custom Chainlink development (jobs/external adapters/smart contracts) * Custom API integration * DevOps node consulting and deployment * Data monetization * Data analytics (AI/ML) # Supported Networks Oracle Labs Chainlink nodes are free to use on any supported testnets. They do not require whitelisting and are operated and monitored in the same way like our production nodes to enable a smooth development and testing process. Currently we support following blockchain networks: * Polygon Mainnet * Ethereum Goerli (testnet) * Polygon Mumbai (testnet) Other mainnets and testnets can be supported on request. # Cloud setup Our services, nodes and application are deployed on a highly reliable cloud infrastructure with guaranteed uptime. We run K8s (Kubernetes) cluster with triple redundancy of all services including Chainlink nodes failover. Our monitoring system is based on following stack: Fluentbit, Loki, Grafana, Prometheus. # How do we monitor? We have set up a failover monitoring that can track all the issues in chainlink nodes or blockchain nodes as well as cloud operator infrastructure.
# Monitoring Endpoints **Cluster-1** | Name | Url | | ------------------- | -------------------------------------------- | | Relay k8s Cluster-1 | | | Grafana | | | Prometheus | | | | | **External Adapters** | Name | Url | | ------------------------------- | ----------------------------------------- | | External Adapters API Cluster-1 | | | | | **Ethereum Goerli Test Network** | Name | Url | | ---------------------- | -------------------------------------------- | | Chainlink Cluster-1 | | | Blockchain Cluster-1 | | | Alchemy RPC (Failover) | | | Infura RPC (Failover) | | **Ethereum Mainnet (under construction)** | Name | Url | | ---------------------- | ---------------------------------------------------------------------------- | | Chainlink Cluster-1 |


(under construction)

| | Blockchain Cluster-1 |


(under construction)

| | Alchemy RPC (Failover) | | | Infura RPC (Failover) | | # Chainlink Guide The following chart explains Chainlink's Direct Request model.
\[0] When a transaction is submitted, it triggers a request on the Consumer smart contract. The JobID and input parameters are specified. \[1] The Consumer smart contract sends this request to a specific Oracle smart contract with the input parameters and payment in LINK. \[2] The Oracle smart contract receives payment in LINK for the request and passed it to a Chainlink Node. \[3] Chainlink Node receives a request and executes it by initiating the job/external adapter specified in the Consumer contract. \[4] Chainlink Node requests a Third-Party API premium or free data with the input parameters and receives the expected response. \[5] Chainlink Node parses and encodes the response, preparing the data to be committed on-chain. \[6] Chainlink Node Wallet invokes the callback function on the Oracle Contract and pays the transaction gas fee on specific blockchain. \[7] Oracle smart contract commits the prepared data on-chain writing it to the Consumer smart contract. \[8] Consumer smart contract receives the fulfilled request making the data available to be read on-chain. # Chainlink Public Jobs ### Supported Networks | Network | Testnet/Mainnet | | -------- | ---------------- | | Ethereum | Testnet (Goerli) | | Ethereum | Mainnet | | Polygon | Testnet (Mumbai) | | Polygon | Mainnet | ### GitHub All public jobs are available in our Github: ### Private & Custom Data Requests * All jobs listed are of the `GET` format * If your project requires a private `POST` format with data inputs, please contact us in our [discord](https://discord.gg/Z7EyGC9j6m) channel. # Solving common problems \#chainlink #node #LINK #faq ### Chainlink transaction reverted Check the following: * Check minimum payment for your chainlink node. Go to Operator UI -> settings -> configuration and find MinContractPayment or MINIMUM\_CONTRACT\_PAYMENT\_LINK\_JUELS. If you’re using jobs on external operator, ask them about this parameter. By default it’s often 0.00001 LINK, but for Goerli 0.1 LINK. * Your Operator Contract Address funded with LINK and have enough for minimum payment to chainlink node. You can check that with call *balanceOf* function on Chainlink Token Contract. * Your contract, that interacting with Operator Contract funded with LINK, has enough LINK for minimum payment to chainlink node and if you set fee for some Operator methods, make sure you set the value more than minimum payment to chainlink node * On Operator Admin Address (you see that in Operator UI) you have enough ETH to run transactions * In Operator Contract, calling function *getAuthorizedSenders* return should contain Operator Admin Address ### How do I know that the LINK balance is sufficient? * Find LINK Token address for your network. * Click on address and go to the scan page * Go to Contract tab * Under the Contract tab you should see tab “Read Contract”. Click that * Click on *balanceOf*, write address you need to check and push *Query* ### How to fund my wallet with testnet LINK and ETH? Testnet LINK is available at [faucets.chain.link](https://faucets.chain.link/goerli) # Chainlink Node Logs Source: (by Linkriver) # Log This service manages log subscription requests for the Chainlink node without creating a new subscription for each request and backfills logs in case of a node crash or restart. ### \[ERROR] Log subscriber could not create subscription to Ethereum node The Chainlink node could not create a new log subscription to listen to on-chain events and will not be able to interact with the target network appropriately. This indicates an issue with the remote RPC endpoint (full node). * Check the blockchain connection of the Chainlink node (e.g. the Full-node-as-a-Service subscription and renew or switch the plan if necessary to prevent RPC rate limits from being hit) * Run an own full node with custom configuration and no performance restrictions\ If you are running an own full node check * Whether an error occurred requiring to resync the node (firewall, bad block, OOM) * Whether it happens due to the node’s performance (hardware specs, disc’s IOPS) ### \[WARN] LogBroadcaster: BlockBackfillSkip is set to true, preventing a deep backfill - some earlier chain events might be missed Backfilling guarantees to check older heads for relevant event logs in case the Chainlink node crashed or has been restarted. This error indicates that the Chainlink node will not get the event logs from older blocks and might skip potential job runs. `BlockBackfillSkip` is set to false by default to avoid this in case of a node crash or restart. * [Official Chainlink documentation](https://docs.chain.link/docs/configuration-variables/#block_backfill_skip) ### \[WARN] LogBroadcaster: Error in the event loop - will reconnect The log subscription has been closed and could not be recreated, this might result from an issue with the remote RPC endpoint (full node) or a changed set of on-chain contracts the Chainlink node should listen to for events. * Check the blockchain connection of the Chainlink node (e.g. the Full-node-as-a-Service subscription and renew or switch the plan if necessary to prevent RPC rate limits from being hit) * Run an own full node with custom configuration and no performance restrictions * Make sure your node address was added to the on-chain contracts it should interact with ### \[WARN] LogBroadcaster: Detected a large block number difference between a log and recently seen head. This may indicate a problem with data received from the chain or major network delays * Check the blockchain connection of the Chainlink node (e.g. the Full-node-as-a-Service subscription and renew or switch the plan if necessary to prevent RPC rate limits from being hit) * Run an own full node with custom configuration and no performance restrictions\ If you are running an own full node check * Whether an error occurred requiring to resync the node (firewall, bad block, OOM) * Whether it happens due to the node’s performance (hardware specs, disc’s IOPS) ### \[ERROR] LogBroadcaster: Backfill - could not fetch latest block header, will retry * Check the blockchain connection of the Chainlink node (e.g. the Full-node-as-a-Service subscription and renew or switch the plan if necessary to prevent RPC rate limits from being hit) * Run an own full node with custom configuration and no performance restrictions\ If you are running an own full node check * Whether an error occurred requiring to resync the node (firewall, bad block, OOM) * Whether it happens due to the node’s performance (hardware specs, disc’s IOPS) ### \[ERROR] LogBroadcaster: Deadline exceeded, unable to backfill a batch of logs. Consider setting EvmLogBackfillBatchSize to a lower value Blocks can be read in batches to avoid hitting the websocket request data limit if the number of blocks to check is high and differs significantly from the latest head. This may delay the processing of the latest log and the Chainlink node may miss job requests. * Set `BlockBackfillSkip` to true if necessary and restart the node * [Official Chainlink documentaiton](https://docs.chain.link/docs/configuration-variables/#eth_log_backfill_batch_size) ### \[ERROR] LogBroadcaster: Inner deadline exceeded, unable to backfill a batch of logs. Consider setting EvmLogBackfillBatchSize to a lower value * Set `EVM_LOG_BACKFILL_BATCH_SIZE` to a lower value, or * Set `BlockBackfillSkip` to `true` if necessary and restart the node ### \[ERROR] LogBroadcaster: Unable to backfill a batch of logs after retries * Set `EVM_LOG_BACKFILL_BATCH_SIZE` to a lower value, or * Set `BlockBackfillSkip` to `true` if necessary and restart the node # Keystore This service enables the generation of new keys or the import, export and erasure of existing ones. ### \[ERROR] key with ID already exists During the first initialization, the Chainlink node generates a key itself, but you can also import a key from a JSON keystore file manually. This error indicates that you are trying to import an already existing key. * Make sure to have followed the [official Chainlink documentation](https://docs.chain.link/docs/miscellaneous/#importing-a-keystore) * If you want to delete an existing key do not forget to make a backup if necessary by exporting it using the following CLI commands ``` chainlink admin login chainlink keys eth list chainlink keys eth export -p ./path/to/pw -o ./output/file.json ``` * Then delete the key using `chainlink keys eth delete ` ### \[ERROR] unable to find eth key with id * Make sure to have followed the [official Chainlink documentation](https://docs.chain.link/docs/miscellaneous/#importing-a-keystore) * Check the existing accounts by using the following CLI commands ``` chainlink admin login chainlink keys eth list ``` ### \[WARN] No P2P\_PEER\_ID set, defaulting to first key in database This message occurs if a Chainlink node is initialized with `FEATURE_OFFCHAINREPORTING=true` for the first time and no `P2P_PEER_ID` is set in the environmental file, so that a new ID is created and automatically used as default one. * The `P2P_PEER_ID` can be found in the key section of the web GUI or displayed using the Chainlink CLI commands ``` chainlink admin login chainlink keys p2p list ``` ### \[ERROR] multiple p2p keys found but peer ID was not set - you must specify a P2P\_PEER\_ID The `P2P_PEER_ID` is used for OCR jobs and needs to be set in the environmental file in order to enable the communication of the Chainlink node within the peer-to-peer network. # HeadTracker The HeadTracker stores the latest block number in a safe manner and reconstitutes the last block number from the data store on reboot. ### \[ERROR] Error in new head subscription, unsubscribed The Chainlink node is not able to create a new head subscription through the JSON-RPC method `eth_subscribe`. This issue might be related to the remote RPC endpoint (full node). * Check the blockchain connection of the Chainlink node (e.g. the Full-node-as-a-Service subscription and renew or switch the plan if necessary to prevent RPC rate limits from being hit) * Run an own full node with custom configuration and no performance restrictions ### \[WARN] HeadTracker: have not received a head for The Chainlink node has not received a head for the mentioned period of time. Block times can sometimes vary depending on the network and its conditions, so this log might be produced occasionally. The default period after which this log is produced is chain-specific for example: Ethereum mainnet: 1min, Polygon mainnet: 15sec * Check the blockchain connection of the Chainlink node (e.g. the Full-node-as-a-Service subscription and renew or switch the plan if necessary to prevent RPC rate limits from being hit) * Run an own full node with custom configuration and no performance restrictions ### \[ERROR] Headlistener: Failed to connect to ethereum node The Chainlink node is not able to establish a websocket connection to the remote RPC endpoint (full node). This connection is established through the JSON-RPC method `eth_subscribe`. * Check the blockchain connection of the Chainlink node (e.g. the Full-node-as-a-Service subscription and renew or switch the plan if necessary to prevent RPC rate limits from being hit) * Run an own full node with custom configuration and no performance restrictions ### \[ERROR] ethereum ChainID doesn't match chainlink config.ChainID * Check the `ETH_CHAIN_ID` setting of the Chainlink node * Check if the Chainlink node is connected to the right RPC endpoint ### \[ERROR] HeadTracker: got very old block with number (highest seen was ). This is a problem and either means a very deep re-org occurred, or the chain went backwards in block numbers. This node will not function correctly without manual intervention * If the Chainlink node has previously synchronized another chain and stored the block headers in its database, it can be helpful to drop the database and re-initialize the node with the correct `ETH_URL` (make sure to backup the private key beforehand if necessary using the following commands) ``` chainlink admin login chainlink keys eth list chainlink keys eth export -p ./path/to/pw -o ./output/file.json ``` * Check the blockchain connection of the Chainlink node (Full-node-as-a-Service subscription and renew or switch the plan to prevent RPC rate limits from being hit if necessary) * Run an own full node with custom configuration and no performance restrictions\ If you are running an own full node check * Whether an error occurred requiring to resync the node (firewall, bad block, OOM) * Whether it happens due to the node’s performance (hardware specs, disc’s IOPS) # GasEstimator The GasEstimator can either run in `BlockHistory`, `FixedPrice` or `Optimism2` mode and manages the setting of the gas price for the Chainlink node’s outgoing transactions. The safest way to prevent stuck or overpaid transactions on most networks is to run the `BlockHistory` mode. ### \[WARN] BlockHistoryEstimator: GAS\_UPDATER\_BLOCK\_HISTORY\_SIZE= is greater than ETH\_FINALITY\_DEPTH=, blocks deeper than finality depth will be refetched on every block history estimator cycle, causing unnecessary load on the eth node. Consider decreasing GAS\_UPDATER\_BLOCK\_HISTORY\_SIZE or increasing ETH\_FINALITY\_DEPTH The BlockHistoryEstimator listens for new heads and updates the base gas price dynamically based on the configured percentile of gas prices in that block. ### \[WARN] BlockHistoryEstimator: error fetching blocks The Chainlink node is not able to fetch the block from its remote RPC endpoint (full node). * Check the blockchain connection of the Chainlink node (e.g. the Full-node-as-a-Service subscription and renew or switch the plan if necessary to prevent RPC rate limits from being hit) * Run an own full node with custom configuration and no performance restrictions ### \[ERROR] BlockHistoryEstimator: cannot fetch, current block height is lower than GAS\_UPDATER\_BLOCK\_DELAY= The block delay variable determines the number of blocks that the block history estimator trails behind the head and thus which block to fetch. * Check the `BLOCK_HISTORY_ESTIMATOR_BLOCK_DELAY` setting and decrease it if necessary * Restart the node to apply the changes to the environmental file ### \[WARN] BlockHistoryEstimator: cannot calculate percentile gas price The Chainlink node cannot set a new default gas price as there are no suitable transactions in the block history to calculate the percentile gas price to choose. ### \[WARN] Calculated gas price of Wei exceeds ETH\_MAX\_GAS\_PRICE\_WEI=, setting gas price to the maximum allowed value of Wei instead The Chainlink node cannot set the new default gas price as its set maximum of `ETH_MAX_GAS_PRICE_WEI` would be exceeded. Transactions will never be sent with a higher gas price, the default values for this setting are chain-specific. * Check if the target network is facing any gas price abnormalities * Increase `ETH_MAX_GAS_PRICE_WEI` if necessary * Decrease `BLOCK_HISTORY_ESTIMATOR_TRANSACTION_PERCENTILE` if necessary * Restart the node to apply the changes to the environmental file ### \[WARN] Calculated gas price of Wei falls below ETH\_MIN\_GAS\_PRICE\_WEI=, setting gas price to the minimum allowed value of Wei instead The Chainlink node cannot set the new default gas price as it would fall below its set minimum of `ETH_MIN_GAS_PRICE_WEI`. Transactions will never be sent with a lower gas price, the default values for this setting are chain-specific. * Decrease `ETH_MIN_GAS_PRICE_WEI` if necessary * Increase `BLOCK_HISTORY_ESTIMATOR_TRANSACTION_PERCENTILE` if necessary * Restart the node to apply the changes to the environmental file # DirectRequest The following log messages are related to the direct request model, which is commonly used in combination with an on-chain oracle or operator contract that the Chainlink node observes for emitted event logs. ### \[ERROR] DirectRequest: OracleRequest log mailbox is over capacity - dropped the oldest log If a consumer sends a Chainlink request to an oracle or operator contract an event log is emitted containing a job ID and other request parameters. The Chainlink node picks up the log and initiates a job run if the job ID is matching one of those in its database. This error message indicates that there is a high amount of requests and the Chainlink node is not able to process all related logs which could lead to missed job runs. * Decrease the direct request frequency for this particular Chainlink node * Do not list your node details publicly if you want to avoid an unpredictable amount of requests * Use the following database commands to whitelist requesters for a single job (always backup your node and database before you manually change anything in the database!) ``` SELECT * FROM initiators; UPDATE initiators SET requesters='$ADDRESS' WHERE job_spec_Id='$JOB_ID'; ``` * Use the following database commands to whitelist requesters for all RunLog jobs ``` UPDATE initiators SET requesters='$ADDRESS1,$ADDRESS2,$ADDRESS3' WHERE type ='runlog'; ``` ### \[ERROR] DirectRequest: failed executing run The Chainlink node checks if the request was sent by a valid requester and if the payment is sufficient to initiate a job run. * Check the `MINIMUM_CONTRACT_PAYMENT_LINK_JUELS` setting of the Chainlink node or relevant job * Use the following database commands to whitelist requesters for a single job (always backup your node and database before you manually change anything in the database!) ``` SELECT * FROM initiators; UPDATE initiators SET requesters='$ADDRESS' WHERE job_spec_Id='$JOB_ID'; ``` * Use the following database commands to whitelist requesters for all RunLog jobs ``` UPDATE initiators SET requesters='$ADDRESS1,$ADDRESS2,$ADDRESS3' WHERE type ='runlog'; ``` # Postgres The Postgres service manages the connection and communication between the Chainlink node and its local or remote PostgreSQL database. ### \[ERROR] could not get advisory lock for classID The advisory lock provides a mechanism that locks the database as an application-oriented session without simultaneously preventing write operations (e.g. manual editing of the database). This means that other applications can be in the queue and have full access to the database as soon as the lock is released. It can be used for automated Chainlink node failover architectures by enabling a secondary node to wait for the lock to be released to jump in whenever the primary node fails (this practice is no longer recommended). This log indicates that the lock could not be closed. * Make sure that the `ETH_DATABASE_URL` environmental variables of the primary and secondary Chainlink nodes are identical * Check if the PostgeSQL database settings for the advisory lock is enabled ### \[WARN] Postgres event broadcaster: disconnected, trying to reconnect… The connection to the PostgreSQL database might have been interrupted and the Chainlink node tries to reconnect to resume the exchange with it. * Check the firewall settings of the Chainlink node’s host machine for outgoing traffic * Check the firewall settings of the PostgreSQL server for incoming traffic * Check the health and availability of the PostgreSQL server ### \[WARN] Postgres event broadcaster: reconnect attempt failed, trying again… The Chainlink node was not able to reconnect to the PostgreSQL database. It will produce this log until it can successfully reconnect. * Check the firewall settings of the Chainlink node’s host machine for outgoing traffic * Check the firewall settings of the PostgreSQL server for incoming traffic * Check the health and availability of the PostgreSQL server # Client The client interface enables interactions with a remote RPC endpoint (full node) and thus the target blockchain a Chainlink node should interact with, it captures the methods the internal Geth client can perform. ### \[ERROR] ethereum url scheme must be websocket The websocket address of the remote RPC endpoint used by the Chainlink node must be set correctly in order to establish a working connection with the target network. * Make sure the `ETH_URL` scheme starts with `ws://` or `wss://` ### \[ERROR] secondary ethereum rpc url scheme must be http(s) The secondary RPC endpoint(s) must be specified correctly in order to relieve the primary full node when sending transactions. * Make sure the scheme looks like this: `ETH_SECONDARY_URLS=http(s)://node.com/1,https://logs.com/2,...` ### \[ERROR] Failed to dial primary client `Dial()` creates a new internal client that connects to the remote RPC endpoint (full node) set with the `ETH_URL` environmental variable. It initiates the blockchain connection of the Chainlink node and automatically tries to reconnect if it is interrupted. If this error message occurs the Chainlink node cannot be (re-)initialized as it does not have a working blockchain connection. * Check `ETH_URL` for correctness * Check the functionality of the connection manually (e.g. by using a program like wscat to establish a `newHeads` subscription) ``` wscat -c ws://$RPC_URL {"id": 2, "method": "eth_subscribe", "params": [newHeads"]} ``` ### \[ERROR] Failed to dial secondary client `Dial()` creates a new internal client that connects to the remote RPC endpoint(s) (full node) set with the `ETH_SECONDARY_URLS` environmental variable. * Check `ETH_SECONDARY_URLS` for correctness * Check the connection’s functionality manually (e.g. by sending a JSON-RPC request to the RPC endpoint) ``` curl -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["latest",true], "id":1}' http://$RPC_URL ``` ### \[WARN] secondary eth client returned error If `ETH_SECONDARY_URLS` is set the Chainlink node broadcasts every transaction to the remote RPC endpoints in parallel. * Make sure to use different full nodes as primary and secondary ones * Check the blockchain connection of the Chainlink node (e.g. the Full-node-as-a-Service subscription and renew or switch the plan if necessary to prevent RPC rate limits from being hit) * Run an own full node with custom configuration and no performance restrictions # BulletproofTxManager This service is responsible for creating, if necessary rebroadcasting and eventually confirming all transactions that the Chainlink node sends on-chain. ### \[WARN] BulletproofTxManager: dropped old transactions from transaction queue The Chainlink node drops the oldest transactions as there are too many queued ones in order to create a new transaction. A large amount of non-broadcast transactions indicates an issue with the remote RPC endpoint (full node), or that a previous transaction is stuck. If the node is not able to rebroadcast it you may need to delete the unconfirmed transactions and pending job runs from the database manually. * Run an own full node with custom configurations and no performance restrictions * Run these commands on your database to delete all unconfirmed transactions and pending job runs (always make a backup of your node and database before you manually change anything in the database!) ``` SELECT * FROM eth_txes; DELETE from eth_txes WHERE state = 'unconfirmed'; DELETE from job_runs WHERE status = 'pending_outgoing_confirmations'; ``` ### \[ERROR] cannot create transaction; too many unstarted transactions in the queue The Chainlink node cannot create a new transaction as the queue limit is exceeded. Many queued transactions indicate that a previous transaction is stuck, if the node is not able to rebroadcast it you may need to delete the unconfirmed transactions and pending job runs from the database manually. * Check the node’s `ETH_MAX_QUEUED_TRANSACTIONS` setting * Run these commands on your database to delete all unconfirmed transactions and pending job runs (always backup your node and database before you manually change anything in the database!) ``` SELECT * FROM eth_txes; DELETE from eth_txes WHERE state = 'unconfirmed'; DELETE from job_runs WHERE status = 'pending_outgoing_confirmations'; ``` ### \[WARN] EthBroadcaster: transaction throttling; transactions in-flight and unstarted transactions pending (maximum number of in-flight transactions is per key) The default value of maximum in-flight transactions is 16, so your Chainlink node may be struggling holding up with the frequency of incoming job requests. If the node successfully broadcasts transactions that do not get confirmed they could be underpriced and not being picked up by the network’s validators. If the node is not able to rebroadcast unconfirmed transactions you may need to delete these and the pending job runs from the database manually. * Check the node’s `GAS_ESTIMATOR_MODE` setting * Check the node’s other [gas related settings](https://docs.chain.link/docs/configuration-variables/#gas-controls) * For high-throughput Chainlink nodes: configure your full node accordingly and increase ETH\_MAX\_IN\_FLIGHT\_TRANSACTIONS (Default: 16) * Run these commands on your database to delete all unconfirmed transactions and pending job runs (always backup your node and database before you manually change anything in the database!) ``` SELECT * FROM eth_txes; DELETE from eth_txes WHERE state = 'unconfirmed'; DELETE from job_runs WHERE status = 'pending_outgoing_confirmations'; ``` ### \[ERROR] EthBroadcaster: transaction gas price was rejected by the eth node for being too high. The Chainlink node tried to submit a transaction with a gas price exceeding the set limit of its remote RPC endpoint (full node). * Check the Chainlink node’s blockchain connection (e.g. the Full-node-as-a-Service subscription and renew or switch the plan if necessary to prevent RPC rate limits from being hit) * Run an own full node with custom configuration and no performance restrictions If you are running an own full node make sure to * Consider to increase `RPCTxFeeCap` * It is recommended to run Geth with no cap, i.e. `--rpc.gascap=0 --rpc.txfeecap=0` ### \[ERROR] EthBroadcaster: tx at gas price Wei was rejected due to insufficient eth The native token balance of the Chainlink node account is too low to submit the transaction. * Fund the Chainlink node account with a sufficient amount of the network’s native token * You can view the node’s account address when the node starts up or on the Keys page of the web GUI ### \[ERROR] default gas price wei was rejected by the eth node for being too low The Chainlink node tried to submit a transaction with an insufficient gas price. * Check the `GAS_ESTIMATOR_MODE` setting * Check the node’s other [gas related settings](https://docs.chain.link/docs/configuration-variables/#gas-controls) ### \[ERROR] Hit gas price bump ceiling, will not bump further. This is a terminal error The Chainlink node repeatedly bumped an unconfirmed transaction and will not continue to bump the gas price any further. This could be due to a network congestion or an internal bumping issue. * Check the `GAS_ESTIMATOR_MODE` setting * Increase `ETH_MAX_GAS_PRICE_WEI` if necessary * Restart the node to apply the changes to the environmental file ### \[ERROR] invariant violation: could not increment nonce because no rows matched query The Chainlink node cannot increment the nonce of the account, this may result from a modification of the nonce while using the account with another Chainlink instance or an external wallet. * Make sure not to send transactions from an account that is used by a Chainlink node except through the Chainlink CLI itself, otherwise it could lead to unrecoverable nonce issues ### \[ERROR] EthConfirmer batchFetchReceipt failed The remote RPC endpoint used by the Chainlink node is not able to transfer the status of a broadcast transaction using the `eth_getTransactionReceipt` method. The Chainlink node needs the receipt to check if a transaction has already been mined or if further actions like resubmitting with a bumped gas price are necessary. * Check the blockchain connection of the Chainlink node (e.g. the Full-node-as-a-Service subscription and renew or switch the plan if necessary to prevent RPC rate limits from being hit) * Run an own full node with custom configuration and no performance restrictions\ If you are running an own full node make sure to * Enable the `--txpool.locals` flag so that the full node can keep better track of the Chainlink node transactions that are sent to the mempool * `--txpool.locals $CHAINLINK_NODE_ADDRESS` * Increase `--txpool.globalslots` (Default: 2000) ### \[WARN] Transaction reverted on chain Reverted on-chain transactions are often related to chain reorgs and the settings for minimum incoming confirmations. The Chainlink node may submit multiple transactions if it does not find the previous ones in the mempool so that they become “stale”, this may be due to a chain reorg or issues with the remote RPC endpoint (full node). Reverted transactions can also occur if several nodes from the same OCR network transmit their results for a certain round in a short time interval and only the transaction that is confirmed first is not reverted. * Check the `MIN_INCOMING_CONFIRMATIONS` setting (Default values have been tested and adjusted for each network, values < 3 may result in multiple transactions) * Only set `ETH_HTTP_URL` if you run your own full node to be sure to use the same full node on the backend (`ETH_URL` and `ETH_HTTP_URL` need to point to the same full node)\ If you are running an own full node make sure to * Enable the `--txpool.locals` flag and add the addresses of your Chainlink node(s) * `--txpool.locals $CHAINLINK_NODE_ADDRESS` * Test different settings for `--peers` depending on the network (Default = 100) * Less peers: probably less, but deeper reorgs * More peers: probably more, but less deep reorgs ### \[ERROR] EthConfirmer: eth\_tx with ID expired without ever getting a receipt for any of our attempts Chainlink requires exclusive ownership of its private keys, sharing them across multiple Chainlink instances or using the keys with an external wallet is not supported and will lead to missed transactions. * Make sure not to send transactions from an account that is used by a Chainlink node except through the Chainlink CLI itself, otherwise it could lead to unrecoverable nonce issues ### \[WARN] EthConfirmer: transactions to rebroadcast which exceeds limit of Too many transactions need to be rebroadcast and the limit of in-flight transactions is exceeded, so the Chainlink node is struggling to hold up with the frequency of incoming job requests. If the node successfully broadcasts transactions that do not get confirmed they may be underpriced and not be picked up by the network’s validators. * Check the `GAS_ESTIMATOR_MODE` setting * Check the node’s other [gas related settings](https://docs.chain.link/docs/configuration-variables/#gas-controls) * For high-throughput Chainlink nodes: configure the full node accordingly and increase `ETH_MAX_IN_FLIGHT_TRANSACTIONS` (Default: 16) ### \[ERROR] Failed to bump gas The Chainlink node does not bump the gas for an unconfirmed transaction, instead it tries to resubmit the previous attempt until it gets accepted. * Check the `GAS_ESTIMATOR_MODE` setting * Increase `ETH_MAX_GAS_PRICE_WEI` if necessary * Restart the node to apply the changes to the environmental file ### \[WARN] EthConfirmer: chain length supplied for re-org detection was shorter than EvmFinalityDepth. If this happens a lot, it could indicate a problem with the remote RPC endpoint, a compatibility issue with a particular blockchain, heads table being truncated too early, or some other problem * Check the blockchain connection of the Chainlink node (e.g. the Full-node-as-a-Service subscription and renew or switch the plan if necessary to prevent RPC rate limits from being hit) * Run an own full node with custom configuration and no performance restrictions * Make sure the network the node is connecting to is supported by Chainlink ### \[WARN] EthResender: failed to resend unconfirmed transactions This message may occur if gas bumping is disabled or the network is experiencing abnormally long block times. It may also be related to issues with the remote RPC endpoint (full node). * Check the `GAS_ESTIMATOR_MODE` setting * Check the node’s other [gas related settings](https://docs.chain.link/docs/configuration-variables/#gas-controls) * Check the blockchain connection of the Chainlink node (e.g. the Full-node-as-a-Service subscription and renew or switch the plan if necessary to prevent RPC rate limits from being hit) * Run an own full node with custom configuration and no performance restrictions ### \[WARN] NonceSyncer: address has been used before, either by an external wallet or a different Chainlink node. Local nonce is but the on-chain nonce for this account was. It's possible that this node was restored from a backup. If so, transactions sent by the previous node will NOT be re-org protected and in rare cases may need to be manually bumped/resubmitted. Chainlink requires exclusive ownership of its private keys, sharing them across multiple Chainlink instances or using the keys with an external wallet is not supported and will lead to missed transactions. * Make sure not to send transactions from an account that is used by a Chainlink node except through the Chainlink CLI itself, otherwise it could lead to unrecoverable nonce issues # BalanceMonitor The BalanceMonitor checks the network's native token balance for each key on every new head the Chainlink node receives from the remote RPC endpoint. ### \[ERROR] BalanceMonitor: error getting keys The Chainlink node is not able to retrieve the keys for its available accounts from the keystore. * Make sure you have followed the instructions for setting up a Chainlink node with either newly generated or correctly imported keys from an existing account * [Official Chainlink documentation](https://docs.chain.link/docs/miscellaneous/#importing-a-keystore) ### \[ERROR] BalanceMonitor: error getting balance for key The Chainlink node is not able to retrieve the needed data object from the remote RPC endpoint it is connected to. The functioning of this service is mandatory for the monitoring of the Chainlink node account balance. * Check the blockchain connection of the Chainlink node (e.g. the Full-node-as-a-Service subscription and renew or switch the plan if necessary to prevent RPC rate limits from being hit) * Run an own full node with custom configuration and no performance restrictions # FluxMonitor The FluxMonitor job type enables features like decentralized data feeds by empowering Chainlink nodes to read on-chain smart contract states (e.g. latest price stored in an aggregator contract) and examining a calculated median of freshly aggregated off-chain data for price deviations. Each node participating in such a decentralized oracle network gets the data by polling external adapters for different data provider APIs and submits the result on-chain when the predefined conditions are met.\ [Official Chainlink documentation](https://docs.chain.link/docs/jobs/types/flux-monitor/) ### \[ERROR] unable to determine hibernation status * Check the contract address * Check the blockchain connection of the Chainlink node (e.g. the Full-node-as-a-Service subscription and renew or switch the plan if necessary to prevent RPC rate limits from being hit) * Run an own full node with custom configuration and no performance restrictions ### \[ERROR] failed to get list of oracles from FluxAggregator contract The Chainlink node needs to get the list of oracles that are allowed to submit observations to the AccesControlledAggregator in order to set the oracle address which matches the key of the node. * Make sure to give access to all oracles that are supposed to submit to the aggregator contract by correctly adding the node addresses to the corresponding smart contract ### \[WARN] None of the node's keys matched any oracle addresses, using first available key. This flux monitor job may not work correctly * Make sure to give access to all oracles that are supposed to submit to the aggregator contract by correctly adding the node addresses to the corresponding smart contract ### \[ERROR] Error determining if log was already consumed It should be determined if a log is a duplicate of an already known one which may happen due to the backfilling feature and should result in ignoring this particular log. This error may occur after a node crash or restart, or indicate issues with the remote RPC endpoint (full node). * Check the blockchain connection of the Chainlink node (e.g. the Full-node-as-a-Service subscription and renew or switch the plan if necessary to prevent RPC rate limits from being hit) * Run an own full node with custom configuration and no performance restrictions ### \[ERROR] Error determining if flag is still raised The Chainlink node checks if both flags enabling the hibernation mode are lowered after receiving a `FlagsFlagRaised` log, this error indicates an issue with the remote RPC endpoint (full node). * Check the contract address * Check the blockchain connection of the Chainlink node (e.g. the Full-node-as-a-Service subscription and renew or switch the plan if necessary to prevent RPC rate limits from being hit) * Run an own full node with custom configuration and no performance restrictions ### \[ERROR] could not fetch oracleRoundState The Chainlink node cannot check if a round has successfully closed with a new answer because it’s unable to consume the AnswerUpdated log. This may be related to an issue with the remote RPC endpoint (full node). * Check the blockchain connection of the Chainlink node (e.g. the Full-node-as-a-Service subscription and renew or switch the plan if necessary to prevent RPC rate limits from being hit) * Run an own full node with custom configuration and no performance restrictions ### \[ERROR] error determining round stats / run status for round The Chainlink node cannot determine the run status of a new round, this may be related to an issue with the remote RPC endpoint (full node). * Check the blockchain connection of the Chainlink node (e.g. the Full-node-as-a-Service subscription and renew or switch the plan if necessary to prevent RPC rate limits from being hit) * Run an own full node with custom configuration and no performance restrictions ### \[ERROR] Ignoring new round request: error fetching eligibility from contract The Chainlink node always checks its eligibility to submit to a new round by checking the round state, if it has already submitted to it and if the aggregator contract can pay it. In this case it is not able to fetch the needed data from the contract, which indicates an issue with the remote RPC endpoint (full node). * Check the blockchain connection of the Chainlink node (e.g. the Full-node-as-a-Service subscription and renew or switch the plan if necessary to prevent RPC rate limits from being hit) * Run an own full node with custom configuration and no performance restrictions ### \[ERROR] error executing new run for job ID name * Make sure the on-chain configs and job spec are correct * Make sure your external adapters are operational ### \[WARN] LogBroadcaster is not connected to Ethereum node, skipping poll The Chainlink node is not able to interact with the target network as it is not connected with a remote RPC endpoint (full node). * Make sure to reestablish a functioning blockchain connection * Check out existing automated failover solutions for the remote RPC endpoint of the Chainlink node, for example: [Fiews/ChainlinkEthFailover](https://github.com/Fiews/ChainlinkEthFailover) # PeriodicBackups The Chainlink node database can be backed up automatically at regular time intervals. [Official Chainlink documentation](https://docs.chain.link/docs/configuration-variables/#database_backup_mode) ### \[ERROR] Invalid path for DATABASE\_BACKUP\_DIR - please set it to a valid directory path * Make sure an existing directory path is used * Default: Chainlink root directory ### \[ERROR] Database backup frequency is too small. Please set it to at least * Set a positive regular time interval (at least 1m) for dumping the database * Examples with valid time unit (e.g. „m“, „h“): `DATABASE_BACKUP_FREQUENCY=7h` ### \[ERROR] DatabaseBackup: Failed * Make sure a valid directory path is used * Make sure there is enough disk space for storing the backup file # OCR The off-chain reporting protocol enables features like decentralized data feeds by empowering Chainlink nodes to generate signed reports of aggregated data within an off-chain peer-to-peer network. Each Chainlink node aggregates data from a set of data sources, the individual observations are then compared, sorted and finally approved by all participating oracles. If the predefined conditions are met (e.g. minimum price deviation or maximum elapsed time), the final report is transmitted on-chain by one single Chainlink node, the aggregator smart contract then verifies that it was signed by a quorum of oracles, so that the median value of the report can be exposed to consuming contracts. The main purpose of the log messages in this section is to provide a better understanding of the functioning of the OCR protocol and the different roles a single Chainlink node can take. [Official Chainlink documentation](https://docs.chain.link/docs/off-chain-reporting/) [Chainlink Labs OCR research paper](https://research.chain.link/ocr.pdf) ### \[ERROR] Pacemaker: Timeout while restoring state from database OCR reports are generated in epochs, the pacemaker algorithm drives the report generation by keeping track of the state and message handling of an oracle to ensure continuous progress of the OCR protocol. * Check the health and availability of the PostgreSQL server ### \[ERROR] Pacemaker: error while restoring state from database * Check the health and availability of the PostgreSQL server ### \[WARN] non-leader round sender The report generation follower algorithm manages observation requests received from the leader of the current report generation instance. In this case the leader seems to be incorrect, thus the protocol halts as an oracle may be trying to usurp the lead. * Ensure the integrity of the network participants ### \[WARN] out of bounds round round rMax msgRound This message warns of a potentially malicious leader as the round value is unusually high and should prevent denial-of-service attacks, the oracles exit the report generation protocol and emit a `changeleader` event. * Ensure the integrity of the network participants ### \[ERROR] messageObserveReq: could not make SignedObservation observation After receiving an observe request the oracle collects a new observation (calls its external adapters to fetch the needed data) and sends a signed message to the leader. The observation needs to be signed with the `offChainPrivateKey` of the oracle in order to produce a valid observe message for which it gets paid. ### \[ERROR] MakeSignedObservation produced invalid signature This error indicates an issue with the `offChainPrivateKey` of the node which is necessary to produce a valid observe message for which the oracle gets paid. ### \[WARN] messageReportReq after report sent If the report generation leader has received sufficient valid observe messages for a certain round, it sorts the observations by their values and sends a report request message to the report generation follower nodes. In this case a report has already been sent and will not be submitted again. ### \[WARN] messageReportReq after round completed The report request will not be processed further because the round is no longer up-to-date, which may result from network delays. ### \[ERROR] messageReportReq: could not validate report sent by leader The Chainlink node could not validate that all signatures in the received report are valid. ### \[ERROR] messageReportReq: failed to sign report The Chainlink node could not validate the report containing the individual observations and oracle identities by signing the compressed report. ### \[ERROR] ReportGeneration: DataSource errored The Chainlink node could not get a successful response from its assigned data sources and may therefore not be able to send an observation to the current leader. * Make sure your external adapters are operational * Make sure you did not hit your data provider rate limits * Check if the used APIs are responsive ### \[ERROR] shouldReport: blockchain interaction timed out, returning true This error may indicate a deep reorg of the target network or issues with the remote RPC endpoint (full node) and prevents the Chainlink node from transmitting reports on-chain. However, it does not prevent the node from contributing to the report generation protocol. * Check the blockchain connection of the Chainlink node (e.g. the Full-node-as-a-Service subscription and renew or switch the plan if necessary to prevent RPC rate limits from being hit) * Run an own full node with custom configuration and no performance restrictions ### \[ERROR] shouldReport: Error during LatestTransmissionDetails There may be too many reports created to determine whether a new final report should be sent, which depends on the predefined conditions of the data feed (e.g. price deviation, elapsed time since the last on-chain update). ### \[ERROR] shouldReport: Error during LatestRoundRequested There may be too many reports created to determine whether a new final report should be sent, which depends on the predefined conditions of the data feed (e.g. price deviation, elapsed time since the last on-chain update). ### \[ERROR] messages not sorted by value The observations were not sorted by value, which indicates issues during the report generation and could lead to incorrect results. ### \[ERROR] given oracle ID of is out of bounds (only have public keys) The report request errors as there is an invalid oracle ID not matching the number of known public keys. * Ensure that untrustworthy participants are denied access to the peer-to-peer network ### \[ERROR] not enough observations in report; got , need more than In order to protect results against manipulation, a minimum number of oracles is required to report their observations in order to generate a valid report. * Ensure that all Chainlink nodes that contribute to the protocol are protected against single points of failure and therefore highly available and responsive ### \[WARN] verifyAttestedReport: dropping final report because it has too few signatures * Make sure that all Chainlink nodes that contribute to the protocol are protected against single points of failures and therefore highly available and responsive ### \[WARN] ReportGeneration: new round number would be larger than RMax + 1. Looks like your connection to more than f other nodes is not working. The report generation protocol produces one report per round and repeats this process until it halts and the next protocol instance is started, this warning indicates delays of the peer-to-peer network and prevents the start of a new round. * Make sure at least one bootstrap node is operational * Make sure that all Chainlink nodes that contribute to the protocol are protected against single points of failures and therefore highly available and responsive ### \[ERROR] ReportGeneration: round overflows, cannot start new round There are too many rounds that did not end with a valid report, which may result from delays of the peer-to-peer network. * Make sure at least one bootstrap node is operational * Make sure that all Chainlink nodes that contribute to the protocol are protected against single points of failures and therefore highly available and responsive ### \[WARN] Non-leader received MessageObserve Observe messages are generated by each oracle participating as report generation follower and are exclusively sent to the current leader, who constructs a report after receiving the observations. This indicates irregularities in the message flow of the report generation leader protocol instance. ### \[WARN] MessageObserve carries invalid SignedObservation The report generation leader could not validate the signed observation sent by one of the oracles. ### \[ERROR] leader's phase conflicts tGrace timeout The `T_observe` grace period is started once the leader has received enough observations to construct a report and gives slower oracles the time to submit their observations. In this case the leader wants to move on to the report phase without the grace period being over. ### \[WARN] messageReport: dropping MessageReport due to not being leader of the current epoch The oracle that is supposed to send out the final report to get the signatures of the other participants is no longer the leader of the current epoch, either due to an insufficient number of progress events or the predetermined number of rounds being reached, which causes the pacemaker protocol to abort this report generation instance and start a new one. ### \[WARN] messageReport: dropping MessageReport due to having already received sender's report Messages that have been sent more than once may be a result of network delays. ### \[ERROR] could not validate signature The report generation leader could not validate the `OnChainSigningAddress` of the message sender, thus its identity cannot be verified and the report will not be used for the final attested report. ### \[ERROR] contractEpoch() failed during eventTransmit The transmission protocol handles the updating of the value that is stored in the aggregator contract by causing an oracle to send the final report through a transaction on-chain. Reports are only allowed if the aggregator contract has seen a report as recent as the latest incoming one to prevent similar transmissions. The latest contract epoch and thus the eligibility to transmit the final report could not be determined, which may be related to issues with the remote RPC endpoint (full node). * Check the blockchain connection of the Chainlink node (e.g. the Full-node-as-a-Service subscription and renew or switch the plan if necessary to prevent RPC rate limits from being hit) * Run an own full node with custom configuration and no performance restrictions ### \[ERROR] Failed to serialize contract report The latest report received by the aggregator contract determines the eligibility to transmit the final report, a failed serialization of it may be related to issues with the remote RPC endpoint (full node). * Check the blockchain connection of the Chainlink node (e.g. the Full-node-as-a-Service subscription and renew or switch the plan if necessary to prevent RPC rate limits from being hit) * Run an own full node with custom configuration and no performance restrictions ### \[ERROR] could not compute median The median of the latest incoming report must deviate by at least the predefined threshold from the median of the final report in order to be transmitted on-chain while avoiding needless transmissions, in this case it could not be computed. ### \[ERROR] could not take median of observations The median of the latest incoming report must deviate by at least the predefined threshold from the median of the final report in order to be transmitted on-chain while avoiding needless transmissions, in this case it could not be computed. ### \[ERROR] Error while persisting pending transmission to database A pending transmission cannot be written to the database. * Check the health and availability of the PostgreSQL server ### \[ERROR] Database.StorePendingTransmission timed out * Check the health and availability of the PostgreSQL server ### \[ERROR] eventTTransmitTimeout: Error while deleting pending transmission from database * Check the health and availability of the PostgreSQL server ### \[ERROR] Database.DeletePendingTransmission timed out * Check the health and availability of the PostgreSQL server ### \[ERROR] eventTTransmitTimeout: contractState() failed The transmit event timed out as the contract state could not be determined, this may be related to issues with the remote RPC endpoint (full node). * Check the blockchain connection of the Chainlink node (e.g. the Full-node-as-a-Service subscription and renew or switch the plan if necessary to prevent RPC rate limits from being hit) * Run an own full node with custom configuration and no performance restrictions ### \[ERROR] eventTTransmitTimeout: Transmit timed out The transaction containing the final report could not be transmitted on-chain, this may be related to issues with the remote RPC endpoint (full node). * Check the blockchain connection of the Chainlink node (e.g. the Full-node-as-a-Service subscription and renew or switch the plan if necessary to prevent RPC rate limits from being hit) * Run an own full node with custom configuration and no performance restrictions ### \[ERROR] eventTTransmitTimeout: Error while transmitting report on-chain The transaction containing the final report could not be transmitted on-chain, this may be related to issues with the remote RPC endpoint (full node). * Check the blockchain connection of the Chainlink node (e.g. the Full-node-as-a-Service subscription and renew or switch the plan if necessary to prevent RPC rate limits from being hit) * Run an own full node with custom configuration and no performance restrictions # Articles # External Initiators \#chainlinknode #externalinitiators #nodeoperator ### Overview To better understand why we need External Initiators and what role they play, it's worth getting a better understanding of the Job Spec. The Job Spec conceptually consists of two parts: * Initiator - When the Job is to be executed * Task - How and in which order the Job is to be executed Under the hood, Chainlink Node has a built-in set of Initiators and Tasks.\ If the built-in tools are not suitable for your task, the External Initiators and External Adapters come to the rescue. ### External Initiator The External Initiator has a bi-directional action. This means that we can both send data to the client and receive data from the data sources. For connected External Initiators, the Chainlink Node will notify them about the events of creating a new Job or deleting an existing Job whose whitelist contains a specific External Initiator.

Example of External Initiator 1.1. Querying an external database and starting a Job according to specific criteria

Example of External Initiator 1.2.0. Blockchain polling. Sending SMS messages to a client based on Blockchain events. Sending SMS messages to a client to create transactions on Blockchain

Example of External Initiator 1.2.1 operation

#### Creating External Initiator The External Initiator is a service (e.g. written in go, nodejs, etc.) that must have a connection to the Postgres database (different from the database used by the Chainlink Node). In order to communicate with the Job deployed on the Chainlink Node, the External Initiator must have cryptographic pairs: * External Initiator Outgoing Access Key / External Initiator Outgoing Secret - To interact External Initiator → Chainlink Node * External Initiator Incoming Access Key/External Initiator Incoming Secret - For interaction Chainlink Node → External Initiator These pairs are automatically generated on the Chainlink Node side when you add the External Initiator. You will also need to add certain env variables to the External Initiator service after that. #### Adding External Initiator to the Chainlink Node The basic process is described in the documentation: What you should pay attention to: The .env file with variables like `EI_*` must be plugged into the service that acts as the External Initiator. Important! The database specified in the variable `EI_DATABASEURL` must be different from the Chainlink Node database. ### Sources # Configuring logging in a kubernetes cluster \#loki #fluentbit #grafana #chainlink ### Requirements * helm 3 ### Stack used * fluent-bit * loki * grafana ### Introduction The fluent-bit -> loki <- grafana stack is great if you are using prometheus, alertmanager, grafana, etc. for monitoring. Why fluent-bit and not promtail? fluent-bit is a very lightweight, high performance log collector. Out of the box it supports the main data sources for reading and sending, and has integration with kubernetes. ### Configuring loki ```shell helm repo add grafana https://grafana.github.io/helm-charts ``` The default loki chart is pretty heavyweight and includes prometheus operator, grafana, the loki cluster deployment, and a few other things. We will stick to configuring these services separately, as well as a single-node loki configuration to make operation and understanding easier. To do this, we need to change some default values via values.yaml #### Example of a values.yaml file to configure using nfs storage Note! We are using Basic Auth. Change your password. gateway.basicAuth.password ```yaml # disable service monitor from internal prometheus operator serviceMonitor: enabled: false # disable enterprise feature enterprise: enabled: false loki: auth_enabled: false server: http_listen_port: 3100 grpc_listen_port: 9095 podAnnotations: ### if you have annotations configured for prometheus, uncomment below # prometheus.io/scrape: "true" # prometheus.io/port: "3100" # prometheus.io/path: "metrics" # prometheus.io/scheme: "http" limits_config: enforce_metric_name: false reject_old_samples: true reject_old_samples_max_age: 168h max_cache_freshness_per_query: 10m split_queries_by_interval: 15m max_global_streams_per_user: 10000 commonConfig: # using single binary loki instance, not cluster replication_factor: 1 storage: type: "filesystem" readinessProbe: httpGet: path: /ready port: 3100 initialDelaySeconds: 30 timeoutSeconds: 1 gateway: # Enable basic auth basicAuth: enabled: true username: "admin" password: "your_password_here" singleBinary: # Use nfs storage persistence: enabled: true type: pvc storageClass: ssd-nfs-storage accessModes: - ReadWriteOnce size: 2Gi # Disable cluster targetModule: "all" test: enabled: false # Disable internal prometheus operator monitoring: dashboards: enabled: false rules: enabled: false alerting: false serviceMonitor: enabled: false selfMonitoring: enabled: false grafanaAgent: installOperator: false lokiCanary: enabled: false ``` #### **Uploading the chart** ```shell helm upgrade --install loki grafana/loki \ --values ./loki/values.yaml \ --namespace YOUR_NAMESPACE ``` ### Fluentbit Fluent Bit allows you to collect logs and metrics from multiple sources, enrich them with filters, and send them to any specific destination. Adding a chart repository ```shell helm repo add fluent https://fluent.github.io/helm-charts ``` #### Configuring log collection Use the values.yaml file to modify the default chart values. ```yaml image: repository: cr.fluentbit.io/fluent/fluent-bit # Overrides the image tag whose default is {{ .Chart.AppVersion }} # tag: "1.9.10" pullPolicy: IfNotPresent service: type: ClusterIP port: 2020 annotations: ### if you have annotations configured for prometheus, uncomment below # prometheus.io/scrape: "true" # prometheus.io/port: "2020" # prometheus.io/path: "/api/v1/metrics/prometheus" metricsPort: 2020 flush: 1 serviceMonitor: enabled: false prometheusRule: enabled: false livenessProbe: httpGet: path: / port: http readinessProbe: httpGet: path: /api/v1/health port: http resources: limits: cpu: 50m memory: 128Mi requests: cpu: 100m memory: 128Mi config: service: | [SERVICE] Daemon Off Flush {{ .Values.flush }} Log_Level {{ .Values.logLevel }} Parsers_File parsers.conf Parsers_File custom_parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port {{ .Values.metricsPort }} Health_Check On inputs: | [INPUT] Name tail Tag kube.* Path /var/log/containers/*.log Mem_Buf_Limit 5MB Skip_Long_Lines On Skip_Empty_Lines Off Refresh_Interval 60 filters: | [FILTER] Name parser Match kube.* Key_Name log Parser chainlink Parser cri Preserve_Key On Reserve_Data On [FILTER] Name kubernetes Match kube.* Kube_URL https://kubernetes.default.svc:443 Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token Kube_Tag_Prefix kube.var.log.containers. Merge_Log On Merge_Log_Trim On Keep_Log Off tls.verify On K8S-Logging.Parser On K8S-Logging.Exclude On Labels On Annotations On Use_Kubelet Off outputs: | [OUTPUT] name loki match * host loki.your-namespace.svc port 3100 http_user admin http_passwd your_loki_password labels $service_logtag auto_kubernetes_labels on customParsers: | [PARSER] Name chainlink Format regex Regex ^(?